Still Simpler Way of Introducing Interior-Point method for Linear Programming

Transcription

1 Stll Smpler Way of Introducng Interor-Pont method for Lnear Programmng Sanjeev Saxena Dept. of Computer Scence and Engneerng, Indan Insttute of Technology, Kanpur, INDIA October 9, 05 Abstract Lnear Programmng s now ncluded n Algorthm undergraduate and postgraduate courses for Computer Scence majors. It s possble to teach nteror-pont methods drectly wth just mnmal knowledge of Algebra and Matrces. A full and more complete verson of ths s avalable as (K.Mehlhorn and S.Saxena, A Stll Smpler Way of Introducng the Interor-Pont Method for Lnear Programmng, at vxra: and arxv: [cs.ds]). Introducton Terlaky [7] and Lesaja [3] have suggested smple ways to teach nteror-pont methods. In ths paper a stll smpler way s beng suggested. Most materal requred to teach nteror-pont methods s avalable n popular text books [5, 8]. However, these books assume knowledge of Calculus, whch s not really requred. In ths paper, t s suggested f approprate materal s selected from these books then t becomes very easy to teach nteror-pont methods as the frst or only method for Lnear programmng n Computer Scence Courses. A full and more complete (but slghtly dfferent) verson of ths s at[4]. Canoncal Lnear Programmng Problem s to mnmse cx T subject to Ax b and x 0. Here A s an n m matrx, b and c are n-dmensonal and x s an m-dmensonal vector. Remark maxmse cx T s equvalent to mnmse cx T. Remark Constrants of type α x +...+α n x n β can be replaced by α x +...+α n x n +γ β wth a new (slack) varable γ 0. Smlarly constrants of type α x α n x n β can be E-mal: ssax@tk.ac.n

2 replaced by α x α n x n γ β wth (surplus) varable γ 0. Thus, we assume that there are n constrants and m varables, wth m > n (more varables and fewer constrants) bascally slack or surplus are added or subtracted to convert nequaltes nto equaltes. We frst use pvotng to make frst term of all but the frst equaton as zero. Bascally, we multply th equaton by a a and subtract the frst equaton. In smlar way we make frst two terms of all but the frst two equatons as zero multply th equaton (for ) by a a and subtract the second equaton. And so on. In case, f n any equaton all coeffcents become zero, we drop those equatons. As a result, n the end all remanng equatons wll be lnearly ndependent. Or the resultng matrx wll have full row rank. Remark We may have to nterchange two columns (nterchange two varables), n case, for example, f a dagonal term of an equaton becomes zero. From convexty, t s suffcent to obtan a locally optmal soluton, as local optmalty wll mply global optmalty. We consder another problem, the dual problem whch s maxmse by T subject to A T y + s c, wth slack varables s 0 and varables y are unconstraned. Clam by T cx T. The equalty wll hold f and only f, s x 0 for all s. Remark Thus f value of both prmal and dual are the same, then both are optmal. Proof: s c A T y, or x T s x T c x T (A T y) c T x (x T A T )y c T x b T y. As x,s 0, we have x T s 0 or c T x b T y. Equalty wll hold f x T s 0 or s x 0 but as s,x 0, we want each term (product) s x 0. Thus, f we are able to fnd a soluton of followng equatons (last one s not lnear, else, an nverson of matrx would have been suffcent), we wll be gettng optmal solutons of both the orgnal and the dual problems. subject to x 0,s 0. Ax b,a T y + s c,x s 0 We wll relax the last condton to get somethng lke (dualty gap): x s wth parameter 0. Thus, we wll be solvng (the exact last equaton wll be derved n the next secton): Ax b,a T y + s c,x s subject to x 0,s 0. Remark Thus, by T cx T m.

3 Use of Newton Raphson Method Let us assume that we have an ntal soluton (x,s,y) (we wll dscuss how to get an ntal soluton n Secton 5) whch satsfes Ax b and A T y + s c We wll use the Newton-Raphson method[5] to get better soluton. Let us choose the next values as x + h,y + k,s + f. Then we want:. A(x + h) b or Ax + Ah b but as Ax b, we get Ah 0.. A T (y +k)+(s+f) c, from A T y +s c, we get A T k +f c A T y s 0 or A T k +f 0 3. (x +h )(s +f ) or x s +h s +f x +h f. Or approxmately, x s +h s +f x (neglectng the non-lnear h f term). Thus, the equaton we wll be solvng s h s + f x x s Thus, we have a system of lnear equatons for h,k,f. We next show that these can be solved by nvertng a matrx. But frst observe that from the thrd equaton, Observaton (x + h )(s + f ) + h f Theorem Followng equatons have a unque soluton:. Ah 0. A T k + f 0 3. h s + f x x s Proof: We wll follow Vanderbe[8] and use captal letters (e.g. X) n ths proof (only) to denote a dagonal matrx wth entres of the correspondng row vector (e.g. n X the dagonal entres wll be x,x,...,x m ). We wll also use e to denote a column vector of all ones (usually of length m). Then n the new notaton, the last equaton s: Let us look at ths equaton n more detal. Sh + Xf h + S Xf h + S Xf h + S Xf Ah + AS Xf Sh + Xf e XSe e XSe S e S XSe pre-multply by S S e X S Se dagonal matrces commute S e x as Xe x AS e Ax pre-multply by A AS Xf AS e b but Ax b and Ah 0 AS XA T k AS e b usng f A T k b AS e (AS XA T )k 3

4 As XS s dagonal wth postve tems and as A has full rank, f W XS (each dagonal term s x /s ) then AS XA T (AW)(AW) T AW A T s nvertble (see appendx). The last equaton can thus be used to get the value of matrx k after nvertng the matrx AS XA T, or ( ) k (AS XA T ) b AS e Then we can fnd f from f A T k. And to get h we use the equaton: h + S Xf S e x,.e., h XS f + S e x Thus, the above system has a unque soluton. Clam h f 0 or equvalently h T f f T h 0 Proof: As A T k +f 0, we get h T A T k +h T f 0 but h T A T (Ah) T 0, hence h T f 0 follows. 3 Invarants n each Iteraton We wll mantan followng nvarants:. Ax T b, wth x > 0 (strct nequalty). A T y + s c wth s > 0 (strct nequalty) 3. If s the approxmate dualty gap then σ 3 5 where σ ( x s ). At( end of ths teraton we want dualty gap ( δ). We wll see that δ can be chosen as δ Θ m ). We frst show that strct nequalty nvarants hold (n σ we have x + h,s + f and same ): Fact If σ < then x + h > 0 and s + f > 0 Proof: We frst show that the product (x +h )(s +f ) s term-wse postve. From Observaton, (x + h )(s + f ) + h s. From σ < we get σ <. But (usng Observaton ): σ ( ) (x + h )(s + f ) ( ) h s < As the sum s at most one, t follows that each term of the summaton must be less than one, or < or < h f <. In partcular + h f > 0. h f Thus the product (x + h)(s + f) s term-wse postve. 4

5 Assume for contradcton that both x + h < 0 and s + f < 0. But as s > 0 and h > 0, we have s (x + h ) + x (s + f ) < 0, or + x s < 0. Whch s mpossble as,x,s are all non-negatve, a contradcton. We have to stll show that the approxmate dualty gap decreases as desred. Let us defne two new varables: H F h s x f x s Observe that H F h f 0 (see Clam ). H + F x s h x + f s x s (h s + f x ) x s ( x s ) ( x ) s x s ( + x ) s x s From, the proof of Fact, we also observe that σ ( ) h f, or σ (H F ) And fnally σ (H F ) (H + F ) 4 usng AM-GM nequalty-see Appendx ( (H + F )) more postve terms 4 ( ) (H + F ) from Clam 4 ( 4 x s ( max ) x s 4 ( σ4 max 4 x s ( ) ) x s ) ( ( x s ) 5 )

6 Thus, As σ ( x s ), each ndvdual term s at most σ or x s σ σ x s σ σ x s + σ In partcular x s σ or max x s σ Thus, σ ( ) σ 4 σ 4 or σ σ σ We summarse our observatons as: Observaton σ σ σ For σ <, σ σ < or σ < σ or σ + σ < 0 or σ ± 4 ( ) Remark Observe that as for σ <, denomnator s a decreasng functon of σ and numerator s σ an ncreasng functon of σ, hence, wll be an ncreasng functon of σ. Ths can also be seen σ σ as σ σ σ ( + σ + σ +...), σ σ wll be an ncreasng functon of σ. Moreover for σ 3, σ σ, thus we need σ < 3, Hence the choce of σ 3 5. For σ 3 5 we have σ 9 0. Let us fnally try to get bounds on δ (and hence ). Let us assume ( δ) then f σ corresponds to x + h,s + f and we have σ ( ) (x + h )(s + f ) ( δ) ( (x + h )(s + f ) ( δ) ) ( δ) ( (x + h )(s + f ) ( δ) ( h f ( δ) + δ δ ( h f ( δ) + δ ( δ) ( δ) ( ( h f ( ( h f 6 + δ ) δ ) From Observaton ) ) + mδ + δ ) h f ) ) + mδ From Clam

7 ( ( δ) σ + mδ ) Thus observe that Observaton 3 σ σ δ + mδ We want to choose δ such that σ 3 5. We want to choose δ such that σ 3 5. As for σ 3 5, σ 9 0, we have σ δ σ + mδ σ + mδ mδ. For ths to be bounded by 3 5, or mδ Thus, we choose δ 4 m whch s on the safe sde for m 4. Remark For m 4, δ (/4 8 4) 7. and σ δ σ + mδ 8 7 (8/400) + (/6) / < Summary Let us assume that ntal dualty gap s 0 and fnal dualty gap s f, as after each teraton, ( δ), thus after r teratons, f ( δ) r 0, or log 0 f r log( δ) r( δ) or r O ( δ log ) 0 O f ( ) m 0 log f As σ x s + σ, we have (n last nequalty we use σ < 3 ). ( σ) x s ( + σ) < 5 3 Thus, when becomes very small, even the products x s s wll be very small. The above method wll gve a polynomal tme algorthm even f 0 m and f m. 5 Intal Soluton: Bg M Method Let us frst assume that there s a number (say) W such that there s an optmal soluton x for whch each x W; we wll see later (see Secton 5.) how to fnd such a number n case all enteres of A and b are ntegers. If e s a column vector (of length m) of all ones, then e T x < mw. Thus an optmal soluton of the problem [, p430] (see also [, p8-9]) 7

8 mnmse cx T subject to Ax b, e T x < mw and x 0. wll also be a soluton of the orgnal problem (wthout e T x < mw constrant). Let us replace (scale) varables x by mwx m + then the problem becomes: mnmse mw m+ cx T subject to Ax b, e T x < m+ and x 0 wth b b(m+)/(mw) We add a new varable x m+ and replace et x < m + by e T x + x m+ m + (wth x m+ 0). Or droppng prmes, the problem s equvalent to: mnmse cx T subject to Ax b, e T x + x m+ m + and x 0. Consder a startng soluton x 0 s.t. all components of x 0 are strctly postve (say all x or x e, x m+ ). Defne a vector ρ b Ae. Let x m+ be one more new varable. Then Ax + x m+ ρ b wth x 0,x m+ 0 has a soluton wth x e and x m+ x m+. For ths choce, e T x + x m+ + x m+ m + s also true. We want a soluton n whch x m+ 0. Thus, we try to mnmse cx T + Mx m+ for a large M. We thus consder the artfcal prmal problem: mnmse cx T + Mx m+ subject to Ax + ρx m+ b, e T x + x m+ + x m+ m + and x 0,x m+ 0 and x m+ 0. The dual problem (wth new dual varable y n+,s m+ and s m+ s: maxmse by T + (m + )y n+ subject to A T y + ey n+ + s c, ρ T y + y n+ + s m+ M y n+ + s m+ 0 wth slack varables s 0,s m+ > 0,s m+ > 0 and varables y are unconstraned. To get an ntal soluton, from last equaton s m+ y n+ > 0 (say). The smplest choce wll be to choose all other y 0 then from frst equaton s c+e whch s agan a postve number (f s larger than all c s). To satsfy the second equaton we must choose s m+ M y n+ M +. Observe that all slack varables are postve. For ths choce, x s c + or x s c and x m+s m+ and x m+s m+ M. Thus σ ( c + M ). We can make σ < 4 by choosng 4 ( c + M). 5. Integer Case Ths secton assumes some more knowledge of algebra determnants and Cramer s rule and some knowledge of geometry. 8

9 If A s an n n matrx then det A π ( )π a π() a π()...a nπ(n) wll be sum of all possble (products) of permutatons π (wth approprate sgn). Clearly det A π ( )π a π() a π()...a nπ(n). a π() a π()...a nπ(n) If each a j U, then det A n!u n. π Cramer s rule says that soluton of equaton Ax b (for n n non-sngular matrx A) s x det(a )/det(a) where A s obtaned by replacng th column of A by b. As all constrants are lnear, soluton space wll be a convex polytope and (by convexty) for optmal soluton t s suffcent to look at corner ponts. At each corner pont exactly n components of x wll be non-zero; remanng m n x wll be zero. Thus, at optmal soluton x det(a )/det(a ), where A s obtaned by keepng only n columns of A. If we assume that each b U, the maxmum value of the determnant can be n!u n. If all entres are ntegers, then determnant has to be at least one f t s non-zero. Thus, each x s between /(n!u n ) and n!u n. Or we can choose W n!u n < (nu) n. Acknowledgement Ths work was nspred by an nformal lecture gven by Nsheeth Vshno at IIT Kanpur. I also wsh to thank students of CS60 (04-05 batch) for ther helpful comments and questons when I was teachng ths materal. Thanks also to Kurt Mehlhorn for pontng out that method for fndng ntal soluton (of earler verson) may not work and for hs suggeston of usng method of [, p430] nstead n Secton 5. References [] D.Bertsmas and J.N.Tstskls, Introducton to lnear optmzaton, Athena Scentfc, 997. [] H.Karloff, Lnear Programmng, Brkhauser, 99. [3] Goran Lesaja, Introducng Interor-Pont Methods for Introductory Operatons Research Courses and/or Lnear Programmng Courses, The Open Operatonal Research Journal, 009, 3, -. [4] K.Mehlhorn and S.Saxena, A Stll Smpler Way of Introducng the Interor-Pont Method for Lnear Programmng, vxra: and arxv: [cs.ds]. [5] C.Roos,T Terlaky and J-P Val, Interor Pont Methods for Lnear Optmzaton, nd Ed, 006, Sprnger. [6] Romesh Sagal, Lnear Programmng, A Modern Integrated Analyss, Kluwer, 995. [7] Tamas Terlaky, An easy way to teach nteror-pont methods, European J of Operatonal Research, 30 (00), -9 [8] R.J.Vanderbe, Lnear Programmng: Foundatons and Extenson, st Ed: Kluwer Academc Publshers, 997 (3rd Ed: Sprnger). (see also e.g. [, pp ], [, p75] or [6, pp 43-44]) 9

10 Appendx: Result from Algebra Assume that A s n m matrx and rank of A s n, wth n < m. Then all n rows of A are lnearly ndependent. Or α A + α A α n A n 0 (here 0 s a row vector of sze m) has only one soluton α 0. Thus, f x s any n matrx (a column vector of sze n), then xa 0 mples x 0. As A s n m matrx, A T wll be m n matrx. The product AA T wll be an n n square matrx. Let y T be an n matrx (or y s a row-vector of sze n). Consder the equaton (AA T )y T 0. Pre-multplyng by y we get yaa T y T 0 or (ya)(ya) T 0 or the dot product < ya,ya > 0 whch, for real vectors (matrces) means, that each term of ya s (ndvdually) zero, or y s dentcally zero. Thus, the matrx AA T has rank n and s nvertble. Also observe that f X s a dagonal matrx (wth all dagonal entres non-zero) and f A has full row-rank, then AX wll also have full row-rank. Bascally f entres of X are x,x,...,x n then the matrx AX wll have rows as x A,x A,...,x n A n (.e., th row of A gets scaled by x ). If rows of AX are not ndependent then there are βs (not all zero) such that: β x A + β x A β n x n A n 0, or there are αs (not all zero) such that: α A + α A α n A n 0 wth α β x. Arthmetc-Geometrc Inequalty follows from: (x + y) (x + y) (x y) 4xy or takng square roots (of the frst and the last) x + y xy 0