Ellipsoid Method. ellipsoid method. convergence proof. inequality constraints. feasibility problems. Prof. S. Boyd, EE392o, Stanford University

Transcription

1 Ellipsoid Method ellipsoid method convergence proof inequality constraints feasibility problems Prof. S. Boyd, EE392o, Stanford University

2 Challenges in cutting-plane methods can be difficult to compute appropriate next query point localization polyhedron grows in complexity as algorithm progresses can get around these challenges... ellipsoid method is another approach developed in 70s by Shor and Yudin used in 1979 by Khachian to give polynomial time algorithm for LP Prof. S. Boyd, EE392o, Stanford University 1

3 Ellipsoid algorithm idea: localize x in an ellipsoid instead of a polyhedron 1. at iteration k we know x E (k) 2. set x (k+1) := center(e (k) ); evaluate f(x (k+1) ) (or g (k) f(x (k+1) )) 3. hence we know (a half-ellipsoid) x E (k) {z f(x (k+1) ) T (z x (k+1) ) 0} 4. set E (k+1) := minimum volume ellipsoid covering E (k) {z f(x (k+1) ) T (z x (k+1) ) 0} Prof. S. Boyd, EE392o, Stanford University 2

4 E (k+1) x (k+1) E (k) f(x (k+1) ) compared to cutting-plane method: localization set doesn t grow more complicated easy to compute query point but, we add unnecessary points in step 4 Prof. S. Boyd, EE392o, Stanford University 3

5 Properties of ellipsoid method reduces to bisection for n = 1 simple formula for E (k+1) given E (k), f(x (k+1) ) E (k+1) can be larger than E (k) in diameter (max semi-axis length), but is always smaller in volume vol(e (k+1) ) < e 1 2n vol(e (k) ) (note that volume reduction factor depends on n) Prof. S. Boyd, EE392o, Stanford University 4

6 Example x (0) x (1) x (2) lacements Prof. S. Boyd, EE392o, Stanford University 5

7 x (3) x (4) x (5) cements Prof. S. Boyd, EE392o, Stanford University 6

8 Updating the ellipsoid E(x, A) = { z (z x) T A 1 (z x) 1 } E E + x g x + Prof. S. Boyd, EE392o, Stanford University 7

9 (for n > 1) minimum volume ellipsoid containing is given by E { z g T (z x) 0 } x + = x 1 n + 1 A g A + = n 2 n 2 1 ( A 2 n + 1 A g gt A ) where g = g / gt Ag Prof. S. Boyd, EE392o, Stanford University 8

10 Stopping criterion x E k, so simple stopping criterion: f(x ) f(x (k) ) + f(x (k) ) T (x x (k) ) f(x (k) ) + inf f(x (k) ) T (x x (k) ) x E (k) = f(x (k) ) f(x (k) ) T A (k) f(x (k) ) f(x (k) ) T A (k) f(x (k) ) ɛ Prof. S. Boyd, EE392o, Stanford University 9

11 f f(x (k) ) f(x (k) ) f(x (k) ) T A (k) f(x (k) ) k Prof. S. Boyd, EE392o, Stanford University 10

12 more sophisticated stopping criterion: U k L k ɛ, where U k = min L k = max i k i k f(x(i) ) ( f(x (i) ) ) f(x (i) ) T A (i) f(x (i) ) Prof. S. Boyd, EE392o, Stanford University 11

13 f U k L k k Prof. S. Boyd, EE392o, Stanford University 12

14 Basic ellipsoid algorithm ellipsoid described as E(x, A) = { z (z x) T A 1 (z x) 1 } given ellipsoid E(x, A) containing x, accuracy ɛ > 0 repeat 1. evaluate f(x) (or g f(x)) 2. if f(x) T A f(x) ɛ, return(x) 3. update ellipsoid 3a. g := f(x) / f(x)t A f(x) 3b. x := x 1 3c. A := n2 n 2 1 n+1 A g ( ) A 2 n+1 A g gt A Prof. S. Boyd, EE392o, Stanford University 13

15 Interpretation change coordinates so uncertainty (E) is unit ball take gradient (or subgradient) step with fixed length 1/(n + 1) properties: can propagate Cholesky factor of A; get O(n 2 ) update not a descent method often slow but robust in practice Prof. S. Boyd, EE392o, Stanford University 14

16 Proof of convergence assumptions: f is Lipschitz: f(y) f(x) G y x E (0) is ball with radius R suppose f(x (i) ) > f + ɛ, i = 0,..., k then f(x) f + ɛ = x E (k) since at iteration i we only discard points with f f(x (i) ) Prof. S. Boyd, EE392o, Stanford University 15

17 from Lipschitz condition, x x ɛ/g = f(x) f + ɛ = x E (k) so B = {x x x ɛ/g} E (k) hence vol(b) vol(e (k) ), so β n (ɛ/g) n e k/2n vol(e (0) ) = e k/2n β n R n (β n is volume of unit ball in R n ) therefore k 2n 2 log(rg/ɛ) Prof. S. Boyd, EE392o, Stanford University 16

18 E (0) x x (k) E (k) B = {x x x ɛ/g} f(x) f + ɛ conclusion: for K > 2n 2 log(rg/ɛ), min i=0,...,k f(x(i) ) f + ɛ Prof. S. Boyd, EE392o, Stanford University 17

19 Interpretation of complexity since x E 0 = {x x x (0) R}, our prior knowledge of f is f [f(x (0) ) GR, f(x (0) )] our prior uncertainty in f is GR after k iterations our knowledge of f is f [ ] min i=0,...,k f(x(i) ) ɛ, min i=0,...,k f(x(i) ) posterior uncertainty in f is ɛ Prof. S. Boyd, EE392o, Stanford University 18

20 iterations required: 2n 2 log RG ɛ = 2n 2 log prior uncertainty posterior uncertainty efficiency: 0.72/n 2 bits per gradient evaluation (degrades with n) Prof. S. Boyd, EE392o, Stanford University 19

21 Inequality constrained problems minimize subject to f 0 (x) f i (x) 0, i = 1,..., m same idea: maintain ellipsoids E (k) that contain x decrease in volume to zero Prof. S. Boyd, EE392o, Stanford University 20

22 case 1: x (k) feasible, i.e., f i (x (k) ) 0, i = 1,..., m then do usual update of E (k) based on f 0 (x (k) ) rules out halfspace of points with larger function value than current point case 2: x (k) infeasible, say, f j (x (k) ) > 0; then f j (x (k) ) T (x x (k) ) 0 = f j (x) > 0 = x infeasible so update E (k) based on f j (x (k) ) rules out halfspace of infeasible points Prof. S. Boyd, EE392o, Stanford University 21

23 Example f 1 (x) = 0 lacements x (0) x (1) f 0(x (1) ) f 0 (x (2) ) x (2) f 1 (x (0) ) Prof. S. Boyd, EE392o, Stanford University 22

24 cements x (3) f 0 (x (4) ) x (4) x (5) f 0 (x (5) ) f 1 (x (3) ) Prof. S. Boyd, EE392o, Stanford University 23

25 Stopping criterion if x (k) is feasible, we have a lower bound on f as before: f f(x (k) ) f(x (k) ) T A (k) f(x (k) ) if x (k) is infeasible, we have for all x E (k) f j (x) f j (x (k) ) + f j (x (k) ) T (x x (k) ) f j (x (k) ) + inf x E (k) f j (x (k) ) T (x x (k) ) = f j (x (k) ) f j (x (k) ) T A (k) f j (x (k) ) Prof. S. Boyd, EE392o, Stanford University 24

26 hence, problem is infeasible if for some j, f j (x (k) ) f j (x (k) ) T A (k) f j (x (k) ) > 0 stopping criteria: if x (k) is feasible and f 0 (x (k) ) T A (k) f 0 (x (k) ) ɛ (x (k) is ɛ-suboptimal) if f j (x (k) ) f j (x (k) ) T A (k) f j (x (k) ) > 0 (problem is infeasible) Prof. S. Boyd, EE392o, Stanford University 25

27 Ellipsoid method for feasibility abstract feasibility problem: find x C R n or determine C = separating hyperplane oracle: for any x, oracle either confirms x C, or returns g 0 s.t. z C g T (z x) 0 E (k+1) x (k) g (k) E (k) C Prof. S. Boyd, EE392o, Stanford University 26

28 start with E (0) which intersects C 1. If x (k) := center(e (k) ) C, quit. Else, compute g 0, s.t. x C g T (x x (k) ) 0 2. E (k+1) := minimum volume ellipsoid covering E (k) {z g T (z x (k) ) 0} Prof. S. Boyd, EE392o, Stanford University 27

29 Example Sfrag replacements Sfrag replacements Prof. S. Boyd, EE392o, Stanford University 28