Ellipsoid Method. ellipsoid method. convergence proof. inequality constraints. feasibility problems. Prof. S. Boyd, EE364b, Stanford University

Transcription

1 Ellipsoid Method ellipsoid method convergence proof inequality constraints feasibility problems Prof. S. Boyd, EE364b, Stanford University

2 Ellipsoid method developed by Shor, Nemirovsky, Yudin in 1970s used in 1979 by Khachian to show polynomial solvability of LPs each step requires cutting-plane or subgradient evaluation modest storage (O(n 2 )) modest computation per step (O(n 2 )), via analytical formula efficient in theory; slow but steady in practice Prof. S. Boyd, EE364b, Stanford University 1

3 Motivation in cutting-plane methods serious computation is needed to find next query point (typically O(n 2 m), with not small constant) localization polyhedron grows in complexity as algorithm progresses (we can, however, prune constraints to keep m proportional to n, e.g., m = 4n) ellipsoid method addresses both issues, but retains theoretical efficiency Prof. S. Boyd, EE364b, Stanford University 2

4 Ellipsoid algorithm for minimizing convex function idea: localize x in an ellipsoid instead of a polyhedron 1. at iteration k we know x E (k) 2. set x (k+1) := center(e (k) ); evaluate g (k) f(x (k+1) ) (g (k) = f(x (k) ) if f is differentiable) 3. hence we know (a half-ellipsoid) x E (k) {z g (k+1)t (z x (k+1) ) 0} 4. set E (k+1) := minimum volume ellipsoid covering E (k) {z g (k+1)t (z x (k+1) ) 0} Prof. S. Boyd, EE364b, Stanford University 3

5 g (k+1) E (k) E (k+1) x (k+1) compared to cutting-plane methods: localization set doesn t grow more complicated easy to compute query point but, we add unnecessary points in step 4 Prof. S. Boyd, EE364b, Stanford University 4

6 Properties of ellipsoid method reduces to bisection for n = 1 simple formula for E (k+1) given E (k), g (k+1) E (k+1) can be larger than E (k) in diameter (max semi-axis length), but is always smaller in volume vol(e (k+1) ) < e 1 2n vol(e (k) ) (volume reduction factor degrades rapidly with n, compared to CG or MVE cutting-plane methods) Prof. S. Boyd, EE364b, Stanford University 5

7 Example x (0) x (1) x (2) Prof. S. Boyd, EE364b, Stanford University 6

8 x (3) x (4) x (5) Prof. S. Boyd, EE364b, Stanford University 7

9 Updating the ellipsoid E(x, P) = { z (z x) T P 1 (z x) 1 } E E + x + x g Prof. S. Boyd, EE364b, Stanford University 8

10 (for n > 1) minimum volume ellipsoid containing half-ellipsoid is given by E { z g T (z x) 0 } x + = x 1 n + 1 P g P + = where g = (1/ g T Pg)g n 2 n 2 1 ( P 2 n + 1 P g gt P ) Prof. S. Boyd, EE364b, Stanford University 9

11 Simple stopping criterion f(x ) f(x (k) ) + g (k)t (x x (k) ) f(x (k) ) + inf z E (k) g (k)t (z x (k) ) = f(x (k) ) g (k)t P (k) g (k) second inequality holds since x E k simple stopping criterion: g (k)t P (k) g (k) ǫ = f(x (k) ) f(x ) ǫ Prof. S. Boyd, EE364b, Stanford University 10

12 Basic ellipsoid algorithm ellipsoid described as E(x, P) = {z (z x) T P 1 (z x) 1} given ellipsoid E(x, P) containing x, accuracy ǫ > 0 repeat 1. evaluate g f(x) 2. if g T Pg ǫ, return(x) 3. update ellipsoid 3a. g := g 1 g T Pg 3b. x := x 1 n+1 P g 3c. P := n2 n 2 1 ( P 2 n+1 P g gt P ) Prof. S. Boyd, EE364b, Stanford University 11

13 Interpretation change coordinates so uncertainty is isotropic (same in all directions), i.e., E is unit ball take subgradient step with fixed length 1/(n + 1) Shor calls ellipsoid method gradient method with space dilation in direction of gradient (which, strangely enough, didn t catch on) Prof. S. Boyd, EE364b, Stanford University 12

14 Example PWL function f(x) = max m i=1 (at i x + b i), with n = 20, m = f 0 f(x(k) ) f(x (k) ) p g (k)t P (k) g (k) k Prof. S. Boyd, EE364b, Stanford University 13

15 f (k) best f k Prof. S. Boyd, EE364b, Stanford University 14

16 Improvements keep track of best upper and lower bounds: u k = min i=1,...,k f(x(i) ), stop when u k l k ǫ l k = max (f(x (i) ) ) g (i)t P (i) g (i) i=1,...,k can propagate Cholesky factor of P (avoids problem of P 0 due to numerical roundoff) Prof. S. Boyd, EE364b, Stanford University 15

17 3 2 f 1 U k 0 L k k Prof. S. Boyd, EE364b, Stanford University 16

18 Proof of convergence assumptions: f is Lipschitz: f(y) f(x) G y x E (0) is ball with radius R suppose f(x (i) ) > f + ǫ, i = 0,...,k then f(x) f + ǫ = x E (k) since at iteration i we only discard points with f f(x (i) ) Prof. S. Boyd, EE364b, Stanford University 17

19 from Lipschitz condition, x x ǫ/g = f(x) f + ǫ = x E (k) so B = {x x x ǫ/g} E (k) hence vol(b) vol(e (k) ), so α n (ǫ/g) n e k/2n vol(e (0) ) = e k/2n α n R n (α n is volume of unit ball in R n ) therefore k 2n 2 log(rg/ǫ) Prof. S. Boyd, EE364b, Stanford University 18

20 E (0) x x (k) E (k) B = {x x x ǫ/g} f(x) f + ǫ conclusion: for k > 2n 2 log(rg/ǫ), min i=0,...,k f(x(i) ) f + ǫ Prof. S. Boyd, EE364b, Stanford University 19

21 Interpretation of complexity since x E 0 = {x x x (0) R}, our prior knowledge of f is f [f(x (0) ) GR, f(x (0) )] our prior uncertainty in f is GR after k iterations our knowledge of f is f [ ] min i=0,...,k f(x(i) ) ǫ, min i=0,...,k f(x(i) ) posterior uncertainty in f is ǫ Prof. S. Boyd, EE364b, Stanford University 20

22 iterations required: 2n 2 log RG ǫ = 2n 2 log prior uncertainty posterior uncertainty efficiency: 0.72/n 2 bits per gradient evaluation Prof. S. Boyd, EE364b, Stanford University 21

23 Deep cut ellipsoid method minimum volume ellipsoid containing ellipsoid intersected with halfspace with h 0, is given by x + E { z g T (z x) + h 0 } = x 1 + αn n + 1 P g P + = n2 (1 α 2 ) n 2 1 ( P ) 2(1 + αn) (n + 1)(1 + α) P g gt P where g = g gt Pg, α = h gt Pg (if α > 1, intersection is empty) Prof. S. Boyd, EE364b, Stanford University 22

24 Ellipsoid method with deep objective cuts 10 0 deep cuts shallow cuts 10 1 f (k) best f k Prof. S. Boyd, EE364b, Stanford University 23

25 Inequality constrained problems minimize f 0 (x) subject to f i (x) 0, i = 1,...,m if x (k) feasible, update ellipsoid with objective cut g T 0 (z x (k) ) + f 0 (x (k) ) f (k) best 0, g 0 f 0 (x (k) ) f (k) best is best objective value of feasible iterates so far if x (k) infeasible, update ellipsoid with feasibility cut assuming f j (x (k) ) > 0 g T j (z x (k) ) + f j (x (k) ) 0, g j f j (x (k) ) Prof. S. Boyd, EE364b, Stanford University 24

26 Stopping criterion if x (k) is feasible, we have lower bound on p as before: p f 0 (x (k) ) g (k)t 0 P (k) g (k) 0 if x (k) is infeasible, we have for all x E (k) f j (x) f j (x (k) ) + g (k)t j (x x (k) ) f j (x (k) ) + inf z E (k) g (k)t (z x (k) ) = f j (x (k) ) g (k)t j P (k) g (k) j Prof. S. Boyd, EE364b, Stanford University 25

27 hence, problem is infeasible if for some j, f j (x (k) ) g (k)t j P (k) g (k) j > 0 stopping criteria: if x (k) is feasible and if f j (x (k) ) g (k)t 0 P (k) g (k) 0 ǫ (x (k) is ǫ-suboptimal) g (k)t j P (k) g (k) j > 0 (problem is infeasible) Prof. S. Boyd, EE364b, Stanford University 26

28 Epigraph ellipsoid method use deep cut ellipsoid method to solve problem with variables (x, t) minimize t subject to f 0 (x) t, f i (x) 0, i = 1,...,m when (x (k), t (k) ) infeasible for epigraph problem, use standard deep feasibility cut if f 0 (x (k) ) > t (k), use cut t g T 0 (x x (k) ) + f 0 (x (k) ) if f j (x (k) ) > 0, use cut g T j (x x(k) ) + f j (x (k) ) 0 when (x (k), t (k) ) feasible for epigraph problem, use cut t f 0 (x (k) ) Prof. S. Boyd, EE364b, Stanford University 27

29 Epigraph ellipsoid example 10 0 epigraph method non-epigraph deep cuts 10 1 f (k) best f k Prof. S. Boyd, EE364b, Stanford University 28