0.1 Gradient descent for convex functions: univariate case

Transcription

1 prnceton unv. F 16 cos 51: Advanced Algorthm Desgn Lecture 14: Gong wth the slope: offlne, onlne, and randomly Lecturer: Sanjeev Arora Scrbe:Sanjeev Arora hs lecture s about gradent descent, a popular method for contnuous optmzaton (especally nonlnear optmzaton). We start by recallng that allowng nonlnear constrants n optmzaton leads to NPhard problems n general. For nstance the followng sngle constrant can be used to force all varables to be 0/1. x (1 x ) = 0. Notce, ths constrant s nonconvex. We saw n earler lectures that the Ellpsod method can solve convex optmzaton problems effcently under farly general condtons. But t s slow n practce. Gradent descent s a popular alternatve because t s smple and t gves some knd of meanngful result for both convex and nonconvex optmzaton. It tres to mprove the functon value by movng n a drecton related to the gradent (.e., the frst dervatve). For convex optmzaton t gves the global optmum under farly general condtons. For nonconvex optmzaton t arrves at a local optmum. Fgure 1: optmum For nonconvex functons, a local optmum may be dfferent from the global We wll frst study unconstraned gradent descent where we are smply optmzng a functon f( ). Recall that the functon s convex f f(λx + (1 λ)y) λf(x) + (1 λ)f(y) for all x, y and λ [0, 1]. 0.1 Gradent descent for convex functons: unvarate case he gradent for a unvarate functon f s smply the dervatve: f (x). If ths s negatve, the value decreases f we ncrease x a lttle, and ncreases f we decrease f. Gradent descent conssts of evaluatng the dervatve and movng a small amount to the rght (.e., ncrease 1

2 x) f f (x) < 0 and to move to the left otherwse. hus the basc teraton s x x ηf (x) for a tny η called step sze. he functon s convex f between every two ponts x, y the graph of the functon les below the lne jonng (x, f(x)) and (y, f(y)). It need not be dfferentable everywhere but when all dervatves exst we can do the aylor expanson: f(x + η) = f(x) + ηf (x) + η f (x) + η3 3! f (x). (1) If f (x) 0 for all x then the the functon s convex. hs s because f (x) s an ncreasng functon of x. he mnmum s attaned for x where f (x) = 0 snce f (x) s +ve to the rght of t and ve to the left. hus movng both left and rght of ths pont ncreases f and t never drops. he functon s concave f f (x) 0 for all x; such functons have a unque maxmum. Examples of convex functons: ax + b for any a, b R; exp(ax) for any a R; x α for x 0, α 1 or α 0. Another nterestng example s the negatve entropy: x log x for x 0. Examples of concave functons: ax + b for any a, b R; x α for α [0, 1] and x 0; log x for x 0. Fgure : Concave and Convex Functon o mnmze a convex functon by gradent descent we start at some x 0 and at step update x to x +1 = x + ηf (x) for some small η < 0. In other words, move n the drecton where f decreases. If we gnore terms that nvolve η 3 or hgher, then f(x +1 ) = f(x ) + ηf (x ) + η f (x ). and the best value for η (whch gves the most reducton n one step) s η = f (x)/f (x), whch gves f(x +1 ) = f(x ) (f (x )) f (x ). hus the algorthm makes progress so long as f (x ) > 0. Convex functons that satsfy f (x) > 0 for all x are called strongly convex. he above calculaton s the man dea n Newton s method, whch you may have seen n calculus. Provng convergence requres further assumptons.

3 3 0. Convex multvarate functons A convex functon on R n, f t s dfferentable, satsfes the followng basc nequalty, whch says that the functon les above the tangent plane at any pont. f(x + z) f(x) + f(x) z x, y. () Here f(x) s the vector of frst order dervatves where the th coordnate s f/ x and called the gradent. Sometmes we restate t equvalently as f(x) f(y) f(x) (x y) x, z (3) 3.1 Basc propertes and examples 69 f(y) f(x)+ f(x) (y x) (x, f(x)) Fgure 3. If f s convex and dfferentable, then f(x)+ f(x) (y x) f(y) for all x, y dom f. Fgure 3: A dfferentable convex functon les above the tangent plane f(x)+ f(x) (y x) s gven by { 0 x C If hgher dervatves also Ĩ C(x) exst, = the x multvarate C. aylor expanson for an n-varate functon f s he convex functon ĨC s called the ndcator functon of the set C. f(x + y) = f(x) + f(x) y + y f(x)y +. (4) We can play several notatonal trcks wth the ndcator functon ĨC. For example the problem of mnmzng a functon f (defned on all of R n, say) on the set C s the same as mnmzng the functon f + ĨC over all of Rn. Indeed, the functon f + ĨC s (by our conventon) f restrcted to the set C. Here f(x) denotes the n n matrx whose, j entry s f/ x x j and t s called the Hessan. It can be checked that f s convex f the Hessan s postve semdefnte; ths In a smlar way we can extend a concave functon by defnng t to be outsde ts doman Frst-order condtons Suppose f s dfferentable (.e., ts gradent f exsts at each pont n dom f, whch s open). hen f s convex f and only f dom f s convex and f(y) f(x)+ f(x) (y x) (3.) holds for all x, y dom f. hs nequalty s llustrated n fgure 3.. he affne functon of y gven by f(x)+ f(x) (y x) s, of course, the frst-order aylor approxmaton of f near x. he nequalty (3.) states that for a convex functon, the frst-order aylor approxmaton Fgure s n fact 4: a global he underestmator Hessan of the functon. Conversely, f the frst-order aylor approxmaton of a functon s always a global underestmator of the functon, then the functon s convex. he nequalty (3.) shows that from local nformaton about a convex functon (.e., ts value and dervatve at a pont) we can derve global nformaton (.e., a global underestmator of t). hs s perhaps the most mportant property of convex functons, and explans some of the remarkable propertes of convex functons and convex optmzaton problems. As one smple example, the nequalty (3.) shows that f f(x) = 0, then for all y dom f, f(y) f(x),.e., x s a global mnmzer of the functon f. means y fy 0 for all y. Example 1 he followng are some examples of convex functons. Norms. Every l p norm s convex on R n. he reason s that a norm satsfes trangle nequalty: x + y x + y x, y.

4 4 f(x) = log(e x 1 + e x + + e xn ) s convex on R n. hs fact s used n practce as an analytc approxmaton of the max functon snce max {x 1,..., x n } f(x)+ max {x 1,..., x n } + log n. urns out ths fact s at the root of the multplcatve weght update method; the algorthm for approxmately solvng LPs that we saw n Lecture 10 can be seen as dong a gradent descent on ths functon, where the x s are the slacks of the lnear constrants. (For a lnear constrant a z b the slack s a z b.) f(x) = x Ax = j A jx x j where A s postve semdefnte. Its Hessan s A. Some mportant examples of concave functons are: geometrc mean ( n =1 x ) 1/n and logdetermnant (defned for X R n as log det(x) where X s nterpreted as an n n matrx). Many famous nequaltes n mathematcs (such as Cauchy-Schwartz) are derved usng convex functons. Example (Lnear equatons wth PSD constrant matrx) In lnear algebra you learnt that the method of choce to solve systems of equatons Ax = b s Gaussan elmnaton. In many practcal settngs ts O(n 3 ) runnng tme may be too hgh. Instead one does gradent descent on the functon 1 x Ax b x, whose local optmum satsfes Ax = b. If A s postve semdefnte ths functon s also convex snce the Hessan s A, and gradent descent wll actually fnd the soluton. (Actually n real lfe these are optmzed usng more advanced methods such as conjugate gradent.) Also, f A s dagonal domnant, a stronger constrant than PSD, then Spelman and eng (003) have shown how to solve ths problem n tme that s near lnear n the number of nonzero entres. hs has had surprsng applcatons to basc algorthmc problems lke max-flow. Example 3 (Least squares) In some settngs we are gven a set of ponts a 1, a,..., a m R n and some data values b 1, b,..., b m taken at these ponts by some functon of nterest. We suspect that the unknown functon s a lne, except the data values have a lttle error n them. One standard technque s to fnd a least squares ft: a lne that mnmzes the sum of squares of the dstance to the dataponts to the lne. he objectve functon s mn Ax b where A R m n s the matrx whose rows are the a s. (We saw n an earler lecture that the soluton s also the frst sngular vector.) hs objectve s just x A Ax (Ax) b+b b, whch s convex. In the unvarate case, gradent descent has a choce of only two drectons to move n: rght or left. In n dmensons, t can move n any drecton n R n. he most drect analog of the unvarate method s to move dametrcally opposte from the gradent. he most drect analogue of our unvarate analyss would be to assume a lowerbound of y fy for all y (n other words, a lowerbound on the egenvalues of f). hs wll be explored n the homework. In the rest of lecture we wll only assume ().

5 5 0.3 Gradent Descent for Constraned Optmzaton As studed n prevous lectures, constraned optmzaton conssts of solvng the followng where K s a convex set and f( ) s a convex functon. mn f(x) s.t. x K. Example 4 (Spam classfcaton va SVMs) hs example wll run through the entre lecture. Support Vector Machne s the name n machne learnng for a lnear classfer; we saw these before n Lecture 6 (Lnear hnkng). Suppose we wsh to tran the classfer to classfy emals as spam/nonspam. Each emal s represented usng a vector n R n that gves the frequences of varous words n t ( bag of words model). Say a 1, a,..., a N are the emals, and for each there s a correspondng bt b { 1, 1} where b = 1 means X s spam. SVMs use a lnear classfer to separate spam from nonspam. If spam were perfectly dentfable by a lnear classfer, there would be a functon W x such that W a 1 f a s spam, and W a 1 f not. In other words, 1 b W a 0 (5) Of course, n practce a lnear classfer makes errors, so we have to allow for the possblty that (5) s volated by some a s. he obvous thng to try s to fnd a W that satsfes as many of the constrants as possble, but that leads to a nonconvex NP-hard problem. (Even approxmatng ths weakly s NP-hard.) hus a more robust verson of ths problem s mn Loss(1 W (b a )) (6) W n (scalng constrant) where Loss( ) s a functon that penalzes unsatsfed constrants accordng to the amount by whch they are unsatsfed. (Note that W s the vector of varables, and the scalng constrant gves meanng to the separaton of 1 n (5) by sayng that W s a vector n the sphere of radus n, whch s a convex constrant.) he most obvous loss functon would be to count the number of unsatsfed constrants but that s nonconvex. For ths lecture we focus on convex loss functons; the smplest s the hnge loss: Loss(t) = max {0, t}. Applyng t to 1 W (b a ) nsures that correctly classfed emals contrbute 0 to the loss, and ncorrectly classfed emals contrbute as much to the loss as the amount by whch they fal the nequalty. he functon n (6) s convex because the functon nsde Loss() s lnear and thus convex, and Loss() preserves convexty snce t can only lft the value of the lnear functon even further. If x K s the current pont and we use the gradent to step to x η x then n general ths new pont wll not be n K. hus one needs to do a projecton. Defnton 1 he projecton of a pont y on K s x K that mnmzes y x. (It s also possble to use other norms than l to defne projectons.) A projecton oracle for the convex body a black box that, for every pont y, returns ts projecton on K.

6 6 Often convex sets used n applcatons are smple to project to. Example 5 If K = unt sphere, then the projecton of y s y/ y. Here s a smple algorthm for solvng the constraned optmzaton problem. he algorthm only needs to access f va a gradent oracle and K va a projecton oracle. Defnton (Gradent Oracle) A gradent oracle for a functon f s a black box that, for every pont z, returns f(z) the gradent valuated at pont z. (Notce, ths s a lnear functon of the form gx where g s the vector of partal dervatves evaluated at z.) he same value of η wll be used throughout. Gradent Descent for Constraned Optmzaton Let η == D G. Repeat for = 0 to y (+1) x () η f(x () ) x (+1) Projecton of y (+1) on K. At the end output z = 1 x(). Let us analyse ths algorthm as follows. Let x be the pont where the optmum s attaned. Let G denote an upperbound on f(x) for any x K, and let D = max x,y K x y be the so-called dameter of K. o ensure that the output z satsfes f(z) f(x ) + ɛ we wll use = 4D G. ɛ Snce x () s a projecton of y () on K we have x (+1) x y (+1) x = x () x η f(x () ) = x () x + η (f)(x () ) η f(x () ) (x () x ) Reorganzng and usng defnton of G we obtan: f(x () ) (x x () ) 1 η ( x () x x (+1) x ) + η G Usng (3), we can lowerbound the left hand sde by f(x () ) f(x ). We conclude that f(x () ) f(x ) 1 η ( x () x x (+1) x ) + η G. (7) Now sum the prevous nequalty over = 1,,..., and use the telescopng cancellatons to obtan =1 (f(x () ) f(x )) 1 η ( x (0) x x ( ) x ) + η G.

7 Fnally, by convexty f( 1 x() satsfes 1 x() ) 1 f(x() ) so we conclude that the pont z = f(z) f(z ) D η + η G. Now set η = D G DG to get an upperbound on the rght hand sde of. Snce = 4D G ɛ we see that f(z) f(x ) + ɛ Onlne Gradent Descent In onlne gradent descent we deal wth the followng scenaro. here s a convex set K gven va a projecton oracle. For = 1,,..., we are presented at step a convex functon f. At step we have to put forth our guess soluton x () K but the catch s that we do not know the functons that wll be presented n future. So our onlne decsons have to be made such that f x s the pont w that mnmzes f (w) (.e. the pont that we would have chosen n hndsght after all the functons were revealed) then the followng quantty (called regret) should stay small: f (x () ) f (x ). hs noton should remnd you of multplcatve weghts, except here we may have general convex functons as payoffs. Example 6 (Spam classfcaton aganst adaptve adversares) We return to the spam classfcaton problem of Example 4, wth the new twst that ths classfer changes over tme, as spammers learn to evade the current classfer. hus there s no fxed dstrbuton of spam emals and t s frutless to tran the classfer at one go. It s better to have t mprove and adapt tself as new emals arrve. At step t the optmum classfer f t may not be known and s presented usng a gradent oracle. hs functon just corresponds to the term n (6) correspondng to the latest emal that was classfed as spam/nonspam. he goal s to do as well as the best sngle classfer we would want to use n hndsght. Znkevch notced that the analyss of gradent descent apples to ths much more general scenaro. Specfcally, modfy the above gradent descent algorthm to ths problem by replacng f(x () ) by f (x () ). hs algorthm s called Onlne Gradent Descent. he earler analyss works essentally unchanged, once we realze that the left hand sde of (7) has the regret for tral. Summng over gves the total regret on the left sde, and the rght hand sde s analysed and upperbounded as before. hus we have shown: heorem 1 (Znkevch 003) If D s the dameter of K and G s an upperbound on the norm of the gradent of any of the presented functons, and η s set to then the regret per step after steps s at most DG. D G

8 Case Study: Onlne Shortest Paths he Onlne Shortest Paths problem models a commuter tryng to fnd the best path wth fewest traffc delays. he traffc pattern changes from day to day, and she wshes to have the smallest average delay over many days of expermentaton. We are gven a graph G = (V, E) and two nodes s, t. At each tme perod, the decson maker selects one path p from the set P s,t of all paths that connect s, t (the choce for the day s commute). hen, an adversary ndependently chooses a weght functon w : E R (the traffc delays). he decson maker ncurs a loss equal to the weght of the path he or she chose: e p w (e). he problem of fndng the best would be natural to consder ths problem n the context of expert advce. We could thnk of every element of P s,t as an expert and apply the multplcatve weghts algorthm we have seen before. here s one major flaw wth ths approach: there may be exponentally many paths connectng s, t n terms of the number of nodes n the graph. So the updates take exponental tme and space n each step, and furthermore the algorthm needs too long to converge to the best soluton. Onlne gradent descent can solve ths problem, once we realze that we can descrbe the set of all dstrbutons x over paths P s,t as a convex set K R m, wth O( E + V ) constrants. hen the decson maker s expected loss functon would be f (x) = w x. he followng formulaton of the problem as a convex polytope allows for effcent algorthms wth provable regret bounds. x e = x e = 1 Flow value s 1. e=(s,w),w V e=(w,t),w V w V, w u, v, e w x e = 0 Flow conservaton. e E, 0 x e 1 Capacty constrants. What s the meanng of the decson maker s move beng a dstrbuton over paths? It just means a fractonal soluton. hs can be decomposed nto a combnaton of paths as n the lecture on approxmaton algorthms. She pcks a random path from ths dstrbuton; the expected regret s unchanged. 0.5 Stochastc Gradent Descent Stochastc gradent descent s a varant of the algorthm n Secton 0.3 that works wth convex functons presented usng an even weaker noton: an expected gradent oracle. Gven a pont z, ths oracle returns a lnear functon gx + f that s drawn from a probablty dstrbuton D z such that the expectaton E g,f Dz [gx + f] s exactly the gradent of f at z. Example 7 (Spam classfcaton usng SGD) Returnng to the spam classfcaton problem of Example 4, we see that the functon n (6) s a sum of many smlar terms. If we randomly pck a sngle term and compute just ts gradent (whch s very quck to do!) then by lnearty of expectatons, the expectaton of ths gradent s just the true gradent. hus the expected gradent oracle may be a much faster computaton than the gradent oracle

9 9 (a mllon tmes faster f the number of emal examples s a mllon!). In fact ths settng s not atypcal; often the convex functon of nterest s a sum of many smlar terms. Stochastc gradent descent can be analysed usng Onlne Gradent Descent (OGD). Let g x be the gradent at step. hen we use ths functon whch s a lnear functon and hence convex as f n the th step of OGD. Let z = 1 =1 x(). Let x be the pont n K where f attans ts mnmum value. heorem E[f(z)] f(x ) + DG, where D s the dameter as before and G s an upperbound of the norm of any gradent vector ever output by the oracle. Proof: E[f(z) f(x )] 1 E[ (f(x () ) f(x ))] by convexty of f 1 E[ f(x () ) (x () x )] usng () = 1 E[g (x () x )] Snce expected gradent s the true gradent = 1 E[f (x ()) f (x )] Defn. of f = 1 E[ (f (x () ) f (x )] and the theorem now follows snce the expresson n the E[ ] s just the regret, whch s always upperbounded by the quantty gven n Znkevch s theorem, so the same upperbound holds also for the expectaton. 0.6 Portfolo Management va Onlne gradent descent (hs was actually covered at the start of Lecture 17) Let s return to the portfolo management problem dscussed n context of multplcatve weghts. We are tryng to nvest n a set of n stocks and maxmse our wealth. For t = 1,,..., let r (t) be the vector of relatve prce ncrease on day t, n other words r (t) = Prce of stock on day t Prce of stock on day t 1. Some thought shows (confrmng conventonal wsdom) that t can be very suboptmal to put all money n a sngle stock. A strategy that works better n practce s Constant Rebalanced Portfolo (CRB): decde upon a fxed proporton of money to put nto each stock, and buy/sell ndvdual stocks each day to mantan ths proporton.

10 10 Example 8 Say there are only two assets, stocks and bonds. One CRB strategy s to put splt money equally between these two. Notce what ths mples: f an asset s prce falls, you tend to buy more of t, and f the prce rses, you tend to sell t. hus ths strategy roughly mplements the age-old advce to buy low, sell hgh. Concretely, suppose the prces each day fluctuate as follows. Stock r (t) Bond r (t) Day 1 4/3 3/4 Day 3/4 4/3 Day 3 4/3 3/4 Day 4 3/4 4/ Note that the prces go up and down by the same rato on alternate days, so money parked fully n stocks or fully n bonds earns nothng n the long run. (Asde: hs knd of fluctuaton s not unusual; t s generally observed that bonds and stocks move n opposte drectons.) And what happens f you splt your money equally between these two assets? Each day t ncreases by a factor 0.5 (4/3 + 3/4) = 0.5 5/ hus your money grows exponentally! Exercse: Modfy the prce ncreases n the above example so that keepng all money n stocks or bonds alone wll cause t to drop exponentally, but the CRB ncreases money at an exponental rate. CRB uses a fxed splt among n assets, but what s ths splt? Wouldn t t be great to have an angel whsper n our ears on day 1 what ths magc splt s? Onlne optmzaton s precsely such an angel. Suppose the algorthm uses the vector x (t) at tme t; the th coordnate gves the proporton of money n stock at the start of the tth day. hen the algorthm s wealth ncreases on t by a factor r (t) x (t). hus the goal s to fnd x (t) s to maxmze the fnal wealth, whch s r (t) x (t). akng logs, ths becomes t log(r (t) x (t) ) (8) t For any fxed r (1), r (),... ths functon happens to be concave, but that s fne snce we are nterested n maxmzaton. Now we can try to run onlne gradent descent on ths objectve. By Znkevch s theorem, the quantty n (8) converges to where x s the best money allocaton n hndsght. log(r (t) x ) (9) t

11 11 hs analyss needs to assume very lttle about the r (t) s, except a bound on the norm of the gradent at each step, whch translates nto a weak condton on prce movements. In the next homework you wll apply ths smple algorthm on real stock data. 0.7 Hnts of more advanced deas Gradent descent algorthms come n dozens of flavors. (he Boyd-Vandenberghe book s a good resource. and Nesterov s lecture notes are terser but stll have a lot of ntuton.) We know the optmal runnng tme (.e., number of teratons) of gradent descent n the oracle model; see the books by Hazan and Bubeck. Surprsngly, just gong along the gradent (more precsely, dametrcally opposte drecton from gradent) s not always the best strategy. Steepest descent drecton s defned by quantfyng the best decrease n the objectve functon obtanable va a step of unt length. he catch s that dfferent norms can be used to defne unt length. For example, f dstance s measured usng l 1 norm, then the best reducton happens by pckng the largest coordnate of the gradent vector and reducng the correspondng coordnate n x (coordnate descent). he classcal Newton method s a subcase where dstance s measured usng the ellpsodal norm defned usng the Hessan. Gradent descent deas underle recent advances n algorthms for problems lke Spelman- eng style solver for Laplacan systems, near-lnear tme approxmaton algorthms for maxmum flow n undrected graphs, and Madry s faster algorthm for maxmum weght matchng. Bblography 1. Convex Optmzaton, S. Boyd and L. Vandenberghe. Cambrdge Unversty Press. (pdf avalable onlne.). Introductory Lectures on Convex Optmzaton: A Basc Course. Y. Nesterov. Sprnger Onlne Convex Programmng and Generalzed Infntesmal Gradent Ascent. M. Znkevch, ICML Book draft Onlne convex optmzaton. Elad Hazan. 5. Lecture notes on onlne optmzaton. S. Bubeck.