2.1 Rademacher Calculus... 3

Transcription

1 COS 598E: Unsupervsed Learnng Week 2 Lecturer: Elad Hazan Scrbe: Kran Vodrahall Contents 1 Introducton 1 2 Non-generatve pproach Rademacher Calculus Spectral utoencoders 4 1 Introducton What I want to do today s frst contnue wth some theoretcal observatons about unsupervsed learnng, and prove generalzaton bounds (non-generatve) for unsupervsed learnng. Ths s as far as I can tell the only way to gve generalzaton error. Last tme we talked about unsupervsed learnng and phlosophy, and we talked about a specfc defnton. We wll come up wth convex relaxatons and effcent algorthms for solvng the problem n ths model. 2 Non-generatve pproach Consder a hypothess class composed of two functons (f, g) H, one s a decoder and the other s an encoder. We measure reconstructon error by E x D [x g f(x)]. If ths value s zero, we ddn t lose any nformaton. We have D s an unknown dstrbuton, lke n the PC model. famly of encoder-decoders s learnable f we can gve a bound on the number of examples requred to get generalzaton error whch s small. We defne the loss L D (h) = E x D [ x h(x) ] (1) Just lke n PC learnng, we take a sample S D m where S = {x 1,, x m }, and we defne sample loss L S (h) = E x S [ x h(x) ]. It s learnable f there exsts an (effcent) algorthm where after m(ɛ, δ) samples t fnds h s.t. wth probablty 1 δ, L D ( h) L D (h )+ɛ. Ths vew looks at learnng as compresson wth low reconstructon error. We can sometmes add a term to the loss dependent on the number of bts requred to express f(x). Now, how do we analyze PC n ths framework? What knd of guarantees can we hope to prove for PC n terms of generalzaton? Theorem 2.1. PC s learnable wth sample complexty d m. Ths d can be mproved to k (k-pc), x R d, but we won t show ths. Therefore, m(ɛ, δ) d ɛ 2. We omt the 1/δ. Proof. Last tme, we defned Rademacher complexty. 1

2 Defnton 2.2. Rademacher complexty. Consder a famly of mappngs F : X R. The Rademacher complexty s R S (F ) = E σ1,,σ m[sup 1 m m {+1, 1} σ f(x )] (2) f F m and =1 R m [F ] = E S =m [R S (F )] (3) If H : X X, loss l : X R, F = H l, then h H, log 1/δ L S (h) L D (h) + 2R S (H) + O( m ) (4) wth probablty 1 δ. Now how does ths help us? Example 2.3. Consder hypothess class H = {w w T x} of lnear separators. Suppose we bound w 2 B wth x 2 r (note that ths mples that the dstrbuton s bounded by the sphere). Then we clam that Theorem 2.4. R S (H) rb m The man thng that wll gve us somethng new s lookng at polynomal kernels rather than lnear kernels later on. Proof. mr S (H) = E σ1,,σ m [sup σ w T x ] w ( = E σ1,,σ m [sup w T σ x )] w E σ1,,σ m [sup w 2 σ x 2 ][ by Cauchy-Schwarz] w B E σ1,,σ m [ σ x ] ( B E[ 1/2 σ x ]) 2 [ by Jensen] = B E σ1,,σ m [ σ σ j x T x j],j (5) Note that lne 6 happens = B x 2 B mr 2 Note that n ths course, we care about real-valued functons, not 0 1 loss as much (n that case t s bounded by VC-dmenson whch s bounded by dmenson of the space). To reterate, n the case that the numbers become large, the loss can also become large t makes sense you need more samples. But f you have a bound on the sze, than the loss becomes manageable as well. 2

3 2.1 Rademacher Calculus Ths wll help us get generalzaton bounds beyond 0 1 losses. Let R m. Thnk of each a as a vector whch has the losses of all x : [l(h(x 1 )), l(h(x 2 )),, l(h(x m ))]. Defne Defnton 2.5. R() = E σ1,,σ m [sup α a ] (6) a Now consder what happens f you take R(c 0 +c 1 )? If you just add a constant, nothng changes n expectaton (±1 n expectaton s 0). However, multplyng mght do somethng: R(c 0 + c 1 ) = c 0 R(), assumng that σ a s non-negatve, or that for every postve a j there s a negatve a j (symmetrc). Lemma 2.6. Talagrand s Lemma. Let l : R R whch s ρ-lpschtz ( l (x) l (y) ρ(x y)). Ths s satsfed f l s dfferentable and ts dervatve s bounded by ρ. But the Lpschtz property s more general, and doesn t requre dfferentablty, only contnuty. Then, let l () = {(l (a 1 ), l (a 2 ),, l (a m )) a }. Consder the example l (w T x) = (w T x y) 2. Then, we have that R(l ()) ρr() (7) Note that l need not be convex! Ths s a useful property to know to get generalzaton error bounds. Now we wll prove generalzaton error bounds for PC. What s the hypothess class H for PC? It s the set of all pars (f, g) ndexed by such that f (x) = x, and g (y) = 1 y, wth R k d. We have l(h, x) = x 1 x 2 = x g f(x) 2 2, the reconstructon error. Now the queston we want to ask s how many examples do we need to see before we can characterze the reconstructon error of ths class? We also assume x 2 1 for smplcty. Theorem 2.7. R S (Hk P C )?. Proof. m R S (Hk P C ) = E σ s[sup = E σ [sup = E σ [sup E σ [sup Tr m σ x 1 x 2 ] =1 ( σ x T I 1 ) 2 x ] σ Tr( ( I 1 ) 2 x x T )] ( (I 1 ) ) 2 σ x x T ] (8) 3

4 Then note that, B = T rb op B so by Holder s nequalty (any prmal dual norm par). Here we take op = Frobenus and as the Frobenus norm (ths s just Cauchy- Schwarz). E σ [sup I 1 F σ x x T F ] de σ [ σ x x T ] ( ) 2] de σ [ σ x x T 1/2 = d E σ [ σ σ j x, x j 2 ],j (9) = d x 2 = md You can optmze the norm pars correctly to get k nstead of d. Now for all L S (h ) L D (h ) 2R m (Hk P C log 1/δ ) +. ɛ Now why s ths nterestng? We ve proved somethng stronger than generalzaton bound; we have agnostc learnng. You re competng wth best lnear encodng for ths data (even f your data s not from the subspace). 3 Spectral utoencoders In PC, we have x x. Instead, let s consder a polynomal verson of ths: x [p 1 (x), p 2 (x),, p k (x)]. Quadratc would mean take x 2 = [x 1,, x d, x 1 x 2,, x x j, ]. Then R k (d2 /2+d). Now how do you defne an unsupervsed learnng problem that can learn a manfold of square degree? We can try to learn the best encodng-decodng wth mnmum reconstructon error. Let s consder the hypothess class parametrzed by matrces H = {, f (x) = x 2, g(y) = v max ( 1 y)}. Even more natural would be to pck g(y) = argmn x R d x 2 y 2 2. There are many thngs you can thnk of, and these gve rse to nonconvex optmzaton problems whch are hard to globally optmze. Note that before you take the top egenvector, you have to do a reshapng operaton on 1 y n order to get a matrx. (ntuton here s that the matrx form of x 2 s xx T, whch has top egenvector x. We also have a lnear map n ths case whch smears thngs, t turns out you can prove that ths approach s robust to nose). Theorem 3.1. H s learnable wth sample complexty dk ɛ 2 usng a smlar approach. The man queston s can we come up wth effcent algorthms to solve these problems. Ths s where convex relaxaton can help. Theorem 3.2. H s effcently learnable wth sample complexty O( d2 k ɛ 4 ). Ths theorem doesn t contradct the NP-hardness of the problem, snce we re allowed to generate a hypothess whch does not come from H. 4

5 The roadmap for provng ths theorem s (1): Gve class Ĥ whch contans H, whch has sample complexty n the form of O(d 2 k/ɛ 4 ). Then, (2) effcently learn Ĥ. Let s defne H = {f, g, f(x) = x, g(y) = By} where we don t restrct B to be the nverse (relaxng the set). Our loss wll be l(h, x) = x 2 Bx 2 2 spectral. Ths loss functon s a relaxaton of our prevous loss. It turns out that x v max ( 1 x 2 ) 2 2 l(h, x) (10) so f we mnmze l(h, x), we mnmze the other too. Now to get ths effcent thng, we have to prove sample complexty for Ĥ you could do Rademacher, but that s knd of non-ntutve. However n ths case, we can get smultaneous generalzaton and optmzaton, va regret mnmzaton. We have Ĥ, a set of hypotheses h, and loss functons l x(h) = x 2 Bx Now, we are gong to do a further relaxaton and call h B = h D, where D = B R d2 d 2. nd the way we can relax s say D k (bound the trace norm). We have l(h 0 ) s convex n D. {D} such that Tr(D) k s also convex. Both condtons must hold for dong smultaneous optmzaton and generalzaton. We defne an algorthm: 1. 0 > D 1, for t = 1, 2, 3,, T, do: 2. sample x D, denote l x (D) = l t (D). ] 3. update: D t+1 = Π K [D t 1 t l t (D t ) 4. return D = 1 T T t=1 D t, the average over all teratons. where Π s the projecton operaton wth respect to Eucldean norm. Theorem 3.3. Onlne Gradent Descent. lt (D t ) mn lt (D t ) 2d2 κ (11) D K T where κ s the rank of the matrx and d s the dmenson. s a corollary, Theorem 3.4. E[L G ( D)] mn L G(h G ) + 2d2 κ (12) h G H T 5

6 Proof. E[L G ( D)] = E x G [l x ( D)] (13) = E x G [l x ( 1 T Dt )] (14) E x [ 1 T lt (D t )] (15) E x [l x (h )] + 2d2 κ T (16) = mn L G(h ) + 2d2 κ (17) h H T snce loss functons are convex, we can apply Jensen. Lne 4 uses the OGD theorem. Ths holds for any h snce regert s worst-case, whch s why we get the mnmum h over H. Every tme you choose example x, t s ndependent of the D t. You choose x from dstrbuton. So you have to wrte a summaton of condtonal expectatons and then do a martngale analyss, but what we have wrtten s precse because t comes out to the same thng. So far, we have dscussed one approach to unsupervsed learnng: compresson based, and we ve argued why t makes sense, and how you can get generalzaton error bounds from these defntons whch are non-standard for unsupervsed learnng. We saw how to get these from Rademacher complexty, and dscussed how to do t for PC and spectral autoencoders and generalzaton by optmzaton. One of the strong ponts of ths framework s that you can use convex relaxaton. Ths has been done for other classes of unsupervsed learnng, lke dctonary learnng. 6