Supplementary material for Non-conjugate Variational Message Passing for Multinomial and Binary Regression

Transcription

1 Supplementary materal for Non-conjugate Varatonal Message Passng for Multnomal and Bnary Regresson October 9, Alternatve dervaton We wll focus on a partcular factor f a and varable x, wth the am of calculatng an exponental famly message m a (x ; φ), parametersed by the natural parameter φ. Consder the local exclusve KL wth exponental famly q (x ; θ), wth natural parameter θ. Here q \a (x) s the cavty dstrbuton and q \ (x \ ) j q j(x j ) s the current varatonal dstrbuton over varables other than x. KL(q (x ; θ)q \ (x \ ) f a (x)q \a (x)) q (x ; θ) log q (x ; θ)dx (1) q (x ; θ)q \ (x \ ) log f a (x)dx () q (x ; θ)q \ (x \ ) log q \a (x)dx + const. (3) The cavty dstrbuton tself factorses as q \a (x) q (x ; θ \a )q \a, (x \ ), where q (x ; θ \a ) s the product of all the other ncomng messages to x. KL(θ) H[q (x ; θ)] (4) q (x ; θ) log f a (x) q(x )dx (5) q (x ; θ) log q (x ; θ \a )dx + const. (6) θ T κ κ κ(θ) S(θ) θ\a + κ(θ\a ) + const. (7) where we have used the fact that the expectaton of the suffcent statstcs of an exponental famly are gven by the dervatves of κ. The varatonal posteror q (x ; θ) wll be updated to m a (x ; φ)q (x ; θ \a ), so we have the relatonshp 1

2 θ θ \a + φ. We assume that θ \a s fxed (whch s at least true once the algorthm has converged), so dfferentatng wrt to θ and φ s equvalent: KL φ KL κ H(θ)θ + κ S(θ) H(θ)θ \a (8) H(θ)φ S(θ) where H(θ) s the Hessan of κ(θ). Settng ths dervatve to zero corresponds to a fxed pont scheme for φ, and recovers the update for φ, the gradent matchng scheme for an exponental famly message. NCVMP as moment matchng Gradent matchng can be seen as analogous to moment matchng n EP. The gradent of the true S s S(θ) q (x ; θ) log f a (x) q(x )dx log q (x ; θ) q (x ; θ) log f a (x) q(x )dx (u(x ) u(x ) q(x ;θ))q (x ; θ) log f a (x) q(x )dx. (9) Whereas the gradent of the approxmate S s S(θ, φ) q (x ; θ) log m a (x ; φ)dx log q (x ; θ) q (x ; θ) log m a (x ; φ)dx (u(x ) u(x ) q(x ;θ))q (x ; θ) log m a (x ; φ)dx. We see that matchng gradents s equvalent to matchng moments of the true and approxmate log factors, gven the current varatonal posteror. 3 NCVMP s parametersaton nvarant NCVMP s based on matchng gradents at the current estmate S(θ; φ) S(θ) θθ (t) θθ (t)

3 Now f we reparameterse n terms of ψ wth a bjectve mappng θ g(ψ) then we would work n terms of S ψ (ψ) S(g(ψ)) and S ψ (ψ; φ) S(g(ψ); φ): S ψ (ψ; φ) S(g(ψ); φ) S(θ; φ) S ψ(ψ) S(g(ψ)) S(θ) The Jacoban matrx s full rank snce g s bjectve, so the orgnal gradent matchng scheme s recovered. 4 Optmsng the varatonal parameter for the quadratc softmax bound The quadratc softmax bound s log e x k x k a t k a + + λ(t k )[(x k a) t k] log σ( t k ) (10) [ ] where t are new varatonal parameters and λ(t) 1 1 t 1+e 1 t. Takng the expectaton wrt to x we have log e x k m k a t k F (a) a + + λ(t k )[(m k a) + v k t k] log σ( t k ) (11) Settng the dervatves of wrt a and t equal to zero gves the followng fxed pont updates for a and t to make the bound as tght as possble: a K m kλ(t k ) + K/ 1 K λ(t k) (1) t k m k + v k m k a + a k (13) For small dmensonalty and counts these fxed pont teratons converge very fast. However, for large counts and dmensonalty K we found that the couplng between t and α was very strong and co-ordnate-wse optmzaton was hghly neffcent. In ths regme an effectve soluton s to substtute the expresson for t k n Equaton 13 nto the objectve functon to gve a unvarate optmzaton problem n α, whch can solved effcently usng Newton s method. See the supplementary materal for detals. The overall bound for the factor s [ ] xk a t k log f(d x) d k x k a + λ(t k )[(x k a) t k] log σ( t k ) (14) 3

4 Calculatng the messages to x and p, and the evdence, are now conjugate operatons. The message to x k wll have precson Sλ(t k ) and mean tmes precson d k 1 ( 1 aλ(t k)). At the mnmum we have t k (a) (m k a) + v k (15) Usng ths expresson we can smplfy Equaton 11 to get F (a) a + m k a t k log σ( t k ) (16) The dervatves of t k wrt a are Usng the chan rule we now fnd: t k(a) (m k a)/t k (a) (17) t k(a) 1/t k (a) (m k a) /t k (a) 3 (18) F (a) 1 + k (1 + t k(a))/ + t k(a)σ(t k (a)) (19) F (a) k t k(a)(σ(t k (a)).5) + t k(a) σ(t k (a))σ( t k (a)) (0) We can then use a Newton algorthm wth LM lne search to cope wth small F (a). 5 Dervaton of tlted bound The tlted bound can be derved as follows, analogously to the unvarate bound log e x log e j ajxj e j ajxj e x (1) j a j m + log e x j ajxj () 1 a jv j + log j e m+(1 a)v/ : T (m, v, a) (3) Takng dervatves wrt a k gves ak T (m, v, a) a k v k v k σ k [ m + 1 (1 a) v ] Settng ths expresson equal to zero results n the fxed pont update a : σ [m + 1 ] (1 a) v (4) (5) 4

5 6 Taylor seres expanson for log-sum-exp We can use a Taylor seres expanson about the mean of x. Ths wll not gve a bound, but may be more accurate and s cheap to compute. log e x log e m + (x m )σ (m) + 1 (x m ) σ (m)[1 σ (m)] We gnore the cross terms of the Hessan because we are usng a fully factorsed varatonal posteror for x. Takng expectatons we fnd log e x q log e m + 1 v σ (m)[1 σ (m)] (6) Ths approxmaton s smlar n sprt to Laplace s approxmaton, expect that we calculate the curvature around an approxmaton mean (calculated usng VMP) rather than the MAP. Usng the notaton n the paper the messages to x k wll be gven by: 1 (d. K)σ k (m)(1 σ k (m)) (7) m kf d k 1 + m k (d. K)σ k (m) (8) Ths message wll always be proper (have postve varance) but there s no guarantee of global convergence snce ths approxmaton s not a bound. 7 Bohnng s bound Bohnng s bound has the same form as the Taylor seres expanson, only wth a dfferent approxmaton to the Hessan matrx H of log exp, specfcally usng the bound the followng bound on log exp: log e x log e m + H 1 (I 11T /K) : H B (9) (x m )σ (m) + 1 (x m )(x j m j )(δ j 1 4 K ) In the case of a fully factorsed dstrbuton on x takng expectatons we have: log e x q log e m + 1 ( 1 1 ) v (30) 4 K j 5

6 Analogously to the Taylor seres expanson, we have the followng message to x k : 1 1 ( (d. K) 1 1 ) (31) K m kf d k 1 + m k (d. K)σ k (m) (3) Note here that the varance s constant and depends only on d. and K, and s always less than or equal to the varance of the message calculated usng the Taylor seres expanson. 6