Gaussian-log-Gaussian wavelet trees, frequentist and Bayesian inference, and statistical signal processing applications

Transcription

1 Aalborg Unverstet Gaussan-log-Gaussan wavelet trees, frequentst and Bayesan nference, and statstcal sgnal processng applcatons Møller, Jesper; Jacobsen, Robert Dahl Publcaton date: 014 Document Verson Early verson, also known as pre-prnt Lnk to publcaton from Aalborg Unversty Ctaton for publshed verson (APA): Møller, J., & Jacobsen, R. D. (014). Gaussan-log-Gaussan wavelet trees, frequentst and Bayesan nference, and statstcal sgnal processng applcatons. Department of Mathematcal Scences, Aalborg Unversty. Research Report Seres, No. R General rghts Copyrght and moral rghts for the publcatons made accessble n the publc portal are retaned by the authors and/or other copyrght owners and t s a condton of accessng publcatons that users recognse and abde by the legal requrements assocated wth these rghts.? Users may download and prnt one copy of any publcaton from the publc portal for the purpose of prvate study or research.? You may not further dstrbute the materal or use t for any proft-makng actvty or commercal gan? You may freely dstrbute the URL dentfyng the publcaton n the publc portal? Take down polcy If you beleve that ths document breaches copyrght please contact us at vbn@aub.aau.dk provdng detals, and we wll remove access to the work mmedately and nvestgate your clam.

2 AALBORG UNIVERSITY Gaussan-log-Gaussan wavelet trees, frequentst and Bayesan nference, and statstcal sgnal processng applcatons by Jesper Møller and Robert Dahl Jacobsen R Aprl 014 Department of Mathematcal Scences Aalborg Unversty Fredrk Bajers Vej 7 G DK - 90 Aalborg Øst Denmark Phone: Telefax: URL: e ISSN On-lne verson ISSN

3 Gaussan-log-Gaussan wavelet trees, frequentst and Bayesan nference, and statstcal sgnal processng applcatons Aprl 4, 014 Jesper Møller and Robert Dahl Jacobsen Department of Mathematcal Scences, Aalborg Unversty Abstract We ntroduce a promsng alternatve to the usual hdden Markov tree model for Gaussan wavelet coeffcents, where ther varances are specfed by the hdden states and take values n a fnte set. In our new model, the hdden states have a smlar dependence structure but they are jontly Gaussan, and the wavelet coeffcents have log-varances equal to the hdden states. We argue why ths provdes a flexble model where frequentst and Bayesan nference procedures become tractable for estmaton of parameters and hdden states. Our methodology s llustrated for denosng and edge detecton problems n two-dmensonal mages. Key words: condtonal auto-regresson; EM algorthm; hdden Markov tree; ntegrated nested Laplace approxmatons. 1 Introducton To model statstcal dependences and non-gaussanty of wavelet coeffcents n sgnal processng, Crouse, Nowak & Baranuk (1998) ntroduced a model where the wavelet coeffcents condtonal on a hdden Markov tree are ndependent Gaussan varables, wth the hdden states takng values n a fnte set (n applcatons, each hdden varable s often bnary) and used for determnng the varances of the wavelet coeffcent. We refer to ths as the Gaussan-fnte-mxture (GFM) wavelet tree model or just the GFM model. The GFM model and a clever mplementaton of the EM-algorthm have been wdely used n connecton to e.g. mage segmentaton, sgnal classfcaton, denosng, and mage document categorzaton, see e.g. Crouse et al. (1998), Po & Do (006), and Cho & Baranuk (001). Accordng to Crouse et al. (1998), the three standard problems (page 89) are tranng (.e. parameter estmaton), lkelhood determnaton 1

4 (.e. determnng the lkelhood gven an observed set of wavelet coeffcents), and state estmaton (.e. estmaton of the hdden states); they focus on the two frst problems, but menton that state estmaton s useful for problems such as segmentaton (page 893). In the present paper, we propose an alternatve model called the Gaussan-log- Gaussan (GLG) wavelet tree model or just the GLG model where the hdden states are jontly Gaussan and descrbed by a smlar dependence structure as n the GFM model, and where the wavelet coeffcents condtonal on the hdden states are stll ndependent Gaussan varables but the log-varance for each wavelet coeffcent s gven by the correspondng hdden state. In comparson wth the GFM model, n many cases the GLG model provdes a flexble model and a better ft for wavelet coeffcents, t s easy to handle for parameter estmaton n a frequetst settng as well as n a Bayesan settng, where state estmaton s also possble n the latter case, and t works well for denosng and edge detecton problems. The paper s organzed as follows. Secton provdes further detals of the GFM and GLG models. Secton 3 studes the moment structure of parametrc GLG models and explots the tractablty of the lower-dmensonal dstrbutons of the GLG model to develop composte lkelhoods so that the EM-algorthm becomes feasble for parameter estmaton. Secton 4 concerns fast Bayesan procedures for margnal posteror estmaton of parameters and hdden states n the GLG model, where we use ntegrated nested Laplace approxmatons (Rue, Martno & Chopn 009). Secton 5 demonstrates how our methods n Sectons 3 and 4 apply for denosng and edge detecton n two-dmensonal mages. Secton 6 contans concludng remarks. Techncal detals are deferred to Appendx A-D. Matlab and R (R Core Team 013) codes for our statstcal nference procedures are avalable at Wavelet tree models For both the GFM and the GLG model, we consder wavelet coeffcents w = (w 1,..., w n ), where the unts 1,..., n represent an abstract sngle ndex system. The unts are dentfed wth the nodes of a tree wth root 1 (the coarsest level of the wavelet transform) and edges correspondng to the parent-chld relatons of the wavelet coeffcents at the coarsest to the fnest level, see Fgure 1. Condtonal on hdden states s = (s 1,..., s n ), the wavelet coeffcents are ndependent Gaussan dstrbuted, where each w depends only on s (n Secton 5. we modfy the GFM and GLG models and consder wavelet coeffcents wth nose). For smplcty and snce t s frequently the case n applcatons, we assume that each condtonal mean E[w s ] = 0 s centered. However, for the two models, the condtonal varance Var[w s ] depends on and s n dfferent ways; the detals are gven n Sectons.1 and.. The condtonal ndependence structure for the hdden states s the same for the two models and gven by the tree structure,.e. s s vewed as a drected graphcal model (see e.g. Laurtzen (1996)): For = 1,..., n, denote c() {1,..., n} the chldren of, where each chld j c() s at one level lower than (see Fgure 1); f s at the fnest wavelet

5 1 1st level ρ(j) nd level j c() 3rd level Fgure 1: Illustraton of a bnary tree structure correspondng to a one-dmensonal sgnal wth l = 3 levels of wavelet coeffcents. Node j has one parent ρ(j) and node has two chldren c(). level, has no chldren (c() = ); else c(). Typcally n applcatons, f s not at the fnest wavelet level, has d chldren, where d s the dmenson of the sgnal/mage. Now, the jont densty for hdden states and wavelet coeffcents factorzes as p(s, w) = p 0 (s 1 ) n [ q (w s ) =1 j c() ] p (s j s ) where p 0 ( ), p ( ), and q ( ) are ether probablty mass functons or probablty densty functons, wth the true nature beng obvous from the context, and where we set j c() p (s j s ) = 1 f c() =. We refer to (1) as a wavelet tree model. We shall consder parametrc models, usng θ as generc notaton for the unknown parameters, and to stress the dependence of θ we wrte e.g. p(s, w θ) for the densty p(s, w). When we later dscuss parameter estmaton (Sectons 3 and 4), we consder k ndependent pars (s (1), w (1) ),..., (s (k), w (k) ) wth densty p(s, w θ), where we suppose only the wavelet coeffcents w (1),..., w (k) are observed..1 The GFM model For the GFM model, t s assumed n Crouse et al. (1998) that the state space of each s s a fnte set {1,..., m} (often m = ), (1) q ( s ) s a Gaussan densty where both the mean µ,s depend on the ndex and the argument s. and the varance σ,s Crouse et al. (1998) remark that nstead of a sngle Gaussan dstrbuton, the m-state Gaussan mxture dstrbuton for each wavelet coeffcent s needed because of the compressng property... resultng n a large number of small coeffcents and a small number of large coeffcents (page 887); and the condtonal dependence structure s used to characterze the key dependences between the wavelet coeffcents (page 887),.e. t matches both the clusterng and persstence propertes of the wavelet transform 3

6 (page 891) so that f one coeffcent s n a hgh-varance (low-varance) state, then ts neghbor s very lkely to also be n a a hgh-varance (low-varance) state (page 891). The parameters of the GFM model are the varance σ,s of q ( s ) N(0, σ,s ), s = 1,..., m, = 1,..., n, the ntal probabltes p 0 ( ) and the unknown transton probabltes p ( ) of the hdden state varables wth c() and = 0,..., n (settng c(0) = {1}). Usually the varances and transton probabltes are assumed only to depend on the level of the nodes. Then, denotng l the number of levels n the tree, the number of parameters s r GFM = ml + m 1 + m(m 1)(l 1). (). The GLG model In the GLG model, each wavelet coeffcent w condtonal on s s zero-mean Gaussan wth varance exp(s ),.e. q ( s ) = q( s ) does not depend on, and q(w s ) N(0, exp(s )); (3) the hdden states are jontly Gaussan,.e. p 0 (s 1 ) = p(s 1 µ 0, σ0 ) and p (s j s ) = p(s j s, α, β, κ ) for j c(), where p(s 1 µ 0, σ 0) N(µ 0, σ 0), (4) p(s j s, α, β, κ ) N(α + β s, κ ), (5) where µ 0, α, and β are real parameters and σ 0 and κ are postve parameters. The densty (3) s completely determned by the varance exp(s ), and t appears to be a more flexble model for wavelet coeffcents than the m-state Gaussan mxture model used n Crouse et al. (1998): In the GFM model, wavelet coeffcents from all trees and assocated to the same parent (or to the root) are sharng the same set of m possble varances, whle n the GLG model, each wavelet coeffcent w (t) for each tree t s havng ts own log-gaussan hdden state s (t). In Secton 3.1. and further on we assume as n the GFM model tyng wthn levels, that s the parameters on each level are equal (detaled later n (15)). Then the number of parameters n the GLG model for a tree wth l levels s r GLG = 3l 1. In comparson the GFM model wth m = s specfed by r GFM = 4l 1 parameters, whle the dfference wll be even larger as m grows, cf. (). 4

7 3 Parameter estmaton usng composte lkelhoods and the EM-algorthm Secton 3.1 descrbes the frst and hgher order moment structure of the hdden states and the wavelets coeffcents under the GLG model. In partcular we clarfy the meanng of tyng wthn levels, whch s assumed when we n Secton 3. dscuss parameter estmaton usng composte lkelhoods and the EM-algorthm. 3.1 Mean and varance-covarance structure Full parametrzaton Ths secton consders a full parametrzaton of the GLG model (4),.e. when µ 0 R, σ 0 > 0, (α, β, κ ) R R (0, ) (6) for 0 and c(). For each node j 1 n the tree structure, let ρ(j) denote the parent to j (see Fgure 1), and set ρ(1) = 0. By (4) and (5), each hdden state s j s Gaussan dstrbuted wth a mean and varance whch are determned by the means and the varances of ts ancestors: p j (s j ) = p(s j µ ρ(j), σ ρ(j) ) N(µ ρ(j), σ ρ(j) ) (7) for j = 1,..., n, where the mean and the varance are determned recursvely from the coarsest level to the second fnest level by µ = α + β µ ρ(), σ = κ + β σ ρ() (8) for 1 and c(). Conversely, the GLG model s parametrzed by (µ 0, σ 0 ) (, ) (0, ) and (µ, σ, β ) (, ) (0, ) (, ) for all 1 wth c(), snce α = µ β µ ρ() and κ = σ β σρ() (9) whenever 1 and c(). Set κ 0 = σ 0 and denote σ,j = Cov(s, s j ), the covarance of s and s j. Note that σ, = σ ρ() ; a general expresson for σ,j s gven by (34) n Appendx A. In partcular, σ h,j = κ ρ(h) β h f j c(h), σ,j = κ ρ(h) β h (10) f, j c(h) and j. [ ] Moments of the form E w awb j for a = 0, 1,... and b = 0, 1,... can be derved by condtonng on the hdden states and explotng well-known moment results for the log- Gaussan dstrbuton, see e.g. (30)-(3) n Appendx A. In partcular, lettng c(0) = {1}, 5

8 then for any h 0 and j c(h), η () h := E [ w j ] = exp ( µh + σ h /), (11) η (4) h := E [ wj 4 ] ( = 3 exp µh + σh), (1) := E [ w wj ] ( ) = exp µ h + κ ρ(h) β h + σ h f c(h) and j, (13) η (,) h ξ (,) h,j := E [ w j w h] = exp ( µh + µ ρ(h) + κ ρ(h) β h + σ h / + σ ρ(h) /). (14) 3.1. Tyng wthn levels For each node n the tree structure, denote l() the level of,.e. l() s the number of nodes n the path from the root to, and let l be the number of levels (see Fgure 1). For convenence, defne l(0) = 0. Henceforth we assume tyng wthn levels of the parameters n (4) and (5), that s α = α(l()), β = β(l()), κ = κ(l()), 1 l() < l. (15) Thus the unknown parameters are µ 0 R, σ 0 > 0, (α(1),..., α(l)) R l, (β(1),..., β(l)) R l, (κ(1),..., κ(l)) (0, ) l. Note that for 1 l() < l, by (8), (15), and nducton, (µ, σ ) depends on the node only through ts correspondng level l(),.e. µ = µ(l()), σ = σ (l()). (16) Furthermore, for 0 l(h) < l and j c(h), we obtan from (11)-(15) that (η () h, η(4) h, η (,) h, ξ (,) h,j ) depends on h only through l(h),.e. η () h = η () (l(h)), η (4) h = η (4) (l(h)), η (,) h = η (,) (l(h)), ξ (,) h,j = ξ (,) (l(h)). (17) 3. Parameter estmaton For parameter estmaton n the GFM model, Crouse et al. (1998) propose to use the EM-algorthm. Here the man dffculty s the calculaton of p (s, s j w) for j c(), the two-dmensonal margnal probabltes of any s and ts chld s j condtonal on the wavelet coeffcents, where the calculaton has to be done for each E-step of the EMalgorthm and each wavelet tree w = w (t), t = 1,..., k. Crouse et al. (1998) solve ths problem usng an upward-downward algorthm whch s equvalent to the forwardbackward algorthm for hdden Markov chans. Durand, Gonçalvès & Guédon (004) mprove on the numercal lmtatons on ths algorthm. Modfyng the upward-downward algorthm n Crouse et al. (1998) to the GLG model s not leadng to a computatonally feasble algorthm manly because, for each s, we have replaced ts fnte state space {1,..., m} under the GFM model by the real lne 6

9 under the GLG model, and numercal ntegraton would be repeatably needed at the varous (many) steps of the algorthm. As notced n Appendx C, the Gauss-Hermte quadrature rule provdes good approxmatons wth few quadrature nodes when consderng trees wth no more than two levels. However, snce the ntegrants nvolved n the transton between levels of the upward-downward algorthm are not suffcently smooth, we propose nstead an EM algorthm for estmatng θ based on composte lkelhoods for the jont dstrbuton of wavelets correspondng to each parent and ts chldren. These jont dstrbutons are relatvely easy to handle. Further detals are gven n the sequel Margnal lkelhoods When defnng composte lkelhoods n connecton to the EM-algorthm n Secton 3.., we use the followng margnal lkelhoods gven n terms of the full parametrzaton (4) and (5). Combnng (3) and (4), we obtan the densty of (s 1, w 1 ), exp p(s 1, w 1 µ 0, σ0) = ( 1 and hence the margnal densty of the root wavelet, q(w 1 µ 0, σ 0) = [ w 1 exp(s 1 ) + s 1 + (s 1 µ 0 ) σ 0 ]) πσ 0 (18) p(s 1, w 1 µ 0, σ 0) ds 1. (19) The margnal log-lkelhood based on the root wavelet vector w 1 = (w (1) 1,..., w(k) 1 ) for the k trees s gven by l 0 (µ 0, σ 0 w 1 ) = k t=1 log q(w (t) 1 µ 0, σ 0). (0) Consder any {1,..., n} wth c(). Denote w,c() the vector consstng of w and all w j wth j c(), and s,c() the vector consstng of s and all s j wth j c(). Usng (3)-(5) and (7), we obtan the densty of (s,c(), w,c() ), p(s,c(), w,c() µ ρ(), σρ(), α, β, κ ) = ( {[ ] exp 1 w exp(s ) + s + (s µ ρ() ) + σρ() j c() [ (π) 1+ c() σ ρ() κ c() ]}) wj exp(s j ) + s j + (s j α β s ) κ where c() denotes the number of chldren to. Hence the densty of w,c() s gven by the ntegral q(w,c() µ ρ(), σρ(), α, β, κ ) = j c() (1) p(s,c(), w,c() µ ρ(), σ ρ(), α, β, κ ) ds j ds. () 7

10 Fnally, denotng w,c() the vector of the th wavelets w (1),..., w (k) w (1) j,..., w (k) j, j c(), the log-lkelhood based on w,c() s l (µ ρ(), σ ρ(), α, β, κ w,c() ) = 3.. EM-algorthm k t=1 j c() and ther chldren log q(w (t), w (t) j µ ρ(), σρ(), α, β, κ ). (3) Ths secton shows how the EM-algorthm apples on composte lkelhoods (Gao & Song 011) defned from the margnal lkelhoods n Secton 3..1 under the assumpton of tyng wthn levels, cf. (15). We proceed from the coarsest to the fnest level, where parameter estmates are calculated by the EM-algorthm as detaled n Appendx C 1. Apply the EM-algorthm for the (margnal) log-lkelhood (0) to obtan an estmate ( µ 0, σ 0 ).. For r = 1,..., l 1, denotng w (r) the vector of all w,c() wth l() = r, the log-composte lkelhood gven by the sum of the log-lkelhoods (3) based on all w,c() wth l() = r s l (r) (µ(r 1), σ (r 1), α(r), β(r), κ(r) w (r) ) = l (µ(r 1), σ (r 1), α(r), β(r), κ (r) w,c() ). :l()=r Now, suppose we have obtaned an estmate ( µ(r 1), σ (r 1)). Then we apply the EM-algorthm on l (r) ( µ(r 1), σ (r 1), α(r), β(r), κ(r) w (r) ) to obtan an estmate ( α(r), β(r), κ (r)). Thereby, usng (8) and (15), an estmate ( µ(r), σ (r)) s also obtaned. These composte lkelhoods for our GLG model can be handled manly because of the condtonal ndependence structure and snce the margnal dstrbuton of s s Gaussan. In contrast, for the GFM model, unless n s small or all except a few nodes have at most one chld, t s not feasble to handle margnal dstrbutons and correspondng composte lkelhoods. For the ntal values used n steps 1 and, moment-based estmates obtaned as descrbed n Appendx B are used. If such an estmate s not meanngful (see Remark 1 n Appendx B), we replace the parameter estmate by a fxed value whch makes better sense. Each teraton of step 1 leads to an ncrease of the margnal log-lkelhood (0), so the value returned by the EM algorthm s a local maxmum; and each teraton of step leads to an ncrease of the log-composte lkelhood, so the value returned by the EM algorthm s a local maxmum when (µ(r 1), σ (r 1)) = ( µ(r 1), σ (r 1)) s fxed. In each step, as usual when applyng the EM-algorthm, there s no guarantee that the global maxmum wll be found. 8

11 4 Bayesan nference For the GFM model, Bayesan methods are not feasble: Indeed Crouse et al. (1998) derve recursons for calculatng the condtonal denstes p(s (t) w (t), θ) and p(s (t) j, s(t) w (t), θ), j c(), but t s not possble to calculate or satsfactory approxmate the margnal posteror denstes for (any subparameter of) θ or (any subvector) of s (t). For nstance, f w = (w (1),..., w (k) ) s the vector of wavelets from all the trees, p(s (t) w) = (t) p(s w (t), θ)p(θ w) dθ where under the GFM model we do not know what p(θ w) s and t seems hopeless to evaluate ths hgh dmensonal ntegral. Usng a Bayesan approach for the GLG model, wth a pror mposed on all the r GLG unknown parameters, also leads to a complcated posteror dstrbuton. In prncple t could be handled by Markov chan Monte Carlo (MCMC) methods, but MCMC samplng remans panfully slow from the end user s pont of vew (page 3 n Rue et al. (009)). However, approxmate Bayesan methods based on Laplace approxmatons (Terney & Kadane 1986, Rue & Martno 007, Rue et al. 009) are feasble for GLG submodels when the number of unknown parameters s not hgh, as n our GLG submodel ntroduced n Secton 4.1. Furthermore, Secton 4. consders ntegrated nested Laplace approxmatons (INLA) to obtan margnal posteror dstrbutons for θ and the hdden states (Rue et al. 009). 4.1 Condtonal auto-regressons Romberg, Cho & Baranuk (001) consder GFM submodels where θ s of dmenson nne, and they demonstrate that the submodels are acceptable for denosng mages wth a hgh degree of self-smlarty, e.g. as found n mages of natural scenes. Encouraged by these results and because of the larger flexblty n modellng the varances of sngle wavelet coeffcents n the GLG model, we consder the followng GLG submodel. Frst, notce that by (4), s s a Gaussan Markov random feld or n fact a condtonal auto-regresson (CAR; Besag (1974, 1975); Rue & Held (005, Chapter 1)). The Gaussan dstrbuton of s s specfed by the mean µ ρ() of each s and the precson matrx (the nverse of the varance-covarance matrx of s) whch has (, j)th entry 1 + c() β κ f = j, c(), cf. Appendx A. κ ρ() 1 κ,j = β ρ() κ ρ(j) β ρ(j) κ ρ() f = j, c() =, f = ρ(j), f j = ρ(), 0 otherwse, (4) 9

12 Second, consder the homogeneous GLG model specfed by that α = α, β = β, κ = κ, whenever l() < l. Then the free parameters are θ = (µ 0, σ0, α, β, κ ) (, ) (0, ) (, ) (, ) (0, ). By (8) and (16), we obtan a specal mean and varance structure for the hdden states: For level r = 1,..., l, α βr 1 µ(r) = β 1 + βr µ 0 f β 1, rα + µ 0 f β = 1, and σ κ βr 1 (r) = β 1 + βr σ0 f β 1, rκ + σ0 f β = INLA Integrated nested Laplace approxmatons (INLA) s a general framework for performng approxmate Bayesan nference n latent Gaussan models where the number of parameters s small (see Rue et al. (009) and Martns, Smpson, Lndgren & Rue (013)). Rue et al. (009) notce that The man beneft of INLA s computatonal: where Markov chan Monte Carlo algorthms need hours or days to run, INLA provdes more precse estmates n seconds or mnutes. Ths ncludes estmates of the posteror margnals for θ and for the hdden states. Parsmonous GLG submodels ft the INLA assumptons. We have mplemented the homogeneous GLG model n INLA, where pror specfcaton s largely handled automatcally n INLA. Specfc calls used n the experments reported n the sequel can be seen n our released code. 5 Examples of applcatons Ths secton compares results usng the GLG and GFM models for wavelet coeffcents n real mages. The GFM model has proven to be useful for modellng dfferent knds of multscale transforms (Crouse et al. 1998, Romberg et al. 001, Po & Do 006), but our results are only for the standard wavelet transform, where n both the GFM and the GLG model the drectons of the wavelet transform are modelled ndependently. Secton 5.1 dscusses how well GLG and GFM models descrbe standard wavelet coeffcents, Secton 5. consders denosng of mages, and Secton 5.3 concerns edge detecton. 5.1 Modellng standard wavelet coeffcents n mages For llustratve purposes, n ths and the followng sectons, we use three test mages from the USC-SIPI mage database avalable at Lena, 10

13 mandrll, and peppers, see Fgure. These mages are 51-by-51 pxels represented as 8 bt grayscale wth pxel values n the unt nterval, and we have ftted the GFM and GLG models to wavelet transforms usng the correspondng EM algorthms. Fgure 3 shows four hstograms of the wavelet coeffcent from a sngle subband along wth the ftted margnal dstrbutons. The fgure llustates that no model s fttng better than the other n all cases: For level 1 of the vertcal subband of Lena (upper left panel) and for level of the vertcal subband of mandrll (upper rght panel), the GLG model provdes the best ft; for level 3 of the vertcal subband of mandrll (lower left panel), the GLG model s too hghly peaked at zero and the GFM model provdes a better ft; and for level 3 of the dagonal subband of mandrll (lower rght panel), the two models ft equally well. 5. Denosng Consder an mage corrupted wth addtve whte nose,.e. we add an ndependent term to each pxel value from the same zero-mean normal dstrbuton. Recall that when workng wth orthonormal wavelets, the dstrbuton and the ndependence propertes of the nose are preserved by the wavelet transform, and the procedure for denosng wth wavelets works as follows: nosy data nosy wavelets nose-free wavelets nose-free data. Thus, a wavelet tree w = (w 1,..., w n ) s also observed wth addtve whte nose: v = w + ε, = 1,..., n, (5) where ε N(0, σε), the ε are mutually ndependent and ndependent of (s, w), and we assume that the nose varance σɛ s known. The dependence structure n the tree wth nosy observatons s llustrated n Fgure 4. From ths and (5) we obtan n p(w v, s, θ) = p(w v, s, θ). =1 Below we dscuss estmaton of w. In the frequentst setup, we estmate w by E[w v, θ], wth θ replaced by ts estmate obtaned by the approprate EM-algorthm (see Secton 3.. and Crouse et al. (1998)). For an m-state GFM model, E[w v, θ] = v m j=1 σ,j p(s = j v ) σ,j + σ ε (see Crouse et al. (1998)). Under the GLG model, we have E[w v, θ] = v c(v µ ρ(), σρ() ) ( [ exp(s ) (exp(s ) + σε) exp 1 v 3/ exp(s ) + σε + (s ]) µ ρ() ) σρ() ds (6) 11

14 Fgure : The three test mages: Lena, mandrll, and peppers. where we use the Gauss-Hermte quadrature rule for approxmatng the ntegral. Equaton (6) s derved n Appendx D. In the Bayesan setup for the homogeneous GLG model, we work wth the posteror dstrbuton p(w v ) from whch we can calculate varous pont estmates. We have p(w v ) = p(w v, s )p(s v )ds, (7) E(w v ) = E(w v, s )p(s v )ds, where p(s v ) s calculated n INLA. Snce p(w s ) N(0, exp(s )) and p(v w ) 1

15 Fgure 3: Hstograms of wavelet coeffcents from one scale of the 3 level wavelet transform wth a Daubeches 4 wavelet. The probablty densty functons of the ftted GLG model (sold lne) and the ftted GFM model (gray lne) are shown. Upper left panel: Level 1 of the vertcal subband of Lena. Upper rght panel: Level of the vertcal subband of mandrll. Lower left panel: Level 3 of the vertcal subband of mandrll. Lower rght panel: Level 3 of the dagonal subband of mandrll. N(w, σ ɛ ), we obtan and ( v exp(s ) p(w v, s ) p(w s )p(v w ) N σɛ + exp(s ), σɛ ) exp(s ) σɛ + exp(s ) E(w v ) = v exp(s ) σ ɛ + exp(s ) p(s v )ds. We apply the two denosng schemes wth a three level wavelet transform usng the Daubeches 4 flter to nosy versons of the three test mages n Fgure. To estmate the performance of a denosng scheme, we calculate the peak sgnal-to-nose rato (PSNR) 13

16 s 1 w 1 s s 3 ε 1 w v 1 ε w 3 ε 3 v v 3 Fgure 4: Graphcal model of a bnary tree wth two levels and nosy observatons. The rectangular nodes are observed varables and the round nodes are unobserved varables. n decbels between a test mage I and a nosy or cleaned mage J. For mages of sze N N, the PSNR n decbels s defned as PSNR = 0 log 10 N(max{I(x)} mn{i(x)}) I J where the maxmum and the mnmum are over all pxels x and s the Frobenus norm. Table 1 shows for the test mages and dfferent nose levels σ ε, the PSNR between each test mage and ts nosy or denosed verson: For the frequentst results, the mages denosed usng the GLG model have PSNRs that are consstently hgher than those denosed usng the GFM model. The Bayesan results yelds the lowest PSNR values, but they are also based on a more parsmonous model. An example of the vsual appearance of denosng usng frequentst means s seen n Fgure 5; agan the GLG model performs best, where detals around e.g. the stem of the center pepper are more crsp. The medan (the 50% quantle) of the posteror dstrbuton s only one possble pont estmate of the posteror dstrbuton. However, usng other quantles or the posteror mean are not provdng better results, see Fgure Edge detecton Edge detecton n an mage s performed by labellng each pxel as beng ether an edge or a non-edge. Turnng to the wavelet transform for ths task has the advantage that wavelet coeffcents are large near edges and small n the homogeneous parts of an mage; the dffculty les n quantfyng large and small. Another advantage s that a multresoluton analyss allows us to search for edges that are present at only selected scales of the mage, thereby gnorng edges that are ether too coarse or too fne. In ths secton, for each tree t = 1,..., k, we focus on how to label the wavelet coeffcent w (t) by an ndcator varable f (t), where f (t) = 1 means w (t) s large, and f (t) = 0 means w (t) s small. Labellng of wavelet coeffcents usng the GFM model s ntroduced n 14

17 Table 1: For the three test mages and three nose levels, peak sgnal-to-nose ratos n db between the mage and ts nosy verson ot ts denosed verson obtaned usng ether the GFM model and the EM-algorthm, the GLG model and the EM-algorthm, or the homogeneous GLG model and INLA. In the latter case, the PSNR s calculated usng the medan of the posteror mage. For each mage, a three level Daubeches 4 wavelet transform s used. test mage PSNR nose level σ ε nosy GFM GLG hom. GLG Lena Mandrll Peppers Sun, Gu, Chen & Zhang (004); we recap ths labellng algorthm and afterwards modfy t to work wth the GLG model. Fnally, we dscuss how to transfer these labels to the pxels and show examples. The labellng n Sun et al. (004) conssts of three steps. Frst, usng the EM algorthm of Crouse et al. (1998), an estmate θ of the parameter vector θ of a -state GFM model s obtaned from the data {w (t) } k t=1. Second, usng an emprcal Bayesan approach, the maxmum a posteror (MAP) estmate of the hdden states s (t) = argmax p(s w (t), θ) = argmax p(s, w (t) θ), (8) s s t = 1,..., k, s computed usng the Vterb algorthm (Durand et al. 004). Thrd, the MAP estmate s used to defne f (t) = (ŝ(t) ). The dea of labellng wavelet coeffcents wth the GLG model s overall the same as presented above for the GFM model, wth the dfferences arsng from the contnuous nature of the hdden states and dfferent algorthms beng appled for parameter estmaton and state estmaton. Frst, the EM algorthm n Secton 3.. s used to provde an estmate θ of the parameter vector of the GLG model. Second, n analogy wth (8) we compute the MAP estmate ŝ(t). However, the Vterb algorthm cannot be used here: The Vterb algorthm computes the MAP estmate by successvely maxmzng the terms n (1) assocated to each level of the wavelet tree. For the GFM model, t s easy to perform these maxmzaton steps due to the fact that the hdden state space s fnte. For the GLG model, the MAP estmate can be computed at the fnest level, 15

18 Fgure 5: Denosng results for the peppers mage from Table 1 when the standard devaton of the nose s 0.0. Top left panel: The orgnal mage. Top rght panel: The nosy mage (PSNR s 13.57). Bottom left panel: The nosy mage cleaned usng the GFM model and the EM-algorthm (PSNR s 3.70). Bottom rght mage: The nosy mage cleaned usng the GLG model and the EM-algorthm (PSNR s 4.41). but ths estmate s a complcated functon that cannot easly be used n the remanng 16

19 Fgure 6: Denosng the peppers mage usng the posteror dstrbuton (7) and INLA. The orgnal and nosy mages are seen n Fgure 5. The top left, top rght, and bottom left mages are based on the 5%, 75%, and 50% quantles of the posteror dstrbuton, respectvely (the PSNRs are 16.38, 16.41, and 19.18, respectvely). The bottom rght mage s based on the mean of the posteror dstrbuton (PSNR s 19.15). The posteror mean and medan are almost dentcal. maxmzaton steps. Instead, we note that n p(s, w θ) = p(s θ) p(w s ) (9) =1 17

20 where p(s θ) s a multdmensonal Gaussan densty functon wth mean vector µ and precson matrx gven by (4) wth θ = θ. The log of (9) and ts gradent vector and Hessan matrx wth respect to s are log p(s, w θ) 1 n {(s µ) ( ) (s } µ) + w exp( s ) + s, =1 log p(s, w θ) = (s µ) + 1 [ w exp( s ) 1 ] 1 n, H ( log p(s, w) θ) = 1 dag( w exp( s ), 1 n ), where means that an addtve term whch s not dependng on s has been omtted n the rght hand sde expresson. The Hessan matrx s strctly negatve defnte for all (s, w) wth w 0 and hence ŝ(t) can be found by solvng log p(s, w (t) θ) = 0 usng standard numercal tools. Thrd, observe that f the estmate (ŝ(t) ) s large n the estmated dstrbuton N( µ ρ(), σ ρ()) for s, then we expect w (t) to be large. Therefore, denotng z p the p-fractle n N( µ ρ(), σ ρ()) (wth e.g. p = 0.9), we defne = 1 f (ŝ(t) ) z p and zero otherwse. It remans to specfy the transfer of f (t) (defned by one of the two methods above) to the pxel doman (ths ssue s not dscussed n Sun et al. (004)). For specfcty, consder a gray scale mage I = {p j } kn j=1 and {w(t) } k t=1 = W {p j} kn j=1, where W s the used wavelet transform operator. To each pxel j we assocate a bnary varable e j ndcatng f j s part of an edge or not: Snce the wavelet transform does not necessarly map bnary values to bnary values, we defne {ẽ j } kn j=1 = W 1 {f (t) } k t=1 and set { 1 f ẽj 0, e j = 0 otherwse. f (t) The e j s are senstve to the choce of W, and usng the Haar wavelet results n thn edges. As mentoned, the multresoluton analyss of the wavelet transform allows us to consder edges that are present at only specfc scales. To exclude edges at a level l n the wavelet transform, we smply modfy {f (t) } k (t) t=1 by settng f = 0 f l() = l. Fgure 7 compares the results of the two edge detecton algorthms, where we only use the fnest scale n the wavelet transform. The method based on the GLG model classfes fewer pxels as edges; n partcular the GFM model classfcatons nclude many false postves. Whle the mages wthn Fgure 7 are comparable, we notce they are not drectly comparable to the mages presented n Sun et al. (004) who use a non-decmated wavelet transform and an extenson of the GFM model where the dfferent drectons are not modelled ndependently. 18

21 Fgure 7: Examples of edge detecton of the Lena and peppers mages usng the method from Sun et al. (004) (left column) and our varant that uses the GLG model (rght column). A three level Haar wavelet transform s used and only the fnest level of the wavelet transform s consdered. The 90% fractle s used for thresholdng wth the GLG model. 6 Concludng remarks We have ntroduced the GLG model for wavelet trees, developed methods for performng nference, and demonstrated possble applcatons n sgnal and mage processng, where the GLG model outperforms the GFM model of Crouse et al. (1998). However, there 19

22 s stll work to be done. We do not have a procedure for lkelhood determnaton of a full wavelet tree gven the model parameters n the general GLG model (t s possble to compute the lkelhood n INLA, but ths s only for submodels). In the GFM model ths lkelhood s calculated as a by-product of the EM algorthm n Crouse et al. (1998), but as noted we cannot easly modfy ths EM algorthm to the GLG model. As an alternatve method for nference we have consdered a varatonal EM algorthm (see e.g. Khan (01)). The parameter estmates obtaned wth ths varatonal method may be more consstent across the levels of the wavelet transform. We have omtted a further dscusson of ths varatonal method, snce t cannot be used for makng nference wth nosy observatons. Acknowledgment Supported by the Dansh Councl for Independent Research Natural Scences, grant , Mathematcal and Statstcal Analyss of Spatal Data, and by the Centre for Stochastc Geometry and Advanced Bomagng, funded by a grant from the Vllum Foundaton. We are grateful to Håvard Rue for help wth INLA. We thank Peter Cragmle, Morten Nelsen, and Mohammad Emtyaz Khan for helpful dscussons. Appendx A: Moments Usng (1) and (3) and by condtonng on s and explotng the condtonal ndependence structure, we obtan E [ w ] ) = exp (µ ρ() + σρ() /, (30) E [ w 4 ] ( ) 3 ( E [ ]) w = exp σρ(), (31) [ ] E w w j E [ ] [ ] = exp(σ,j ) f j. (3) w E wj For = 1,..., n, let v = s β ρ() s ρ() where β 0 = 0. Then s = j P 1, v j h P j,ρ() β h (33) where P 1, s the path of nodes from 1 to (n the tree, and ncludng 1 and ), P j,ρ() s the path of nodes from j to ρ() f j P 1, \ {}, and we set h P j,ρ() β h = 1 f j =. Note that v 1,..., v n are ndependent Gaussan dstrbuted and v N(α ρ(), κ ρ() ) where κ 0 = σ 0. Hence we mmedately obtan from (33) that [ ][ σ,j = β h1 β h ]. (34) h 0 P 1, P 1,j κ ρ(h 0 ) h 1 P h0,ρ() h P h0,ρ(j) 0

23 Fnally, because of the smple one-to-one lnear relatonshp between (v 1,..., v n ) and (s 1,..., s n ), (4) s straghtforwardly derved. Appendx B: Estmatng equatons based on moment relatons Assume c() 1, = 1,..., n; ths condton s n general satsfed n wavelet applcatons. Usng mean value relatons for the full parametrzaton (4) we descrbe a smple and fast procedure whch provdes consstent estmates for the parameters under (15) as the number of wavelet trees tends to nfnty. Let n r denote the number of nodes on level r {1,..., l}. Frst, by (11), (1) and (16), for each level r = 0,..., l 1, there s a one-to-one correspondence between (µ(r), σ (r)) and (η () (r), η (4) (r)), where ( ) µ(r) = log σ (r) = log η () (r) σ (r)/, ) log ( η (4) (r)/3 ( η () (r) Combnng these relatons wth unbased estmates gven by η (a) (r) = 1 kn r k t=1 :l()=r j c() ). ( (t)) a, w a =, 4, r < l, we obtan consstent estmates ( ) µ(r) = log η () (r) σ (r)/, (35) ( ) ( ) σ (r) = log η (4) (r)/3 log η () (r), r < l, (36) Second, by (11)-(14), for each node h 1 wth c(h), ( ) ( ) log η (,) h log η () h β h = ( ) ( ) log log log ξ (,) h,j j η () h ( ) η () ρ(h) whenever j c(h). Ths combned wth (15) and (17), the unbased estmates gven above, and the consstent estmates ξ (,) (r) = 1 kn r k t=1 :l()=r j c() ( (t)) ( (t)) w w j and η (,) (r) = 1 kn r k t=1 :l()=r c() ( c() 1) j 1,j c() j 1 <j ( (t)) ( (t)) w j 1 w j 1

24 provde consstent estmates log [ η β(r) (,) (r) ] log [ η () (r) ] = log [ ξ(,) (r) ] log [ η () (r) ] log [ η () (r 1) ] (37) for r = 0,..., l 1. Fnally, usng (9) and (35)-(37), we obtan consstent estmates ( α h, κ h ) = ( α(r), κ (r) ) for 0 r = l(h) < l. Remark 1 The estmatng equaton (35) does not guarantee that σ (r) > 0; n fact, for small wavelet datasets, we have observed that σ (r) may be negatve. For σ (r) to be postve s equvalent to requre that η (4) (r) > 3 ( η () (r) ). (38) As η (4) s the fourth moment and η () the second moment of the same random varable, (38) s a much stronger condton than the usual condton for varance estmaton, namely wth 3 replaced by 1. Remark The estmaton procedure s mmedately modfed to GLG submodels. In case of the homogeneous GLG model, defne η () = exp ( µ 0 + σ 0), η (4) = 3 exp ( µ 0 + σ 0), and n accordance wth (11)-(14) correspondng unbased estmates Thereby η (a) = ( µ 0 = log η ()) σ /, h 1: c(h) η(a) h, a =, 4, c ( ) ( σ = log η (4) /3 log η ()), provde consstent estmates. Appendx C: EM-algorthm for the margnal lkelhoods The EM-algorthm (Dempster, Lard & Rubn 1977, Gao & Song 011) s an teratve estmaton procedure whch apples for steps 1 n Secton 3.. as descrbed below. We start by notcng that the condtonal densty of s 1 gven w 1 s p(s 1 w 1, µ 0, σ 0 ) = p(s 1, w 1 µ 0, σ 0 ) q(w 1 µ 0, σ 0 ) exp ( 1 [ w1 exp(s 1 ) + s 1 + (s 1 µ 0 ) ]) σ0 (39)

25 where n the expresson on the rght hand sde we have omtted a factor whch does not depend on the argument s 1 of the condtonal densty. Note also that for l() = r < l, the condtonal densty of s,c() gven w,c() s p(s,c() w,c(), µ(r 1), σ (r 1), α(r), β(r), κ (r)) t=1 = p(s,c(), w,c() µ(r 1), σ (r 1), α(r), β(r), κ (r)) q(w,c() µ(r 1), σ (r 1), α(r), β(r), κ (r)) ( exp 1 {[ w exp(s ) + s + (s µ(r 1)) ] σ + (r 1) [ wj exp(s j ) + s j + (s j α(r) β(r)s ) ]}) κ. (40) (r) j c() In step 1, suppose ( µ 0, σ 0 ) s the current estmate. We consder the condtonal expectaton wth respect to (39) when (µ 0, σ0 ) s replaced by ( µ 0, σ 0 ). Then the next estmate for (µ 0, σ0 ) s the maxmum pont for the condtonal expectaton of the log-lkelhood whch s based on both w 1 and s 1 ; ths log-lkelhood s gven by [ ] k log p(s (t) 1, w(t) 1 µ 0, σ0) 1 k log(σ 0) + (s(t) 1 µ 0) σ0 where means that an addtve term whch s not dependng on (µ 0, σ0 ) has been omtted n the rght hand sde expresson, cf. (18). It follows mmedately that ths maxmum pont s gven by σ 0 = µ 0 = 1 k [ 1 k k E t=1 k t=1 t=1 [ ] E s (t) 1 w(t) 1, µ 0, σ 0, [ ( s (t) 1 ) w (t) 1, µ 0, σ 0 ] ] µ 0, where the condtonal expectaton s calculated usng (39). We do not have a closed expresson for the margnal densty nor ts moments. Snce the jont densty s the product of a Gaussan densty and a smooth functon, the Gauss-Hermte quadrature rule (see e.g. Press, Teukolsky, Vetterlng & Flannery (00)) s well-suted for approxmatng the ntegrals usng few quadrature nodes. The teraton s repeated wth ( µ 0, σ 0 ) = ( µ 0, σ 0 ) untl convergence s effectvely obtaned, whereby a fnal estmate ( µ 0, σ 0 ) s returned. In step, suppose ( α(r), κ (r)) s the current estmate, whch we use together wth the estmate ( µ(r 1), σ (r 1)) to obtan the next estmate for (α(r), κ(r)): Replacng (µ(r 1), σ (r 1), α(r), β(r), κ(r)) by ( µ(r 1), σ (r 1), α(r), β(r), κ(r)), ths estmate s the maxmum pont for the condtonal expectaton wth respect to (40) of each term 3

26 n the followng sum: k t=1 :l()=r log p(s (t),c(), w(t),c() µ(r 1), σ (r 1), α(r), β(r), κ (r)) 1 k t=1 :l()=r j c() [log(κ (r)) + (s j α(r) β(r)s ) ] κ (r) where addtve terms whch do not depend on (α(r), κ(r)) have been omtted, cf. (1). Now, calculate s(r) defned as the average of the followng condtonal means: s(r) = 1 kn r 1 k t=1 :l()=r It s easly seen that the maxmum pont s gven by β(r) = k t=1 :l()=r j c() k t=1 :l()=r α(r) = s(r) β s(r), [ κ 1 k (r) = E [ s (t) w (t),c(), µ(r 1), σ (r 1), α(r), β(r), κ (r) ]. E [ s (t) j (s(t) s(r)) w (t),c(), µ(r 1), σ (r 1), α(r), β(r), κ (r) ] c() E [ (s (t) kn r 1 t=1 :l()=r [ (s (t) E j 1 c() s(r)) w (t),c(), µ(r 1), σ (r 1), α(r), β(r), κ (r) ] j c() (t)) w (t) β(r)s,c(), µ(r 1), σ (r 1), α(r), β(r), κ (r)] ] α(r). The teraton s repeated wth ( α ( r), κ (r)) = ( α(r), κ (r)) untl convergence s effectvely obtaned, whereby a fnal estmate ( α(r), κ (r)) s returned. Appendx D: Condtonal expectaton of nosy observatons under the GLG model Let the stuaton be as n Secton 5. and consder the GLG model. The jont densty of (s, v ) s found just as n the nose-free case n Secton 3..1, ( [ ]) exp 1 v p(s, v µ ρ(), σρ() ) = p(v s )p(s µ ρ(), σρ() ) = exp(s + (s µ ρ() ) )+σε σρ() πσ ρ() exp(s ) + σε and the margnal densty of the wavelet wth nose s q(v µ ρ(), σρ() ) = p(s, v µ ρ(), σρ() ) ds., 4

27 We do not have a closed form expresson for ths ntegral, but due to the form of the ntegrant we approxmate the ntegral wth the Gauss-Hermte quadrature rule, see e.g. Press et al. (00). The condtonal densty of s gven v s ( [ p(s v, µ ρ(), σρ() ) = p(s, v µ ρ(), σρ() ) exp 1 q(v µ ρ(), σρ() ) = v exp(s )+σ ε ]) + (s µ ρ() ) σρ() c(v µ ρ(), σ ρ() ) exp(s ) + σ ε where c(v µ ρ(), σρ() ) = πσ ρ()q(v µ ρ(), σρ() ). Furthermore, from well-known results about the bvarate normal dstrbuton we obtan Hence E[w s, v, θ] = Corr[w, v s ] whereby we obtan (6). References Var[w s ] Var[v s ] v = Var[w s ] Var[v s ] v = exp(s ) exp(s ) + σε v. E[w v, θ] = E [ E[w s, v, θ] v, θ ] [ exp(s ) = v E exp(s ) + σε ] v, θ Besag, J. (1974). Spatal nteracton and the statstcal analyss of lattce systems (wth dscusson), Journal of the Royal Statstcal Socety: Seres B 36(): Besag, J. (1975). Statstcal analyss of non-lattce data, Statstcan 4(3): Cho, H. & Baranuk, R. (001). Multscale mage segmentaton usng wavelet-doman hdden Markov models, IEEE Transactons on Image Processng 10(9): Crouse, M. S., Nowak, R. D. & Baranuk, R. G. (1998). Wavelet-based statstcal sgnal processng usng hdden Markov models, IEEE Transactons on Sgnal Processng 46(4): Dempster, A. P., Lard, N. M. & Rubn, D. B. (1977). Maxmum lkelhood from ncomplete data va the EM algorthm, Journal of the Royal Statstcal Socety: Seres B 39(1): Durand, J.-B., Gonçalvès, P. & Guédon, Y. (004). Computatonal methods for hdden Markov tree models an applcaton to wavelet trees, IEEE Transactons on Sgnal Processng 5(9): Gao, X. & Song, P. X.-K. (011). Composte lkelhood EM algorthm wth applcatons to multvarate hdden Markov model, Statstca Snca 1(1): Khan, M. E. (01). Varatonal Learnng for Latent Gaussan Models of Dscrete Data, PhD thess, The Unversty of Brtsh Columba. 5

28 Laurtzen, S. L. (1996). Graphcal Models, Clarendon Press, Oxford. Martns, T. G., Smpson, D., Lndgren, F. & Rue, H. (013). Bayesan computng wth INLA: new features, Computatonal Statstcs and Data Analyss 67: Po, D. D.-Y. & Do, M. N. (006). Drectonal multscale modelng of mages usng the contourlet transform, IEEE Transactons on Image Processng 15(6): Press, W. H., Teukolsky, S. A., Vetterlng, W. T. & Flannery, B. P. (00). Numercal Recpes n C++, edn, Cambrdge Unversty Press. R Core Team (013). R: A Language and Envronment for Statstcal Computng, R Foundaton for Statstcal Computng, Venna, Austra. ISBN URL: Romberg, J. K., Cho, H. & Baranuk, R. G. (001). Bayesan tree-structured mage modelng usng wavelet-doman hdden Markov models, IEEE Transactons on Image Processng 10(7): Rue, H. & Held, L. (005). Gaussan Markov Random Felds: Theory and Applcatons, Chapman and Hall, London. Monographs on Statstcs and Appled Probablty, vol Rue, H. & Martno, S. (007). Approxmate Bayesan nference for herarchcal Gaussan Markov random feld models, Journal of Statstcal Plannng and Inference 137(11): Rue, H., Martno, S. & Chopn, N. (009). Approxmate Bayesan nference for latent Gaussan models by usng ntegrated nested Laplace approxmatons, Journal of the Royal Statstcal Socety: Seres B 71(): Sun, J., Gu, D., Chen, Y. & Zhang, S. (004). A multscale edge detecton algorthm based on wavelet doman vector hdden Markov tree model, Pattern Recognton 37(7): Terney, L. & Kadane, J. B. (1986). Accurate approxmatons for posteror moments and margnal denstes, Journal of the Amercan Statstcal Assocaton 81(393):