STA 114: Statistics. Notes 10. Conjugate Priors

Transcription

1 STA 114: Statistics Notes 10. Conjugate Priors Conjugate family Once we get a /pmf ξ(θ x) by combining a model X f(x θ) with a /pmf ξ(θ) on θ Θ, a report can be made by summarizing the. It helps to have the /pmf in a recognizable form so that we can easily compute its mean, spread, quantiles etc. This is not guaranteed to happen in general. For example, the model X Binomial(n, p) and the ξ(p) = e p /(e 1), p [0, 1] lead to the ξ(p x) = const p x (1 p) nx e p, p [0, 1], with a constant term that is fairly difficult to compute, making it difficult to get summaries of this (recall Lab 4). However, for certain models, certain s do lead to s that are analytically tractable. We already saw one in the female birth rate analysis, a uniform for the binomial model gives a beta. In fact more is true for this model: any beta leads to a beta! This phenomenon is called conjugacy. A formal definition is given below. Definition 1 (Conjugacy). A collection of s (or pmfs) is called a conjugate family for a model X f(x θ), θ Θ, if whenever a ξ(θ) is chosen from the collection, it leads to a ξ(θ x) that is also a member of the collection, for every observation X = x. Conjugacy in itself is not a very useful property. For example the collection of all s on Θ is surely conjugate to the model. It becomes useful when a small collection of s exhibit conjugacy to a certain statistical model. By a small collection we usually mean a collection of s/pmfs G = {g(θ a) : a A} indexed by a low-dimensional vector a. G is conjugate to a statistical model X f(x θ) if ξ(θ) = g(θ a) for some a A means for every x, ξ(θ x) = g(θ a ) for some a A. As mentioned before, the collection of beta s {Beta(a, b) : a > 0, b > 0} is a (-dimensional) conjugate family to the binomial model X Binomial(n, p), p [0, 1]. Table?? below gives a list of other common models with known, low-dimensional conjugate families. We will establish conjugacy for three of the listed models; you re required to do the maths for the remaining ones in HW5. The binomial-beta conjugacy The of bet(a, b) distribution, for a > 0, b > 0. equals g(p) = p a1 (1 p) b1 /B(a, b), p [0, 1], where B(a, b) = 1 0 qa1 (1 q) b1 dq is known as the Beta function. If we take ξ(p) = Beta(a, b) (for some a > 0, b > 0) as the for a binomial model X Binomial(n, p), p [0, 1], then for any observations x {0, 1,, n}, ξ(p x) = const f(x p)ξ(p) = const p x (1p) nx p a1 (1p) b1 = const p a+x1 (1p) b+nx1. 1

2 Model Parameter Prior Posterior X Binomial(n, p) 0 p 1 Beta(a, b) Beta(a, b ) a > 0, b > 0 a = a + x b = b + n x X = (X 1,, X n ) λ > 0 Gamma(a, b) Gamma(a, b ) X i Poisson(λ) a > 0, b > 0 a = a + n x b = b + n X = (X 1,, X n ) λ > 0 Gamma(a, b) Gamma(a, b ) X i Exponential(λ) a > 0, b > 0 a = a + n b = b + n x X = (X 1,, X n ) < < Normal(a, b ) Normal(a, b ) Normal(, σ ) < a < a = nb x+σ a nb +σ σ known b > 0 b = σ b nb +σ X = (X 1,, X n ) < < Nχ (m, k, r, s) Nχ (m, k, r, s ) Normal(, σ ) σ > 0 < m < m = km+n x X i X i k > 0, r > 0, s > 0 s = k+n k = k + n r = r + n rs+ kn k+n ( xm) +(n1)s x r+n Table 1: Conjugate and for some common models. But the of Beta(a+x, b+nx) (note: a+x > 0, b+nx > 0) is p a+x1 (1p) b+nx1 /B(a+ x, b + n x). Therefore ξ(p x) is a constant multiple of the Beta(a + x, b + n x). But if two s are constant multiples of each other, they must be identical (and the constant must be 1). So ξ(p x) = Beta(a + x, b + n x). The normal-normal conjugacy Next we show that for the model X 1,, X n Normal(, σ ), (, ), σ fixed, the ξ() = Normal(a, b ) gives a ξ( x) = Normal(a, b ) for some a and b [which we shall identify]. It suffices to show that ξ( x) is a constant multiple of the Normal(a, b ) density. This is equivalent to showing log ξ( x) = const + ( a ) b

3 by going in the log-scale. Now, by definition log ξ( x) = const + l x () + log ξ() n i=1 = const (x i ) ( a) σ b n i=1 = const (x i x) + n( x ) σ = const 1 [ ] n( x ) ( a) + σ b = const 1 ( nb x+σ a nb +σ ) b σ nb +σ ( a) b and therefore, ξ( x) = Normal( nb x+σ a completion of squares identity (give it a try!): b, σ ). nb +σ nb +σ The last equality above follows from a n( x ) ( a) + σ b = (nb + σ nb x+σa )( ) nb +σ + b σ A conjugate family for the full normal model n( x a) nb + σ. For the full normal model X 1,, X n Normal(, σ ), (, σ ) (, ) (0, ), we need a bivariate ξ(, σ ) on (, ) (0, ). There are several choices here. For example we could take ξ(, σ ) = g()h(σ ) where g() is a on (, ) and h(σ ) is a on (0, ). This is in fact widely used, with g() usually taken to be a normal and h(σ ) taken to be an inverse-gamma (i.e., the of 1/σ is a gamma ). However, the family of such s are not conjugate to the model. In particular, the ξ(, σ x) does not factor into a product g( x)h(σ x). There is however, a conjugate family of s, known as the normal-inverse-chi-square s, denotes Nχ (m, k, r, s) with parameters m (, ), k > 0, r > 0 and s > 0. The of this distributionis given by: g(w, v) = const. v r+3 exp ( k(w m) + rs v where the constant equals (rs/)r/ πγ(r/). Here are two important results. ), (w, v) (, ) (0, ) Result 1. A pair of random variables (W, V ) has a Nχ (m, k, r, s) if and only if 1. rs V χ (r), and. [W V = v] Normal(m, v/k). Result. If (W, V ) Nχ (m, k, r, s) then W m t(r). s/k 3

4 The first result can be proved by direct calculations (you re welcome to try it). The second result follows once you note that if (W, V ) Nχ (m, k, r, s) then by Result 1, we can write W = m + V/kZ where Z Normal(0, 1) is independent of V. Also U = rs V χ (r). Then, W m V/kZ = = Z t(r). s/k s/k U/r With this as the prelude, we re now ready to prove that for ξ(, σ ) = Nχ (m, k, r, s) we get ξ(, σ x) = Nχ (m, k (s), r, s ) where m = km+n x k+n k = n + k r = n + r s = 1 kn (rs + ( x n+r k+n m) + (n 1)s x). Working in the log-scale we have log ξ(, σ x) = l x (, σ ) + log ξ(, σ ) = const n n log σ i=1 (x i ) r + 3 log σ k( m) + rs σ σ = const n + r + 3 n log σ i=1 (x i ) + k( m) + rs σ = const n + r + 3 log σ (n 1)s x + n( x ) + k( m) + rs. σ There is another square-completion identity that we need (try this): ( n( x ) + k( m) = (k + n) Plugging this in we get log ξ(, σ x) = const r + 3 and hence ξ(, σ x) = Nχ (m, k, r, s ). Putting it to use ) km + n x + kn k + n k + n ( x m). log σ k ( m ) + r s σ It is nice to have low-dimensional conjugate family for a statistical model one is interested in. But we still have the job of deciding upon one member of the family to ultimately use as the /pmf. If a singular choice is hard to justify, we can run the analysis for a multitude of reasonable choices and present the analysis side by side. This can be done both graphically, and more important via tables. To summarize a or a, we can report a number representing the center of the, such as the mean or the median. To summarize the spread of the, we can 4

5 Prior Prior summary Posterior Posterior summary Mean [.05,.975 ] Mean [.05 (x),.975 (x)] Gamma(,.) 10 (1.1, 7.86) Gamma(15, 10.) 14.9 (1.63, 17.36) Gamma(5,.5) 10 (3.5, 0.48) Gamma(155, 10.5) (1.53, 17.17) Gamma(50, 5) 10 (7.4, 1.96) Gamma(00, 15) (11.55, 15.4) Gamma(500, 50) 10 (9.14, 10.9) Gamma(650, 60) (10.0, 11.68) Table : Poisson-gamma example, and summaries report an interval [a, b] such that the packs most of its area inside the interval. If we want an 1 α area inside, for some small α, then we can take a = the α/-th quantile of the and b = the (1 α/)-th quantile of it. For a ξ(θ), we ll denote its u-th quantile by θ u and for a ξ(θ x) we will use the notation θ u (x). We will usually work with α = 5% and hence report [θ.05, θ.975 ] from the and [θ.05 (x), θ.975 (x)] from the. Example. Consider hurricane counts X 1,, X n for n = 10 consecutive years in the north Atlantic basin, modeled as X i Poisson(). Figure?? and Table?? show - summaries for 4 choices of the from the conjugate family of gamma s. The observed data are x = (1, 14, 15, 1, 16, 14, 7, 10, 14, 16). 5

6 a =, b = 0. a = 5, b = a = 50, b = 5 a = 500, b = Figure 1: Prior- summary for hurricane count data with Gamma(a, b) conjugate s. 6