Ridge, Bayesian Ridge and Shrinkage

Transcription

1 Readings Chapter 15 Christensen Merlise Clyde October 1, 2015

2 Ridge Trace t(x$coef) x$lambda

3 Generalized Cross-validation > select(lm.ridge(employed ~., data=longley, lambda=seq(0, 0.1, ))) modified HKB estimator is modified L-W estimator is smallest value of GCV at > longley.rreg = lm.ridge(employed ~., data=longley, lambda=0.0028) > coef(longley.rreg) GNP.deflator GNP Unemployed Armed.Forces e e e e e-03 Population e-01 Year 1.557e+00

4 Shrinkage Shrinkage Standardized Ridge STA721 Linear Models Duke Standardized University OLS

5 Bayesian Ridge: Prior on k Reparameterization: Y = 1α + (I P 1 )XS 1/2 S 1/2 β + ɛ = 1α + X s β s + ɛ Y c = X s β s + ɛ s ɛ s N(0, (I P 1 )/φ) Ȳ α, φ N(α, 1/(nφ)) U T p Y = Lγ + ɛ p ɛ p N(0, I p /φ) SSE Y T U n p 1 U T n p 1Y G((n p 1)/2, φ/2) Hierarchical prior p(α φ, γ, κ) 1 γ φ, κ N(0, I(φκ) 1 ) p(φ κ) 1/φ prior on κ? Take κ φ G(1/2, 1/2)

6 Posterior Distributions Joint Distribution α, γ, φ κ, Y Normal-Gamma family given Y and κ κ Y not tractable Obtain marginal for γ via Numerical integration MCMC: Full conditionals Pick initial values α (0), β (0), φ (0), Set t = 1 1 Sample κ (t) p(κ α (t 1), γ (t 1), φ (t 1), Y) 2 Sample α (t), γ (t), φ (t) κ(t), Y 3 Set t = t + 1 and repeat until t > T Use Samples α (t), γ (t), φ (t), κ (t) for t = B,..., T for inference Change of variables to get back to β

7 Rao-Blackwellization Model What is best estimate of β from Bayesian perspective? Loss (β a) T (β a) under action a Decision Theory: Take action a that minimizes posterior expected loss which is posterior mean of β. Estimate of posterior mean is Ergodic average of MCMC: i βs(t) /T Posterior mean given κ β s (κ) = (X st X s + κi) 1 X st X s ˆβ s Rao-Blackwell Estimate 1 T (X st X s + κ (t) I) 1 X st X s ˆβ s t

8 Testimators Goldstein & Smith (1974) have shown that if 1 0 h i 1 and γ i = h i ˆγ i 2 γ 2 i Var(ˆγ i ) < 1+h i 1 h i then γ i has smaller MSE than ˆγ i Case: If γ j < Var(ˆγ i ) = σ 2 /l 2 i then h i = 0 and γ i is better. Apply: Estimate σ 2 with SSE/(n - p - 1) and γ i with ˆγ i. Set h i = 0 if t-statistic is less than 1. testimator - see also Sclove (JASA 1968) and Copas ( JRSSB 1983)

9 Generalized Ridge Instead of γ j iid N(0, σ 2 /k) take γ j ind N(0, σ 2 /κ i ) Then Condition of Goldstein & Smith becomes [ 2 γi 2 < σ ] κ i li 2 If l i is small almost any κ i will improve over OLS if li 2 is large then only very small values of κ i will give an improvement. Prior on κ i? Prior that can capture the feature above?

10 Induced prior on β? γ j σ 2, κ j ind N(0, σ 2 /κ j ) β N(0, σ 2 V K 1 V T ) which is not diagonal. Or start with β σ 2, K N(0, σ 2 K 1 ) loss of invarince with linear transformations of X s X s AA 1 β = Zα where A 1 β = α

11 Related Regression on PCA Principal Components of X may be obtained via the Singular Value Decomposition: X = U p LV T the l i are the eigenvalues of X T X Y = 1α + ULV T β + ɛ = 1α + Fγ + ɛ Columns F i U i are the principal components of the data multivariate data X 1,..., X p If the direction F i is ill-defined (l i = 0 or λ i < ɛ then we may decide to not use F i in the model. equivalent to setting γ i = ˆγ i if l i δ γ i = 0 if l i < ɛ

12 Summary OLS can clearly be dominated by other estimators for extimating β Lead to Bayes like estimators choice of penalties or prior hyper-parameters hierarchical model with prior on κ i Shrinkage, dimension reduction & variable selection? what loss function? Estimation versus prediction? Copas 1983

13 Full Conditionals

14 Full Conditionals

15 Full Conditionals