STAT 35 Solutios to Homework 3: 30 poits Sprig 205 The objective of this Problem Set is to study the Stei Pheomeo 955. Suppose that θ θ, θ 2,..., θ cosists of ukow parameters, with 3. We wish to estimate these parameters with measuremets X,..., X, oe for each parameter θ i, ad we suppose that the measuremets are idepedet, Normally distributed, with X i N θ i, σ 2 where σ 2 is kow. If v is a -dimesioal vector, we deote v 2 v2 i. Part I: 5 poits. Show that the Maximum-Likelihood stimator for θ is ˆθ i,ml X i, i,..., Sice we have X i Nθ i, σ 2, the likelihood fuctio is give by likθ i f θi X i 2πσ 2 e 2σ 2 X i θ i 2 We wat to fid the value of θ i that maximizes the likelihood fuctio or equivaletly the log likelihood fuctio. Takig the log of the likelihood fuctio, we have lθ i loglikθ i 2 log2πσ2 2σ 2 X i θ i 2 To idetify the value of θ i which maximizes the log-likelihood fuctio, we set the derivative with respect to θ i to zero, ad solve for θ i. Thus lθ i θ i σ 2 X i θ i implyig that so σ 2 X i ˆθ i,ml 0 ˆθ i,ml X i as required.
2. Show that the Method of Momets stimator for θ is ˆθ i,mm X i, i,..., Sice X i Nθ i, σ 2, it follows immediately that X i θ i so that ˆθ i,mm X i sice we have oly oe observatio X i Nθ i, σ 2. 3. Recall that the risk of a estimator ˆθ is defied as Rˆθ ˆθ θ 2 ˆθ i θ i 2 Show that the risk of both ˆθ ML ad ˆθ MM is Rˆθ ML Rˆθ MM σ 2 Rˆθ ML Rˆθ MM X i θ i 2 Xi 2 2X i θ i + θi 2 [ X 2 i 2θi X i + θi 2 ] [ σ 2 + θi 2 2θi 2 + θi 2 ] σ 2 σ 2 Note that X 2 i V arx i + X i 2 2
Part II: 0 poits We will ow see that it is possible to fid a estimator for θ with smaller risk. We defie as the Stei estimator of θ. ˆθ i,s X i 2σ2 X i, i,...,. Show that Rˆθ S X θ 2 + 2 2σ 2 θ + 2 2 σ 4 R ˆθiS ˆθ ii,s θ i 2 X i 2σ2 X i θ i 2 X 2σ2 i θ i X i X i θ i 2 2σ2 2σ 2 2 2X i θ i X i + X i X i θ i 2 2 2σ 2 X i θ i X 2 i + 2 2 σ 4 Xi X i θ i 2 + 2 2σ 2 θ X + 2 2 σ 4 2 X 4 X i θ i 2 + 2 2σ 2 θ + 2 2 σ 4 X θ 2 + 2 2σ 2 θ + 2 2 σ 4 2 2. Stei s Lemma says that, whe X N µ, σ 2 ad h is a smooth fuctio, the Use Stei s Lemma to show that X µhx σ 2 h X Rˆθ S σ 2 2 2 σ 4 3
Hit : For every i,...,, cosider hx i Hit 2: Recall that X + Y X + Y. X i X i X 2 i + j i X j 2. Recall that R ˆθiS X θ 2 + 2 2σ 2 θ + 2 2 σ 4 ad from Part I, we kow that X θ 2 σ 2. Thus to show this result, we oly eed to show that 2 2σ 2 θ 2 2 2 σ 4 which we ca do as follows θ θ Xi θ i X i X i X i θ i X 2 i + j i X2 j σ 2 X i X i Xi 2 + j i X2 j σ 2 X 2 i + 2Xi 2 j i X2 j Xi 2 + j i X2 j σ 2 X2 i + j X2 j 2X2 i Xi 2 + 2 j i X2 j X σ 2 2 2X 2 i X 4 [ ] X σ 2 2 2 i X 4 ] σ [ 2 2 X2 i X ] 4 σ [ 2 2 X 2 σ 2 2 Thus we have show that R ˆθiS σ 2 2 2 σ 4 2 Hit 4
3. Show that Rˆθ S < Rˆθ ML if 3 Recall that Rˆθ ML σ 2, so sice 2 2 σ 4 > 0 for 3. Rˆθ S σ 2 2 2 σ 4 < σ 2 Rˆθ ML 4. The Cramer-Rao boud ca be exteded to the multivariate case. I particular, the risk of a ubiased estimator ˆθ satisfies Rˆθ i varˆθ i i Iθ i Justify that ˆθ S is a biased estimator. Note: You do ot eed to compute the bias. Recall that the Fisher iformatio for θ is give by [ 2 ] Iθ 0 2 θ log fx; θ θθ0 We first compute the Fisher iformatio for θ i : fx; θ i 2πσ 2 e 2σ 2 x θ i 2 takig the log yields log fx; θ i 2 log2πσ2 2σ 2 x θ i 2 Differetiatig with respect the θ i, we get Ad differetiatig agai, we get So takig the egative of the expectatio implies that θ i log fx; θ i σ 2 x θ i 2 2 θ i log fx; θ i σ 2 Iθ i σ 2 Thus sice a ubiased estimator, ˆθ, of θ i must have risk satisfyig Rˆθ i σ 2 σ 2 It follows that ˆθ S is biased, sice we showed i previous parts that Rˆθ S < Rˆθ ML σ 2 5
Part III: 5 poits Now, suppose that we have m iid measuremets G,..., G m of the proportio of water i a grapefruit, m iid measuremets P,..., P m of the proportio of water i a potato, ad m iid measuremets S,..., S m of the proportio of water i a sea bass. For simplicity, assume that the measuremets all have the same precisio i.e. the same variace, which is kow i advace.. If you oly care about fruits ad vegetables, how would you estimate the water cotet of both grapefruit ad potato? Note that the Stei estimator simultaeously calculates a estimate of proportio of water i each of grapefruit, potatoes ad sea bass, despite the fact that these quatities are urelated. The ML ad MOM estimates, however, are applied separately to estimate the proportio of water i grapefruit, potatoes ad sea bass respectively. We showed above that the Stei estimator is superior to the ML ad MOM estimators whe 3, so sice i this example we are estimatig oly 2 parameters, we would use ML or MOM to estimate the proportio of water i grapefruit ad potato separately. 2. If you also care about fish, how would you adjust the estimator? Why? As argued i the previous questio, the Stei estimator is superior to the ML ad MOM estimators whe 3. I this example, we have that 3, ad thus we should estimate the proportio of water i grapefruit, potatoes ad sea bass simultaeously usig the Stei estimator. 6
Part IV: 5 poits bous The Stei estimator is also kow as a shrikage estimator. For large, let us see that this is ideed the case. Assume that the parameters θ are modeled as iid samples from the Normal distributio θ i N 0, c 2. We ca rewrite ˆθ S as ˆθ i,s ρx i, with ρ σ 2. 2 X 2. By writig X i θ i + X i θ i, ad usig the Law of Large Numbers, justify that 2 X 2 2 σ2 + 2 c2,. Hit : Observe that Z i θ i X i θ i are iid radom variables, sice θ i N 0, c 2 ad X i θ i N 0, σ 2. Hit 2: Use the fact that if A A ad B B ad A ad B are costats, the A + B A + B. Sice 2 X 2 2 2 2 2 2 X 2 i X 2 i θ i + X i θ i 2 [ θ 2 i + 2θ i X i θ i + X i θ i 2] θi 2 + 2 2 θ i X i θ i + 2 X i θ i 2 LLN 2 θ2 + 2 2 θ X θ + 2 X θ 2 2 c2 + 2 σ2 θ 2 V arθ + θ 2 c 2 + 0 c 2 θ X θ θ X θ 0 X θ 2 V arx θ + X θ 2 σ 2 + 0 σ 2 2. Deduce that 0 < p < i the case. Therefore, the Stei estimator shriks each coordiate towards zero i order to reduce the total expected risk Sice ρ σ 2 2 X 2 7
it follows from oticig that σ 2 > 0, 2 > 0 ad > 0, for 3, that ρ < Next, the previous questio provided a limit that, by oticig 2, yields So ρ σ2 c 2 + σ 2 > σ2 σ 2 0 ρ > 0 8