5. Best Unbiased Estimators

Size: px

Start display at page:

Download "5. Best Unbiased Estimators"

Christopher Arnold
5 years ago
Views:

1 Best Ubiased Estimators 1 of 7 7/16/2009 6:13 AM Virtual Laboratories > 7. Poit Estimatio > Best Ubiased Estimators Basic Theory Cosider agai the basic statistical model, i which we have a radom experimet that results i a observable radom variable X takig values i a set S. Oce agai, the experimet is typically to sample objects from a populatio ad record oe or more measuremets for each item. I this case, the observable radom variable has the form X = (X 1, X 2,..., X ) where X i is the vector of measuremets for the i th item. Suppose that θ is a real parameter of the distributio of X, takig values i a parameter space Θ R. Let f θ deote the probability desity fuctio of X for θ Θ. Note that the expected value, variace, ad covariace operators also deped o θ, although we will sometimes suppress this to keep the otatio from becomig too uwieldy. Suppose ow that λ = λ(θ) is a parameter of iterest that is derived from θ. I this sectio we will cosider the geeral problem of fidig the best estimator of λ amog a give class of ubiased estimators. Recall that if U is a ubiased estimator of λ, the var θ (U) is the mea square error. Thus, if U ad V are ubiased estimators of λ ad var θ (U) var θ (V) for all θ Θ The U is a uiformly better estimator tha V. O the other had, it may be the case that U has smaller variace for some values of θ while V has smaller variace for other values of θ. If U is uiformly better tha ay other ubiased estimator of λ, the U is a Uiformly Miimum Variace Ubiased Estimator (UMVUE) of λ. The Cramér-Rao Lower Boud We will show that uder mild coditios, there is a lower boud o the variace of ay ubiased estimator of the parameter λ. Thus, if we ca fid a estimator that achieves this lower boud for all θ Θ, the the estimator must be a UM VUE of λ. The assumptio that we must make is that if h : S R with θ ( h(x) ) < for θ Θ the d θ (h(x)) = θ ( h(x) dl( f θ(x)), θ Θ

2 Best Ubiased Estimators 2 of 7 7/16/2009 6:13 AM 1. Show that this coditio is equivalet to the assumptio that the derivative operator d ca be iterchaged with the expected value operator θ. Geerally speakig, the fudametal assumptio will be satisfied if f θ (x) is differetiable as a fuctio of θ, with a derivative that is joitly cotiuous i x ad θ, ad if the support set {x S : f θ (x) > 0} does ot deped o θ. 2. Show that θ dl (f θ( X)) = 0 for θ Θ. Hit: Use the basic coditio with h(x) = 1 for x S. 3. Show that cov θ ( h(x), dl( f θ(x)) = d θ(h(x)) a. b. First ote that the covariace is simply the expected value of the product of the variables, sice the secod variable has mea 0 by the Exercise 2. Use the basic coditio. 4. Prove the followig result. Hit: The variable has mea 0. dl( f θ (X)) var θ ( = dl( f θ (X)) θ ( ( 5. Fially, use the Cauchy-Scharwtz iequality to establish the Cramér-Rao lower boud, amed for Harold Cramér ad CR Rao: ( d 2 θ(h(x)) θ ( dl( f 2 ) 6. Suppose ow that λ(θ) is a parameter of iterest ad h(x) is a ubiased estimator of λ(θ). Use the Cramér-Rao lower boud to show that θ ( dl( f 2 ) 7. Show that equality holds i Exercise 6, ad hece h(x) is a UM VUE, if ad oly if there exists a fuctio u(θ) such that (with probability 1) a. b. h(x) = λ(θ) + u(θ) dl( f θ(x)) Equality holds i the Cauchy-Schwartz iequality if ad oly if the radom variables are liear trasformatios of each other. Recall also that dl (f θ( X)) has mea 0. 2 )

3 Best Ubiased Estimators 3 of 7 7/16/2009 6:13 AM The quatity θ ( dl 2 (f θ( X)) that occurs i the deomiator of the lower bouds of Exercise 5 ad ) Exercise 6 is called the Fisher iformatio umber of X, amed after Sir Roald Fisher. The followig exercises gives a alterate versio for the expressio i Exercise 6 that is usually computatioally better. 8. Show that if the appropriate derivatives exist ad if the appropriate iterchages are permissible the dl( f θ (X)) θ( ( 2) = d 2 θ ( 2 l( f θ(x)) ) 9. Combie Exercise 6 ad Exercise 8 to show that if λ(θ) is a parameter of iterest ad h(x) is a ubiased estimator of λ(θ) the Radom Samples θ d2 2 l( f Suppose ow that X = (X 1, X 2,..., X ) is a radom sample of size from the distributio of a radom variable X havig probability desity fuctio g θ. 10. Prove the followig special case of the Cramér-Rao lower boud. Hit: The joit probability desity fuctio is the product of the margial probability desity fuctios. ( d 2 θ(h(x)) θ ( dl(g 2 ) 11. Suppose ow that λ(θ) is a parameter of iterest ad h(x) is a ubiased estimator of λ(θ). Use Exercise 10 to show that θ ( dl(g 2 ) From Exercise 11, ote that the Cramér-Rao lower boud varies iversely with the sample size. 12. I the settig of the previous exercise, show the followig result (assume that the appropriate derivatives exist ad the appropriate iterchages are permissible): θ d2 2 l(g

4 Best Ubiased Estimators 4 of 7 7/16/2009 6:13 AM Examples ad Special Cases We will apply the results above to several parametric families of distributios. First we eed to recall some stadard otatio. Suppose that X = (X 2, X 2,..., X ) is a radom sample of size from the distributio of a real-valued radom variable X with mea μ. The sample mea is M = 1 i =1 X i The special ad stadard versios of the sample variace are, respectively, The Beroulli Distributio W 2 = 1 i =1 (X i μ) 2, S 2 = 1 1 i =1 (X i M) 2 Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the Beroulli distributio with ukow success parameter p ( 0, 1). The basic assumptio is satisfied. 13. Show that 1 p (1 p) is the Cramér-Rao lower boud for the variace of ubiased estimators of p. 14. Show that the sample mea M (which is the proportio of successes) attais the lower boud i the previous exercise ad hece is a UM VUE of p. The Poisso Distributio Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the Poisso distributio with ukow parameter a ( 0, ). The basic assumptio is satisfied. 15. Show that a is the Cramér-Rao lower boud for the variace of ubiased estimators of a. 16. Show that the sample mea M attais the lower boud i the previous exercise ad hece is a UMVUE of a. The Normal Distributio Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the ormal distributio with mea μ R ad variace σ 2 ( 0, ). The basic assumptio is satisfied with respect to both of these parameters. Recall also that the fourth cetral momet is ((X μ) 4 ) = 3 σ Show that σ 2 is the Cramér-Rao lower boud for the variace of ubiased estimators of μ. 18. Show that the sample mea M attais the lower boud i the previous exercise ad hece is a

5 Best Ubiased Estimators 5 of 7 7/16/2009 6:13 AM UMVUE of μ. 19. Show that 2 σ 4 is the Cramér-Rao lower boud for the variace of ubiased estimators of σ Show (or recall) that the sample variace S 2 has variace 2 σ 4 boud i the previous exercise. 1 ad hece does ot attai the lower 21. Show that if μ is kow, the the special sample variace W 2 attais the lower boud i Exercise 19 ad hece is a UMVUE of σ Show that if μ is ukow, o ubiased estimator of σ 2 attais the Cramér-Rao lower boud i Exercise 19. Hit: Use the result i Exercise 7. The Gamma Distributio Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the gamma distributio kow shape parameter k ad ukow scale parameter b ( 0, ). The basic assumptio is satisfied with respect to b. 23. Show that b 2 is the Cramér-Rao lower boud for the variace of ubiased estimators of b. k 24. Show that M k attais the lower boud i the previous exercise ad hece is a UMVUE of b. The Beta Distributio Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the beta distributio with left parameter a > 0 ad right parameter b = 1. The basic assumptio is satisfied with respect to a. 25. Show or recall that the mea ad variace of the distributio are a. b. μ = σ 2 = a a +1 a (a +1) 2 (a +2) 26. Show that the Cramér-Rao lower boud for the variace of ubiased estimators of μ is a 2 (a +1) Show that the sample mea M does ot achieve the Cramér-Rao lower boud i the previous exercise, ad hece is ot a UMVUE of μ. The Uiform Distributio Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the uiform distributio o [ 0, a] where a > 0 is the ukow parameter.

6 Best Ubiased Estimators 6 of 7 7/16/2009 6:13 AM 28. Show that the basic assumptio is ot satisfied. 29. Show that the Cramér-Rao lower boud for the variace of ubiased estimators of a is a 2. Of course, the Cramér-Rao Theorem does ot apply, by the previous exercise. 30. Show (or recall) that V = +1 smaller tha the Cramér-Rao boud i the previous exercise. max {X 1, X 2,..., X } is ubiased ad has variace a 2, which is ( +2) The reaso that the basic assumptio is ot satisfied is that the support set {x R : f a (x) > 0} depeds o the parameter a. Best Liear Ubiased Estimators We ow cosider a somewhat specialized problem, but oe that fits the geeral theme of this sectio. Suppose that X = (X 1, X 2,..., X ) is a sequece of observable real-valued radom variables that are ucorrelated ad have the same ukow mea μ, but possibly differet stadard deviatios. Let σ = (σ 1, σ 2,..., σ ) where σ i = sd(x i ) for i {1, 2,..., }. We will cosider estimators of μ that are liear fuctios of the outcome variables. Specifically, we will cosider estimators of the followig form, where the vector of coefficiets c = (c 1, c 2,..., c ) is to be determied: 31. Show that Y is ubiased if ad oly if i =1 Y = i =1 32. Compute the variace of Y i terms of c ad σ. c i = Use the method of Lagrage multipliers (amed after Joseph-Louis Lagrage) to show that the variace is miimized, subject to the ubiased costrait, whe 2 1 / σ j c j = i =1 c i X i, j {1, 2,..., } 1 / σ 2 i This exercise shows how to costruct the Best Liear Ubiased Estimator (BLUE) of μ, assumig that the vector of stadard deviatios σ is kow. Suppose ow that σ i = σ for i {1, 2,..., } so that the outcome variables have the same stadard deviatio. I particular, this would be the case if the outcome variables form a radom sample of size from a distributio with mea μ ad stadard deviatio σ. 34. Show that i this case the variace is miimized whe c i = 1 mea. for each i ad hece Y = M, the sample

7 Best Ubiased Estimators 7 of 7 7/16/2009 6:13 AM This exercise shows that the sample mea M is the best liear ubiased estimator of μ whe the stadard deviatios are the same, ad that moreover, we do ot eed to kow the value of the stadard deviatio. Virtual Laboratories > 7. Poit Estimatio > Cotets Applets Data Sets Biographies Exteral Resources Key words Feedback

Combining imperfect data, and an introduction to data assimilation Ross Bannister, NCEO, September 2010

Combining imperfect data, and an introduction to data assimilation Ross Bannister, NCEO, September 2010 Combiig imperfect data, ad a itroductio to data assimilatio Ross Baister, NCEO, September 00 rbaister@readigacuk The probability desity fuctio (PDF prob that x lies betwee x ad x + dx p (x restrictio o