Last Lecture Biostatistics 60 - Statistical Iferece Lecture Cramer-Rao Theorem Hyu Mi Kag February 9th, 03 If you kow MLE of, ca you also kow MLE of τ() for ay fuctio τ? What are plausible ways to compare betwee differet poit estimators? 3 What is the best ubiased estimator or uiformly ubiased miimium variace estimator (UMVUE)? 4 What is the Cramer-Rao boud, ad how ca it be useful to fid UMVUE? Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 / 4 Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 / 4 : Cramer-Rao iequality : Cramer-Rao boud i iid case Theorem 739 : Cramer-Rao Theorem Let X,, X be a sample with joit pdf/pmf of f X (x ) Suppose W(X) is a estimator satisfyig EW(X) ] τ(), Ω VarW(X) ] < For h(x) ad h(x) W(x), if the differetiatio ad itegratios are iterchageable, ie d d Eh(x) ] d h(x)f X (x )dx h(x) d x X x X f X(x )dx Corollary 730 If X,, X are iid samples from pdf/pmf f X (x ), ad the assumptios i the above Cramer-Rao theorem hold, the the lower-boud of VarW(X) ] becomes VarW(X) ] τ ()] E { log f X(X )} ] The, a lower boud of VarW(X) ] is τ ()] VarW(X) ] E { log f X(X )} ] Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 3 / 4 Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 4 / 4
: Score Fuctio : Fisher Iformatio Number Defiitio: Score or Score Fuctio for X X,, X iid f X (x ) S(X ) log f X(X ) E S(X )] 0 S (X ) log f X(X ) Defiitio: Fisher Iformatio Number { } ] I() E log f X(X ) E S (X ) ] { } ] I () E log f X(X ) { } ] E log f X(X ) I() The bigger the iformatio umber, the more iformatio we have about, the smaller boud o the variace of ubiased estimates Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 5 / 4 Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 6 / 4 : Simplified Fisher Iformatio - Normal Distributio Lemma 73 If f X (x ) satisfies the two iterchageability coditios d f X (x )dx d x X x X d d f X(x )dx x X x X f X(x )dx f X(x )dx which are true for expoetial family, the { } ] ] I() E log f X(X ) log f X(X ) X,, X iid N (µ, σ ), where σ is kow I(µ) µ µ ] µ log f X(X µ) { ( µ log exp πσ { log(πσ ) { }] (X µ) σ σ )}] (X µ) σ }] (X µ) σ The Cramer-Rao boud for µ is I(µ)] σ Var(X) Therefore X attais the Cramer-Rao boud ad thus the best ubiased estimator for µ Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 7 / 4 Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 8 / 4
Example of Cramer-Rao lower boud attaimet Solutio (cot d) Problem iid X,, X Beroulli(p) Is X the best ubiased estimator of p? Does it attai the Cramer-Rao lower boud? Solutio E(X) p Var(X) p( p) Var(X) { } ] I(p) E log f X(X ) p ] log f X(X ) p Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 9 / 4 f X (x ) p x ( p) x log f X (x ) x log p + ( x) log( p) p log f X(x p) x p x p p log f X(x p) x p x ( p) I(p) X p X ] ( p) p p p + p ( p) p + p p( p) Therefore, the Cramer-Rao boud is I(p) p( p) VarX, ad X attais the Cramer-Rao lower boud, ad it is the UMVUE Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 0 / 4 Regularity coditio for Cramer-Rao Theorem d h(x)f X (x )dx h(x) d x X x X f X(x )dx This regularity coditio holds for expoetial family How about o-expoetial family, such as iid X,, X Uiform(0, )? Usig Leibitz s Rule Leibitz s Rule d b() f(x )dx f(b() )b () f(a() )a () + d a() Applyig to Uiform Distributio d h(x) d 0 f X (x ) / ) dx h() ( 0 d d h(0)f X(0 ) d0 d + ( ) h(x) dx 0 b() a() f(x )dx ( ) h(x) dx The iterchageability coditio is ot satisfied Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 / 4 Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 / 4
Solvig the Uiform Distributio Example Whe is the Cramer-Rao Lower Boud Attaiable? If X,, X iid Uiform(0, ), the ubiased estimator of is T(X) + X () ] + E X () ] + Var X () ( + ) < The Cramer-Rao lower boud (if iterchageability coditio was met) is I() It is possible that the value of Cramer-Rao boud may be strictly smaller tha the variace of ay ubiased estimator Corollary 735 : Attaimet of Cramer-Rao Boud Let X,, X be iid with pdf/pmf f X (x ), where f X (x ) satisfies the assumptios of the Cramer-Rao Theorem Let L( x) i f X(x i ) deote the likelihood fuctio If W(X) is ubiased for τ(), the W(X) attais the Cramer-Rao lower boud if ad oly if log L( x) S (x ) a()w(x) τ()] for some fuctio a() Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 3 / 4 Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 4 / 4 Proof of Corollary 735 Proof of Corollary 735 (cot d) We used Cauchy-Schwarz iequality to prove that Cov{W(X), ] X(X )}] log f VarW(X)]Var log f X(X ) I Cauchy-Schwarz iequality, the equality satisfies if ad oly if there is a liear relatioship betwee the two variables, that is log f X(x ) log L( x) a()w(x) + b() ] E log f X(X ) E S (X )] 0 E a()w(x) + b()] 0 a()e W(X)] + b() 0 a()τ() + b() 0 b() a()τ() log L( x) a()w(x) a()τ() a() W(x) τ()] Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 5 / 4 Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 6 / 4
Revisitig the Beroulli Example Method Usig Corollary 735 Problem iid X,, X Beroulli(p) Is X the best ubiased estimator of p? Does it attai the Cramer-Rao lower boud? L(p x) p x i ( p) x i i log L(p x) log p x i ( p) x i i logp x i ( p) x i ] i x i log p + ( x i ) log( p)] i log p x i + log( p)( x i ) i i Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 7 / 4 Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 8 / 4 Method Usig Corollary 735 (cot d) log L(p x) p i x i i x i p p x ( x) p p ( p)x p( x) p( p) (x p) p( p) a(p)w(x) τ(p)] where a(p) p( p), W(x) x, τ(p) p Therefore, X is the best ubiased estimator for p ad attais the Cramer-Rao lower boud Normal distributio example Problem iid X,, X N (µ, σ ) Cosider estimatig σ, assumig µ is kow Is Cramer-Rao boud attaiable? Solutio ] I(σ ) (σ ) log f X(X µ, σ) p ] f(x µ, σ ) πσ exp (x µ) σ log f(x µ, σ ) log(πσ ) (σ ) log f(x µ, σ ) (x µ) + σ (σ ) (x µ) σ Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 9 / 4 Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 0 / 4
Solutio (cot d) Is Cramer-Rao lower-boud for σ attaiable? (σ ) log f(x µ, σ ) (x µ) (σ ) (σ ) 3 ] I(σ (x µ) ) σ4 σ 6 Cramer-Rao lower boud is ˆσ i (x i x), gives σ 4 + σ 6 E(x µ) ] σ 4 + σ 6 σ σ 4 I(σ ) σ4 Var( ˆσ ) The ubiased estimator of σ4 > σ4 So, ˆσ does ot attai the Cramer-Rao lower-boud Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 / 4 L(σ x) i log L(σ x) log(πσ ) log L(σ x) σ exp (x i µ) ] πσ σ π πσ + σ + σ 4 i (x i µ) i (x i µ) (σ ) σ (x i µ) σ 4 i ( i (x i µ) ) σ a(σ )(W(x) σ ) Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 / 4 Is Cramer-Rao lower-boud for σ attaiable? (cot d) Therefore, If µ is kow, the best ubiased estimator for σ is i (x i µ) /, ad it attais the Cramer-Rao lower boud, ie i Var (X i µ) ] σ4 If µ is ot kow, the Cramer-Rao lower-boud caot be attaied At this poit, we do ot kow if ˆσ i (x i x) is the best ubiased estimator for σ or ot Today : Cramero-Rao Theorem of Cramer-Rao Theorem ad Corollary Examples with Simple Distributios Next Lecture Rao-Blackwell Theorem Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 3 / 4 Hyu Mi Kag Biostatistics 60 - Lecture February 9th, 03 4 / 4