5 Statistical Inference

Size: px

Start display at page:

Download "5 Statistical Inference"

Rudolph Richard
5 years ago
Views:

1 5 Statistical Iferece 5.1 Trasitio from Probability Theory to Statistical Iferece 1. We have ow more or less fiished the probability sectio of the course - we ow tur attetio to statistical iferece. I statistical iferece, we wat to use iformatio provided by the outcome of a sample to make ifereces, or educated guesses, about the uderlyig probability distributio that geerated the sample. As a example, suppose that we have a sample draw from a distributio with ukow mea µ ad ukow variace σ. Ad the sample mea =5ad sample variace S =3. What do you guess the value of µ is? [Poit estimatio of µ] What rage of values do you guess µ lies i? [Iterval Estimatio of µ] Do you thik µ =6or µ<6? [Hypothesis test cocerig the value of µ]. What is statistical iferece? Statistical iferece draws coclusios about a populatio [i.e., probability desity fuctio] from a radom sample that has supposedly bee draw from that populatio. 3. What do we exactly do i statistical iferece? [parametric statistical iferece] The most commo procedure i statistical iferece is parametric statistical iferece. Let a radom variable have a p.d.f. that is of kow fuctioal form but depeds upo oe or more ukow parameters θ. We ca compactly deote this situatio by writig the p.d.f. as f (x; θ) for θ Θ. The set Θ is called the parameter space. We are thus cofroted with a parametric family of distributios {f (x; θ) :θ Θ}. A ivestigator (you!) may wat to kow exactly oe p.d.f. out of this family as beig the p.d.f. of this radom variable. That is, you may eed to have a poit estimatio of θ. Or you may wat to have more iformatio about what regio this parameter θ is more likely to lie i, that is you may wat to have a iterval estimatio of θ. Example 1: The family of all distributios which are N µ, σ for differet values of µ ad σ form a parametric family: θ = µ, σ. Θ = R R ++. Example : The family of Ber(p) distributios for differet values of p form a parametric family: θ = p, Θ =[0, 1]. 57

2 Example 3: The family of all uiform distributios [µ 0.5, θ +0.5] distributios for differet values of µ form a parametric family of distributios: θ = µ, Θ = R. Example 4: The family of all uiform [µ β,µ+ β] distributios for differet values of µ ad β form a parametric family of distributios: θ =(µ, β) ad Θ = R. Sice the probability distributio of is kow oce we kow the ukow parameters θ, it is reasoable to attempt to estimate the values of these parameters by usig a observatio of a sample of iid r.v.s with the same distributio as. 4. Parametric Poit Estimatio: To have a poit estimator of some parameter θ, we start with defiig some statistic u ( 1,..., ) where 1,..., are radomly draw from the iterested p.d.f. [A statistic which is used to estimate a parameter is called a estimator] Ifx 1,...,x are the realizatio of the radom sample, the u (x 1,..., x ) will be a poit estimate of θ. [The value that the estimator takes oce the sample values areobservediscalledaestimate.] That is, a estimator is a r.v. ad a estimate is aspecific value that the r.v. takes o. (S) ca be used as a estimator of the parameter µ σ ad if 1 = x 1,..., = x are the values 1,..., takes o, the 1 x i is a estimate of µ; Aother estimator for µ is, thesamplemedia(orthesamplemode). I fact, there are lots of reasoable estimators for µ : sample mea, sample media, or you might use the average of the largest ad smalles observatios, or you might look at a weighted average of the sample with weights depedig o rak order of the observatio 1 w i i: + ( i): Questio: what estimator should we choose? 5. Some desirable properties of a poit estimator is as follows: 58

3 [Ubiasedess] Ay statistic ˆθ = u ( 1,..., ) whose mathematical expectatio is equal to a parameter θ of iterest is called a ubiased estimator of the parameter θ. Otherwise it is called to be biased ad the bias is defied to be Eˆθ θ. [Efficiecy] If a estimator has the smallest variace amog ubiased estimators, we say that it is a efficiet estimator. [the smaller the spread the better because the we have higher probability of gettig a value of ˆθ which is close to θ] [Liearity] If u ( 1,..., ) is a liear fuctio, the we say that it is a liear estimator. [Cosistecy] Ay statistic tht coverges i probability to a parameter θ is called a cosistet estimator. [BLUE] If a estimator is liear, ubiased, ad has a miimum variace i the class of all liear ubiased estimators of a parameter, it is called a best liear ubiased estimator [BLUE]. 6. Examples 1: Suppose that 1,..., are draw from a p.d.f with mea µ ad variace σ. The statistic (recall sample mea) u ( 1,..., )= is a liear estimator (obvious) of µ. Itisaubiasedestimatorofµ because E = µ It is obviously cosistet by law of large umber. The sample variace S = P i 1 is a ubiased estimator of σ. 7. Example : Suppose 1,..., are a radom sample from a p.d.f which is N µ, σ. We ca defie two statistics as follows: u 1 ( 1,..., ) = P i u ( 1,..., ) = media ( 1,... ) 59

4 It ca be show that both have expecatio equal to µ, which meas that both are ubiased ad cosistet estimators of µ. But they have differece variace. The variac of u 1 is give by σ /, ad that of u is give by πσ /. So we kow that u 1 is a more efficiet estimator that u. 8. Example 3: Suppose that 1,..., are draw from a p.d.f with mea µ ad variace σ. Let us cosider two statistics, = = P i P i + 1 Note that E = µ, while E = µ/ ( + 1). Therefore is a biased estimator of parameter. But if we cosider, the E µ, therefore is a cosistet estimator of µ. 9. If a estimator is ubiased ad has uiformly miimum variace amog ubiased estimators, the we say that estimator is UMVU (Uiformly miimum variace ubiased). 10. If we have two ubiased estimators ˆθ 1 ad ˆθ,wecadefietherelativeefficiecy of ˆθ 1 to ˆθ to be Varˆθ Varˆθ 1 so ˆθ 1 is relatively more efficiet tha ˆθ if Varˆθ >Varˆθ Some people object to the UMVU criterio because it completely elimiates from cosideratio ay estimator which is biased, eve if the bias is small. For example S = 1 i is ot cosidered as a estimator of σ sice it is biased. However, sometimes it seems thatwemightwattogiveupalittlebiaseifitmeatthatwecouldreducethe spread of the distributio of the estimatio. [Figure Here] A alterative criterio is to choose a estimator ˆθ to miimize m.s.e = E ³ˆθ θ. 60

5 This criterio is called the miimum mea squared error criterio (MSE) Notice the followig m.s.e = E ³ˆθ θ ³ i = E h³ˆθ Eˆθ + Eˆθ θ ³ i = E ³ˆθ Eˆθ +E h³ˆθ Eˆθ Eˆθ θ ³ = Varˆθ + biasˆθ ³ + E Eˆθ θ Hece this criterio allows a tradeoff betwee bias ad variace. The mi m.s.e estimator could be biased, if by allowig a slight bias the variace is decreased by eough to more tha offset the bias. 5. Maximum Likelihood Estimator 1. Defiitio [Likelihood]: Give a radom sample 1,..., from a populatio with p.d.f. f(x; θ), the joit p.d.f of 1,..., regarded as a fuctio of θ is called the likelihood fuctio of the sample. The likelihood fuctio of a sample realizatio x 1,...,x, is writte as L (θ; x 1,..., x ), that is, L(θ; x 1,...,x )= Y i f(x i, θ).. Defiitio [Maximum Likelihood Estimator]: Ifu(x) maximizes a give a likelihood L(θ; x), thewecallu() the maximum likelihood estimator (MLE). Fact: I all cases of practical iterest, MLEs are cosistet although they may ot be ubiased. 3. Example: Cosider i N(µ, 1) i.i.d. i = 1,...,.Here ( ) L(µ; x 1,...,x )=(π) / exp (x i µ) Maximizig L is equivalet to maximizig log L, the log likelihood. We have log L = log(π) (x i µ). 61

6 Settig d log L dµ = i (x i µ) =0 we fid ˆµ(x 1,...,x )= 1 P i x i maximizes L. We thus come up with the statistic ˆµ( 1,..., )= P i as the MLE for µ. Weobservethat E[ˆµ( 1,..., )] = µ so that ˆµ( 1,..., )= is a ubiased estimator for µ. Due to the law of large umbers, we also observe that ˆµ is a cosistet estimator of µ. 4. Example: Let i,i = 1,..., be a radom sample from the expoetial distributio with p.d.f. f (x; θ) = 1 θ e x/θ,x>0, θ > 0. The likelihood fuctio The loglikelihood is Hece L (θ; x 1,...x ) = 1 θ e x 1/θ... 1 θ e x /θ = 1 P θ exp x i θ P l L (θ) = l θ x i θ d l L (θ) dθ = θ + P x i θ =0 yields that P ˆθ = = i is the MLE of θ. Notice the secod order coditio is satisfied. 5. Exercise: Let i,i = 1,..., be a radom sample from N (θ 1, θ ). Fid the MLE of θ 1 ad θ. µ " 1 P # L (θ 1, θ )= exp (x i θ 1 ) πθ θ 6

7 Oe fid the maximizers are give by ˆθ 1 = ˆθ = 1 i. 6. Theorem [Ivariace Priciple of a maximum likelihood estimator]: Sometimeswewish to estimate some fuctio of θ, sayh(θ). For coveiece, assume that η = h(θ) is a 1-1 trasformatio. The the value of η, sayˆη, which maximizes L(θ), equalsh(ˆθ), where ˆθ is the MLE of θ: dh(θ) =h(ˆθ) Example: I the above example, suppose it is desired to estimate µ 3. The m.l.e. of µ 3 equals ˆµ 3. Sometimes it is impossible to fid a closed form expressio for the m.l.e. ad we have to rely o umerical methods. Example: i Γ(α, β) i.i.d. The joit p.d.f. equals µ 1 P L(α, β; x 1,...,x )= Γ(α)β α (x 1...x ) α 1 i exp x i β ad we have to rely o umerical methods to maximize L. 7. Exercise: Let f (x; θ) = 1, 0 <x θ, 0 < θ < θ = 0 elsewhere, ad let 1,..., deote a radom sample from this distributio. Fid the MLE ˆθ of θ. What is the expected value of the MLE ˆθ? Is ˆθ ubiased? 5.3 Method of Momets of Estimator 1. Oe of the early methods of poit estimatio is to simply equate the first sample momet to the first theoretical momet, ad if eeded, equate the secod sample momet to the secod theoretical secod momet etc. 63

8 . [MethodofMometsEstimator:MME]Suppose that 1,..., are i.i.d. There are K parameters of iterest θ 1,...,θ K. Suppose that it is kow that E[ k i ]=M k (θ 1,...,θ K ), k = 1,...,K Also suppose that the map M :(θ 1,...,θ K ) 7 (M 1,...,M K ) is a 1-1 trasformatio, so that the iverse trasformatio M 1 exists. By the law of large umbers, we kow that 1 i k i E[ k i ]=M k (θ 1,...,θ ) Thus, with sufficietly large, we ca practically equate ˆM k 1 P i k i with M k. Usig the iverse trasformatio of M, we ca obtai the method of momets estimator M 1 ( ˆM 1,..., ˆM K ). 3. Example: I the case of the Gamma distributio, we kow that Thus, we have or E[ i ]=αβ, Var( i )=αβ β = Var( i), α = E[ i] E[ i ] Var( i ) β = E[ i ] E[ i], α = E[ i ] E[ i ] E[ i ] E[ i] Replacig E[ i ] ad E[i ] by P 1 i i ad 1 P i i, we obtai a method of momets estimator for α ad β. 4. Example: Let i,i = 1,..., be a radom sample from N µ, σ. Fid the method of momets estimators of µ ad σ. M 1 = E = µ M = E = µ + σ matchig these to empirical first ad secod cetral momets µ = 1 x i µ + σ = 1 64 x i

9 We get ˆµ = 1 ˆσ = 1 x i (x i x). 5. Exercise: Let f (x; θ) = 1, 0 <x θ, 0 < θ < θ = 0 elsewhere, ad let 1,..., deote a radom sample from this distributio. Estimate θ usig themethodofmomets.calclatetheexpectedvalueadthevariaceofthemethod of momet estimator of θ. 6. Exercise: Let 1,..., be a radom sample from the PDF f (x; θ) =θx, 0 < θ x<. Fid the method of momets estimator of θ. 65

10 5.4 Cofidece Itervals Cofidece Iterval for Meas 1. The problem: cosider a radom sample i,i = 1,..., from a ormal distributio N µ, σ. We have earlier show is a ubiased MLE or MOM estimator of µ. But how close is to the ukow parameter µ? To do this we use the distributio of, amely, N µ, σ / to costruct what is called a cofidece iterval for the ukow parameter µ.. Cofidece Iterval: Themaiideaofcostructigcofidece iterval is to realize that a estimator has a samplig distributio. For example if 1,..., are a radom sample from distributio N µ, σ, we kow that ³ ~N µ, σ or equivaletly, µ σ/ N (0, 1). 3. We cosider two cases: (a) CASE I: σ is kow. I this case, we ca for a probability 1 α, fid a umber z α/ from the ormal table such that P µ z α/ µ σ/ z α/ = 1 α which implies that µ µ µ σ σ P z α/ µ + z α/ = 1 α Hece the radom iterval z α/ µ σ, + z α/ µ σ icludes the ukow mea µ with probability 1 α. For example if 1 α =0.95, we have z α/ = z 0.05 =

11 Ocethesampleisobservedadthesamplemeacomputedtoequal x, the iterval µ µ σ σ x z α/, x + z α/ is a kow iterval ad is called the 100(1 α)% cofidece iterval of the ukow mea µ. (b) We do ot kow σ. Weusethefollowigresult: Theorem: If 1,..., are a radom sample of size from N µ, σ, ad the = 1 S = 1 1 i i i. ad S are idepedet; ii. ( 1) S P i σ = σ χ ( 1). PROOF: The proof of (i) proceeds as follows. Oe ca show that S is the product of 1 idepedet ormal variables. Ad it ca be show that is idepedet from these 1 ormal variables. Hece is idepedet from S. [Cautio: ad S are idepedet oly for ormal radom samples.] Proofof(ii):Notethat µ i µ " i + µ W = = σ σ µ i = + µ σ σ because the cross product term is equal to i µ σ =0. Now sice Y i =( i µ) /µ is N (0, 1), hece W χ (). Ad N µ, σ /, hece µ σ = Z χ (1). 67 #

12 Hece (i) implies that ( 1) S σ = are idepedet. Therefore µ i = W Z χ ( 1). µ S/ = µ σ/ σ ad ( 1) S σ µ σ/ q ( 1)S σ / ( 1) t ( 1). We kow that t distributio is symmetric, for ay sigificace level α, we ca fid a value t α/ ( 1) from the statistical table such that µ P t α/ ( 1) µ S/ t α/ ( 1) = 1 α The the (1 α) 100 percet cofidece iterval of µ is give by t α/ ( 1) S, + t α/ ( 1) S. That is, the probability is (1 α) that the above radom iterval cotais the true mea µ. 4. If i,i = 1,... is ot draw from a ormal distributio (with mea µ ad variace σ ). (a) Suppose that σ is kow, the we ca, provided that is large eough (over 30), use the CLT ad obtai µ σ/ N (0, 1) Hece µ µ σ σ x z α/, x + z α/ is the approximate 100(1 α)% cofidece iterval for µ; (b) If σ is ukow but the sample size is 30 or more, the we ca use the fact that µ S/ has a approximate N (0, 1) distributio to obtai a cofidece iterval. 68

13 (c) If σ is ukow ad is small, the we use the fact that µ S/ has a approximate t distributio with d.f Cofidece Iterval for Differece of Two Meas 1. The problem: Let 1,..., ad Y 1,..., Y m be, respectively, two idepedet radom samples of size ad m from two ormal distributios N µ, σ ad N µy, σ Y. Suppose that we are iterested i comparig the meas µ ad µ Y? How should we do it?. CASE I: σ ad σ Y are kow. Sice ad Ȳ are both ormal ad idepedet, we have µ W = Ȳ N µ µ Y, σ + σ Y m Hece Ȳ (µ µ Y ) q σ + σ Ym N (0, 1). Hece P z α/ Ȳ (µ µ Y ) q z α/ = 1 α σ + σ Ym Hece P Ã Ȳ z α/ r σ + σ Y m µ µ Y Ȳ + z α/ r σ + σ Y m Hece " r r # σ ( x ȳ) z α/ + σ Y σ m, ( x ȳ)+z α/ + σ Y m is a 100(1 α)% cofidece iterval for µ µ Y.! = 1 α 3. Case II: if σ ad σ Y are ukow but the sample sizes m ad are large, the we ca replace σ ad σ Y by S ad S Y respectively as the ubiased estimates of the variace. This meas that " ( x ȳ) z α/ r s + s Y m, ( x ȳ)+z α/ is a 100(1 α)% cofidece iterval for µ µ Y. 69 r s + s Y m #

14 4. CASE III: if σ ad σ Y are ukow, the sample sizes are small, ad the samples are from ormal distributios. I geeral this problem is a difficult oe. However if we assume that σ = σ Y, the we ca proceed as follows: We kow that Z = Ȳ (µ µ Y ) q σ + σ m N (0, 1). Moreover, we kow that ( 1) S σ χ ( 1), (m 1) S Y σ χ (m 1). Sice they are idepedet, we have U = ( 1) S σ + (m 1) S Y σ χ (m + ). I additio, the idepedece of the sample meas ad sample variace implies that Z ad U are idepedet as well. The accordig to the defiitio of a t distributed radom variable, we have That is T = r Ȳ (µ µ Y ) q ( 1)S σ σ + σ m + (m 1)S Y σ +m T = q = Z U +m t ( + m ). Ȳ (µ µ Y ) 1 t ( + m ). + m 1 q ( 1)S +(m 1)S Y +m Deote by t α/ ( + m ) to be the value t 0 such that P ( t 0 T t 0 )=1 α we kow that q P ( 1)S Ȳ t +(m 1)SY 0 +m Ȳ + t 0 q ( 1)S +(m 1)S Y +m 1 + m 1 µ µ Y 1 + m 1 = 1 α. Hece giver sample meas x, ȳ ad sample variaces s ad s Y, the 100(1 α)% cofidece iterval for µ µ Y is q x ȳ t ( 1)s α/ ( + m ) +(m 1)s Y q +m ( 1)s x ȳ + t α/ ( + m ) +(m 1)s Y +m 1 + m 1, 1 + m 1 5. OTHER CASES: What if σ 6= σ Y ad the sample size is small? There are approximatio results usig t distributios. (we will omit here).. 70

15 5.4.3 Cofidece Iterval for Variaces 1. Problem I: Suppose that we have a radom sample 1,..., from a ormal distributio N µ, σ. We have show earlier that S is a good estimator of σ. Now, costruct acofidece iterval for σ.. The statistics we use to costruct the cofidece iterval is the χ statistic. Recall that whe the sample is from a ormal distributio, we have ( 1) S σ χ ( 1). Suppose that we wat to costruct a 100(1 α)% cofidece iterval. Let a = χ (1 α/) ( 1) ad b = χ α/ ( 1) [Figure Here], we kow that µ P a ( 1) S σ Hece µ ( 1) S P b Hece give a sample b = 1 α σ ( 1) S = 1 α. a ( 1) s b, ( 1) s a is a 100(1 α)% cofidece iterval of σ ad "r r # ( 1) s ( 1) s, b a is a 100(1 α)% cofidece iterval of σ. 3. Problem II: Suppose that we have a radom sample 1,..., from a ormal distributio N µ, σ ad a radom sample Y1,...,Y m from a ormal distributio N µ Y, σ Y. We wat to compare σ ad σ Y, i.e. fid a cofidece iterval for the ratio σ /σ Y. 4. Howdowedoit?Recallthat ( 1) S σ χ ( 1) (m 1) SY σ χ (m 1) Y 71

16 ad they are idepedet. Recall the defiitio of F distributed radom variable, we kow the F = (m 1)S Y σ Y / (m 1) ( 1)S / ( 1) σ = S Y /σ Y S F (m 1, 1) /σ Let c = F 1 α/ (m 1, 1) =1/F α/ ( 1,m 1) ad d = F α/ (m 1, 1). We have µ µ P c S Y /σ Y S d = P c S /σ SY σ σ d S Y SY = 1 α. Hece a 100(1 α)% cofidece iterval for σ /σ Y is c s s,d s Y s. Y Cofidece Iterval for Proportios 1. The Problem: Cosider a radom sample 1,..., from some distributio F (x). What is the probability that the radom variable will lie i the iterval [a, b], say p = F (b) F (a)? Oe way to estimate it is to fid out the empirical freqecy that the sample x 1,...,x lies i the iterval [a, b], i.e., P ˆp = 1 {x i [a,b]}. Ca we provide a cofidece iterval of p based o this estimator ˆp?. To do this, we covert the radom sample 1,..., ito aother radom sample Y 1,..., Y where Y i = 1 if i [a, b] Y i = 0 otherwise. That is, based o the sample i,i = 1,...,, we create a radom sample from a Beroulli distributio with probability p. The estimator of p is P Ȳ = Y i. Note that EY i = p, V ary i = p (1 p), hece by CLT, we kow that if is large eough, Ȳ p p (0, 1). p (1 p) / N 7

17 Hece Ã P z α/ Ȳ p p p (1 p) / z α/! = 1 α. The problem here is that the variace p (1 p) is ukow (we alluded to this problem earlier). Oe way to proceed is to ote that a cosistet estimator for p is Ȳ (by WLLN), hece Ȳ p qȳ 1 Ȳ / is approximately N (0, 1) as well. Therefore, we have approximately Ȳ p P z α/ z α/ = 1 α. qȳ 1 Ȳ / Hece a 100(1 α)% cofidece iterval for p is give by " # rȳ (1 ȳ ) rȳ (1 ȳ ) ȳ z α/, ȳ + z α/. 3. Example: I a political campaig, oe cadidate has a poll take at radom amog thevoters. Theresultsare185 out of =351 voters favor this cadidate. Eve though Ȳ = 185/351 =0.57, should the cadidate feel very cofidet of wiig? Accordig to our discussio, a approximate 95% cofidece iterval for the fractio p of the voters who favor this cadidate is " r r # , = [0.475, 0.579]. Hece there is a good possibility that p is less tha 50% ad the cadidate should take this ito accout whe campaigig. 73

5. Best Unbiased Estimators

5. Best Unbiased Estimators Best Ubiased Estimators http://www.math.uah.edu/stat/poit/ubiased.xhtml 1 of 7 7/16/2009 6:13 AM Virtual Laboratories > 7. Poit Estimatio > 1 2 3 4 5 6 5. Best Ubiased Estimators Basic Theory Cosider agai