Chapter 6: Point Estimation

Transcription

1 Chapter 6: Point Estimation Professor Sharabati Purdue University March 10, 2014 Professor Sharabati (Purdue University) Point Estimation Spring / 37

2 Chapter Overview Point estimator and point estimate Define point estimator and estimate Unbiased estimators Minimum variance estimators Finding the standard error of an estimator Deriving directly Bootstrapping Methods of point estimation The method of moments The method of maximum likelihood Professor Sharabati (Purdue University) Point Estimation Spring / 37

3 Motivation for Point Estimation Suppose we want to find the ratio p of FIV infected cats in a specific area. It s impossible to check all feral cats in the area. Maybe we can do the following: 1 X = 1 if cat has FIV, and X = 0 if not. 2 p = proportion of FIV infected cats 3 Distribution of X: Bernoulli with p unknown. How to estimate the value of the parameter p of a Bernoulli distribution? RV = Distribution = Unknown parameter of interest Professor Sharabati (Purdue University) Point Estimation Spring / 37

4 Point Estimator and Point Estimate To estimate the value of p, we catch 25 feral cats randomly and check them. Suppose we found that cat number 1,5,10,15,23 are infected with FIV. 1 Bernoulli rv s X 1, X 2,, X 25 form a random sample. 2 Point estimator of p: ˆp = X 1+ +X this is a statistic 3 Point estimate of p: ˆp = 5 25 = this is a value Random sample = Estimator = Estimate Professor Sharabati (Purdue University) Point Estimation Spring / 37

5 Point Estimator and Point Estimate Definition A point estimate of a parameter θ is a single number that can be regarded as a sensible value for θ. A point estimate is obtained by selecing a suitable statistic and computing its value from given sample data. The selected statistic is called the point estimator. Point estimate is a value, point estimator is a statistic. Usually we use ˆθ to denote the point estimator of a parameter θ. Different statistic can be used to estimate the same parameter, i.e., a parameter may have multiple point estimators. Professor Sharabati (Purdue University) Point Estimation Spring / 37

6 Example Assume the breakdown voltage for pieces of epoxy resin to be normally distributed. Now we want to estimate the mean µ of the breakdown voltage. We randomly check 20 breakdown voltages, and denote them as X 1, X 2,, X 20. Suppose the observed voltage values are: {24.46, 25.61, 26.25, 26.42, 26.66, 27.15, 27.31, 27.54, 27.74, 27.94, 27.98, 28.04, 28.28, 28.49, 28.50, 28.87, 29.11, 29.13, 29.50, 30.88} Which point estimators could be used to estimate µ? Answer 1 Sample mean: ˆµ = X 2 Sample median: ˆµ = X 3 Average of the extremes: ˆµ = min(x i)+max(x i ) 2 4 Trimmed mean ˆµ = X tr(10), 10% trimmed mean (discard the smallest and largest 10% of the sample data and then take an average). 5 etc... Professor Sharabati (Purdue University) Point Estimation Spring / 37

7 Example Assume the breakdown voltage for pieces of epoxy resin to be normally distributed. Now we want to estimate the mean µ of the breakdown voltage. We randomly check 20 breakdown voltages, and denote them as X 1, X 2,, X 20. Suppose the observed voltage values are: {24.46, 25.61, 26.25, 26.42, 26.66, 27.15, 27.31, 27.54, 27.74, 27.94, 27.98, 28.04, 28.28, 28.49, 28.50, 28.87, 29.11, 29.13, 29.50, 30.88} Which point estimators could be used to estimate µ? Answer 1 Sample mean: ˆµ = X 2 Sample median: ˆµ = X 3 Average of the extremes: ˆµ = min(x i)+max(x i ) 2 4 Trimmed mean ˆµ = X tr(10), 10% trimmed mean (discard the smallest and largest 10% of the sample data and then take an average). 5 etc... Professor Sharabati (Purdue University) Point Estimation Spring / 37

8 Example We want to know the variance of the elastic modulus of AZ91D alloy. We do not know the population distribution of elastic modulus, but we observed 8 elastic modulus of AZ91D alloy specimens from a die-casting process: {44.2, 43.9, 44.7, 44.2, 44.0, 43.8, 44.6, 43.1} Assume these observations are the result of a random sample X 1, X 2,, X 8 from the population. We actually want to estimate the population variance σ 2. Which point estimators could be used to estimate σ 2? Answer 1 We may use the sample variance: ˆσ 2 = S 2 = 2 We may use: ˆσ 2 = (Xi X) 2 n (Xi X) 2 n 1 Professor Sharabati (Purdue University) Point Estimation Spring / 37

9 Example We want to know the variance of the elastic modulus of AZ91D alloy. We do not know the population distribution of elastic modulus, but we observed 8 elastic modulus of AZ91D alloy specimens from a die-casting process: {44.2, 43.9, 44.7, 44.2, 44.0, 43.8, 44.6, 43.1} Assume these observations are the result of a random sample X 1, X 2,, X 8 from the population. We actually want to estimate the population variance σ 2. Which point estimators could be used to estimate σ 2? Answer 1 We may use the sample variance: ˆσ 2 = S 2 = 2 We may use: ˆσ 2 = (Xi X) 2 n (Xi X) 2 n 1 Professor Sharabati (Purdue University) Point Estimation Spring / 37

10 Choosing an Optimal Estimator A point estimator ˆθ is a random variable, and its value varies from sample to sample error of estimation = ˆθ θ ˆθ = θ + error of estimation Which is the best one? Unbiased Estimator: the one that is closest to the true value of the parameter θ on average. Minimum Variance Estimator: the estimation error to be small. Professor Sharabati (Purdue University) Point Estimation Spring / 37

11 Ideally an estimator should be accurate (low bias) and precise (low variability) Professor Sharabati (Purdue University) Point Estimation Spring / 37

12 Unbiased Estimators and Minimum Variance Estimators Unbiased Estimator Definition Principle of unbiased estimation Some unbiased estimators Minimum Variance Estimator Principle of minimum variance unbiased estimator (MVUE) MVUE for normal mean Professor Sharabati (Purdue University) Point Estimation Spring / 37

13 Unbiased Estimator Definition A point estimator ˆθ is said to be an unbiased estimator of the parameter θ if E(ˆθ) = θ. If ˆθ is not unbiased, the difference E(ˆθ) θ is called the bias of ˆθ. Distribution of θ^1 Distribution of θ^2 E(θ^1) = θ E(θ^2) Professor Sharabati (Purdue University) Point Estimation Spring / 37

14 Some Unbiased Estimators Sample mean X = X 1+X 2 + +X n n population mean µ. is the unbiased estimator of the For continuous, symmetric distributions, sample median X and and trimmed mean are also unbiased estimators of population mean µ. Sample variance S 2 = population variance σ 2. (Xi X) 2 n 1 is the unbiased estimator of the Professor Sharabati (Purdue University) Point Estimation Spring / 37

15 Exercises 1 For X is a binomial(n, p) rv, with p unknown, is the estimator ˆp = X n an unbiased estimator? 2 For normal distribution with mean µ and variance σ 2, given a random sample of size n, X 1, X 2,, X n. Is sample mean X = X 1+ +X n n an unbiased estimator of µ? 3 For any distribution, is sample mean X an unbiased estimator of population mean µ? 4 Given a random sample X 1, X 2,, X n from a continuous uniform distribution f(x) = 1 θ, defined on [0, θ]. Is ˆθ = 2 X = 2(X 1+X 2 + +X n) n an unbiased estimator of θ? Professor Sharabati (Purdue University) Point Estimation Spring / 37

16 Answers to the Exercises 1 For X is a binomial(n, p) rv, with p unknown, is the estimator ˆp = X n an unbiased estimator? Yes [Hint: use the definition of unbiased estimator to check.] 2 For normal distribution with mean µ and variance σ 2, given a random sample of size n, X 1, X 2,, X n. Is sample mean X = X 1+ +X n n an unbiased estimator of µ? Yes [Hint: see PROPOSITION on text p224] 3 For any distribution, is sample mean X an unbiased estimator of population mean µ? Yes [Hint: see PROPOSITION on text p223] 4 Given a random sample X 1, X 2,, X n from a continuous uniform distribution f(x) = 1 θ, defined on [0, θ]. Is ˆθ = 2 X = 2(X 1+X 2 + +X n) n an unbiased estimator of θ? Yes [Hint: check bias = E(ˆθ) θ] Professor Sharabati (Purdue University) Point Estimation Spring / 37

17 = 1 3 (4σ2 σ 2 ) = σ 2 Professor Sharabati (Purdue University) Point Estimation Spring / 37 Principle of Unbiased Estimation When choosing among several different estimators of θ, select the one that is unbiased. Which one is preferred? Let X 1, X 2, X 3, X 4 be a random sample of size 4 from a normal distribution with unknown variance σ 2. To estimate σ 2, we may use ˆσ 2 1 = S 2 = (X i X) 2 or 4 1 ˆσ 2 2 = (X i X) 2 = S2. E( ˆσ 2 1 ) = E(S 2 ) { [ 1 = E 2 Xi ( ]} X i) = 1 { ( ) E X 2 i 1 [ ( 3 4 E ) ]} 2 Xi = 1 { 4σ 2 + 4µ 2 1 ( [ ] [ ( )] )} 2 V ar Xi + E Xi 3 4 = 1 {4σ 2 + 4µ σ2 14 } (4µ)2

18 Why Minimum Variance Estimator? To estimate (unkonwn) p for a binomial distribution with 10 trials? 1 Take a random sample of size 1, and let ˆp 1 = X Take a random sample of size m, and let ˆp 2 = X 1+X 2 +X 3 + +X m 10m Which one is unbiased? ( ) X1 E(ˆp 1 ) = E = E(X 1) = p = p ( ) X1 + + X m E(ˆp 2 ) = E = 1 10m 10m E(X X m ) = 1 10m m 10p = p So both are unbiased. Which one is preferred? Professor Sharabati (Purdue University) Point Estimation Spring / 37

19 Principle of minimum variance unbiased estimator (MVUE) Among all estimators of θ that are unbiased, choose the one that has minimum variance. The resulting ˆθ is called the minimum variance unbiased estimator (MVUE) of θ. While comparing several unbiased estimators, choose the one that has the smallest variance. Which estimator of p is preferred? ( ) X1 V ar(ˆp 1 ) = V ar = 1 p(1 p) (10p(1 p)) = ( ) X1 + + X m V ar(ˆp 2 ) = V ar = 1 10m 10 2 m 2 V ar(x p(1 p) X m ) = 10m Professor Sharabati (Purdue University) Point Estimation Spring / 37

20 MVUE for the Mean of Normal Distribution Theorem Let X 1, X 2,, X n be a random sample from a normal distribution with parameters µ and σ. Then the estimator ˆµ = X is the MVUE for µ. Professor Sharabati (Purdue University) Point Estimation Spring / 37

21 Standard Error of an Estimator Definition of standard error How to compute standard error? Derive the standard deviation directly Use computer intensive methods such as bootstrap Professor Sharabati (Purdue University) Point Estimation Spring / 37

22 Standard Error of an Estimator To denote the precision of the estimator, we may use its variance or standard deviation as a measure. Definition of Standard Error The standard error of an estimator ˆθ is its std. deviation σˆθ = V ar(ˆθ). If the standard error itself involves unknown parameters whose values can be estimated, substitution of these estimators into σˆθ yields the estimated standard error of the estimator. The estimated standard error can be denoted either by ˆσˆθ or by sˆθ. To find the standard error of an estimator, we may: Derive the standard deviation directly. Use computer intensive methods such as bootstrapping. Professor Sharabati (Purdue University) Point Estimation Spring / 37

23 Example Assume that the breakdown voltage for pieces of epoxy resin is normally distributed. To estimate the mean µ of the breakdown voltage, we randomly check 20 breakdown voltages, and denote them as X 1, X 2,, X 20. Suppose the observed voltage values are: {24.46, 25.61, 26.25, 26.42, 26.66, 27.15, 27.31, 27.54, 27.74, 27.94, 27.98, 28.04, 28.28, 28.49, 28.50, 28.87, 29.11, 29.13, 29.50, 30.88} 1 Calculate a point estimate of mean µ of the breakdown voltage and state which estimator you used. 2 Given σ = 1.5, what s the standard error of the estimator? 3 If σ is unknown, what s the estimated standard error of the estimator? Professor Sharabati (Purdue University) Point Estimation Spring / 37

24 Answers 1 We can use estimator X to find an estimate of µ. The estimate of ˆµ = X = 20 = When σ = 1.5 is known, the estimtor X is normal distributed with std. dev. σ X = σ n [Hint: see PROPOSITION on text p224] The standard error of the estimator ˆµ = X is σˆµ = σ X = σ n = = When σ is unknown, we use an estimate of σ (e.g. sample std. dev. s = s 2 = x 2 i ( x i ) = 1.462) to replace σ The estimated standard error of the estimator ˆµ = X is ˆσˆµ = ˆσ X = ˆσ n = = Professor Sharabati (Purdue University) Point Estimation Spring / 37

25 Find Standard Error Using Bootstrap When the form of the estimator ˆθ is too complicated and it is impossible to obtain the expression of the standard error, we use bootstrap. Idea of Bootstrap Suppose the population pdf is f(x; θ) and that data x 1, x 2,, x n gives ˆθ. We use the computer to obtain bootstrap samples from the pdf f(x, ˆθ), and for each sample we calculate a bootstrap estimate ˆθ. 1 First bootstrap: x 1, x 2,, x n. Estimate = ˆθ 1. 2 Second bootstrap: x 1, x 2,, x n. Estimate = ˆθ 2. 3 Repeat B (100 or 200) times, get ˆθ 1, ˆθ 2,, ˆθ B different estimates. 4 Sample mean of the bootstrap estimates θ = ˆθ 1 + +ˆθ B B The boostrap estimate of ˆθ s standard error sˆθ = (ˆθ i θ ) 2 1 B 1 Professor Sharabati (Purdue University) Point Estimation Spring / 37

26 Bootstrap Example Let X be the time to breakdown of an insulating fluid between electrodes at a particular voltage. X follows the exponential distribution with pdf f(x) = λe λx. A random sample (size n = 10) of the breakdown times (min) is {41.53, 18.73, 2.99, 30.34, 12.33, , 73.02, , 4.00, 26.87}. Use ˆλ = 1 X as the estimator. From the sample data, we get the estimate ˆλ = 1 x = Now generate bootstrap samples using the pdf f(x) = e x. 1 First bootstrap sample {11.25,,, 42.65}, ˆλ 1 = 2 Second bootstrap sample{54.61,, 18.63, 5.68}, ˆλ 2 = 3 Repeat B = 100 times = Sample mean of bootstrap estimates: λ = = The bootstrap estimate of ˆλ s standard error: sˆλ = (ˆλ i λ ) 2 = = Professor Sharabati (Purdue University) Point Estimation Spring / 37

27 Methods of Point Estimation Method of moments Definition of moments Method of moments Maximum likelihood estimation Maximum likelihood estimation (mle) examples Estimating functions of parameters Large sample behavior of the mle Professor Sharabati (Purdue University) Point Estimation Spring / 37

28 Definition of Moments Definition (Population moment) Let X follow a specific population distribution. The k-th moment of the population distribution with pmf p(x) or pdf f(x) is: E(X k ). Definition (Sample moment) Let X 1, X 2,, X n be a random sample from a pmf p(x) or pdf f(x). The k-th sample moment is: n i=1 Xk i n Sample moments can be used to estimate population moments. Professor Sharabati (Purdue University) Point Estimation Spring / 37

29 Examples of Moments A random sample: X 1, X 2,, X n. Normal distribution with mean µ and variance σ 2,. Population Moments First E(X) = µ Sample Moments X 1 + +X n n = X Second E(X 2 ) = σ 2 + µ 2 X 2 1 +X X2 n n For uniform distribution with parameters a and b, a < b. f(x) = 1 b a. First Population Moments E(X) = a+b 2 Second E(X 2 ) = a2 +ab+b 2 3 Sample Moments X 1 + +X n n X1 2+X X2 n n = X Professor Sharabati (Purdue University) Point Estimation Spring / 37

30 The Method of Moments Definition Let X 1, X 2,, X n be a random sample from a distribution with pmf or pdf f(x; θ 1,, θ m ) where θ 1,, θ m are parameters whose values are unknown. Then the moment estimators ˆθ 1, ˆθ 2,, ˆθ m are obtained by equating the first m sample moments to the corresponding first m population moments and solving for θ 1, θ 2,, θ m. This is called method of moments. An explanation when m = 2 We need to estimate two parameters θ 1, θ 2 of a distribution. The population moments E(X) and E(X 2 ) are the functions of θ 1, θ 2. Given the observed values x 1,, x n of a random sample X 1,, X n, we can estimate populations moments with sample moments: E(X) = X and E(X 2 X 2 i ) = n. This gives us two equations of θ 1 and θ 2. Solve the equations, we get the estimates of parameters θ 1, θ 2. Professor Sharabati (Purdue University) Point Estimation Spring / 37

31 Using Moments - Example Moment Estimator Given X 1,, X 5, a random sample of uniform distribution and the observed values of the random sample give X 1+ +X 5 5 = 2, X1 2+ +X2 5 5 = Find the moment estimates of parameters a and b in the uniform distribution f(x) = 1 b a, a < b. Hint E(X) = a + b 2 E(X 2 ) = a2 + ab + b 2 3 = X X 5 5 = 2 = X2 1 + X X2 5 5 = 13 3 Professor Sharabati (Purdue University) Point Estimation Spring / 37

32 Using Moments - Exercises Let X 1,, X 10 be a random sample of size 10 of a normal distribution. The observed values of the random sample are {3.92, 3.76, 4.01, 3.67, 3.89, 3.62, 4.09, 4.15, 3.58, 3.75}. Find the moment estimates of the mean µ and standard deviation σ of a normal distribution. Given a random sample X 1, X 2,, X n from some exponential distribution f(x) = λe λx, use the moment estimator to esimate parameter λ. Professor Sharabati (Purdue University) Point Estimation Spring / 37

33 Maximum Likelihood Estimation - Example A sample of ten new bike helmets manufactured by a company is obtained. Let X i = 1 if the ith helmet is flawed, X i = 0 if the ith is flawless. X i s are independent. Let p = P (a helmet is flawed) = P (X i = 1), thus all X i s follow the same Bernoulli distribution with parameter p. The observed values x i s of X i s are: {1, 0, 1, 0, 0, 0, 0, 0, 0, 1}. Now estimate p using MLE. Maximum Likelihood Estimation - MLE 1 What is the probability that we will have the current observed values? P (X 1 = 1, X 2 = 0, X 3 = 1, X 4 = X 9 = 0, X 10 = 1) = P (X 1 = 1)P (X 2 = 0) P (X 10 = 1) = p 3 (1 p) 7 f(p) 2 For what value of p is the observed sample most likely to have occurred? Find p that maximizes f(p).i.e., equivalent to max p ln[f(p)]. Taking derivative of ln[f(p)] and quating it to zero yields d dp {ln[f(p)]} = d dp {3 ln(p) + 7 ln(1 p)} = 3 p 7 1 p = 0 ˆp = 3 10 =.30 Professor Sharabati (Purdue University) Point Estimation Spring / 37

34 Maximum Likelihood Estimation Definition Let X 1, X 2,, X n have joint pmf or pdf f(x 1, x 2,, x n ; θ 1,, θ m ) where parameters θ 1, θ 2, θ m have unknown values. When x 1, x 2,, x n are the observed sample values of X 1,, X n, f(x 1, x 2,, x n ; θ 1,, θ m ) is called the likelihood function. The maximum likelihood estimates (mle s) of θ i s ˆθ 1, ˆθ 2,, ˆθ m are those values of θ i s that maximize the likelihood function. Professor Sharabati (Purdue University) Point Estimation Spring / 37

35 Example Let X 1, X 2,, X n be a random sample from a normal distribution with mean µ and variance σ 2. The observed values are x 1, x 2,, x n. Find the likelihood estimates of µ and σ 2. Answer Likelihood function: ln likelihood function: f(x 1,, x n ; µ, σ 2 1 ) = ( 2πσ 2 ) n 2 e (xi µ) 2 /(2σ 2 ) ln(f(x 1, x 2,, x n ; µ, σ 2 )) = n 2 ln(2πσ2 ) 1 2σ 2 (xi µ) 2 To find the maximizing values of µ, σ, we take partial derivatives of ln[f(x 1, x 2,, x n ; µ, σ 2 )], equate them to zero, and solve the equations. The mle esimates are ˆµ = X and ˆσ 2 = (Xi X) 2 n. Professor Sharabati (Purdue University) Point Estimation Spring / 37

36 Exercise Let X 1, X 2,, X n is a random sample from an exponential distribution with parameter λ. Given the observed values x 1, x 2,, x n. Find the maximum likelihood estimate of λ. Hint Likelihood function: ln likelihood function: f(x 1, x 2,, x n ; λ) = λe λx 1 λe λx2 λe λxn = λ n e λ(x 1+ +x n) ln(f(x 1, x 2,, x n, λ)) = n ln(λ) λ(x x n ) Professor Sharabati (Purdue University) Point Estimation Spring / 37

37 Estimating Functions of Parameters The invariance principle Let ˆθ 1, ˆθ 2,, ˆθ m be the mle s of the parameters θ 1, θ 2,, θ m. Then the mle of any function h(θ 1, θ 2, θ m ) of these parameters is the function h(ˆθ 1, ˆθ 2,, ˆθ m ). Example In the normal distribution case, the mle of µ and σ 2 are ˆµ = X and ˆσ 2 = (X i X) 2 /n. How to obtain the mle of σ? [Hint: use the function h(µ, σ 2 ) = σ 2 = σ] Because of the above invariance principle, we can substitute the mle s into the function: ˆσ = ˆσ 1 2 = (Xi n X) 2 Professor Sharabati (Purdue University) Point Estimation Spring / 37

38 Large Sample Behavior of MLE Proposition Under very general conditions on the joint distribution of the sample, when the sample size n is large, the maximum likelihood estimator of any parameter θ is approximately unbiased E(ˆθ) θ and has variance that is either as small as or nearly as small as can be achieved by any estimator. Short version: i.e., when sample size is large enough, the mle ˆθ is approx. MVUE of θ. Professor Sharabati (Purdue University) Point Estimation Spring / 37

39 Summary Point estimator and point estimate Unbiased estimators Minimum variance estimators Minimum variance unbiased estimator (MVUE) Finding the standard error of an estimator Deriving directly Bootstrapping Methods of point estimation The method of moments The method of maximum likelihood Maximum likelihood estimation (MLE) Estimating functions of parameters Large sample behavior of the mle Professor Sharabati (Purdue University) Point Estimation Spring / 37