Point Estimation. Copyright Cengage Learning. All rights reserved.

Transcription

1 6 Point Estimation Copyright Cengage Learning. All rights reserved.

2 6.2 Methods of Point Estimation Copyright Cengage Learning. All rights reserved.

3 Methods of Point Estimation The definition of unbiasedness does not in general indicate how unbiased estimators can be derived. We now discuss two constructive methods for obtaining point estimators: the method of moments and the method of maximum likelihood. By constructive we mean that the general definition of each type of estimator suggests explicitly how to obtain the estimator in any specific problem. 3

4 Methods of Point Estimation Although maximum likelihood estimators are generally preferable to moment estimators because of certain efficiency properties, they often require significantly more computation than do moment estimators. It is sometimes the case that these methods yield unbiased estimators. 4

5 The Method of Moments 5

6 The Method of Moments The basic idea of this method is to equate certain sample characteristics, such as the mean, to the corresponding population expected values. Then solving these equations for unknown parameter values yields the estimators. 6

7 The Method of Moments Definition Let X 1,..., X n be a random sample from a pmf or pdf f(x). For k = 1, 2, 3,..., the kth population moment, or kth moment of the distribution f(x), is E(X k ). The kth sample moment is Thus the first population moment is E(X) = µ, and the first sample moment is ΣX i /n = The second population and sample moments are E(X 2 ) and ΣX i2 /n, respectively. The population moments will be functions of any unknown parameters θ 1, θ 2,.... 7

8 The Method of Moments Definition Let X 1, X 2,..., X n be a random sample from a distribution with pmf or pdf f (x; θ 1,..., θ m ), where θ 1,..., θ m are parameters whose values are unknown. Then the moment estimators θ 1,..., θ m are obtained by equating the first m sample moments to the corresponding first m population moments and solving for θ 1,..., θ m. 8

9 The Method of Moments If, for example, m = 2, E(X) and E(X 2 ) will be functions of θ 1 and θ 2. Setting E(X) = (1/n) Σ X i (= ) and E(X 2 ) = (1/n) Σ X i 2 gives two equations in θ 1 and θ 2. The solution then defines the estimators. 9

10 Example 12 Let X 1, X 2,..., X n represent a random sample of service times of n customers at a certain facility, where the underlying distribution is assumed exponential with parameter λ. Since there is only one parameter to be estimated, the estimator is obtained by equating E(X) to. Since E(X) = 1/λ for an exponential distribution, this gives 1/λ = or λ = 1/. The moment estimator of λ is then λ 10

11 Maximum Likelihood Estimation 11

12 Maximum Likelihood Estimation The method of maximum likelihood was first introduced by R. A. Fisher, a geneticist and statistician, in the 1920s. Most statisticians recommend this method, at least when the sample size is large, since the resulting estimators have certain desirable efficiency properties. 12

13 Example 15 A sample of ten new bike helmets manufactured by a certain company is obtained. Upon testing, it is found that the first, third, and tenth helmets are flawed, whereas the others are not. Let p = P(flawed helmet), i.e., p is the proportion of all such helmets that are flawed. Define (Bernoulli) random variables X 1, X 2,..., X 10 by 13

14 Example 15 cont d Then for the obtained sample, X 1 = X 3 = X 10 = 1 and the other seven X i s are all zero. The probability mass function of any particular X i is, which becomes p if x i = 1 and 1 p when x i = 0. Now suppose that the conditions of various helmets are independent of one another. This implies that the X i s are independent, so their joint probability mass function is the product of the individual pmf s. 14

15 Example 15 cont d Thus the joint pmf evaluated at the observed X i s is f (x 1,..., x 10 ; p) = p(1 p)p... p = p 3 (1 p) 7 (6.4) Suppose that p =.25. Then the probability of observing the sample that we actually obtained is (.25) 3 (.75) 7 = If instead p =.50, then this probability is (.50) 3 (.50) 7 = For what value of p is the obtained sample most likely to have occurred? That is, for what value of p is the joint pmf (6.4) as large as it can be? What value of p maximizes (6.4)? 15

16 Example 15 cont d Figure 6.5(a) shows a graph of the likelihood (6.4) as a function of p. It appears that the graph reaches its peak above p =.3 = the proportion of flawed helmets in the sample. Graph of the likelihood (joint pmf) (6.4) from Example 15 Figure 6.5(a) 16

17 Example 15 cont d Figure 6.5(b) shows a graph of the natural logarithm of (6.4); since ln[g(u)] is a strictly increasing function of g(u), finding u to maximize the function g(u) is the same as finding u to maximize ln[g(u)]. Graph of the natural logarithm of the likelihood Figure 6.5(b) 17

18 Example 15 cont d We can verify our visual impression by using calculus to find the value of p that maximizes (6.4). Working with the natural log of the joint pmf is often easier than working with the joint pmf itself, since the joint pmf is typically a product so its logarithm will be a sum. Here ln[ f (x 1,..., x 10 ; p)] = ln[p 3 (1 p) 7 ] = 3ln(p) + 7ln(1 p) (6.5) 18

19 Example 15 cont d Thus [the (1) comes from the chain rule in calculus]. 19

20 Example 15 cont d Equating this derivative to 0 and solving for p gives 3(1 p) = 7p, from which 3 = 10p and so p = 3/10 =.30 as conjectured. That is, our point estimate is =.30. It is called the maximum likelihood estimate because it is the parameter value that maximizes the likelihood (joint pmf) of the observed sample. In general, the second derivative should be examined to make sure a maximum has been obtained, but here this is obvious from Figure

21 Example 15 cont d Suppose that rather than being told the condition of every helmet, we had only been informed that three of the ten were flawed. Then we would have the observed value of a binomial random variable X = the number of flawed helmets. The pmf of X is For x = 3, this becomes The binomial coefficient is irrelevant to the maximization, so again =

22 Maximum Likelihood Estimation The likelihood function tells us how likely the observed sample is as a function of the possible parameter values. Maximizing the likelihood gives the parameter values for which the observed sample is most likely to have been generated that is, the parameter values that agree most closely with the observed data. 22

23 Example 16 Suppose X 1, X 2,..., X n is a random sample from an exponential distribution with parameter λ. Because of independence, the likelihood function is a product of the individual pdf s: The natural logarithm of the likelihood function is ln[ f (x 1,..., x n ; λ)] = n ln(λ) λσx i 23

24 Example 16 cont d Equating (d/dλ)[ln(likelihood)] to zero results in n/λ Σx i = 0, or λ = n/σx i = Thus the mle is it is identical to the method of moments estimator [but it is not an unbiased estimator, since 24

25 Example 17 Let X 1,..., X n be a random sample from a normal distribution. The likelihood function is so 25

26 Example 17 cont d To find the maximizing values of µ and σ 2, we must take the partial derivatives of ln(f ) with respect to µ and σ 2, equate them to zero, and solve the resulting two equations. Omitting the details, the resulting mle s are The mle of σ 2 is not the unbiased estimator, so two different principles of estimation (unbiasedness and maximum likelihood) yield two different estimators. 26

27 Estimating Functions of Parameters 27

28 Estimating Functions of Parameters In Example 17, we obtained the mle of σ 2 when the underlying distribution is normal. The mle of σ =, as well as that of many other mle s, can be easily derived using the following proposition. Proposition The Invariance Principle Let θ 1, θ 2,..., θ m. be the mle s of the parameters Then the mle of any function h(θ 1, θ 2,..., θ m ) of these parameters is the function h( ) of the mle s. 28

29 Example 20 Example 17 continued In the normal case, the mle s of µ and σ 2 are To obtain the mle of the function substitute the mle s into the function: The mle of σ is not the sample standard deviation S, though they are close unless n is quite small. 29

30 Large Sample Behavior of the MLE 30

31 Large Sample Behavior of the MLE Although the principle of maximum likelihood estimation has considerable intuitive appeal, the following proposition provides additional rationale for the use of mle s. Proposition Under very general conditions on the joint distribution of the sample, when the sample size n is large, the maximum likelihood estimator of any parameter θ is approximately unbiased and has variance that is either as small as or nearly as small as can be achieved by any estimator. Stated another way, the mle is approximately the MVUE of θ. 31

32 Large Sample Behavior of the MLE Because of this result and the fact that calculus-based techniques can usually be used to derive the mle s (though often numerical methods, such as Newton s method, are necessary), maximum likelihood estimation is the most widely used estimation technique among statisticians. Many of the estimators used in the remainder of the book are mle s. Obtaining an mle, however, does require that the underlying distribution be specified. 32