19 Ubiased estimators I Chapter 17 we saw that a dataset ca be modeled as a realizatio of a radom sample from a probability distributio ad that quatities of iterest correspod to features of the model distributio. Oe of our tasks is to use the dataset to estimate a quatity of iterest. We shall maily deal with the situatio where it is modeled as oe of the parameters of the model distributio or as a certai fuctio of the parameters. We will first discuss what we mea exactly by a estimator ad the itroduce the otio of ubiasedess as a desirable property for estimators. We ed the chapter by providig ubiased estimators for the expectatio ad variace of a model distributio. 19.1 Estimators Cosider the arrivals of packages at a etwork server. Oe is iterested i the itesity at which packages arrive o a geeric day ad i the percetage of miutes durig which o packages arrive. If the arrivals occur completely at radom i time, the arrival process ca be modeled by a Poisso process. This would mea that the umber of arrivals durig oe miute is modeled by a radom variable havig a Poisso distributio with (ukow) parameter µ. The itesity of the arrivals is the modeled by the parameter µ itself, ad the percetage of miutes durig which o packages arrive is modeled by the probability of zero arrivals: e µ. Suppose oe observes the arrival process for a while ad gathers a dataset x 1,x 2,...,x,wherex i represets the umber of arrivals i the ith miute. Our task will be to estimate, based o the dataset, the parameter µ ad a fuctio of the parameter: e µ. This example is typical for the geeral situatio i which our dataset is modeled as a realizatio of a radom sample X 1,X 2,...,X from a probability distributio that is completely determied by oe or more parameters. The parameters that determie the model distributio are called the model parameters. We focus o the situatio where the quatity of iterest correspods
286 19 Ubiased estimators to a feature of the model distributio that ca be described by the model parameters themselves or by some fuctio of the model parameters. This distributio feature is referred to as the parameter of iterest. I discussig this geeral setup we shall deote the parameter of iterest by the Greek letter θ. So, for istace, i our etwork server example, µ is the model parameter. Whe we are iterested i the arrival itesity, the role of θ is played by the parameter µ itself, ad whe we are iterested i the percetage of idle miutes the role of θ is played by e µ. Whatever method we use to estimate the parameter of iterest θ, theresult depeds oly o our dataset. Estimate. A estimate is a value t that oly depeds o the dataset x 1,x 2,...,x, i.e., t is some fuctio of the dataset oly: t = h(x 1,x 2,...,x ). This descriptio of estimate is a bit formal. The idea is, of course, that the value t, computed from our dataset x 1,x 2,...,x, gives some idicatio of the true value of the parameter θ. We have already met several estimates i Chapter 17; see, for istace, Table 17.2. This table illustrates that the value of a estimate ca be aythig: a sigle umber, a vector of umbers, eve a complete curve. Let us retur to our etwork server example i which our dataset x 1,x 2,...,x is modeled as a realizatio of a radom sample from a Pois(µ) distributio. The itesity at which packages arrive is the represeted by the parameter µ. Sice the parameter µ is the expectatio of the model distributio, the law of large umbers suggests the sample mea x as a atural estimate for µ. O the other had, the parameter µ also represets the variace of the model distributio, so that by a similar reasoig aother atural estimate is the sample variace s 2. The percetage of idle miutes is modeled by the probability of zero arrivals. Similar to the reasoig i Sectio 13.4, a atural estimate is the relative frequecy of zeros i the dataset: umber of x i equal to zero. O the other had, the probability of zero arrivals ca be expressed as a fuctio of the model parameter: e µ. Hece, if we estimate µ by x,we could also estimate e µ by e x. Quick exercise 19.1 Suppose we estimate the probability of zero arrivals e µ by the relative frequecy of x i equal to zero. Deduce a estimate for µ from this.
19.2 Ivestigatig the behavior of a estimator 287 The precedig examples illustrate that oe ca ofte thik of several estimates for the parameter of iterest. This raises questios like Ĺ Ĺ Whe is oe estimate better tha aother? Does there exist a best possible estimate? For istace, ca we say which of the values x or s 2 computed from the dataset is closer to the true parameter µ? Theasweriso. The measuremets ad the correspodig estimates are subject to radomess, so that we caot say aythig with certaity about which of the two is closer to µ. Oe of the thigs we ca say for each of them is how likely it is that they are withi a give distace from µ. To this ed, we cosider the radom variables that correspod to the estimates. Because our dataset x 1,x 2,...,x is modeled as a realizatio of a radom sample X 1,X 2,...,X, the estimate t is a realizatio of a radom variable T. Estimator. Let t = h(x 1,x 2,...,x ) be a estimate based o the dataset x 1,x 2,...,x.Thet is a realizatio of the radom variable T = h(x 1,X 2,...,X ). The radom variable T is called a estimator. The word estimator refers to the method or device for estimatio. This is distiguished from estimate, which refers to the actual value computed from a dataset. Note that estimators are special cases of sample statistics. I the remaider of this chapter we will discuss the otio of ubiasedess that describes to some extet the behavior of estimators. 19.2 Ivestigatig the behavior of a estimator Let us cotiue with our etwork server example. Suppose we have observed the etwork for 30 miutes ad we have recorded the umber of arrivals i each miute. The dataset is modeled as a realizatio of a radom sample X 1,X 2,...,X of size =30fromaPois(µ) distributio. Let us cocetrate o estimatig the probability p 0 of zero arrivals, which is a ukow umber betwee 0 ad 1. As motivated i the previous sectio, we have the followig possible estimators: S = umber of X i equal to zero ad T =e X. Our first estimator S ca oly attai the values 0, 1 30, 2 30,...,1, so that i geeral it caot give the exact value of p 0. Similarly for our secod estimator T, which ca oly attai the values 1, e 1/30, e 2/30,.... So clearly, we
288 19 Ubiased estimators caot expect our estimators always to give the exact value of p 0 o basis of 30 observatios. Well, the what ca we expect from a reasoable estimator? To get a idea of the behavior of both estimators, we preted we kow µ ad we simulate the estimatio process i the case of =30observatios. Let us choose µ = l 10, so that p 0 =e µ =0.1. We draw 30 values from a Poisso distributio with parameter µ = l 10 ad compute the value of estimators S ad T. We repeat this 500 times, so that we have 500 values for each estimator. I Figure 19.1 a frequecy histogram 1 of these values for estimator S is displayed o the left ad for estimator T o the right. Clearly, the values of both estimators vary aroud the value 0.1, which they are supposed to estimate. 250 250 200 200 150 150 100 100 50 50 0 0 0.0 0.1 0.2 0.3 0.0 0.1 0.2 0.3 Fig. 19.1. Frequecy histograms of 500 values for estimators S (left) ad T (right) of p 0 =0.1. 19.3 The samplig distributio ad ubiasedess We have just see that the values geerated for estimator S fluctuate aroud p 0 =0.1. Although the value of this estimator is ot always equal to 0.1, it is desirable that o average, S is o target, i.e., E[S =0.1. Moreover, it is desirable that this property holds o matter what the actual value of p 0 is, i.e., E[S =p 0 irrespective of the value 0 <p 0 < 1. I order to fid out whether this is true, we eed the probability distributio of the estimator S. Ofcoursethis 1 I a frequecy histogram the height of each vertical bar equals the frequecy of values i the correspodig bi.
19.3 The samplig distributio ad ubiasedess 289 is simply the distributio of a radom variable, but because estimators are costructed from a radom sample X 1,X 2,...,X, we speak of the samplig distributio. The samplig distributio. Let T = h(x 1,X 2,...,X )bea estimator based o a radom sample X 1,X 2,...,X. The probability distributio of T is called the samplig distributio of T. The samplig distributio of S ca be foud as follows. Write S = Y, where Y is the umber of X i equal to zero. If for each i we label X i =0as a success, the Y is equal to the umber of successes i idepedet trials with p 0 as the probability of success. Similar to Sectio 4.3, it follows that Y has a Bi(, p 0 ) distributio. Hece the samplig distributio of S is that of a Bi(, p 0 ) distributed radom variable divided by. This meas that S is a discrete radom variable that attais the values k/, wherek =0, 1,...,, with probabilities give by ( ) ( k p S =P S = k ) ( ) =P(Y = k) = p k k 0(1 p 0 ) k. The probability mass fuctio of S for the case = 30 ad p 0 = 0.1 is displayed i Figure 19.2. Sice S = Y/ ad Y has a Bi(, p 0 ) distributio, it follows that E[S = E[Y = p 0 = p 0. So, ideed, the estimator S for p 0 has the property E[S =p 0. This property reflects the fact that estimator S has o systematic tedecy to produce p S(a) 0.25 0.20 0.15 0.10 0.05 0.00 0.0 0.2 0.4 0.6 0.8 1.0 Fig. 19.2. Probability mass fuctio of S. a
290 19 Ubiased estimators estimates that are larger tha p 0, ad o systematic tedecy to produce estimates that are smaller tha p 0. This is a desirable property for estimators, ad estimators that have this property are called ubiased. Defiitio. A estimator T is called a ubiased estimator for the parameter θ, if E[T =θ irrespective of the value of θ. The differece E[T θ is called the bias of T ; if this differece is ozero, the T is called biased. Let us retur to our secod estimator for the probability of zero arrivals i the etwork server example: T =e X. The samplig distributio ca be obtaied as follows. Write T =e Z/, where Z = X 1 + X 2 + + X. From Exercise 12.9 we kow that the radom variable Z, beig the sum of idepedet Pois(µ) radom variables, has a Pois(µ) distributio. This meas that T is a discrete radom variable attaiig values e k/,wherek =0, 1,... ad the probability mass fuctio of T is give by p T (e k/) =P (T =e k/) =P(Z = k) = e µ (µ) k. k! The probability mass fuctio of T for the case = 30 ad p 0 = 0.1 is displayed i Figure 19.3. From the histogram i Figure 19.1 as well as from the probability mass fuctio i Figure 19.3, you may get the impressio that T is also a ubiased estimator. However, this ot the case, which follows immediately from a applicatio of Jese s iequality: 0.05 p T (a) 0.04 0.03 0.02 0.01 0.00 0.0 0.2 0.4 0.6 0.8 1.0 a Fig. 19.3. Probability mass fuctio of T.
19.3 The samplig distributio ad ubiasedess 291 E[T =E [e X > e E[ X, where we have a strict iequality because the fuctio g(x) =e x is strictly covex (g (x) =e x > 0). Recall that the parameter µ equals the expectatio of the Pois(µ) model distributio, so that accordig to Sectio 13.1 we have E [ X = µ. We fid that E[T > e µ = p 0, which meas that the estimator T for p 0 has positive bias. I fact we ca compute E[T exactly (see Exercise 19.9): E[T =E [e X =e µ(1 e 1/). Note that (1 e 1/ ) 1, so that E[T =e µ(1 e 1/) e µ = p 0 as goes to ifiity. Hece, although T has positive bias, the bias decreases to zero as the sample size becomes larger. I Figure 19.4 the expectatio of T is displayed as a fuctio of the sample size for the case µ = l(10). For = 30 the differece betwee E[T adp 0 =0.1 equals 0.0038. 0.25 0.20 E[T 0.15 0.10... 0.05 0.00 0 5 10 15 20 25 30 Fig. 19.4. E[T as a fuctio of. Quick exercise 19.2 If we estimate p 0 =e µ by the relative frequecy of zeros S = Y/, the we could estimate µ by U = l(s). Argue that U is a biased estimator for µ. Is the bias positive or egative? We coclude this sectio by returig to the estimatio of the parameter µ. Apart from the (biased) estimator i Quick exercise 19.2 we also cosidered
292 19 Ubiased estimators thesamplemea X ad sample variace S 2 as possible estimators for µ. These are both ubiased estimators for the parameter µ. Thisisadirect cosequece of a more geeral property of X ad S, 2 which is discussed i the ext sectio. 19.4 Ubiased estimators for expectatio ad variace Sometimes the quatity of iterest ca be described by the expectatio or variace of the model distributio, ad is it irrelevat whether this distributio is of a parametric type. I this sectio we propose ubiased estimators for these distributio features. Ubiased estimators for expectatio ad variace. Suppose X 1,X 2,...,X is a radom sample from a distributio with fiite expectatio µ ad fiite variace σ 2.The X = X 1 + X 2 + + X is a ubiased estimator for µ ad S 2 = 1 1 is a ubiased estimator for σ 2. (X i X ) 2 i=1 The first statemet says that E [ X = µ, which was show i Sectio 13.1. The secod statemet says E [ S 2 = σ 2. To see this, use liearity of expectatios to write E [ S 2 1 = E [ (X i 1 X ) 2. i=1 Sice E [ X = µ, wehavee [ Xi X =E[Xi E [ X = 0. Now ote that for ay radom variable Y with E[Y =0,wehave Var(Y )=E [ Y 2 (E[Y ) 2 =E [ Y 2. Applyig this to Y = X i X, it follows that Note that we ca write E [ (X i X ) 2 =Var ( X i X ). X i X = 1 X i 1 X j. j i
19.4 Ubiased estimators for expectatio ad variace 293 The from the rules cocerig variaces of sums of idepedet radom variables we fid that Var ( X i X ) =Var We coclude that E [ S 2 1 = 1 = 1 1 1 X i 1 j i X j ( 1)2 = Var(X i )+ 1 2 Var(X j ) = 2 j i [ ( 1) 2 2 + 1 2 σ 2 = 1 σ2. E [ (X i X ) 2 i=1 i=1 Var ( X i X ) = 1 1 1 σ2 = σ 2. This explais why we divide by 1 i the formula for S 2 ; oly i this case S 2 is a ubiased estimator for the true variace σ2. If we would divide by istead of 1, we would obtai a estimator with egative bias; it would systematically produce too-small estimates for σ 2. Quick exercise 19.3 Cosider the followig estimator for σ 2 : V 2 = 1 (X i X ) 2. i=1 Compute the bias E [ V 2 σ 2 for this estimator, where you ca keep computatios simple by realizig that V 2 =( 1)S2 /. Ubiasedess does ot always carry over We have see that S 2 is a ubiased estimator for the true variace σ 2.A atural questio is whether S is agai a ubiased estimator for σ.thisisot the case. Sice the fuctio g(x) =x 2 is strictly covex, Jese s iequality yields that σ 2 =E [ S 2 > (E[S ) 2, which implies that E[S <σ. Aother example is the etwork arrivals, i which X is a ubiased estimator for µ, wherease X is positively biased with respect to e µ. These examples illustrate a geeral fact: ubiasedess does ot always carry over, i.e., if T is a ubiased estimator for a parameter θ, the g(t ) does ot have to be a ubiased estimator for g(θ).
294 19 Ubiased estimators However, there is oe special case i which ubiasedess does carry over, amely if g(t )=at + b. Ideed,ifT is ubiased for θ: E[T =θ, the by the chage-of-uits rule for expectatios, E[aT + b =ae[t +b = aθ + b, which meas that at + b is ubiased for aθ + b. 19.5 Solutios to the quick exercises 19.1 Write y for the umber of x i equal to zero. Deote the probability of zero by p 0,sothatp 0 =e µ. This meas that µ = l(p 0 ). Hece if we estimate p 0 by the relative frequecy y/, we ca estimate µ by l(y/). 19.2 The fuctio g(x) = l(x) is strictly covex, sice g (x) =1/x 2 > 0. Hece by Jese s iequality E[U =E[ l(s) > l(e[s). Sice we have see that E[S =p 0 =e µ, it follows that E[U > l(e[s) = l(e µ )=µ. This meas that U has positive bias. 19.3 Usig that E [ S 2 = σ 2, we fid that E [ V 2 [ 1 =E S2 = 1 E[ S 2 1 = σ2. We coclude that the bias of V 2 equals E[ V 2 σ 2 = σ 2 / < 0. 19.6 Exercises 19.1 Suppose our dataset is a realizatio of a radom sample X 1,X 2,...,X from a uiform distributio o the iterval [ θ, θ, where θ is ukow. a. Show that T = 3 (X2 1 + X 2 2 + + X 2 ) is a ubiased estimator for θ 2. b. Is T also a ubiased estimator for θ? If ot, argue whether it has positive or egative bias. 19.2 Suppose the radom variables X 1,X 2,...,X have the same expectatio µ.
a. Is S = 1 2 X 1 + 1 3 X 2 + 1 6 X 3 a ubiased estimator for µ? b. Uder what coditios o costats a 1,a 2,...,a is a ubiased estimator for µ? T = a 1 X 1 + a 2 X 2 + + a X 19.6 Exercises 295 19.3 Suppose the radom variables X 1,X 2,...,X have the same expectatio µ. For which costats a ad b is a ubiased estimator for µ? T = a(x 1 + X 2 + + X )+b 19.4 Recall Exercise 17.5 about the umber of cycles to pregacy. Suppose the dataset correspodig to the table i Exercise 17.5 a is modeled as a realizatio of a radom sample X 1,X 2,...,X from a Geo(p) distributio, where 0 < p < 1 is ukow. Motivated by the law of large umbers, a atural estimator for p is T =1/ X. a. Check that T is a biased estimator for p ad fid out whether it has positive or egative bias. b. I Exercise 17.5 we discussed the estimatio of the probability that a woma becomes pregat withi three or fewer cycles. Oe possible estimator for this probability is the relative frequecy of wome that became pregat withi three cycles S = umber of X i 3. Show that S is a ubiased estimator for this probability. 19.5 Suppose a dataset is modeled as a realizatio of a radom sample X 1,X 2,...,X from a Exp(λ) distributio, where λ>0 is ukow. Let µ deote the correspodig expectatio ad let M deote the miimum of X 1,X 2,...,X. Recall from Exercise 8.18 that M has a Exp(λ) distributio. Fid out for which costat c the estimator is a ubiased estimator for µ. T = cm 19.6 Cosider the followig dataset of lifetimes of ball bearigs i hours. 6278 3113 5236 11584 12628 7725 8604 14266 6125 9350 3212 9003 3523 12888 9460 13431 17809 2812 11825 2398 Source: J.E. Agus. Goodess-of-fit tests for expoetiality based o a lossof-memory type fuctioal equatio. Joural of Statistical Plaig ad Iferece, 6:241-251, 1982; example 5 o page 249.
296 19 Ubiased estimators Oe is iterested i estimatig the miimum lifetime of this type of ball bearig. The dataset is modeled as a realizatio of a radom sample X 1,...,X. Each radom variable X i is represeted as X i = δ + Y i, where Y i has a Exp(λ) distributio ad δ>0 is a ukow parameter that is supposed to model the miimum lifetime. The objective is to costruct a ubiased estimator for δ. Itiskowthat E[M =δ + 1 λ ad E[ X = δ + 1 λ, where M = miimum of X 1,X 2,...,X ad X =(X 1 + X 2 + + X )/. a. Check that T = ( ) X M 1 is a ubiased estimator for 1/λ. b. Costruct a ubiased estimator for δ. c. Use the dataset to compute a estimate for the miimum lifetime δ. You may use that the average lifetime of the data is 8563.5. 19.7 Leaves are divided ito four differet types: starchy-gree, sugary-white, starchy-white, ad sugary-gree. Accordig to geetic theory, the types occur with probabilities 1 4 (θ +2), 1 4 θ, 1 4 (1 θ), ad 1 4 (1 θ), respectively, where 0 <θ<1. Suppose oe has leaves. The the umber of starchy-gree leaves is modeled by a radom variable N 1 with a Bi(, p 1 ) distributio, where p 1 = 1 4 (θ + 2), ad the umber of sugary-white leaves is modeled by a radom variable N 2 with a Bi(, p 2 ) distributio, where p 2 = 1 4 θ.the followig table lists the couts for the progey of self-fertilized heterozygotes amog 3839 leaves. Type Cout Starchy-gree 1997 Sugary-white 32 Starchy-white 906 Sugary-gree 904 Source: R.A. Fisher. Statistical methods for research workers. Hafer, New York, 1958; Table 62 o page 299. Cosider the followig two estimators for θ: T 1 = 4 N 1 2 ad T 2 = 4 N 2.
a. Check that both T 1 ad T 2 are ubiased estimators for θ. b. Compute the value of both estimators for θ. 19.6 Exercises 297 19.8 Recall the black cherry trees example from Exercise 17.9, modeled by a liear regressio model without itercept Y i = βx i + U i for i =1, 2,...,, where U 1,U 2,...,U are idepedet radom variables with E[U i =0ad Var(U i )=σ 2. We discussed three estimators for the parameter β: B 1 = 1 ( Y1 + + Y ), x 1 x B 2 = Y 1 + + Y x 1 + + x, B 3 = x 1Y 1 + + x Y x 2 1 + +. x2 Show that all three estimators are ubiased for β. 19.9 Cosider the etwork example where the dataset is modeled as a realizatio of a radom sample X 1,X 2,...,X from a Pois(µ) distributio. We estimate the probability of zero arrivals e µ by meas of T =e X.Check that E[T =e µ(1 e 1/). Hit: write T = e Z/,whereZ = X 1 + X 2 + + X has a Pois(µ) distributio.