1 Estimating the uncertainty attached to a sample mean: s 2 vs.

Size: px

Start display at page:

Download "1 Estimating the uncertainty attached to a sample mean: s 2 vs."

Cynthia Wright
6 years ago
Views:

1 Political Sciece 100a/200a Fall 2001 Cofidece itervals ad hypothesis testig, Part I 1 1 Estimatig the ucertaity attached to a sample mea: s 2 vs. σ 2 Recall the problem of descriptive iferece: We wat to use the data we collect to say somethig about the value of some parameter, like the percet who favor Bush prior to a electio, or the umber killed i civil wars sice But either 1. We do t observe the whole populatio, as i the case of survey research where we oly sample a small fractio of registered voters, etc Or we observe the populatio, but what we are really iterested i is the uderlyig social or ecoomic process that geerates these values, such as umber or magitude of civil wars, ad we believe that there are umerous radom factors that goig ito makig such quatities. (I the simplest case of this sort, we wat to say somethig about a repeatable process like a coi flip or a missile system s accuracy, based o a fiite umber of experimets or observatios.) From the cetral limit theorem ad some of our other results, we have draw some powerful coclusios about the sample mea as a estimator for a uderlyig populatio parameter µ (e.g. the proportio who favor Bush, or the true umber killed i civil wars sice 1945, or the likelihood that a BMD system works o a give trial). I particular, x a N(µ, σ 2 /), or equivaletly, z = x µ σ/ a N(0, 1). You should uderstad what this says: The sample mea x has a approximately ormal distributio with mea µ (it is ubiased) ad variace σ 2 /. But ow the questio is, How ca we apply this result i practice whe we have a buch of sample data? 1 Notes by James D. Fearo, Dept. of Political Sciece, Staford Uiversity, November

2 e.g.: Suppose we sample 25 coutries ad estimate life expectacy i each oe. We kow that x, the sample mea, is a ubiased estimator for life expectacy by coutry aroud the world. But how do we estimate the ucertaity attached to this estimate? Where do we get a estimate for the stadard deviatio of the sample mea, σ/? We ecouter a problem: Our theoretical result requires us to have σ 2, the variace of the populatio variable. This is of course a problem because we do t observe the populatio that is why we are drawig ad examiig a sample i the first place! We have, however, a atural cadidate to use as a estimate for σ 2. Why ot just use the variace of the sample values? Cosider, for example, the variace of the sample, call this σs. 2 σs 2 = 1 (x i x) 2, where x i is the ith value i the sample ad is the size of the sample. i=1 Our ext questio should be: Is this a good estimate of σ 2, the true populatio variace? What makes somethig a good estimator? Oe criterio, as we have discussed, is ubiasedess. If we took may radom samples of 25 coutries, computed σs 2 for each oe, ad the averaged them, would the result be cetered o the true populatio value σ 2? I other words, is it true that E(σ 2 s) = σ 2? The aswer turs out to be No, ot quite. Below, I show that E(σs) 2 = 1 σ2 What does this imply? σ 2 s is a slightly biased estimator of the populatio variace σ2 ; it is a little bit too small o average, although as the sample size gets larger it is almost ubiased. (You could show that it coverges i probability to the right value.) To get a ubiased estimator, all we have to do is multiply by /( 1) so 1 (xi x) 2 1 σ2 s = 1 = 1 1 (xi x) 2. This quatity is what we call the sample variace s 2 = 1 (x i x) 2 1 i=1 2

3 This is the quatity that Stata ad (I thik) most other statistical packages computes whe it shows you variace or stadard deviatio. Note that we ca compute s 2 usig oly iformatio available i our sample. (Recall also that s 2 is a estimate of the populatio variace σ 2, NOT a estimate of the variace of the sample mea x. Importat to keep this straight.) s 2 is a good estimate for the populatio variace σ 2 i the sese that o average (across hypothetical radom samples) it will be right. So, this gives us a ew possibility for estimatig the ucertaity attached to our estimate x. Why do t we just use x a N(µ, s 2 /), istead of x a N(µ, σ 2 /)? 2 Applicatio: Cofidece itervals Suppose we do this. How do we summarize the ucertaity associated with our estimate? Oe valuable approach is to costruct a cofidece iterval. (See FPP, ch21, sectios 2-3). Ituitively, you have already bee exposed to cofidece itervals may times, wheever you hear a poll result reported as (for istace), 34% of Americas do t like Al Gore s beard, accordig to a poll with a plus or mius 3% margi of error. What does this really mea? Draw picture of ormal distributio aroud µ with stadard deviatio σ/.... Idicate a particular sample mea x. Show that i about 95% of all such radom samples, x will lie withi 2 s.d.s o either side of the populatio value, µ. Thus, if you could draw radom samples of this size may, may times, i about 95% of them, a iterval of two s.d.s aroud the sample mea would cover the true value µ. (discuss weird locutio...) More formally, Def : A β% cofidece iterval is the iterval o either side of the sample mea x (symmetric aroud x) that would take i β% of the area uder the probability distributio for the sample mea. e.g.: A 95% cofidece iterval aroud the sample mea exteds 1.96σ/ to either side of x Show with diagram... Why 1.96? 3

4 e.g.: I the above case, our estimate for a 95% cofidece iterval exteds 1.96 s 25 o either side of x (ot usig the fiite sample correctio) What good is this? A cofidece iterval ca be iterpreted as follows: We do t kow what the true populatio mea µ is, but if we draw 25 coutry samples repeatedly, may, may times, 95% of the time the 95% cofidece iterval would cover the true mea, µ. The cofidece iterval is NOT to be iterpreted as follows (at least ot by a frequetist): It is ot true that the probability that the true mea falls withi the 95% cofidece iterval is 95%. It either does or it does t. A cofidece iterval is ofte a fairly ituitive ad helpful way of summarizig ucertaity about a parameter beig estimated tha just givig the stadard error of the sample mea, because it gives the reader a sese of the plausible rage of the error of the estimate. e.g., public opiio polls are reported with a margi of error which is the size of oe side of a cofidece iterval (typically 95%, I thik). Icreasigly, i political sciece research, you fid people reportig cofidece itervals rather tha sigificace levels (about which more later), because cofidece itervals tell you somethig more substative about the parameter of iterest (i.e., the rage i which the parameter likely falls). Example 1: A cofidece iterval for a estimate of life expectacy across coutries... Use Stata to sample 25 coutries, compute mea ad sd of sample, costruct 95% cofidece iterval. Does it cover the true value? Example 2: A cofidece iterval for the umber killed i civil wars sice 1945 Show i Stata data o umbers killed by war for These estimates are icredibly oisy, however. You might thik this would make estimatig the total umber killed by addig up these oisy estimates a eve more oisy, hopeless edeavor. Let x i be a estimate of umber killed for the ith war, ad let S be the sum of a buch of estimates. The we have S = x 1 + x 2 + x x, ad thus var(s) = var(x 1 ) + var(x 2 ) var(x ) 4

5 Notice that we are treatig S as a radom variable that is the sum of a buch of radom variables here. That is, we see just oe set of estimates, but we imagie that these estimates are the product of a stochastic process that could produced quite differet umbers. Further, what ca we say about the probability distributio of the sum S? So we ca estimate the variace (ad thus the sd) of the sum if we ca a estimate for the variace of the estimate for each war. Let s just make a guess: Suppose that the sd of each estimate is proportioal to the umber killed thus, for bigger wars the ucertaity associated with the umber killed is proportioately larger. Let s try assumig that the sd is 10% of the estimate we have ad see what this gives us. ge sdest =.1*iissdth, ge varest = sdest^2, sum varest. Thus our estimate for the sd of the sum S is what? Ok, ow costruct a cofidece iterval... 3 Aother problem with our approach: s 2 is a radom variable Let s retur to the problem of estimatig a cofidece iterval for a populatio mea based o a sample (e.g., life expectacy across coutries). You may be woderig, Is t there somethig circular about usig the sample to estimate the variace of the (uobserved) populatio (the box, i FPP)? What about the ucertaity attached to s 2 as a estimate of σ 2? Should t this be factored i somehow? Yes, it should. Look agai at what we did: we substituted the sample variace s 2 for the true populatio variace σ 2 i our theoretical result, usig x a N(µ, s 2 /), istead of x a N(µ, σ 2 /)? But this was kid of devious. Note that because it is based o a sample, ad would be a little differet for each differet sample we could draw, the sample variace s 2 is a radom variable. If so, the what gives us the licese to stick it ito N(µ, ) ad still believe that what we have is distributed ormally? 5

6 Give that s 2 is a radom variable, would t you expect that this would add to our ucertaity about the sample mea, x? I fact it does, ad to be really correct we eed to accout for this additioal ucertaity itroduced by the fact that we oly have a estimate of σ 2, ot the true populatio value. It turs out that i a large sample (big ), this additioal ucertaity does t really matter. The cetral limit theorem will esure that despite the added ucertaity itroduced by s 2, the distributio of x will become approximately ormal with variace s 2 /. But what about i a small sample? Here the fact that we oly have a estimate of σ 2 that is likely to have error attached to it becomes importat. It is possible (though difficult) to establish the followig result: Th m : If a radom variable X has Normal distributio, ad we draw a sample from X with observatios, the the sample mea x has a t distributio with 1 degrees of freedom. A t distributio looks very much like a ormal distributio except that it has fatter tails. More weight is put o values relatively far from the mea. So usig the t distributio is bit more coservative about how precise a estimate of the sample mea you are gettig. However, as sample size gets larger, a t distributio with 1 degrees of freedom coverges quickly to a ormal distributio, so with a large sample (as low as 25 if the uderlyig variable is ot highly asymmetric) usig the t distributio is essetially equivalet to usig a ormal distributio. (Show with Stata...) I readig political sciece articles usig regressio ad related methods, you will costatly ecouter authors talkig about t statistics. This is what they are referrig to. Whe you are testig hypothesis i a regressio model e.g., that after cotrollig for per capita icome, subsahara Africa states have sigificatly lower life expectacies o average the test statistic spit out by a stadard regressio is a t-statistic. Illustrate with Stata... Questio: Costruct a 95% cofidece iterval for our estimate of the mea life expectacy aroud the globe by coutry, usig the more coservative (ad appropriate, for a 25 coutry sample) t distributio. Stata has the built i fuctio ivt(df, p), where df is the umber of degrees of freedom, which is the sample size mius oe, ad p is the probability you wat. I.e., typig ivt(24,.95) will give the distace from zero you have to go o either side to 6

7 get 95% of the area uder a t distributio with 24 degrees of freedom. Draw... this gives So you eed to go 2.06 stadard uits o either side to get a 95% cofidece iterval. For our problem, a stadard uit is a stadard deviatio of the sample mea, s/ =?/5 =?. So a 95% cofidece iterval is:... 4 Hypothesis testig: the core logic These results cocerig the probability distributio of the sample mea allow us to test hypotheses about uobserved parameters of social sciece iterest, such as the populatio mea of the proportio of likely Gore voters, average life expectacy by coutry for the whole world, or the true probability that two democracies will fight each other. You have already see examples of the logic at work. The most basic example is the coi toss experimet. Questio: You have a coi you suspect may be biased i favor of heads. How ca you decide? You try the experimet of tossig it 10 times, ad you fid that it comes up heads 8 times. You formulate your ull hypothesis that the coi is fair, ad ask: What is the probability that I would see 8 or more heads i te tosses if this coi were i fact fair? Formally, let H 0 = coi is fair, H 1 = coi is biased i favor of heads We will ask what is P (8 or more heads H 0 ). If very small, we will reject the ull. This is a questio we ca aswer without the cetral limit theorem, just by usig probability theory. The probability of 8 heads i 10 tosses of a fair coi is B(8; 10,.5) = ( 10 8 ) /2 10 =.044. Likewise, we ca calculate B(9; 10,.5) ad B(10; 10,.5), add them together to get the probability of gettig 8 or more heads i 10 tosses if the coi were i fact fair: approximately is a example of a p-value. You might see it reported as p =

8 Def : (a bit loose; cf. FPP) The p-value of a hypothesis test is the probability that you would see this data or worse (for the ull hypothesis) if the ull hypothesis were true. So it is fairly ulikely that this data would be produced by a fair coi. But still, i a bit more tha oe i twety such experimets we would see this may heads or worse. What ext? Social scietists (ad scietists much more geerally) have developed covetios here, such as: We reject the ull hypothesis i favor of the alterative hypothesis if the p value is less tha.05, or.01, or.10, etc. I fact, the stadard covetio i political sciece is that you might report a relatioship that has a p value of.10 or less, ad below that how small p is is take as a measure of how decisively the data reject the ull. Notice that it is etirely possible that you could wrogly reject the ull hypothesis. e.g., if you saw 9 heads i 10 tosses, you would coclude that the p value was.01, so you would surely reject the ull i favor of your alterative the coi is biased i favor of heads. But oe i 100 (hypothetical) times this will be a mistake. You will be committig what is called a Type I error wrogly rejectig the ull hypothesis whe the ull is correct. What s the alterative? If we tighte the stadard for rejectig the ull e.g., we will accept the ull for p values of less tha, say,.001 the we will be icreasig the odds of makig a Type II error, which meas wrogly acceptig the ull whe the ull hypothesis is false. There is a tradeoff betwee Type I ad Type II errors. The covetio resolves this coservatively, makig us reluctat to coclude agait the (radom) ull hypothesis uless the data are pretty highly ulikely to observed if the ull is true. 5 Applyig the logic: Usig the ormal approximatio with Beroulli trials Questio: What if you collected more data? Suppose you flip the coi 100 times ad get 62 heads. Agai, our test is to ask what is the probability we observe this may heads or more if the ull hypothesis were true. 8

9 We could ask Stata to figure this out precisely. set obs 101, rage x 0 100, ge bi = comb(100,x)/2 100, graph bi x,s(.), ege p = sum(x) if x > 61. Or much more simply i Stata 7: di Biomial(100,62,.5). But there is aother way that is more useful i geeral because we ca apply it to problems where we do t have such a well-defied uderlyig stochastic process that geerates the data (e.g., life expectacy across coutries): Use the ormal approximatio. By the cetral limit theorem, the sum of the umber of heads i 100 tosses of a fair coi has a approximately ormal distributio. What is E(X) ad var(x), if X is the umber of heads that come up? show that sd(x) = σ, which FPP call square root law, so here sd(x) = 100 p(1 p) = = 5. So the umber of heads i 100 flips has a approx. ormal distributio with mea 50 ad s.d. of 5. What is the probability of observig 62 or more heads? Ask how may stadard uits 62 is away from 50 ad use the ormal table: Approximately z = = 2.4, 5 usig di 1 - ormprob(2.4) i Stata gives.008, which is pretty close to.0104 we calculated directly from the biomial distributio. (I Stata 7, the cumulative ormal fuctio has bee chaged to orm(z).) We ca do slightly better by askig what is the probability of observig 61.5 or more heads (draw diagram). This is what FPP discuss as a cotiuity correctio. The z = ( )/5 = 2.3, so p =.0107 which is very close to the true value ideed. There is a useful ad importat thig to ote about the expressio for z above. z is a test statistic, ad has the geeral form z = observed value value expected H 0 est. stadard error Explai. Be sure to read ad uderstad FPP, chapter 26 o this. Questio: Test the ull hypothesis that global life expectacy by coutry is 65 years agaist the alterative hypothesis that it is less tha 60 years. Compute x 65 z = s/ which is the umber of stadard uits x is away from 65, ad the calculate orm(z). (Discuss oe tailed test...) Note that Stata has a commad that will do t tests automatically: ttest varame = #, for example. Illustrate... (does ot use fiite sample correctio) 9

10 6 Proof that s 2 is a ubiased estimator for σ 2 Why is σ 2 s a biased estimate of σ 2? First a ituitio, the a proof. First, otice that oe of the compoets that goes ito the estimate σ 2 s is the sample mea itself, sice σ 2 s = 1 (x i x) 2 = 1 i=1 (x i 1 which ca be rewritte iside the sum as σs 2 = 1 (x i (1 1 ) 1 x j ) 2 i=1 i=1 j i x j ) 2 Explai rewrite. Notice that this is almost as if we had a ew radom variable x i (1 1/). This variable has to have lower variace tha σ 2 because of the 1 1/ term which is effectively reducig everythig towards the mea. Ituitively, what is happeig is that by usig the sample mea i costructig σ 2 s, we itroduce a ifluece that lowers σ 2 s relative to we are tryig to estimate, σ2. Imagie that i our 25 coutry sample, we happe to get a coutry with very low life expectacy. Note that this pulls our sample mea dow, towards the very low life expectacy umber. I effect, this is reducig the size of our estimate σ 2 s relative to what it would be if we were usig µ istead of x whe calculatig σ 2 s. Now, a proof: 1. First, ote that E(σ 2 s ) = E( 1 (xi x) 2 ) = E( 1 ( x 2 i ) x2 ) j=1 = ( 1 E( x 2 i ) ) E( x 2 ) = E(x 2 i ) E( x2 ) usig a fact about rewritig the expressio for variace you ve see a couple of times, ad the properties of expectatios. 2. Next, ote that E((x i µ) 2 ) = σ 2 by defiitio, so E(x 2 i 2x i µ + µ 2 ) = σ 2 E(x 2 i ) µ2 = σ 2, so E(x 2 i ) = σ2 + µ 2. 10

11 3. I just the same way we ca use the result that to show that E(( x µ) 2 ) = σ2 E( x 2 ) = µ 2 + σ2 4. Now we have expressios for E(x 2 i ) ad E( x2 ), so we ca retur to the result i step 2 above, gettig E(σ 2 s ) = E(x2 i ) E( x2 ) = σ 2 + µ 2 µ 2 σ2 = σ ( ) = σ 2 ( 1 5. This shows that σs 2 is a biased estimate of σ2. It is a little too small o average. 6. A ubiased estimate is: s 2 1 σ2 s = 1 (xi x) 2 1 s (xi x) 2. That takes care of the questio about why σ 2 s is biased. ) 11

Estimating Proportions with Confidence

Estimating Proportions with Confidence Aoucemets: Discussio today is review for midterm, o credit. You may atted more tha oe discussio sectio. Brig sheets of otes ad calculator to midterm. We will provide Scatro form. Homework: (Due Wed Chapter