Deriving & Understanding the Variance Formulas

Size: px

Start display at page:

Download "Deriving & Understanding the Variance Formulas"

Vincent Knight
5 years ago
Views:

1 Dervg & Uderstadg the Varace Formulas Ma H. Farrell BUS 400 August 28, 205 The purpose of ths hadout s to derve the varace formulas that we dscussed class ad show why take the form they do. I class we wet over the formulas at a tutve level, ad dscussed whch features of the data mpacted them ad what ways. But why were those the rght formulas? Ths hadout tres to aswer that questo, whch may also shed some lght o how the formulas work. Oe mportat ssue wll be the formg of predcto tervals, ad the reaso we care about V[e f ] versus V[Ŷf ]. The ma puchle here s ths: a predcto terval s a rage of lkely values for Y f, ot E[Y X = X f ] = β 0 + β X f. Throughout, we wll oly look at smple lear regresso. The formulas are easer to uderstad, ad the tuto s etrely the same. (The formulas are the same too, just reterpretg X as vectors or matres as approprate.) Also, just lke class we wll do everythg codtoal o X. That meas we wll treat the values X, X 2,..., X as fed umbers; they are ot radom. Ths does t chage ay of the tuto ether. Cotets Notato Week 2: Samplg Dstrbutos, Dervato Nocostat varace Week 2: Samplg Dstrbutos, Dervato Weghted Average Formulas for b 0, b, ad Ŷ Varace Dervatos for b 0 ad b Varace Dervatos for Ŷf ad Predcto Itervals Uderstadg Predcto Itervals, Whch are for Y f Week 4: Varace of e j ad leverage Notato Remember that the model s Y = β 0 + β X + ε, ε N(0, σ 2 ). Note that V[Y X ] = σ 2.

2 The sample varaces of X ad Y are: s 2 = The sample covarace s s y = The sample correlato s: r y = s y s s y. (X X) 2 ad s 2 y = (X X)(Y Ȳ ) = (Y Ȳ )2. (X X)Y. 2 Week 2: Samplg Dstrbutos, Dervato I ths secto I gve a farly smple dervato of the samplg dstrbuto of the slope estmate b. Smlarly dervatos apply to the tercept estmate, b 0, a forecast, Ŷf, ad aythg from multple lear regresso. Recall from week that b = corr(x, Y ). var(x) Usg ths, pluggg the defto of Y = β 0 + β X + ε, we get b = (X X)Y (X X) 2 = (X X) β 0 + β X + ε (X X) 2 = β (X X) 0 (X X) 2 + β (X X)X (X X) 2 + (X X)ε (X X) 2 (X X)ε = 0 + β + (X X) 2. The last equalty comes from these two facts: (X X) = 0 (X X)X = (X X) 2, whch you ca verfy by drect calculato (just lke you dd o homework zero!). So we have show that (X X)ε b = β + ( )s 2. The frst term s just β, whch s just some fed umber (eve though we do t kow t). The secod term s Normally dstrbuted, ether by the Cetral Lmt Theorem or because the ε are 2

3 assumed to be Normal. Therefore b s also Normally dstrbuted, but what Normal dstrbuto? A Normal dstrbuto s characterzed by the mea ad varace. The mea s easy: the secod term has mea zero, because the ε have mea zero, ad so the mea of b s just β. To compute the varace, remember that β s just a umber, so t has o varace, ad that we are treatg the X as fed umbers. Therefore: V[b ] = V [ (X X)ε ] ( )s 2 = (( )s 2 ) 2 (X X) 2 V [ε ]. Now, we use the assumpto that the ε have costat varace, σ 2. The we ca pull t out of the summato, ad we get the result from class: V[b ] = σ 2 (( )s 2 ) 2 (X X) 2 = σ 2 (( )s 2 ) 2 ( )s2 = σ 2 ( )s 2. We have thus show that ( ) b N β, σ 2 ( )s 2 Ths eactly matches the result from class ad t matches Equato (6) below. () 2. Nocostat varace To see what happes wth the varace s ot costat, retur to the peultmate step above: V[b ] = (( )s 2 ) 2 (X X) 2 V [ε ]. Suppose every ε ca have a dfferet varace; call t σ 2. The we ca t pull aythg out of summato! We just get: V[b ] = ad therefore ( b N (( )s 2 ) 2 β, (X X) 2 σ 2, (X X) 2 σ 2 ) (( )s 2 ) 2. To deal wth ths, we ether eed to do a varace-stablzg trasformato or do heteroskedastcty robust ferece; both of whch we dscuss Week 4. 3 Week 2: Samplg Dstrbutos, Dervato 2 Here I carefully derve all the varace formulas for b 0, b, ad Ŷ usg a weghted-average represetato that wll prove very useful. Ths s the frst thg that s troduced, the et subsecto. 3

4 3. Weghted Average Formulas for b 0, b, ad Ŷ We wll show that b 0, b, ad Ŷ are all just weghted averages of the outcomes Y. Ths kd of makes sese the followg way: regresso our am s to etract a geeral, o-average tred for Y gve X. Eve more precsely, we are estmatg the codtoal epectato, ad sce epectatos are just averages, t makes sese that our estmators are just averages. Ths s more tha just a ce cocdece, t s a mportat dea terms of the type of estmato we re dog ad how we get the results we do. Beg wth the slope coeffcet, b. Recall from class that b = r y s y /s. Let s wrte out eactly what that meas, ad re-wrte the formula: b = r y s y s = s y s s y s y s = s y s 2 = s y s 2 = (X X)Y (X X) 2 = (X X)Y (X X) 2. Now, we ca recogze that ths s really just a weghted sum of the values Y, =,..., : b = W Y, where W = (X X) (X. (2) X) 2 Ths wll be a very useful formula. It s also mportat to ote that W = (X X) (X X) 2 = X X We ca do the same thg for the tercept ow too: b 0 = Ȳ b X = Y (X X) 2 = X X (X X) 2 = 0. (3) W XY = ( ) W X Y. (4) Ad for a predcto for some ew value X f. Remember that we just read the predcto off the least squares le: Ŷ f = b 0 + b X f. Therefore: Ŷ f = ( ) W X Y + W Y X f = ( ) + W [X f X] Y (5) 3.2 Varace Dervatos for b 0 ad b It s ow really smple to compute the varace of the tercept ad slope coeffcet. Remember that the X are fed umbers ad that the Y are depedet of each other. Usg the weghted sum formula (2): [ ] V[b ] = V W Y = V [W Y ] = V [Y ] = σ 2 4 = σ 2 (X X) 2 ( (X X) 2) 2.

5 Cacelg the umerator ad deomator we have σ 2 V[b ] = (X X) 2 = σ 2 ( )s 2. (6) Ths eactly matches the result from class ad t matches Equato () above. The tercept works almost eactly the same way. Frst, epad as follows. V[b 0 ] = σ 2 ( W X ) 2 = σ 2 ( 2 + W 2 X 2 2 XW = σ X 2 2 X The frst term s just σ 2 /. For the secod, the same cacelg the umerator ad deomator of W 2 that we dd just above shows that ths term s gog to be σ 2 X2 /( (X X) 2 ) = σ 2 X2 /[( )s 2 ]. Fally, the thrd term s zero because (3) tells us that W = 0. Puttg ths together we get W. V[b 0 ] = σ2 + σ2 X2 ( )s 2. (7) We foud the same thg class, but stckg to the orgal formula b 0 = Ȳ + b X. From ths, t follows that V[b 0 ] = V[Ȳ ] + X 2 V[b ] + 2 XCov(Ȳ, b ) (remember X s fed because we are codtog o all the X ). The frst two terms are the same as (7), ad the thrd we argued tutvely was zero because the average dd t tell us aythg about the slope, just shfted the le up or dow parallel. We have ow formalzed that dea. 3.3 Varace Dervatos for Ŷf ad Predcto Itervals Frst, let us derve V[Ŷf ] ad V[e f ], ad the talk about what a predcto terval s. From (5) ad usg the eact same trcks as the prevous subsecto: V[Ŷf ] = ( ) 2 + W [X f X] σ 2 = σ σ2 (X f X) σ2 (X f X) W. The frst term s σ 2 /. The secod term, after the same cacelg the umerator ad deomator of W 2 that we dd above, becomes σ 2 (X f X) 2 /[( )s 2 ]. As before, the thrd term s zero because of (3). Therefore ( V[Ŷf ] = σ 2 + (X f X) 2 ) ( )s 2. (8) The estmator of ths s called s 2 ft class. I class, we derved ths straght from the predcto Ŷ f = b 0 + b X f, ad got the eact same aswer. There we had to use the covarace of b 0 ad b, whch ca be derved usg these same trcks aga. 5

6 Fally, because the ew observato X f ad Y f s depedet of all the others, ad because e f = Y f Ŷf, we have: V[e f ] = V[Y f ] + V[Ŷf ] 2Cov(Y ( f, Ŷf ) = σ (X f X) 2 ) ( )s 2. (9) The estmator of ths s called s 2 pred class. 3.4 Uderstadg Predcto Itervals, Whch are for Y f What s a predcto terval ad why do we eedv[e f ]? A predcto s just lke a cofdece terval, t s rage of lkely values. But lkely values of what? Ths s the key: t s a rage of lkely values for Y f. We thk of 95% predcto/cofdece tervals as beg of the form: somethg ± 2 (the std err of that somethg). I fact, ths s eactly how we form cofdece tervals for β : [ ( P β b ± 2 )] V[b ] = Why? Because b s our estmator of β, ad sce β s a fed (but ukow) umber, V[b ] = V[b β ]. That s, the varace of our estmator s the same as the varace of the error we make. What happes f we try to apply the same logc to Ŷf? Well, Ŷf s our estmator of Y f, but Y f s ot a fed value! It s radom, ad we have t observed t. So t s ot true that V[Ŷf ] = V[Y f Ŷf ]. So what we really wat s a rage of lkely values for Y f that s cetered at Ŷf. That s our predcto terval, ad t s formed as Ŷ f ± 2 V[e f ]. What would happe f we stead used Ŷf ± 2 V[Ŷf ]? Ths s a 95% cofdece terval for E[Y X = X f ] = β 0 + β X f, that s, the average value for Y gve that X s X f. But we do t wat a rage of lkely values for for the average value of Y f, we wat oe for the actual of Y f. The actual value (that we have t observed yet) s the average value wth the dosycratc shock up or dow, ad t s the ucertaty of ths shock that we capture wth the etra varablty. 4 Week 4: Varace of e j ad leverage [Note: class we looked at the subscrpt pot, but here I m chagg that to j to ot get cofused wth other otato, lke.] To derve the varace of a gve resdual e j, we have to re-walk the steps that we dd for V[e f ], but wth oe crucal, yet subtle dfferece: for e f, the ew observato X f ad Y f s depedet 6

7 of the data used to form the guess Ŷf, but ow, e j s based o oe of the pots already our data set. Go back to Equato (9): the covarace term wll ot cacel, ad we ow have V[e j ] = V[Y j ] + V[Ŷj] 2Cov(Y j, Ŷj). (0) We already kow V[Y j ] = σ 2 from the defto of the model, ad from Equato (8) (appled to Y j stead of Y f ) we kow that ( V[Ŷj] = σ 2 + (X j X) 2 ) ( )s 2. () So we just eed to fgure out the covarace term. Usg the weghted average formula for Ŷj from Equato (5) (appled to Y j stead of Y f ), we fd that ( ( ) ) 2Cov(Y j, Ŷj) = 2Cov Y j, + W [X j X] Y ( ) = 2 + W [X j X] Cov (Y j, Y ). But because all the observatos are depedet, Cov(Y j, Y ) = 0 uless = j, ad the you get Cov(Y j, Y j ) = V[Y j ] = σ 2. So oly oe term of the s left, ad we get ( ) ( 2Cov(Y j, Ŷj) = 2 + W j[x j X] σ 2 = 2 + (X j X) 2 ) ( )s 2 σ 2 (2) where the secod equalty just plugs the defto of W j. Pluggg Equatos () ad (2) to (0), we get the fal aswer: V[e j ] = V[Y j ] + V[Ŷj] 2Cov(Y j, Ŷj) = σ 2 + σ 2 ( + (X j X) 2 whch s eactly the formula we have from week 4. ( )s 2 ( = σ [ 2 + (X j X) 2 ( )s 2 ) ( 2 )], + (X j X) 2 ( )s 2 ) σ 2 7

1036: Probability & Statistics

1036: Probability & Statistics 036: Probablty & Statstcs Lecture 9 Oe- ad Two-Sample Estmato Problems Prob. & Stat. Lecture09 - oe-/two-sample estmato cwlu@tws.ee.ctu.edu.tw 9- Statstcal Iferece Estmato to estmate the populato parameters