Kernel Density Estimation. Let X be a random variable with continuous distribution F (x) and density f(x) = d

Size: px

Start display at page:

Download "Kernel Density Estimation. Let X be a random variable with continuous distribution F (x) and density f(x) = d"

Bathsheba Cox
5 years ago
Views:

1 Kerel Desity Estimatio Let X be a radom variable wit cotiuous distributio F (x) ad desity f(x) = d dx F (x). Te goal is to estimate f(x). Wile F (x) ca be estimated by te EDF ˆF (x), we caot set ˆf(x) = d dx ˆF (x) sice ˆF (x) is a step fuctio. Te stadard oparametric metod to estimate f(x) is based o smootig usig a kerel. Wile we are typically iterested i estimatig te etire fuctio f(x), we ca simply focus o te problem were x is a specific fixedumber,adteseeowtemetodgeeralizesto estimatig te etire fuctio. So cosider x fixed. Defiitio 1 K(u) is a kerel fuctio if K(u) =K( u) (symmetric about zero), R K(u)du = 1 ad R K(u)du =0. We will focus o te case were K(u) 0, so tat K(u) is a symmetric desity wit zero mea. We K(u) 0 it is called a secod-order kerel ad tese are te most commo used i applicatios. Te kerel will be used as a weigtig fuctio. Te most commo coices are te Gaussia kerel K(u) =φ(u) = 1 2π exp µ u2, 2 te Epaecikov kerel K(u) = u 2, u 1 0 u > 1 ad te Biweigt or Quartic kerel K(u) = u 2 2, u 1 0 u > 1. Te most importat coice is te badwidt >0 wic cotrols te amout of smootig. If is large, tere is a lot of smootig, ad if is small tere is less smootig. Let K (u) = 1 K ³ u Note tat K (u) is a kerel fuctio. If K(u) is a desity, te so is K (u). Te differece is tat tevariaceofk is tat of K, multiplied by 2. So as gets small, te desity K cocetrates about its mea, zero. Now cosider te radom variable Y = K (X x) were X is te origial radom variable, x is a fixed umber, ad is a badwidt. Y as mea EY = EK (X x) = K (z x) f(z)dz = K (u) f(x + u)du = K (u) f(x + u)du Te secod equality uses te cage-of variables u =(z x)/ wic as Jacobia. Te last expressio sows tat Y is a average of f(z) locally about x..

2 Tis itegral (typically) is ot aalytically solvable, so we approximate it usig a secod order Taylor expasio of f(x + u) i te argumet u about u =0, wic is valid as 0. Tus f (x + u) ' f(x)+f 0 (x)u f 00 (x) 2 u 2 ad tus EY ' K (u) µf(x)+f 0 (x)u + 12 f 00 (x) 2 u 2 du = f(x) K (u) du + f 0 (x) K (u) udu f 00 (x) 2 K (u) u 2 du = f(x)+ 1 2 f 00 (x) 2 κ sice R K (u) du =1,ad R K (u) udu =0, wit κ = R u 2 K (u) du, tevariaceoftekerelk(u). Wile for ay fixed, EY 6= f(x), as 0, EY f(x). Tus we propose estimatig f(x) by tesamplemeaoftey usig a small value of. Te sample value of Y is Y i = K (X i x), wit sample average ˆf(x) = 1 K (X i x). Te is te classic oparametric kerel desity estimator of te desity f(x). Itisteaverageof asetofweigts.ifalargeumberofx i are ear x, te te weigts are relatively large ad ˆf(x) is larger. Coversely, if oly a few X i are ear x, te te weigts are small ad ˆf(x) is small. Te badwidt cotrols te meaig of ear. We derived ˆf(x) as te estimator of f(x) for fixed x. Butitalsoisteestimatorofteetire fuctio. Iterestigly, ˆf(x) is a valid desity we K(u) is a desity. Tat is, sice K(u) 0, te ˆf(x) 0 for all x, ad ˆf(x)dx = 1 K (X i x) dx = 1 K (X i x) dx = 1 were te secod-to-last equality makes te cage-of-variables u =(X i x)/. We ca also calculate te momets of te desity ˆf(x). Te mea is x ˆf(x)dx = 1 xk (X i x) dx = 1 = 1 = 1 X i X i (X i + u) K (u) du K (u) du + 1 uk (u) du K (u) du =1 tesamplemeaoftex i. Agai we used te cage-of-variables u =(X i x)/. Note: tis is te mea of te desity ˆf(x), ot te expectatio E ˆf(x).

3 Te secod momet of te desity is x 2 1 ˆf(x)dx = x 2 K (X i x) dx = 1 (X i + u) 2 K (u) du = 1 Xi X i K(u)du + 1 = 1 Xi κ 2 u 2 K (u) du It follows tat te variace of te desity ˆf(x) is x 2 ˆf(x)dx µ 2 x ˆf(x)dx = 1 Xi κ Ã 1! 2 X i =ˆσ κ Tus te variace of te estimated desity is iflated by te factor κ relative to te sample momet. We ow explore te samplig properties of ˆf(x). Specifically, we calculate te bias, variace ad MSE. Tebiasiseasytocalculate.Weave so E ˆf(x) = 1 EK (X i x) =f(x)+ 1 2 f 00 (x) 2 κ Bias(x) = 1 2 f 00 (x) 2 κ. We see tat te bias of ˆf(x) at x depeds o te secod derivative f 00 (x). Tesarpertederivative, te greater te bias. Ituitively, te estimator ˆf(x) smoots data local to X i = x, so is estimatig a smooted versio of f(x). Te bias results from tis smootig, ad is larger te greater te curvature i f(x). Te itegrated squared bias (a global measure of bias) is Bias(x) 2 dx = 4 κ 2 R(f 00 ) 4 were R(f 00 )= f 00 (x) 2 dx is te Rougess of f 00 or f. It is called te rougess because it idexes te amout of wiggles i f. Not surprisigly, te global bias is iger we te rougess is greater. Furtermore, we ca see tat for ay x ad globally i x, te bias teds to zero as teds to zero. Tus for te bias to asymptotically disappear, must go to zero as. Tis is a miimal requiremet for cosistet estimatio.

4 We ow examie te variace of ˆf(x). Sice it is a average of iid radom variables, usig firstorder Taylor approximatios ad te fact tat 1 is of smaller order ta () 1 we 0 as, Var(x) = 1 Var(K (X i x)) Teitegratedvariaceis = 1 EK (X i x) 2 1 (EK (X i x)) 2 ' 1 µ z x 2 2 K f(z)dz 1 f(x)2 = 1 K (u) 2 f (x + u) du ' f (x) K (u) 2 du f (x) R(K) =. ³ Var ˆf(x) dx ' f (x) R(K) dx = R(K). We see tat for fixed x or globally, te variace teds to zero if as. Togeter, te asymptotic mea-squared error (AMSE) for fixed x is te sum of te approximate squared bias ad approximate variace AMSE (x) = 1 4 f 00 (x) 2 4 κ 2 + ad te mea itegrated squared error (AMISE) is AMISE = 4 κ 2 R(f 00 ) 4 f (x) R(K) + R(K). (1) Asufficiet coditio for cosistet estimatio is tat te MSE teds to zero as. Tis occurs iff 0 yet as. Tat is, must ted to zero, but at a slower rate ta 1. Equatio (1) is a asymptotic approximatio to te MSE. We defie te asymptotically optimal badwidt 0 as te value wic miimizes tis approximate MSE. Tat is, 0 =argmiamise It ca be foud by solvig te first order coditio yieldig d d AMISE = 3 κ 2 R(f 00 ) R(K) 2 =0 0 = µ R(K) 1/5 κ 2 R(f 00. (2) )

5 Tis solutio takes te form 0 = c 1/5 were c is a fuctio of K ad f, but ot of. We tus say tat te optimal badwidt is of order O( 1/5 ). Note tat tis declies to zero, but at a very slow rate. I practice, ow sould te badwidt be selected? Tis is a difficult problem, ad tere is a large ad cotiuig literature o te subject. We see tat te optimal coice is give i (2). Sice is give, ad K (ad tus R(K) ad κ) are selected by te researcer, all compoets are kow except R(f 00 ). Te obvious trouble is tat tis is ukow, ad could take ay value! A classic simple solutio proposed by Silverma as come to be kow as te referece badwidt or Silverma s Rule-of-Tumb. It uses formula (2) but replacig te ukow f wit te N(0, ˆσ 2 ) distributio, were ˆσ 2 is a estimate of σ 2. Tis coice for gives a optimal rule we f(x) is ormal, ad gives a early optimal rule we f(x) is close to ormal. Te dowside is tat if te desity is very far from ormal, te rule-of-tumb ca be fairly iefficiet. Workig troug te itegrals, te rule-of-tumb coice is a simple fuctio of, depedig o te kerel K beig used. Gaussia Kerel: rule =1.06 1/5 Epaecikov Kerel: rule =2.34 1/5 Biweigt (Quartic) Kerel: rule =2.78 1/5 Uless you delve more deeply ito kerel estimatio teory, my recommedatio is to use te rule-of-tumb badwidt, peraps adjusted by visual ispectio of te resultig esitmate ˆf(x). Wile tere are oter approaces, te advatages ad disadvatages are delicate. I ow discuss some of tese coices. Te plug-i approac is to estimate R(f 00 ) i a firststep,adteplug tis estimate ito te formula (2). Tis is more treacerous ta may first appear, as te optimal for estimatio of te rougess R(f 00 ) is quite differet ta te optimal for estimatio of f(x). However, tere are moder versios of tis estimator wic appear to work well. Aoter popular coice for selectio of is kow as cross-validatio. Tis works by costructig a estimate of te MISE usig leave-oe-out estimators. Tere are some desireable properties of cross-validatio badwidts, but tey are also kow to coverge very slowly to te optimal values. Tey are also quite ill-beaved we te data as some discretizatio (as is commo i ecoomics), i wic case te cross-validatio rule ca sometimes selected very small badwidts, leadig to dramatically udersmooted estimates. Fortuately tere are remedies, wic are kow as smooted crossvalidatio wicisaclosecousioftebootstrap. Computatio Typically, we calculate ˆf(x) i order to ave a grapical represetatio of te desity fuctio. I tis case, we start by defiig a set of gridpoits {x 1,..., x g } were we will calculate ˆf(x). Some researcers set te gridpoits equal to te sample values. Oters set a uiform grid betwee te mi ad max of te data or a selected quatile. At eac poit x j, te desity estimate ˆf(x j )= 1 K (X i x j ) is calculated. A easy way to do tis is to write te computer code to loop across te x j, ad te compute ˆf(x j ) at eac poit by a simple sample average of te kerel weigts. (Tis is ot a efficiet computatioal algoritm, but ease of programmig ofte outweigs umerical efficiecy.) Oce tese ave bee all calculated, te pairs {x j, ˆf(x j )} ca be plotted.

14.30 Introduction to Statistical Methods in Economics Spring 2009

14.30 Introduction to Statistical Methods in Economics Spring 2009 MIT OpeCourseWare http://ocwmitedu 430 Itroductio to Statistical Methods i Ecoomics Sprig 009 For iformatio about citig these materials or our Terms of Use, visit: http://ocwmitedu/terms 430 Itroductio