4452 Mathematical Modeling Lecture 17: Modeling of Data: Linear Regression

Transcription

1 Mah Modeling Lecure 17: Modeling of Daa: Linear Regression Page 1 5 Mahemaical Modeling Lecure 17: Modeling of Daa: Linear Regression Inroducion In modeling of daa, we are given a se of daa poins, and we wan o fi a funcion wih adjusable parameers o he daa poins. Obviousl, we wan he funcion o approximae he daa as well as possible, and o do ha ou have o choose cerain values for he parameers in our funcion. You choose hese parameer values b firs designing a meri funcion which ou wish o minimize. When he meri funcion is minimized, he agreemen beween he funcion and he daa will have close agreemen. You can see ha fiing a funcion o daa becomes a problem of minimizaion in man dimensions (he number of adjusable parameers in our funcion is he dimension of he problem). Once we have fi he funcion o he daa, we need o assess how good he fi acuall is. There has o be some sor of saisical analsis of he fi. The bulk of his discussion was based on Reference [1], which is an excellen firs resource for a varie of applied numerical analsis. General Se Up We have N daa poins ( i, i ), i = 1,,..., N which we wan o fi o a model funcion which has M adjusable parameers, f() = f(; α 1,..., α M ). We acuall have a grea deal of choice in wha pe of funcion we wan o minimize. I can be anhing ha will measure he relaion of he daa o he model funcion. The vecor which compares he daa o he model funcion a each poin is given b 1 f( 1 ; α 1,..., α M ) f( ; α 1,..., α M )... N f( N ; α 1,..., α M ) We can minimize his vecor based on a varie of differen norms: l 1 norm: l p norm: l norm: i f( i ; α 1,..., α M ) ( ) 1/p ( i f( i ; α 1,..., α M )) p N max ( i f( i ; α 1,..., α M )) Wha is picall done is ha he l norm is used, since i is he Euclidean space norm, and he square of

2 Mah Modeling Lecure 17: Modeling of Daa: Linear Regression Page he norm is minimized (hence he name: leas squares fi): minimize F (α 1,..., α N ) = ( i f( i ; α 1,..., α M )) The above resul assumes we know he daa poins onl. However, wha if we know he daa poins and for each daa poin a sandard deviaion, σ i? How could his be incorporaed ino he funcion we wish o minimize? We can do he following minimize F (α 1,..., α N ) = ( ) i f( i ; α 1,..., α M ) (1) σ i which assumes ha each daa poin has a measuremen error which is independenl random and disribued as a normal disribuion around he acual model f(). This resul is based on a grea deal of saisics, and he idea ha random deviaions will converge o a normal disribuion. Of course, his ma no be he case in pracice. Frequenl, i is rue ha σ i = σ is he same for all he daa poins. In ha case, σ can be facored ou of he sum and he σ does no appear in he soluion for α 1 and α. Since his is he case, if ou are given daa which does no have an associaed error σ i which depends on he daa poin ou can simpl se σ i = 1 and proceed wih he analsis. Minimizing Eq. (1) is jus a mulivariable unconsrained minimizaion procedure, which ields he ssem of equaions = ( ) ( ) i f( i ; α 1,..., α M ) f( i ; α 1,..., α M ), k = 1,..., M () α k σ i which mus be solved for he M unknowns α i. Linear Regression Linear regression does no mean fiing daa o a sraigh line! The linear refers o he models dependence on he parameers α k. However, for now we are ineresed in fiing o a sraigh line. In his case, our fiing funcion becomes f(; α 1, α ) = α 1 + α. The ssem of equaions in Eq. () becomes N 1 α 1 σ i i α 1 σ i + α N i σ i + α N i σ i = = i σ i i i σ i (3)

3 Mah Modeling Lecure 17: Modeling of Daa: Linear Regression Page 3 We can simplif he noaion if we use he following: S = 1, S = σ i i σ i, S = i σ i, S = i σ i, S = i i σi, = SS S. The soluion o Eq. (3) is given b α 1 = S S S S α = SS S S The Correlaion Coefficien How Good is Our Model Funcion? All we need now is an esimae of how good our linear fi is. Reference [1] has a significanl expanded discussion on deermining how good our linear regression model is. We will consider he case where σ i σ for all i. This is frequenl no a resricive assumpion, since he sources of error in measuring he daa ha lead o σ i are frequenl he same for all measuremens. We calculae wha is called he correlaion coefficien, R, which is a raio of he model sum of he squares o he oal sum of he squares R = (f( i) ȳ) ( i ȳ), ȳ = 1 N i. Thus, if R 1 he model is a good represenaion of he daa, and he daa are represenaive of a linear funcion. If R he daa are esseniall random, and a linear funcion canno represen he daa well. The resuls of his analsis for a paricular daa se (conained in he corresponding Mahemaica file) are shown in Fig. 1. The daa se is given for compleeness in he Appendix Figure 1: Linear fi, f() = , o a daa se. For his fi, he correlaion coefficien was found o be R =.9, which indicaes ha he daa does represen a linear funcion, and he linear funcion we found represens he daa well.

4 Mah Modeling Lecure 17: Modeling of Daa: Linear Regression Page Exrapolaion Once we have found he linear model of he daa, wha is i good for? Man imes, wha is of ineres is he slope of he curve, or he -inercep. Or, he model can be used for exrapolaion. In an case, we would like an esimae of he sandard deviaion of he model from he daa. We can ge an esimae b compuing σ = N ( i f( i )) () If our daa is normall disribued abou he model funcion (which i ma ver well no be!), we would expec measuremens will be wihin ±σ of he model funcion % of he ime, and wihin ±σ 95% of he ime. Figure shows his resul Figure : An example of he number of daa poins conained wihin ±σ (lef) and ±σ (righ) of he model funcion, wih σ = 1.15 from Eq. (). We ge % wihin ±σ and 95% wihin ±σ. We can use his σ o esimae he error in exrapolaion. Since we are assuming he model funcion is accurael represening he underling linear dependence of he daa, he random flucuaions in measuring he daa would sill accoun for error in fuure measuremen. We would expec, for example, since f(5) = and σ = 1.15 ha if we measured he ssem a = 5 we would find % of he ime. Ouliers The average of all he daa poins is he poin (, ȳ) where = 1 N i, ȳ = 1 N i The model funcion will alwas go hrough his poin if we have σ i = 1. In a sense, his represens he cenroid of he daa se.

5 Mah Modeling Lecure 17: Modeling of Daa: Linear Regression Page 5 An oulier is a daa poin which lies far off he regression line. The effec of his oulier is o inordinael affec he underling model funcion. Figure 3 shows a daa se wih an oulier, and he fi which was obained. 3 Figure 3: Linear fi o a daa se conaining an oulier. For his fi, he correlaion coefficien was found o be R =.9, which should make us quesion wheher or no he fi is accurael represening he daa. The oulier is srong enough o make all he oher poins from o lie beneah he model funcion. I is imperaive ha an analsis of daa ses which uses linear regression includes some analsis of ouliers, especiall for small daa ses, like he one picured, for which he effec of one oulier can be enormous. If an oulier can be idenified, i should be deleed from he daa se and he linear regression redone. You can deec ouliers eiher visuall (if he are obviousl ouliers, like in m example), or b a more ssemaic analsis of he quani i f( i ). If his quani is exremel large for a few daa poins, he ma be ouliers. A more saisical wa o find ouliers i is o search he daa for poins for which i f( i ) > σ. Obviousl, deleing daa from our daa se should be done wih exreme cauion, and mus be repored full in an use of he daa se or model funcion which is creaed afer he removal of ouliers. Oher Forms of Linear Regression As menioned before, he linear in linear regression comes no from he fac ha we are fiing a sraigh line o he daa poins, bu ha we are fiing a model funcion which depends linearl on he fi parameers α k. We could be sines, cosines, or oher powers of. Le s look a an example of using a differen fi model. Consider he daa se shown in Fig.. From looking a he daa, we hink ha i looks quadraic. So our model funcion should be chosen o be f(; α) = α. Noe ha if we waned o chose a model funcion ha looked like α 1 α his would be a nonlinear regression problem, since he parameers α k no longer appear in a linear manner.

6 Mah Modeling Lecure 17: Modeling of Daa: Linear Regression Page Figure : Daa se which is no modeled well b a sraigh line (op lef); linear regression model funcion f() =.3793, σ =.3 (op righ); ±σ inerval conains 7% of he poins (boom lef); ±σ inerval conains 95% of he poins (boom righ). We can use he linear regression ideas o deermine α. The ssem in Eq. () is now simpl he equaion = ( i α i ) i which means he parameer α is given b α = i i. i I again esimaed σ as σ = N ( i f( i )). Final Though M analsis in his lecure is saisfacor for mos applicaions of linear regression ou will run ino. However, if ou ge ino serious linear regression applicaions, ou should undersand he deeper saisical heor behind i. You will need o have a deeper undersanding if ou use saisics packages o do our regression for ou, since ou should alwas undersand wha i is ha he compuer is elling ou. Mahemaica produces

7 Mah Modeling Lecure 17: Modeling of Daa: Linear Regression Page 7 a remendous amoun of oupu wih is linear regression package, and I have o sa ha I undersand onl a small porion of i. If I use i, I onl use he packages I undersand. Appendix Daa se used in Figs. 1 and : {{1, 7.137}, {, }, {3,.75}, {, 9.39}, {5, 7.577}, {,1.5}, {7, 1.35}, {, 11.73}, {9, 1.73}, {, }, {11, 1.99}, {1, }, {13, 19.91}, {1, }, {15,.1733}, {1,.57}, {17, 1.7}, {1,.735}, {19, 5.39}, {, 5.9}} Daa se wih oulier, used in Fig. 3: {{1, 7.137}, {, }, {3,.75}, {, 9.39}, {5, 7.577}, {, 1.5}, {7, 1.35}, {, 11.73}, {9, 1.73}, {, }, {11, 1.99}, {1, }, {13, 19.91}, {1, }, {15,.1733}, {1,.57}, {17, 1.7}, {1,.735}, {19, 5.39}, {, 5.9}} Daa se used in Fig. : {{,.731}, {1,.151}, {,.9979}, {3,.577}, {,.533}, {5,.79571}, {, }, {7,.977}, {,.9}, {9,.7139}, {, 3.99}, {11,.5951}, {1, 5.73}, {13, 5.599}, {1,.9377}, {15, }, {1,.39}, {17, 9.57}, {1,.}, {19, 11.7}, {, 1.53}} References [1] W. H. Press, S. A. Teukolsk, W. T. Veerling & B. P. Flanner, Numerical Recipes in Forran 77, nd ed, Cambridge Universi Press (New York) 199. [] M. Meerschaer, Mahemaical Modelling, nd ed., Academic Press (San Diego) 1999.