PERFORMANCE OF RIDGE REGRESSION ESTIMATOR METHODS ON SMALL SAMPLE SIZE BY VARYING CORRELATION COEFFICIENTS: A SIMULATION STUDY

Transcription

1 Journal of Mathematcs and Statstcs 10 (1: 5-9, 014 ISSN: Scence Publcatons do: /jmss Publshed Onlne 10 (1 014 (htt:// PERFORMANCE OF RIDGE REGRESSION ESTIMATOR METHODS ON SMALL SAMPLE SIZE BY VARYING CORRELATION COEFFICIENTS: A SIMULATION STUDY 1,,3 Anwar Ftranto and 1 Lee Ceng Y 1 Deartment of Mathematcs, Facult of Scence, Unverst Putra Malasa, Serdang, Malasa Insttute for Mathematcal Research, Unverst Putra Malasa, Serdang, Malasa 3 Deartment of Statstcs, Bogor Agrcultural Unverst, Bogor, Indonesa Receved ; Revsed ; Acceted ABSTRACT When ndeendent varables have hgh lnear correlaton n a multle lnear regresson model, we can have wrong analss. It haens f we do the multle lnear regresson analss based on common Ordnar Least Squares (OLS method. In ths stuaton, we are suggested to use rdge regresson estmator. We conduct some smulaton stud to comare the erformance of rdge regresson estmator and the OLS. We found that Hoerl and Kennard rdge regresson estmaton method has better erformance than the other aroaches. Kewords: Multcollneart, Multle Lnear Regresson, Rdge Regresson 1. INTRODUCTION haen due to the resent of hgh leverage onts. Recent researches focused on hgh leverage onts corresond to Man goal of multle lnear regresson model s to outler are Bagher and Md (009. determne the best set of arameters, β, so that the Rdge regresson estmator methods have been redcted value of deendent varables close to the real roosed as alternatves to the OLS estmators when the values (Orlov, In multle lnear regresson ndeendent assumton has not been satsfed n the models, we normall assume that the ndeendent analss. Several methods have been roosed for varables are ndeendent. However, n ractce, the estmatng the rdge arameter and consder a crteron exlanator varables ma be correlated between each for comarson of the estmators. We resent several other. Ths nter-relaton between the exlanator methods based of rdge regresson estmators. varables s called multcollneart. It s the undesrable Hence, rdge regresson estmator has been roosed stuaton and can haen when the correlatons among as an alternatve to the OLS estmators when the the ndeendent varables are strong. It has several ndeendent assumton has not been satsfed n the effects as has been descrbed b Judge et al. (1988. One analss. The rdge estmator constrans the length of the of them s that t ncreases the standard errors of the coeffcents. In ths stuaton, the ndeendent regresson coeffcent of the estmator n the resence of assumtons are no longer vald n multle lnear multcollneart. Rdge regresson wll be able to regresson models. The regresson coeffcents whch are mnmze the varance of the estmators when the desgn based on Ordnar Least Square estmator (OLS tends to matrx s not nvertble. The modfcaton of desgn become unstable n the resence of multcollneart. matrx to mae ts determnant dfferent form 0 causes Wethrll (1986 also mentoned that multcollneart s a the estmator to be based. Ths method s sgnfcantl serous roblem when we mae nferences for a model reduces the varance of the estmators. Through ths so that t must be handled aroratel. research, we want to observe how are the arameters of Excet due to strong natural lnear correlaton rdge regresson estmator n dfferent level of correlaton between ndeendent varables, multcollneart can coeffcents b usng Monte Carlo rocedure. Corresondng Author: Anwar Ftranto, Deartment of Mathematcs, Facult of Scence, Unverst Putra Malasa, Serdang, Malasa Scence Publcatons 5

2 Anwar Ftranto and Lee Ceng Y / Journal of Mathematcs and Statstcs 10 (1: 5-9, Rdge Regresson Model Multcollneart refers to a stuaton n whch two or more redctor varables n a multle regresson model are hghl correlated. Multcollneart occurs when there s a lnear relatonsh between one or more of the ndeendent varables. In ths stuaton, the regresson coeffcents change sgnfcantl n resonse to small changes n the model. The regresson coeffcents cannot be estmated wth great accurac because the coeffcents ossess large varance. The rdge regresson estmator s much more stable than the OLS estmator n the resence of multcollneart. The rdge estmator restrcts the length of the coeffcents estmator n order to reduce the effects of multcollneart (Hocng et al., In the resence of multcollneart, Hoerl and Kennard (1970 ntroduced the rdge estmator as an alternatve to the OLS estmator when the ndeendent assumton s not longer vald. The rdge estmator s shown as follow Equaton (1 (Hoerl and Kennard, 1970: β = X X + Ι X ( 1 Scence Publcatons (1 where, the I denotes an dentt matrx and s nown as rdge arameter. The MSE of ˆβ s shown as follow: MSE β = σ + β X X + Ι β ( = 1 ( + ( ( The MSE( β n Equaton ( deends on unnown arameters, β and σ, whch can t be calculated n ractce. As ncrease from zero to nfnt, the regresson estmates wll aroxmatel equal to zero. It MSE β ˆ comared to the OLS elds mnmum ( estmator, although these estmator results n bas, for a certan value of (Hoerl and Kennard, In ractce, we have to estmate from the real data nstead. Standard model of a multle lnear regresson can be exressed nto canoncal form. An orthogonal matrx D exsts such that: D CD = Λ where, C = X X and Λ = dag ( 1,,, contans the egenvalues of the matrx C, then the canoncal form of the model (1 s Equaton (3: 6 * = Xα + ε (3 where, X* = XD and α = D β. The general form of OLS estmator s shown as follows Equaton (4: 1 * α = Λ X Then, the rdge estmator s wrtten as Equaton (5: ( 1 * * * α( = X X + K X (4 (5 where, K = dag ( 1,,,, >0. The rdge estmator n Equaton (4 s nown as general form of rdge regresson (Hoerl and Kennard, Accordng to Hoerl and Kennard (1970, the value of 1 whch mnmzes the Equaton (6: MSE α( Is: ( = σ + = 1 ( + = 1 ( + α (6 σ = (7 α where, σ denotes the error varance of model Equaton (1, α s the th element of α. Equaton (7 shows that the values of full deends on the unnown σ and α. Snce σ and α are unnown, these values must be estmated from the observed data. Bhar and Guta (001 roosed a new crteron of detectng outler n exermental desgns whch s based on average Coostatstc. Meanwhle, Zhou and Zhu (003 realzed the fact that n ractce, exerments ma eld unusual observatons (outlers. In the resence of outlers n a data, estmaton methods such as ANOVA, truncated ANOVA, Maxmum Lelhood (ML and modfed ML do not erform well, snce these estmates are greatl nfluenced b outler. Zhou and Zhu (003 verfed that wth robust desgns, one can get effcent and relable estmates for varance comonents regardless of outlers whch ma haen n an exerment. Then Gou (006 conducted further research regardng outler n an exerment who descrbed how to dscover an outler and estmate ts true value and recentl, Ftranto and Md (013 who comared classcal and robust aroach n exermental desgn. The method s based on the use of a dnamc varable and the small effects of the Danel s dagram.

3 Anwar Ftranto and Lee Ceng Y / Journal of Mathematcs and Statstcs 10 (1: 5-9, Method for Estmatng Rdge Regresson Parameters Several methods have been roosed n order to defne a new estmator that can erform better comared to the exstng methods. In ths art, we resent some methods for estmatng rdge arameter. Hoerl and Kennard (1970 found that the best method to estmate α( s to use = for all and the suggested s to be HK (or HK where Equaton (8: ˆ HK σˆ = ( αˆ Scence Publcatons If σ and α are nown, then HK s suffcent to gve rdge estmators havng smaller MSE than the OLS estmator. Hocng et al. (1976 defned a new method for choosng arameter. The suggested an estmator of b usng HSL (or HSL, whch roduces the followng estmator Equaton (9: ˆ ( αˆ = 1 HSL = σ ( αˆ = 1 Recentl, Alhams and Shuur (007 suggested a new aroach for choosng the rdge arameters b addng 1/ to some well-nown estmators, where s the largest egenvalues of X X. The aled the modfcaton to the revous estmator whch was roosed b Hocng et al. (1976 n order to defne a new estmator NHSL (or NHSL Equaton (10: ˆ ( α = 1 NHSL = σ + = HSL + ( ˆ = 1 α ˆ 1 ˆ 1 Snce 1 > 0, NHSL, s greater than HSL The use of Monte Carlo Smulaton (8 (9 (10 Monte Carlo method s a stochastc technque whch s used to nvestgate roblems based on the use of random numbers and the robablt statstcs. We can use Monte Carlo method to solve hscal roblems, for examle t allows us to examne more comlex sstems. Wth Monte Carlo method, we can samle the large sstem n a number of random confguratons. Bagher and Md (009 also conducted Monte Carlo smulaton stud n a robust aroach n the resence of multcollneart. 7 Smulaton stud wll be dscussed to comare the erformance of rdge estmators under several degrees of multcollneart. Dfferent rdge estmators corresondng to dfferent values of rdge arameter are consdered. McDonald and Galarneau (1975 and several other researchers used the followng equaton to generate the exlanator varables Equaton (11: ( 1 x = 1 γ z + γ z, = 1,,...,n, j = 1,,..., (11 j j where, z j s ndeendent standard normal seudo-random numbers and γ s lnear correlaton between an two exlanator varables. The n observatons for the deendent varable are determned b Equaton (1: = β + β x + β x β x + ε, = 1,, n (1 where, ε are ndeendent normal (0,σ seudo-numbers. The comarson s based on the MSE crtera. The MSEs of rdge estmators are evaluated b Equaton (7.. METERIALS AND METHODS.1. Smulaton Desgn The smulaton s conducted b usng SAS release 9.. To acheve dfferent degrees of correlaton, the exlanator varables were generated usng the Equaton (11. Sze of samle to be consdered n ths research s small samle of sze 0 wth number of exlanator varables of equal to 10. Dfferent values of correlatons are consdered n the smulaton stud are 0.5, 0.7 and 0.9. These three values of to reresent low, moderate and hgh correlatons between exlanator varables. The exlanator varables need to be standardzed so that the wll be n correlaton form. Meanwhle, fve dfferent values of standard devaton to be consdered n ths stud, whch are 0.1, 0.5, 1.0, 5.0 and Performance Measures of the Estmators For gven values of, σ and γ, we reeated the exerment b 1000 tmes. For each relcaton, r = 1,,3,..,1000, the values of these three rdge estmators and the corresondng arameters, wll be estmated usng the standardzed varables and then the estmated coeffcents are transformed bac to the orgnal model. The values were comuted based on ts corresondng method. The erformance of the estmators s evaluated n terms of the averaged mean square error (Dorugade and Kashd, 010 wth the followng equaton:

4 Anwar Ftranto and Lee Ceng Y / Journal of Mathematcs and Statstcs 10 (1: 5-9, 014 ( ( ( α α ( α( α R r r 1000 r = 1 MSE α ˆ = ˆ ˆ The comarson between estmated MSEs are then based on the values of, σ and γ. 3. RESULTS AND DISCUSSION Results of the estmators erformance are dslaed n Table 1. The table dslas the MSEs of each estmator under several levels of correlatons corresondng to dfferent values of σ. The frst column of the table contans σ whch has fve dfferent values. The second column of the table contans the correlaton coeffcent, γ. We comare the MSEs of each estmator under three levels of correlatons where γ corresondng to dfferent values of σ. From Table 1, we notced that the HK and HSL estmators are better than OLS estmator for all levels of correlatons corresondng to dfferent values of σ. Ths result n accordance and strengthens the revous research whch have been conducted b Al-Hassan (010. Table 1. Estmated MSEs of each rdge regresson estmator at three levels of correlatons corresond to dfferent values of σ Estmaton method Std dev (σ γ OLS HK HSL NHSL Fg. 1. Plot of estmated MSEs obtaned b dfferent rdge regresson methods of each rdge regresson estmator at three levels of correlatons corresond to dfferent values of σ Scence Publcatons 8

5 Anwar Ftranto and Lee Ceng Y / Journal of Mathematcs and Statstcs 10 (1: 5-9, 014 The MSEs of OLS estmator are lower than the MSEs of NHSL estmator for all levels of correlaton when the value of σ = 0.1. However, the NHSL estmator erforms better than OLS estmator when σ>0.1. From Table 1, we can conclude that HK estmator erforms better than the OLS and other rdge estmators. We dsla grahcal lot of MSE (of each rdge regresson method versus the level of correlaton coeffcents b dfferent values of standard devatons on Fg. 1. We comare the MSEs of each estmator grahcall b varng the standard devatons and the correlatons between the exlanator varables. The results of dfferent levels of correlatons corresondng to standard devaton as shown below. In Fg. 1 we can see that when the data has small varablt (whch are reresented b σ = 0.1 and σ = 0.5 the MSE values between rdge regresson method are about the same so that the small dfferences between them can be neglected. But when the value of standard devaton s at least one, we can observe that the MSE values ncreases as the standard devaton ncreases, regardless the methods. Moreover, we can see that the OLS estmator has the hghest MSE comare to rdge estmators and wthn the regresson estmaton methods, the HK estmator erforms better than the HSL and NHSL estmators for all levels of correlatons. Scence Publcatons 4. CONCLUSION In ths artcle, we dd smulaton studes of several methods for estmatng the rdge arameters. The erformance of each rdge estmator deend on the standard devaton (σ and the correlatons between of exlanator varables (γ. For σ = 0.1, HK estmator and HSL estmator have smaller MSE than the OLS estmator for all levels of correlatons. However, the OLS estmator s reasonabl better than NHSL estmator for all levels of correlatons for ths gven value of. HK estmator mght be recommended to be used to estmate the rdge arameter. Further nvestgaton of rdge estmators s needed n future n order to mae an defnte statement. 5. ACKNOWLEDGEMENT The reachers are grateful to Unverst Grants Scheme b Unverst Putra Malasa for awardng a research grant for suortng the research REFERENCES Al-Hassan, Y.M., 010. Performance of a new rdge regresson estmator. J. Assoc. Arab Unv. Basc Aled Sc., 9: 3-6. DOI: /j.jaubas Alhams, M.A. and G. Shuur, 007. A monte carlo stud of recent rdge arameters. Commun. Stat., Smulat. Comut., 36: DOI: / Bhar, L. and V.K. Guta, 001. A useful statstc for studng outlers n exermental desgns. Ind. J. Stat., 63: Dorugade, A.V. and D.N. Kashd, 010. Alternatve method for choosng rdge arameter for regresson. Aled Mathem. Sc., 4: Ftranto, A. and H. Md, 013. A comarson between classcal and robust method n a factoral desgn n the resence of outler. J. Math. Stat., 9: DOI: /jmss Gou, J., 006. Factoral exermental desgn: Detectng an outler wth the dnamc varable and the Danel s dagram. Chemomet. Intell. Laborator Sst., 80: DOI: /j.chemolab Hocng, R.R., F.M. Seed and M.J. Lnn, A class of based estmators n lnear regresson. Technometrcs, 18: DOI: / Hoerl, A.E. and R.W. Kennard, Rdge regresson: Based estmaton for nonorthogonal roblems. Technometrcs, 1: DOI: / Judge, G.G., R.C. Hll, W.E. Grffths, H. Luteohl and T.C. Lee, Introducton to the Theor and Practce of Econometrcs. nd Edn., John Wll and Sons, New Yor, ISBN-10: , : McDonald, G.C. and D.I. Galarneau, A Monte Carlo evaluaton of some rdge-te estmators. J. Am. Statst. Assoc., 70: DOI: / Orlov, M.L., Multle Lnear Regresson Analss Usng Mcrosoft Excel. 1st Edn., Oregon State Unverst. Wethrll, H.H., Evaluaton of ordnar rdge regresson. Bull. Mathem. Statst., 18: Zhou, J. and H. Zhu, 003. Robust estmaton and desgn rocedures for the random effects model. Canadan J. Stat., 31: DOI: / Bagher, A. and H. Md, 009. Robust estmatons as a remed for multcollneart caused b multle hgh leverage onts. J. Math. Statst., 5: DOI: /jmss