The Identification of Good and Bad High Leverage Points in Multiple Linear Regression Model

Size: px

Start display at page:

Download "The Identification of Good and Bad High Leverage Points in Multiple Linear Regression Model"

Letitia Hampton
6 years ago
Views:

1 Mathematcal Methods and Systems n Scence and Engneerng he Identfcaton of Good and Bad Hgh Leverage Ponts n Multple Lnear egresson Model HABSHAH MIDI 1 and MOHAMMED A. MOHAMMED 2 Faculty of Scence and Insttute For Mathematcal esearch Unversty Putra Malaysa UPM Serdang Selangor, Malaysa Malaysa habshahmd@gmal.com 1 ; mohammedam23@yahoo.com 2 Abstract: - Much ork has been carred out on the detecton of hgh leverage ponts thout payng much attenton to classfyng them nto good and bad leverage ponts. It s crucal to detect bad leverage pont as t has unduly effect on the parameter estmates. In ths paper, e propose a ne technque to dentfy good and bad leverage ponts. We nvestgate the performance of our proposed method by employng some ell-knon data sets. Key-Words: - DGP, MGDFFIS, Outler, Hgh Leverage Ponts, Studentzed esdual 1 Introducton here are several versons of outlers n regresson problem. Observatons are judged as resdual outlers on the bass of ho unsuccessful the ftted regresson equaton s n accommodatng them. As such any observaton that has large resdual s referred to as resdual outler. Observatons hch are extreme or outlyng n the Y-coordnate, s call Y-outler. Addtonally, hgh leverage ponts are those observatons hch are outlyng n the X varables. It can be classfed nto good and bad leverage ponts. Good leverage ponts and bad leverage ponts are those outlyng observatons n the explanatory varables that follo and do not follo the pattern of the majorty of the data, respectvely. Bad leverage pont has a larger mpact on the computed values of varous estmates. On the other hand, good leverage pont contrbute to the effcency of an estmate. In ths respect, only bad leverage ponts need to be don eghted and good leverage pont should not be gven lo eght n the computaton of eghtng functon n any robust method. Hoever, t s no evdent that most robust methods attempt to reduce the effect of outlers by don-eghtng the outlers rrespectve of hether they are good or bad leverage ponts. here are a number of good papers n the lterature for the detecton of hgh leverage pont [see [2], [8], [10]. Nonetheless, those dentfcaton methods are mostly focused on the detecton of hgh leverage ponts thout pn- pontng the good and bad leverage ponts. It s very mportant to detect and classfy the good and bad leverage ponts as only the bad leverage ponts are responsble for the msleadng concluson about the fttng of regresson model. It s easy to capture the exstence of several versons of outlers n regresson analyss by usng graphcal method. If only one ndependent varable s beng consdered, the four type of outlers can easly be observed from a scatter plot of y aganst the x varables. For more than one ndependent varables, ousseeu and Van Zomeren (1990 [16] have proposed a robust dagnostc plot or outler map hch s more effcent than the non-robust plot for classfyng observatons nto four type of data ponts specfcally, regular or good observatons, good leverage ponts, bad leverage ponts and vertcal outlers. he dentfcaton of multple vertcal outlers and multple hgh leverage ponts are presented n Chapter 2. In Chapter 3, e proposed an mproved procedure of classfyng ISBN:

2 Mathematcal Methods and Systems n Scence and Engneerng observatons nto the four categores. o numercal examples are presented n Chapter 4 to sho the mert of our proposed method. 2 Identfcaton of multple y-outlers and multple hgh leverage ponts Consder a multple lnear regresson model: Y Xβ + ε (1 here Y s an n 1 vector of observaton of dependant varables, X s an n p matrx of ndependent varables, β s p 1 vector of unknon regresson parameters, ε s an n 1 vector of random errors th dentcally normal 2 dstrbuton as ε ~ NID (0, σ, and p s the number of ndependent varables. he lnear regresson model n (1 can be re-rtten as follos; y x β + ε, 1,2,..., n, (2 here y s the observed of dependent varable and x s a p 1 vector of predctors. he OLS estmates n (1 s gven by ˆ ( β X X X Y (3 and the th resduals can be expressed n term of the true dsturbance as: ˆ Y Y (1 W ε (4 here W X ( X X X s knon as hat matrx. he dagonal elements of leverage matrx s called the hat values (see [1],[3],[8],[11] and denoted as, gven by x ( X X x, 1,2,..., n (5 he hat matrx s often used as dagnostcs to dentfy leverage ponts [10]. Unfortunately, the hat matrx suffers from the maskng effect. So, often fal to detect hgh leverage ponts. Had (1992 [8] suggested a sngle case deleted measure so called Potentals. he dagonal elements of Potental denoted as (p s gven by (see[3],[8] p x ( X ( X ( x, 1,2,..., n (6 here X ( s the matrx X th exclude the th ro. We can rerte p as a functon of as p /( 1, 1,2,..., n (7 Unfortunately, all of these dagnostc measures are not successful n dentfyng multple hgh leverage ponts. o rectfy ths problem, Imon (2005 [10] proposed the generalzed potentals (GP denoted as p.. he GP dagnostc method able to detect multple hgh leverage ponts, but t s not adequately effectve n dentfyng the exact number of outlers. hs s due to the choce of the ntal basc subset. Habshah et al. (2009 [14] has developed Dagnostc obust Generalzed Potental (DGP to mprove the rate of detecton of hgh leverage ponts. he DGP conssts of to steps hereby n the frst step, robust method s used to dentfy the suspected hgh leverage ponts. On the second step, the generalzed Potental dagnostc approach s used to confrm our suspcon. Habshah et al. noted that the lo leverage ponts( f any are put back nto the estmaton subset sequentally (observaton th the least p ll be substtuted at frst and to re-compute p values. hs process s contnued untl all member of the deleton set s checked hether or not they can be declared as hgh leverage ponts. he suspect hgh leverage ponts are determned by the robust Mahalanobs dstance (MD, based on the mnmum volume ellpsod (MVE developed by ousseeu [11] as MD ( X ( X here ( X and ( X C C ( X ( X ( X, for 1,2,. n are robust locatons and shape estmates of the MVE, respectvely. Imon [10] suggested a cut-off value for the robust Mahalanobs dstances as Medan (MD + 3MAD (MD (8 Let us denote a set of good cases remanng n the analyss by and a set of bad cases deleted by D. Also suppose that contans (n d cases ISBN:

3 Mathematcal Methods and Systems n Scence and Engneerng after d < (n p cases n D are deleted. Once the remanng set s determned, then the second steps of DGP are carred out to confrm the suspected hgh leverage ponts by usng the generalzed potentals (GP denoted p defned as p here ( ( 1 ( D n hch the threshold, D D ( D for D for X ( X X X. Observatons p values larger than the follong p > Medan( P + c MAD( P here c can be taken as a constant value of 2 or 3, are declared as hgh leverage ponts. he Studentzed resduals (nternally Studentzed resduals and -Student resdual (externally Studentzed resduals are dely used measures for the dentfcaton of outlers (see [5]. he Studentzed resduals s defned as r, 1,2,..., n (9 ˆ σ 1 here ˆ σ [ / n p ] s the resduals mean square. he specal case (8 s: t, 1,2,..., n (10 ˆ σ 1 ( s called the -student, here σ ( s the resduals mean square excludng the th case. hs to measures also not be able to detect multple outlers. Imon [10] suggested a generalzed studentzed resdual (Gt based on a group of deleton to dentfy multple outlers or vertcal outlers. he generalzed verson of regresson dagnostcs frst requres to select deleton group D that contans all suspect nfluental cases. he suspect nfluental cases consder outlers and hgh leverage ponts separately hereby outlers and hgh leverage ponts are ˆ dentfed usng the robust LS resduals (ousseeu& Leroy 1987 [11] and Generalzed Potentals (Imon, 2002 [9], respectvely. he unon of set of suspected outlers and set of suspected hgh leverage ponts become members of the deleton set, hch has d observatons. Nevertheless, the ntal basc subset of Gt s not very stable and suffer from maskng effects. In ths regards, the DGP s employed to rectfy ths problem. Subsequently, the vector of estmated parameters n the remanng groups, denoted as β are defned as ˆ ( ˆ 1 β ( ( X X X Y he th deleton observaton s gven by ˆ ( y x β (, 1,2,..., n, (11 (12 he th externally Studentzed resdual t for the remanng groups, s gven by ˆ y x β ( t (13 ˆ σ 1 ( he dagonal element of the hat matrx s gven by ( x ( X ( X ( x, 1,2,..., n (14 By utlzng the results of ao (1965, an addtonal pont n set s defned as ˆ β ( + ( + x ( X ( X X X + x x + x x ( X x ( 1+ ( (15 ˆ ( X X β + ( (16 1+ ( hs lead to the formulaton of the externally studentzed resdual for defned as Y + x y ˆ ˆ y x β ( + ε ( t (17 ˆ σ 1 ( + ˆ σ 1+ ( Subsequently, the Modfed Generalzed Studentzed resduals (MGt for the hole data set can be developed by combnng Equaton (13 and Equaton (17 as follos; ISBN:

4 Mathematcal Methods and Systems n Scence and Engneerng MGt ˆ σ ˆ σ ( 1 ( 1+ ( (, for, for (18 3 Ne Dagnostc Plot For Classfyng Observaton nto four categores ousseeu and Van Zomeren (1990 [16] proposed a robust dagnostc plot hch s more effectve than the non-robust plot for classfyng observaton nto regular observaton, vertcal outlers, good and bad hgh leverage ponts. he proposed plot, plots the standardzed LMS resdual aganst the robust dstant based on MVE. he non-robust plot, plots the standardzed OLS resdual aganst the Mahalanobs dstant (MD. We suspect that the robust dagnostc plot s not very effectve n classfyng the observatons nto respectve categores snce t s based on robust mahalanobs dstant hch suffers from maskng and sampng effects. As such e proposed to mprove the classfcaton method by plottng the MGt versus DGP by the follong rule: - egular Observaton (O: Any observaton s declared as O f MGt 2. 5 and p < Medan( P + c MAD( P - Vertcal Outlers (VO: Any observaton s declared as VO f MGt > 2. 5 and p < Medan( P + c MAD( P - Good Leverage Pont (GLP: Any observaton s declared GLP f MGt 2. 5 and p Medan( P + c MAD( P v- Bad Leverage Pont (BLP: Any observaton s declared BLP f MGt > 2. 5 and p Medan( P + c MAD( P 4 Example and dscusson 4.1 Stackloss Data Our frst example s the ell-knon Stack loss dataset hch s taken from Bronlee (1965 [13], that contans 21 observatons th three ndependent varables (Ar flo, Coolng ater and Acd concentraton. Many researchers ponted out ths dataset has one vertcal outler ( Case 4 and four hgh leverage ponts (Cases 1, 2, 3, 21. It can be seen from able 1 that all exstng methods fal to dentfy those observatons correctly. he DGP can only detects four hgh leverage ponts correctly but not be able to dentfy Case 4 as vertcal outler. he MGt successfully detects the fve observatons as outlers but t does not specfy to hch category are those outlers belong to. Next, e ould lke to see the classfcaton made by OLS-MD, LMS-MD and MGt- DGP plots. Due to space constrant, the plots are not dsplayed. he OLS-MD plot fals to locate any outler. Both LMS-MD and MGt- (19 DGP can classfy the observatons nto ther respectve categores; here cases (1,2,3,21 and case 4 are classfed as hgh leverage pont and vertcal outler, respectvely. Next, e ould lke to observe the effect of removng those outlers (vertcal and bad leverage ponts from the data set. he results of able 2 sho that those fve outlers have a ISBN:

5 Mathematcal Methods and Systems n Scence and Engneerng very sgnfcant effect on the parameter estmates. emovng them from the data set has reduced the standard errors of all estmates able1: MD, MD, DGP, t, t and MDt for Stackloss data Case No. MD (3.05 MD (3.05 DGP (0.73 t (2.5 t (2.5 MGt ( able 2: he egresson estmates for Stackloss Data 4.2 Arcraft Data Our second example s the Arcraft dataset hch s taken from Gray (1985 [7]. hs dataset has four predctor varables (aspect rato, lft to drag rato, eght of the plane and maxmal thrust and the response varable s cost and contan 23 cases. able 3 sho that t fals to dentfy any outler. he DGP, MGt detect 3 outlers but one of the detected outler s not the same. It can be observed From Fgures 1 that the non-robust plot cannot dentfy any outler but only detects cases 22 and 14 as good leverage ponts. he LMS- MD plot detects 1 observaton (case 22 as bad hgh leverage pont and 3 good leverage ponts (cases 15, 20, 14, hle the MGt-DGP dentfes a bad leverage ponts (Cases 19 and 22, one vertcal outler (case 16 and one good leverage pont (case 21. Next, e ould lke to justfy hch plot has dentfed or has classfed the observatons correctly. he correct plot s the one that hen deletng the dentfed bad leverage ponts and vertcal outlers causes sgnfcant changes to the parameter estmates, and able to reduce more standard errors of the estmates than the other plots. It can be observed from able 4 that the standard errors of the estmates hen removng observatons 16, 22 and 19 (dentfed by MGt- DGP s lesser than hen removng only one observaton (dentfed by LMS-MD able3: MD, MD, DGP, t, t and MDt for Arcraft data Case No. MD (3.34 MD (3.34 DGP (0.36 t (2.5 t (2.5 MGt ( able 4: he egresson estmates for Arcraft Data ISBN:

6 Mathematcal Methods and Systems n Scence and Engneerng Fgure 3: he Modfed Generalzed std. resdual aganst DGP for Arcraft Data Fgure 1: he OLS std. resdual aganst MD for Arcraft Data 5 Concluson In ths paper e proposed a ne method for the dentfcaton of bad leverage ponts by means of dagnostc plot. he classcal dagnostc plot fals to dentfy the bad leverage ponts. he robust LMS-MD plot also not very successful n classfyng observatons nto four categores. In ths regards e propose MGt- DGP hch s very successful n classfyng observatons nto regular observaton, vertcal outlers, good and bad leverage ponts. Fgure 2: he std. LMS resdual aganst MD for Arcraft Data eferences: [1] Atknson, A. C. Fast very robust methods for the detecton of multple outlers, Journal of the Amercan Statstcal Assocaton, Vol. 89, 1994, pp [2] Belsley DA, Kuh E, Welsch E, egresson dagnostcs, Identfyng nfluental data and sources of collnearty. J Wley, Ne York, [3] Chatterjee, S. & Had, A. S. Senstvty Analyss n Lnear egresson, J Wley, Ne York, [4] Cook,. D. Detecton of nfluental observatons n lnear regresson, echnometrcs, Vol. 19, 1977, pp ISBN:

7 Mathematcal Methods and Systems n Scence and Engneerng [5] Cook,. D. & Wesberg, S. esduals and Influence n egresson, Chapman & Hall, Ne York, [6] Ellenberg, J. H. estng for a sngle outler from a general regresson, Bometrcs, Vol. 32, 1976, pp [7] Gray, G. B., Graphcs for egresson Dagnostcs, In Amercan Statstcal Assocaton Proceedngs of the Statstcal Computng Secton, Washngton, D. C, ASA, 1985, pp [8] Had, A. S. A ne measure of overall potental nfluence n lnear regresson, Computatonal Statstcs and Data Analyss, Vol. 14, 1992, pp [9] Imon, A. H. M. Identfyng multple hgh leverage ponts n lnear regresson, Journal of Statstcal Studes, Specal Volume n Honour of Professor Mr Masoom Al, 2002, pp [10] Imon, A. H. M., Identfyng multple nfluental observatons n lnear regresson, Journal of Appled Statstc. Vol. 32, 2005, pp [11] ousseeu, P. J. & Leroy, A. obust egresson and Outler Detecton, J Wley, Ne York, [12] Vnoth B. and ajarathnam, Outlers Detecton n Smple lnear egresson Models and obust A case study on Wheat Produacton Data, Journal of Statstcs, Vol. 3, 2014, pp [13] Bronlee, K. A., Statstcal heory and Methodology n Scence and Engneerng, J Wley, Ne York, [14] Habshah, M, Norazan M.. and Imon A.H.M., he performance of Dagnostc- obust Generalzed Potentals for the dentfcaton of multple hgh leverage ponts n lnear regresson, Journal of Appled Statstc. Vol. 5, 2009, pp [15] Narula, S. C., Saldva, P. N., Andre, C. D. S., Elan, S. N. A. F. and Capelozz, V., he mnmum sum of absolute errors regresson: a robust alternatve to the least squares regresson. Statstcs n Medcne. Vol. 18, 1999, pp [16] ousseeu, P., and van Zomeren, B., Unmaskng Multvarate Outlers and Leverage Ponts, Journal of the Amercan Statstcal Assocaton, Vol. 85, 1990, pp ISBN:

Abstract The R chart is often used to monitor for changes in the process variability. However, the standard

Abstract The R chart is often used to monitor for changes in the process variability. However, the standard An Alternatve to the Stanar Chart chael B.C. Khoo an H.C. Lo School of athematcal Scences, Unverst Sans alaysa, 800 nen, Penang, alaysa Emal: mkbc@usm.my & hclo@cs.usm.my Abstract The chart s often use