Regression Review and Robust Regression. Slides prepared by Elizabeth Newton (MIT)

Transcription

1 Regression Review and Robust Regression Slides prepared by Elizabeth Newton (MIT)

2 S-Plus Oil City Data Frame Monthly Excess Returns of Oil City Petroleum, Inc. Stocks and the Market SUMMARY: The oilcity data frame has 29 rows and 2 columns. The sample runs from April 979 to December 989. This data frame contains the following columns: VALUE: Oil monthly excess returns of Oil City Petroleum, Inc. stocks. Market monthly excess returns of the market. E Newton 2 This output was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

3 Oil City Data (continued) Returns relative change in the stock price over a one month interval Excess returns are computed relative to the monthly return of a 90-day US Treasury bill at the risk-free rate Financial economists use least squares to fit a straight line predicting a particular stock return from the market return. Beta estimated coefficient of the market return. Measures the riskiness of the stock in terms of standard deviation and expected returns. Large beta -> stock is risky compared to market, but also expected returns from the stock are large. E Newton 3

4 Plot of Market returns vs. month oilcity$market Month E Newton 4 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

5 Plot of Oil City Petroleum return vs. month Oil month E Newton 5 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

6 Histogram of Market Returns Market E Newton 6 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

7 Histogram of Oil City Returns Oil E Newton 7 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

8 Plot of Oil City vs. Market Returns 94 Oil City Market E Newton 8 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

9 Plot of Oil City vs. Market Returns without observation 94 Oil City Market E Newton 9 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

10 > summary(oilcity) Oil Market Min.: Min.: st Qu.: st Qu.: Median: Median: Mean: Mean: rd Qu.: rd Qu.: Max.: Max.: E Newton 0 This code was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

11 Summary oil.lm Call: lm(formula Oil ~ Market, data oilcity) Residuals: Min Q Median 3Q Max Coefficients: Value Std. Error t value Pr(> t ) (Intercept) Market Residual standard error: on 27 degrees of freedom Multiple R-Squared: 0.07 F-statistic: 5.24 on and 27 degrees of freedom, the p-value is Correlation of Coefficients: (Intercept) Market E Newton This code was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

12 Plot of residual vs. fit for oil.lm Residuals Fitted : Market E Newton 2 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

13 Plot of Cooks Distance vs. Index 94 Cook's Distance E Newton 3 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

14 Plot of hat matrix diagonals for oil.lm hat(model.matrix(oil.lm)) month E Newton 4 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

15 Summary of model without observation 94 Call: lm(formula Oil ~ Market, data oilcity94) Residuals: Min Q Median 3Q Max Coefficients: Value Std. Error t value Pr(> t ) (Intercept) Market Residual standard error: on 26 degrees of freedom Multiple R-Squared: F-statistic: 3. on and 26 degrees of freedom, the p-value is Correlation of Coefficients: (Intercept) Market E Newton 5 This code was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

16 Plot of residual vs fit for model without observation 94 Residuals Fitted : Market E Newton 6 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

17 Weighted Least Squares Used when observations, y, have unequal y Xβ + 2 E( ) 0, Var ( ) σ V V is non - singular positive definite V is diagonal if errors are uncorrelated, V is always symmetric nxn non - singular symmetric matrix,r such that R'R RR V R is sometimes called the square root of V i variances E Newton 7

18 Weighted least squares (continued) 0 ) ( ) ( y or, becomes, X, y Define new variables : β β β R E E X R X R y R X y R X R y R E Newton 8

19 Weighted least squares (continued) I RRR R VR R R E R R R E E E E E Var ) ' ( ) ' ( ) ' ( } )]' ( )][ ( {[ ) ( σ σ σ E Newton 9

20 Weighted Least Squares (continued) Q( β ) ' V ( y Var ( ˆ) β 2 σ (X' WX) 2 σ ( X' WX ) Xβ )' W ( y - Least squares normal equations are (X' WX) ˆ β The solution is : ˆ β (X' WX) - Xβ ) X' WW W, (X' WX) X' W - W V X' Wy WX( X' WX ) - var( y) WX( XWX ) weights X' Wy E Newton 20

21 Robust Regression Used to reduce influence of outliers LAR Regression : minimize L n i y i x β i n i e i LMS Regression : minimize : median{[y i x β ] i 2 } median{e 2 i } M estimators : minimize : n i g(y i x β ) i n i g(e ), i g a function of residuals E Newton 2

22 Robust Regression (continued) IRLS, iteratively reweighted least squares Minimize e We W is a diagonal matrix of weights, inversely proportional to magnitude of scaled residuals, u i u i e i /s, smadmedian{ e i -median(e i ) } Procedure:. Obtain initial coefficient estimates from OLS 2. Obtain weights from scaled residuals 3. Obtain coefficient estimates from WLS 4. Return to 2. Convergence usually rapid. E Newton 22

23 (See Figure 0.4, and Equations 0.44 and 0.45 in Neter et al. Applied Linear Statistical Models.) Neter et al. Applied Linear Statistical Models 23

24 Plot of residuals in oil.rreg oil.rreg$resid E Newton 24 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

25 Plot of weights in robust regression for oil city data set Weights Month E Newton 25 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

26 Plot of sqrt(weights)resid/s in oil.rreg (sqrt(oil.rreg$w) oil.rr E Newton 26 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

27 Coefficient table for oil.rreg > x<-cbind(,market) > beta<-solve(t(x)%%diag(w)%%x)%%t(x)%%diag(w)%%oil > r<-oil-x%%beta > s<- median(abs(r-median(r))).4826 > covm<-solve(t(x)%%diag(w)%%x)s^2 > se<-sqrt(diag(covm)) > tvaluebeta/se > prob<-2(-pt(abs(tvalue),27)) > cbind(beta,se,tvalue,prob) beta se tvalue prob (Intercept) x Covariance matrix is approximate. E Newton 27 This code was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

28 Plots of fitted regression lines for oil city data 94 Oil oil.lm oil.lm94 oil.rreg Market E Newton 28 This graph was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.

29 Least Trimmed Squares Regression Minimizes where q is : q i e 2 i chosen, to be between n/2 and n Based on a genetic algorithm for finding a subset of data with minimum SSE. High breakdown point: fits the bulk of the data well, even if bulk is only a little more than half the data. Resulting weights are or 0 E Newton 29

30 > summary(oil.lts) Method: [] "Least Trimmed Squares Robust Regression." Call: ltsreg(formula Oil ~ Market) Coefficients: Intercept Market Scale estimate of residuals: Robust Multiple R-Squared: Total number of observations: 29 Number of observations that determine the LTS estimate: 6 Residuals: Min. st Qu. Median 3rd Qu. Max Weights: E Newton 30 This code was created using S-PLUS(R) Software. S-PLUS(R) is a registered trademark of Insightful Corporation.