Topic 8: Model Diagnostics

Transcription

1 Topic 8: Model Diagnostics

2 Outline Diagnostics to check model assumptions Diagnostics concerning X Diagnostics using the residuals

3 Diagnostics and remedial measures Diagnostics: look at the data to diagnose situations where the assumptions of our model are violated Violations inference cannot be trusted Remedies: changes in analytic strategy to fix / adjust for these problems

4 Look at the data Before trying to describe the relationship between a response variable (Y) and an explanatory variable (X), we should look at the distributions of these variables We should always look at X If Y depends on X, looking at Y alone may not be very informative looking marginally at Y is a common mistake

5 Diagnostics for X If X has many values, use Proc Univariate to get numerical summaries (e.g., mean, median, quartiles) If X has only a few values, use Proc Freq or the Freq option in Proc Univariate to get summaries (e.g., percentages, counts)

6 Diagnostics for X Examine the distribution of X Is it skewed? Are there outliers? Do the values of X depend on time (i.e., the order in which they were collected)? Suggests possible confounding factors

7 What s the concern? Model estimates based on means and sums of squares These numerical summaries are not robust to outliers Can inflate variance or influence trend Confounders may alter the relation between X and Y

8 Important Statistics Mean Standard deviation Skewness Kurtosis Range

9 Example: Toluca lot size data toluca; infile../data/ch01ta01.txt'; input lotsize hours; seq=_n_; proc univariate data=toluca plot; var lotsize; run;

10 Crude Plots Stem Leaf # Boxplot *--+--* Multiply Stem.Leaf by 10**+1

11 Moments Moments N 25 Sum Weights 25 Mean 70 Sum Observations 1750 Std Deviation Variance 825 Skewness Kurtosis Uncorrected SS Corrected SS Coeff Variation Std Error Mean

12 Location and Spread Basic Statistical Measures Location Variability Mean Std Deviation Median Variance Mode Range Interquartile Range

13 Quantiles (Definition 5) Quantile Estimate 100% Max % % % % Q % Median 70 25% Q % 30 5% 30 1% 20 0% Min 20

14 Extreme Observations Lowest Highest Value Obs Value Obs

15 SAS CODE FOR TREND IN ORDER? symbol1 v=circle i=sm70; proc gplot data=a1; plot lotsize*seq; run;

16

17 Normal distributions Regression model does not state that X come from a single Normal distribution X can follow any distribution Regression model does not state that Y come from a single Normal distribution Assume Y X i is Normal In some cases, however, X and/or Y may be Normal and it can be useful to know this

18 Common plots Histograms Bell-shaped? Symmetric? Can often fit Normal curve on top of histogram to assess fit Box plots Normal quantile plots

19 Normal quantile plots Consider n=5 observations iid N(0,1) From Table B.1, we find P(z -.84) =.20 P(-.84 < z -.25) =.20 P(-.25 < z.25) =.20 P(.25 < z.84) =.20 P(z >.84) =.20

20 Normal quantile plots So we expect One observation -.84 One observation in (-.84, -.25) One observation in (-.25,.25) One observation in (25,.84) One observation >.84

21 Normal quantile plots Use similar idea to define expected Normal scores for given sample size n Z i = Φ -1 ((i-.375)/(n+.25)), i=1 to n Plot the order statistics X (i) vs Z i KNNL plots X (i) vs s Z i Doesn t affect nature of the plot

22 Normal quantile plots The standardized X variable is z = (X - μ)/σ So, X = μ + σ z If the data are approximately Normal, the relationship will be approximately linear with slope close to σ and intercept close to μ.

23 SAS CODE proc univariate data=toluca plot; var lotsize; qqplot lotsize; run;

24

25 Diagnostics for residuals Model: Y i = β 0 + β 1 X i + e i Predicted values: Ŷ i = b 0 + b 1 X i Residuals: e i = Y i Ŷ i So, Y i = Ŷ i + e i The e i should be similar to the e i The model assumes e i iid N(0, σ 2 )

26 Plot Plot Plot PLOT PLOT PLOT Plot

27 Questions addressed by diagnostics for residuals Is the relationship linear? Is the variance constant? Are the errors Normal? Are the errors dependent? Are there outliers?

28 Is the Relationship Linear? Plot Y vs X Plot e vs X (residual plot) Residual plot better emphasizes deviations from linear pattern Recall Topic 2 plots of scatterplot and residual plot

29 SAS CODE: Fake #1 libname xxx../data ; Data xxx.a100; do x=1 to 30; y=x*x-10*x+30+25*normal(0); output; end; run; Generates data set where Y=X 2-10X+30 Errors are Normally distributed with s=25

30 SAS CODE proc reg data=xxx.a100; model y=x; output out=a2 r=resid; run;

31 OUTPUT Analysis of Variance Source DF Sum of Squares Mean Square F Value Pr > F Model <.0001 Error Corrected Total Parameter Estimates Variable DF Parameter Estimate Standard Error t Value Pr > t Intercept <.0001 x <.0001 A significant positive relationship!!

32 SAS CODE: Visual Checks symbol1 v=circle i=rl; proc gplot data=a2; plot y*x; run; symbol1 v=circle i=sm60; proc gplot data=a2; plot y*x; proc gplot data=a2; plot resid*x/ vref=0; run; Scatterplot with regression line Scatterplot with smoothed curve Residual plot

33 Does not appear to be linear

34

35 Nonlinear behavior easier to see here?!

36 Does the variance differ across X? Plot Y vs X Plot e vs X Plot of e vs X will emphasize problems with the variance assumption

37 SAS CODE: Fake #2 libname xxx../data'; Data xxx.a100a; do x=1 to 100; y=30+100*x+10*x*normal(0); output; end; run; Generates data set where Y= X Errors are Normally distributed with s=10x

38 SAS CODE proc reg data=xxx.a100a; model y=x; output out=a2 r=resid; run;

39 OUTPUT Analysis of Variance Source DF Sum of Squares Mean Square F Value Pr > F Model <.0001 Error Corrected Total Parameter Estimates Variable DF Parameter Estimate Standard Error t Value Pr > t Intercept x <.0001 A significant positive relationship!! Estimate close to the true value!!

40 SAS CODE: Visual Checks symbol1 v=circle i=sm60; proc gplot data=a2; plot y*x; Scatterplot with smoothed curve proc gplot data=a2; plot resid*x / vref=0; run; Residual plot

41

42

43 So what?! Why is non-constant variance an issue here? Trend appears linear Estimates close to the truth Answer:

44 Are the errors Normal? The real question is whether the distribution of the errors is far enough away from Normal to invalidate our confidence intervals and significance tests Look at the residuals distribution Use a Normal quantile plot Be wary of tests of Normality

45 SAS CODE data a1; infile..\data\ch01ta01.txt'; input lotsize hours; proc reg data=a1; model hours=lotsize; output out=a2 r=resid; proc univariate data=a2 plot normal; var resid; histogram resid / normal kernel; qqplot resid;

46

47

48 Univariate Output Fitted Normal Distribution for resid Parameters for Normal Distribution Parameter Symbol Estimate Mean Mu 0 Std Dev Sigma Goodness-of-Fit Tests for Normal Distribution Test ----Statistic p Value Kolmogorov-Smirnov D Pr > D >0.150 Cramer-von Mises W-Sq Pr > W-Sq >0.250 Anderson-Darling A-Sq Pr > A-Sq >0.250 No obvious deviations from normality as P-values are greater than 0.05

49 Dependent Errors Usually we see this in a plot of residuals vs time order (KNNL) or seq (our SAS variable) We can have trends and/or cyclical effects in the residuals Observations not independent If you are interested read KNNL pgs

50 Are there outliers? Plot Y vs X Plot e vs X Plot of e vs X should emphasize an outlier

51 SAS CODE: Fake #3 Data xxx.a100b1; do x=1 to 100 by 5; y=30+50*x+200*normal(0); output; end; x=50; y=30+50* ; d='out'; output; run; Generates data set where Y=30+50X Errors are Normally distributed with s=200

52 SAS CODE proc reg data=xxx.a100b1; model y=x; where d ne 'out'; run; proc reg data=xxx.a100b1; model y=x; output out=a2 r=resid; run;

53 Without Outlier Analysis of Variance Source DF Sum of Squares Mean Square F Value Pr > F Model <.0001 Error Corrected Total Parameter Estimates Variable DF Parameter Estimate Standard Error t Value Pr > t Intercept x <.0001 s=217.8

54 With Outlier Analysis of Variance Source DF Sum of Squares Mean Square F Value Pr > F Model Error Corrected Total Parameter Estimates Variable DF Parameter Estimate Standard Error t Value Pr > t Intercept x s=2250.2

55 SAS CODE: Visual Checks symbol1 v=circle i=rl; proc gplot data=a2; plot y*x; proc gplot data=a2; plot resid*x/ vref=0; run;

56

57

58 Different kinds of outliers The outlier in the last example influenced the intercept but not the slope It inflated all of our standard errors Here is an example of an outlier that influences the slope

59 SAS CODE Data xxx.a100c1; do x=1 to 100 by 5; y=30+50*x+200*normal(0); output; end; x=100; y=30+50* ; d='out'; output; run;

60 SAS CODE proc reg data=xxx.a100c1; model y=x; where d ne 'out'; run; proc reg data=xxx.a100c1; model y=x; output out=a2 r=resid; run;

61 Without Outlier Analysis of Variance Source DF Sum of Squares Mean Square F Value Pr > F Model <.0001 Error Corrected Total Parameter Estimates Variable DF Parameter Estimate Standard Error t Value Pr > t Intercept x <.0001

62 With Outlier Analysis of Variance Source DF Sum of Squares Mean Square F Value Pr > F Model Error Corrected Total Parameter Estimates Variable DF Parameter Estimate Standard Error t Value Pr > t Intercept x

63 SAS CODE: Visual Checks symbol1 v=circle i=rl; proc gplot data=a2; plot y*x; proc gplot data=a2; plot resid*x/ vref=0; run;

64

65

66 Background Reading Program topic8.sas has code for the proc univariate diagnostics of X Program residualchecks.sas have the residual analysis The permanent sas data sets are a100.sas7bdat, a100a.sas7bdat, a100b1.sas7bdat, and a100c1.sas7bdat. Read Sections 3.8 and 3.9