Decision 411: Class 6

Transcription

1 Decision 411: Class 6 Fitting regression models to time series data Economic interpretation of coefficients How to model seasonality with regression Log-log (constant elasticity) models Automatic stepwise variable selection What will be on Tuesday s quiz? Quiz will be open-book, open-notes, notes, 1st hour Manual calculation of forecast and confidence limits for mean or RW model Tentative model identification based on exploratory plots Interpretation of output: model comparisons, insignificant variables, diagnostic tests Criteria for choosing the best model 1

2 Y Regression analysis: recap A simple regression model merely fits a straight line to a scatterplot of Y (the dependent variable) versus X (the independent variable) X The correlation between X and Y, together with their means and standard deviations, determines the slope and intercept of the regression line. Multiple regression equation General multiple regression equation for predicting a dependent variable Y from independent variables,, X k : independent variables X 1, X 2,, Yˆ = ˆ β + ˆ β X + ˆ β X ˆ β X t 0 1 1t 2 2t k kt Constant term is the baseline that would be obtained if all X s were zero at the same time (if that is logically possible). More generally it just moves the regression line up or down to hit the center of the Y data. The rest of the prediction equation consists of a weighted sum of the X s. Hence the predicted pattern of Y is a weighted sum of the patterns in the X s. In general, the weights (coefficients) may be positive or negative. 2

3 If Y and X are time series,, the time dimension in the data ought to be taken into account When the forecasts for Y are plotted versus time, they no longer lie on a straight line: they are just a rescaled version of X! X Y FCST This is the essence of a linear model. If there s more than one X variable, the predictions for Y are a sum of rescaled copies of the X s, plus the intercept. (The scaling factors may be positive or negative.) When plotted versus time, the dependent variable to be predicted (Y) might look something like this: utpe ot 1.6 Variables Y T 3

4 An independent variable (X) to be used for predicting Y might itself be another random variable: utpe ot 1.2 Variables X T or it could be a time trend ( slope ) variable: 20 p T A time trend variable actually serves to detrend detrend Y and all the X s prior to estimating the other coefficients of the model, i.e., it corrects for differences in trend among all the variables. 4

5 or X could be a change in trend at a specific point in time: 10 p T or a nonlinear (e.g. quadratic) curve: p T 5

6 or a dummy variable for occasional or periodic events (e.g., seasons of the year): 1.5 p T or a dummy variable for a step-change at a specific point in time: 1.5 p T 6

7 or a lagged value of the dependent variable: Variables Y LAG(Y,1) LAG(Y,2) T or a lagged value of an independent variable: utpe ot Variables X LAG(X,1) LAG(X,2) T 7

8 How to fit regression models In principle, all you need to do is find the right independent variable(s) ) for predicting your dependent variable. In practice, you may also need to transform the variables to improve the linearity of the relationship or the distribution of the errors (e.g., by lagging, logging, deflating, differencing, multiplying or dividing variables, etc. etc.). You may also need to select the appropriate amount of past data to use in model fitting. Model-fitting steps 1. Collect data for dependent & independent variables 2. Identify potentially useful data transformations 3. Fit preliminary models; ideally hold out data for out-of of-sample testing. 4. Screen out insignificant variables & look for other ways to simplify or fine-tune. 5. Check residual diagnostics to test assumptions 6. Compare performance against simpler models (e.g., random walk or other time series models that do not use exogenous X variables). 8

9 What s the bottom line? R-squared is not the bottom line it it may not even be comparable among models fitted with different data samples and/or transformations, and there is no universal standard for good. Residual diagnostics & t-stats are not the bottom line, just red flags that may wave to indicate problems with model assumptions. Error measures (RMSE, MAPE, MAE) are the bottom line when compared in the same units provided that they can be trusted,, i.e., provided that model assumptions appear to be valid. Simplicity & sound logic are also important: can you explain or sell the model to your boss or client? Over- and under-fitting If too few potential regressors are considered, there may be omitted variables whose effects are not captured or which will load onto other variables (proxy effects). If too many potential regressors are considered, there are dangers of over-fitting from too much data mining ( spurious regressors may be found). In either case, the model will predict the future less well than it fitted the past data. You need to exercise judgment to pre-screen potential regressors for relevance. Think before you compute! 9

10 Tools for identifying potential regressors Scatterplot matrices and correlation matrices may help to identify variables related to Y. Scatterplots also indicate whether outliers are present* and whether nonlinear transformations may be useful. Autocorrelation plots show whether lags of Y may be useful as regressors. Crosscorrelation plots show whether lags of X s may be useful as regressors. *Outliers should not be removed from the analysis just because they are outliers, but they should be given special attention: are they due to data entry errors, weird events that will not be repeated, or are they useful natural experiments? Example of causal modeling: 3 years of monthly sales and advertising data for a weight-loss product (unit sales, $ advertising) Time series plot shows somewhat- aligned peaks and valleys Variables Sales Advertising Index but what is the bang for the buck? 10

11 X-Y scatterplot shows a positive, roughly linear relationship, so let s estimate the slope via simple regression Plot of Sales vs Adv 37 32?? Sales Adv Simple regression of sales on advertising 11

12 Simple regression suggests that $1 advertising yields additional units sold, but autocorrelations and residual-vs vs-time plot are bad. Perhaps lagged variables should be considered? 50% confidence limits for predictions were selected via pane options for interval plot Which plots to look at? When fitting time series, the 3 most interesting plots in the Multiple Regression procedure are usually: 1. Residuals vs. predicted values (ideally the patterns are random and errors have the same variance for small or large predictions) 2. Residuals versus row number (ideally the pattern is random and does not show evidence of autocorrelation or heteroscedasticity) 3. Interval plots (data and forecasts plotted versus row number, or versus independent variables, with optional confidence limits superimposed) 12

13 Tools for identifying useful lags: auto- and cross-correlations correlations of sales and advertising Time Series/Descriptive Methods procedure shows that Sales has a significant autocorrelation at lag 1 and the cross- correlation of between sales and advertising is significant at lag 1 as well as lag 0. First try adding lagged advertising to model 13

14 Updated results Much better error stats! This model suggests that the effect of advertising carries over into two periods with a total impact of = unit per $. But residuals have an upward trend, even worse autocorrelation. Add lagged sales next? After adding lagged Sales, autocorrelations now look much better, and error stats are further improved. The economic interpretation of this model is a bit trickier. The sum of advertising coefficients ients is only 0.227, but this is amplified by the autoregressive sales factor. The total impact is 0.227( )

15 Where did that formula come from??!! Adding lag(sales,1) to the model implies that increases in Sales have momentum of their own. Its coefficient of evidently means that boosting sales by 1 unit this period yields a further boost of in the next period (independent of this period s or next period s advertising!), which in turn boosts sales 2 periods ahead by , etc. Hence the direct impact of an advertising $, which is estimated to be over two periods, gets amplified by a factor of = 1/( ) = 1.57 by the geometric series formula. Hence the direct effect plus the momentum effect is a factor of x units of sales per $ adv. What s the real bottom line? Different regression models may yield different estimates of the economic impact of decisions, based on different underlying assumptions about cause and effect e.g., e.g., lagged response, long-term momentum, etc. The last two models suggest that the economic impact of advertising lies somewhere between 0.30 and 0.36 unit per $. Note: rather than adding lagged sales to the model, we could have added one or two more lags of advertising as another way to capture effects of advertising that extend more than one period into the future. When lag(advertising,2) is added rather than lag(sales,1), it turns out to be marginally significant (t=1.6), and the sum of advertising coefficients rises to 0.35, about the same as above. 15

16 The same models can be fitted in the Forecasting procedure: Mean model + Regression variables Note: to get exactly the same results for these lagged-variable models as in the Multiple Regression procedure, you must de-select the first row (by using Index >1 in the Select field on the input panel), otherwise Statgraphics will try to backforecast the missing lagged values in row 1, This yields head-to to-head model comparisons: A random walk model has been included as a reference point note that it is as good as the original simple regression. Also, the simple regression results are slightly different from the original ones, because the first data row was excluded from all models for these comparisons. 16

17 When using the Forecasting procedure to fit and compare regression models, you cannot request forecasts for future time periods unless future data is available for the regressors. This is an inherent limitation of regression forecasting models, in comparison to purely extrapolative time series models. Dummy variables A dummy variable is an independent variable whose values are all 1 s and 0 s, indicating the presence or absence of some condition. The estimated coefficient of the dummy variable is a constant to be added to the forecast when the condition is present Dummy variables are often used to model seasonal effects as well as the effects of unusual events or structural changes 17

18 Modeling seasonality with regression Suppose Y is a seasonal time series that is correlated with some other X variables Examples: advertising, promotions, prices, competition, interest rates, economic indicators One modeling approach would be to seasonally adjust Y before fitting a regression model. However... Potential problems If the seasonal indices are estimated from the data without controlling for effects of other nonseasonal variables, the seasonal indices could be in error and/or data could be overfitted. If the seasonal indices are estimated from other (e.g., more highly aggregated data), then seasonal adjustment of the data may distort the effects of the other variables if those effects are really constant over time. 18

19 Possible solutions Use seasonal dummy variables as additional regressors to estimate the seasonal effects while controlling for other variables. Use an externally-supplied seasonal index as an additional regressor. Use seasonal lags and/or seasonal differences,, i.e., base this period s forecast on what happened one year ago, not one period ago. Additive or multiplicative? Regression models are inherently additive models, therefore coefficients of seasonal dummy variables represent an additive seasonal pattern. If the seasonal pattern is multiplicative,, a log transformation may be helpful: then the coefficients of the seasonal dummies can be converted to equivalent multiplicative seasonal factors. 19

20 Gap revisited For purposes of illustration, we ll look at trend-line and curve-fitting models combined with seasonal dummy variables. We ll also see how a dummy variable can be used to model a trend change ( kink ) in a series. The variable NETSALES consists of quarterly net sales in $1000 s. (X 1.E5) 53 Time Series Plot for NETSALES 43 NETSALES Q3/97 Q3/99 Q3/01 Q3/03 Q3/05 Q3/07 20

21 Gap data file: the QUARTER variable will be used to construct seasonal s dummy variables on-the the-fly: the expression QUARTER=1 is a variable with a 1 in every 1 st quarter and 0 s elsewhere, i.e., a dummy variable for the 1 st quarter. An estimated regression coefficient for QUARTER=1 will be a constant to be added to the forecast in every 1st quarter. Similarly, QUARTER=2 and QUARTER=3 will be the dummies for the 2 nd and 3 rd quarters. (We won t use a 4 th quarter dummy the coefficients of the others will measure changes relative to the 4 th quarter.) To get regression forecasts for future time periods: The QUARTER variable has been extended 12 quarters into the future, because it will be used to construct dummy variables in the regressions, essions, and regression models can t generate forecasts for the future unless future values for the all the independent variables are available. 21

22 Linear trend + seasonal dummies Obviously not a good fit does not respond to the general economic downturn in and the flattening out of sales growth afterward, so residual time series and autocorrelation plots are very bad. The e coefficient of QUARTER=1 is -1,202,000 (in $1000 s), hence 1 st quarter sales are predicted to be $1,202M less than 4 th quarter sales, other things equal. For later reference, note that RMSE=$383M and MAPE=10% for this model. 22

23 (X 1.E6) 8 Time Sequence Plot for NETSALES Linear trend = E E4 t + 3 regressors 6 NETSALES Q2/97 Q2/00 Q2/03 Q2/06 Q2/09 Q2/12 The straight-line trend of this model is clearly inappropriate. The coefficient of t is the estimated long-term trend: $56M per quarter. (X 1.E5) 9 Residual Plot for adjusted NETSALES Linear trend = E E4 t + 3 regressors 5 Residual Q2/97 Q2/00 Q2/03 Q2/06 Q2/09 The residual plot shows shows a kink (change in trend) at Q4/00 23

24 Let s create a new variable called TRENDCHANGE whose value is zero up to row 15 and which ramps up (1, 2, 3, ) afterward. The estimated coefficient of this variable will represent a change in the trend beginning at row 16. Here are the results of adding TRENDCHANGE to the regression. The T fit is dramatically improved: RMSE has dropped from $383M to $185M and MAPE has dropped from 10% to 4.3% for this model. Residual time series plot and autocorrelation plot are still not great because the dip in still has not captured accurately. 24

25 (X 1.E6) 6 Time Sequence Plot for NETSALES Linear trend = -2.49E E5 t + 4 regressors 5 NETSALES Q2/97 Q2/00 Q2/03 Q2/06 Q2/09 Q2/12 The fit to the flatter recent trend has been much improved. There is still a slight upward trend in the long-range forecasts. The coefficient of t, which is now $143M per quarter, is the estimated ed trend up to Q4/00, at which point the TRENDCHANGE variable kicks in. Note the much tighter confidence intervals due to the reduction in RMSE. The coefficient of TRENDCHANGE is -$122M per quarter, so the estimated overall trend since Q4/00 is $144M - $122M = $22M per quarter. This is the slope of the long-range forecasts on the forecast plot. 25

26 Time Series Plot for adjusted NETSALES 15.5 adjusted NETSALES The preceding models assumed an additive seasonal pattern, and the coefficients of the quarterly dummies were the additive seasonal indices. Alternatively,we 14 could estimate multiplicative seasonal indices by fitting Q2/97 the Q2/00 same model Q2/03 with a natural Q2/06log transformation. Q2/09 Here is a plot of the logged data. The seasonal pattern does look more additive in these terms, and the trend- change at Q4/00 is still evident. This model assumes that sales grew at a constant percentage rate up to Q4/00 and at a different percentage rate afterward. The slope coefficient of is the prior growth rate (5.6% per quarter), and the TRENDCHANGE coefficient of is the change in the growth rate after Q4/00 (minus 5.1%). Hence the estimated growth rate after Q4/00 is 5.6% - 5.1% = 0.5% per quarter. 26

27 The error stats of this model are slight better than those of the unlogged model (MAPE of 4.0% here vs. 4.3% earlier), although the residual-vs vs-time and residual autocorrelation plots don t look quite as good. They are not the bottom line, though. Here they indicate more room for improvement via some kind of fine-tuning. Tournament results: the unlogged and logged regression models with the trend change (Models B and C) compare favorably with Holt s and Winters models, despite the autocorrelation in the residuals. 27

28 Equation of the logged model In a model with a natural log transformation, conversion of the forecasts back to original units and interpreting the coefficients requires unlogging unlogging by applying the EXP function. In this model, the unlogged forecasts have the equation f(t) ) = EXP( t) up to Q4/00 and f(t) ) = EXP( t) afterward, before the seasonal terms are factored in. By EXP ing the coefficients of the dummy variables, we obtain multiplicative seasonal indices relative to Q4=100. Seasonal indices Quarter 1 index: EXP(-0.335) = 71.5% Quarter 2 index: EXP(-0.303) = 73.9% Quarter 3 index: EXP(-0.206) = 81.4% Estimated coefficients of the dummy variables in logged model: Estimated multiplicative indices from seasonal decomposition procedure: Q2 Q3 Q4 Q1 28

29 Comparison with seasonal indices obtained by ratio-to to-moving-average method* Quarter Q1 Q2 Q3 Q4 Seasonal index Rescaled to Q4= From logged regression Almost identical! * using Time Series/Seasonal Decomposition procedure on NETSALES Just for fun, let s try a quadratic trend line with the 3 seasonal dummies. RMSE and MAPE are $191M and 4%, much better than the linear trend model and not much worse than the trend-change model. However, this sort of polynomial curve- fitting is not a recommended method of predicting the future. 29

30 (X 1.E5) 53 Time Sequence Plot for NETSALES Quadratic trend = E E6 t t^2 + 3 regressors 43 NETSALES Q2/97 Q2/00 Q2/03 Q2/06 Q2/09 Q2/12 A 3-year 3 extrapolation of a downward quadratic curve is not very credible! Take-aways aways This analysis has illustrated how regression can be used for the estimation of seasonal patterns via the use of dummy variables, and changes in trend via ramp variables, as well as for curve-fitting. Either additive or multiplicative seasonal indices can be estimated within a regression model, depending on whether or not a log transformation is used. Curve-fitting is usually not a good way to forecast outside the sample! Before using any model to forecast into the future, make sure its fit to the recent past is good, and make sure you believe that its assumptions will continue to hold in the future. 30

31 Another approach: seasonal lags As an alternative to seasonal adjustment or dummy variables, seasonality can be captured by using a seasonal lag,, i.e., the one-year year-prior value of the dependent variable, as a regressor. If s is the number of periods in a season (e.g., s=4 for quarterly data), it is often worth trying lags 1, s, and s+1. In this approach, there is no explicit estimation of seasonal indices. Instead, last year s seasonal pattern is used as a model for predicting the future seasonal pattern. Here is a mean model with lags 1, 4, and 5 of sales used as independent variables. The model type has been set to ARIMA with zeroes in all the input fields (i.e., no differencing or AR/MA factors), which is exactly equivalent to a mean model, except that it allows us to suppress the otherwise automatic backforecasting of missing values of lagged variables. 31

32 Not bad! RMSE=186, MAPE=3.6%, decent-looking residual time series & autocorrelation plots, even with no special treatment for (X 1.E5) 53 Time Sequence Plot for NETSALES ARIMA(0,0,0) with constant + 3 regressors 43 NETSALES Q2/97 Q2/00 Q2/03 Q2/06 Q2/09 This model bases the forecast entirely on what has happened in the last 5 quarters. 32

33 Updated tournament results: model with lagged variables is Model D. How does it work, and why does it make sense? We ll look deeper into models like this when we get to the ARIMA part of the course, but here s the basic logic of this model: The coefficient of lag(netsales,4) is almost exactly equal to 1, the coefficients of lag(netsales,1) and lag(netsales,5) are opposite in sign and roughly equal in magnitude (±0.7),( and the constant is 430. The forecasting equation is therefore: yˆ = y ( y y ) t t 4 t 1 t 5 In words, the predicted increase over the same quarter last year is equal to 430 plus 0.7 times the previous quarter s increase over the same quarter last year, thus it predicts a general upward season-to to-season change,but adapts to recent same-quarter results. 33

34 Effects of pricing and promotions Here the variables are pounds sold of two sizes of bags of chips (XL and XXL), as well as price-per per-bag and pounds-on on-display. (104 weekly observations at a regional supermarket chain) Scatterplot matrix INDEX LBXL LBXXL PRXL PRXXL DISPXL Significant negative price effect for the XL size but is it DISPXXL linear? Also, a positive display effect. (3 outliers at top of display scatterplot correspond to very low prices.) 34

35 Correlation matrix Significant correlations with both price and display Auto- and cross-correlation correlation plots for LBXL: possibly a significant auto-correlation at lag 1 and a cross-correlation correlation with DISPXL at lag 1 as well as lag 0? Not useful The most significant cross- correlation is at lag -1, but this is not useful for predicting LBXL from DISPXL (instead the reverse!) 35

36 Regression with all likely suspects, including lagged variables Lagged variables turn out not to be significant Tools for selecting regressors After a model has been fitted, t-statistics of coefficients (and their p-values) indicate whether some variables can be removed. Rule of thumb: t less than 2 in magnitude (p>( p>.05) suggests variable can be removed. It s not required to remove a marginal variable if it is strongly supported by intuition, but beware of including several marginal variables at once. 36

37 After manually removing the insignificant lagged variables Dsplay variables now have dubious significance when both are included in the model Automatic stepwise selection Stepwise regression (forward and backward) is available as a model option. OK when used to automate what you would have done anyway by hand, but use with care. Backward stepwise automates the process of sequentially removing the least significant variable until only significant variables remain Forward stepwise automates the process of sequentially adding the variable that would be most significant upon being the next one entered 37

38 Automatic stepwise selection This is a right-mouse mouse- button Analysis Option in the Multiple Regression procedure Automatic stepwise selection F-to-enter and F-to-remove determine the t-stats needed for a variable to enter or stay in model. F = t-squared, hence F=4 corresponds to t=2. Resist the urge to lower the threshold in order to find or keep more variables danger of overfitting! 38

39 Automatic methods, continued Automatic model-fitting techniques aren t a substitute for your own good judgment. They are only as good (or bad) as the set of variables you give them to work with. They won t find omitted variables or suggest transformations of variables Dangerous for data snooping in large, uncritically- chosen sets of variables: out-of of-sample validation becomes very crucial At the end of the day, you (not the computer) are responsible for the model! Backwards stepwise eliminates one more variable After removing DISPXL (the least significant variable in the 4-variable model), DISPXXL rises just above the t=2 threshold of significance. 39

40 Residual-vs vs-predicted and residual probability plot are suggestive of a nonlinear relationship and/or non-normal normal errors percentage Normal Probability Plot RESIDUALS Try logging both pounds-sold sold and price INDEX LOG(LBXL) LOG(LBXXL) LOG(PRXL) The pounds- sold vs. price relationship now appears more linear LOG(PRXXL) Note: we can t DISPXL directly log the display variables since they DISPXXL contain zeroes 40

41 Correlations are similar to those obtained earlier Auto- and cross-correlation correlation plots are also similar. However, the time series plot now looks much more normal (less spiky) 41

42 Regression results Coefficients of logged variables can now be interpreted as elasticities: a 1% increase in PRXL yields a 1.8% decrease in LBXL Interestingly, the cross-elasticity of PRXXL with LBXL is almost exactly opposite (+1.88), as if the products are substitutes Residual-versus versus- predicted and residual probability plot now look better! Normal Probability Plot percentage LRESIDUALS 42

43 Not-logged model refitted in the Forecasting procedure Logged model refitted in the Forecasting procedure 43

44 Head-to to-head model comparisons in original units Here model C is obtained by applying an AR(1) autocorrelation correction to model B (a very slight improvement) With 20 points held out for validation Error statistics of all three models are significantly higher in the validation period. Have we overfitted the data, or are the last 20 points exceptional? 44

45 With 50 points held out for validation Now the error statistics are quite similar. Evidently the models are not overfitted. Coefficient estimates for model with 50 points held out Price coefficients are almost the same as before (good!). The display coefficients are also in the same ballpark as before. They are no longer technically significant, but this is because the standard errors are larger when a smaller sample is used to estimate them. 45

46 Class 6 recap Fitting regression models to time series data Economic interpretation of coefficients How to model seasonality with regression Log-log (constant elasticity) models Stepwise variable selection 46