Malliaris Training and Forecasting the S&P 500. DECISION SCIENCES INSTITUTE Training and Forecasting the S&P 500 on an Annual Horizon: 2004 to 2015

Transcription

1 DECISION SCIENCES INSTITUTE Training and Forecasting the S&P 500 on an Annual Horizon: 2004 to 2015 (Full Paper Submission) Mary E. Malliaris Loyola University Chicago ABSTRACT Forecasting the S&P 500 direction has been of interest to investors since its inception. This paper uses several variables that have demonstrated relationships to the S&P 500 and studies their influence over time. We build C5.0 decision tree models for each of nine years and test them on the following year. The models have good explanatory power during the time period on which they are trained, but due to the fluctuation in variable importance year to year, fail to explain the out-of-sample data. Due to economic changes, models may need to be retrained often to remain effective. KEYWORDS: Forecasting, Decision Tree, S&P 500, Data Mining INTRODUCTION The stock market is driven by the anticipation of dividends and capital appreciation. Both dividends and capital appreciation depend upon the performance of the economy. Economic growth, price stability and interest rates contribute to the performance of the stock market. However, economic growth, price stability and interest rates all experience volatility as various economic and non-economic shocks disturb the economy. The challenge is to identify variables driving the economic system, to distinguish between patterns that endure over some time period, and then use those patterns to see if the patterns found remain stable. In other words, when modelling the behavior of the stock market, the success of such modelling depends on selecting variables whose importance is not neutralized by unexpected shocks to the system. The period from 2004 to the beginning of 2015 started with relative calm and mild growth in the S&P 500, moved into and through a crisis, and returned, with the help of quantitative easing, to a period of growth. The dramatic event during this period is the Global Financial Crisis. The prevailing hypothesis of this crisis is that rapid increases in housing prices both in the U.S. and several other countries during the period fueled this crisis. What caused such increases in housing during this period was relatively low interest rates. Some have argued that interest rates were low because of an easy monetary policy in the U.S. while others attribute the low interest rates to excessive saving, primarily from China. Thus in our investigation we consider both 30- year mortgage rates and the rates of 10-year Treasury notes. When the housing bubble burst during and the Great Recession occurred from December 2007 to June 2009, monetary policy considered drastic initiatives beyond reducing the Federal funds rate to essentially zero; these drastic steps included what has since been called quantitative easing. We acknowledge the importance of quantitative easing by considering the weekly asset purchases of the Fed. Xu et al (2012) have studied the relationship between the Fed policy and

2 the 30-year mortgage rates and found that surprises in Fed policy have a major impact on the 30-year rate. Simultaneous and significant interactions have been identified by Serfling and Miljkovic (2001) among the S&P 500, the 10-year treasury note and the money supply. We thus concentrate the focus of our analysis on the behavior of the S&P 500 Index by considering the variables of past S&P 500 movement, the 10-year note, 30-year mortgage rates, and fed assets. Also included are variables derived from these series, including skewness and kurtosis of the S&P 500. Eastman and Lucey (2008) find that skewness matters in returns, as does Bates (2000). Equities and bonds together are often considered when analyzing investments (Bowe, 2007; Underwood, 2009; Connolly, 2005; Fleming, 1998). There are several other variables that are important in explaining the behavior of stock prices but not considered in this paper. For example, Uryasev (2011) has found that market volatility is an important ingredient in constructing a portfolio strategy that will give very good outcomes in bear markets and good outcomes in bull markets. Dolvin and Pyles (2009) focused on momentum and reversal in overall markets and discovered that market returns show a measurable response to extreme movements. DATA Weekly mortgage rates and federal assets were downloaded from the St. Louis FRED database using series WRMORTG and WALCL. Mortgage rates are released weekly on Thursday, and Federal Assets are released on Wednesday. The 10-year notes were downloaded from daily yield curve values, and the daily S&P 500 was sourced from Yahoo finance. For both daily series, the Thursday value was used as the weekly representative since this is the first day of the week in which values from all four base series were available. Table 1 lists each of the base and derived variables used in training the models for this data set. Table 1. Variables used in training the models Variable Meaning Example MortScl Scaled weekly value of 30-year mortgage rates NoteScl Scaled weekly value of 10-year notes FAScl Scaled weekly value of Fed assets SPScl Scaled weekly value of the S&P SPSkew14 14-week S&P skewness SPKurt14 14-week S&P kurtosis SPSkew14Dir Is the current skewness up or down from last week D SPKurt14Dir Is the current kurtosis up or down from last week D PChgFA Percent change in Fed Assets PChgSP Percent change in S&P MortDir Is this week's Mortgage Rate up or down D NoteDir Is this week's Ten Year Note up or down from last week D FedDir Is this week's Fed Asset up or down from last week D SPDir Is this week's S&P up or down from last week U SPDirTp1 Direction of S&P next week (Target variable) U

3 These four base variables were scaled to be between 0 and 1 over the entire time period. Other non-target values were calculated from these four base variables. The derived variables shown in Table 1 include measures for skewness and kurtosis of the S&P over a running 14-week period (about 3 months), whether this skewness or kurtosis measure was up or down compared to the previous measure, the percent change from week to week in the Fed assets and the S&P, and the direction each of the four base variables moved from last week to this week. The target variable is the direction that the S&P 500 moved from the current week to the following week. The graph of Figure 1 shows the scaled values of the four base data series. Mortgage rates and ten-year notes have scaled values that are close for much of the period. However, we see a distinct divergence between these two in late 2007 and early In early 2010, they also move apart, but remain close for the rest of the period. Federal assets indicate very little relationship to any of the other series until the beginning of quantitative easing. When quantitative easing begins, there is a large jump in the amount of Fed assets. After this jump, their scaled value forms a rough top border to the S&P 500 path. During the period of the crash, all series but the Fed assets are traveling in a negative direction. Figure 1. Scaled Values of the Four Series MortScl NoteScl FAScl SPScl The values of 3-month skewness and kurtosis values are shown in Figure 2, along with the path of the scaled S&P 500 index. The largest spike in kurtosis comes during the crash, but numerous small spikes also occur over this nine-year period. There are more frequent occurrences of spikes in kurtosis over a value of 2 after the market crash of 2008.

4 Figure 2. Skewness and Kurtosis variables vs the S&P 500 MODELS AND RESULTS For each of the eleven years from 2004 through 2014, a C5.0 decision tree model was constructed using the direction of the S&P 500 for the following week as the target variable. A decision tree model begins with a target variable and all rows of data in one group called the root node. At the first step, each of the input variables is checked to see which one of them, if used to divide the data, would give the best split, best being defined as most pure on the values of the target. This variable is used to form the first division of the data into two parts, or nodes. Each node is inspected and if it has only one value of the target variable, the process stops for that node. If it has more than one value, then all variables are checked to see which one would give the best split of the data, in its current mix. That variable is then used to form another split of that set of data. This process continues until one of two events occurs: either a set of data has only one value of the target, or there is no variable that could give a better split of the data in that node. Of particular interest in this study are three items, the accuracy or fit of the model to the data, the variables identified by the model as being most important in the decision tree, and the accuracy of the trained model on the following year. Each C5.0 model was run in IBM s SPSS Modeler 15.0 data mining package. The variable importance feature in Modeler ranks the variables in relative importance to their impact on the target. All variables used by the model sum in importance to 1. The larger the value, the more important the variable is to the final value the target assumes, and typically, the higher up on the tree it appears. The values taken on by each of these variables used by the models, are summarized in Table 2. When a cell is blank, the variable was considered unimportant by that specific model. The median number of variables used in each model was 6, with 2 variables being the fewest and 9 the most. We see no variable was used every year and each variable was used at least once. One variable, MortDir, appeared with non-zero values in eight of the eleven years. The

5 direction of movement of the S&P skewness was used in seven of the models. The least used variable was the percent change in the S&P 500 which appeared only in the 2010 model. Another way to look at the impact of the variables is to sum the values of the base and derived variables for each of the base variables. This can give us an overview of the overall importance of each variable to the way the model structures its forecasts. These are shown in Table 3. Table 2. Relative Variable Importance over the years. Inputs MortScl NoteScl FAScl SPScl SPSkew SPKurt SPSkew14Dir SPKurt14Dir PChgFA PChgSP.16 MortDir NoteDir FedDir SPDir In four of the eleven years, the S&P was moved most by variables outside the S&P itself. But, in only four of the years was the impact of the S&P variables greater than 50%. There is little consistency over the periods. But we also need to ask how well these models did within each of the periods on which they were trained. In no period is the effect of all four base variables the same. Rather, their influence fluctuates over the years. Table 3. Sum of base and derived variable impact. Total Impact Mortgage Tot Note Tot Fed Assets Tot S&P 500 Tot Table 4 shows the counts of the actual direction of the S&P the following week versus the direction predicted by the C5.0 model for both the training and validation sets. These are separated into Up and Down predictions. With the values where the actual direction of movement and the forecasted direction matched shown in bold type.

6 Table 4. Count of correct and incorrect forecasts, correct forecasts in bold. Actual Tp1 Actual Tp1 Training Set Direction Validation Set Direction Forecasts for Year Down Up Forecasts for Year Down Up 2004 Down Down Up 1 20 Up Down Down Up 4 23 Up Down Down 7 17 Up 1 29 Up Down Down Up 3 26 Up Down Down 12 8 Up 3 26 Up Down Down 3 15 Up 4 29 Up Down Down Up 9 30 Up Down Down 20 1 Up 1 25 Up Down Down 10 6 Up 3 28 Up Down Down 11 9 Up 5 33 Up Down Down 1 4 Up 6 27 Up 1 7 The model trained on one year was used to predict the values for the following year. For example, the 2004 trained model was used to predict the 2005 values. We notice that in the majority of the models, forecasting the Down direction correctly in the validation set is more challenging year to year than forecasting Up. This information is summarized as overall percent accuracy in Table 5. For each year, the percent of correct overall model directions over the training sets ranged from a high of 98% in 2006 to a low of 78% in Seven of the eleven training years were above 90% accuracy in their model estimates. So we can see that the model in each year does a good job explaining that data even though the selection of variables varies greatly from one year to the next. The year 2010 was one in which the model did not use any of the scaled values of the base variables in constructing its forecast. In addition, it is the only year in which the percent change of the S&P appears as an important variable. It also used only four variables in the model forecast. In spite of these characteristics, it was one of the models that did the best job in generalizing, thus enabling it to do rather well on the forecast for the following year where it was correct 59% of the time.

7 Overall accuracy drops when the models are applied to the following year. We see that the validation set accuracy ranged from a low of 37% using the 2009 trained model on the 2010 data to a high of 59% when using the 2010 model with the 2011 data does show a higher accuracy, but the data only goes through mid-april. If we calculate the correlation between the percent correct on the training and validation sets, we get This can be an indication that the models are learning the training data a little too well. This can make generalization to a new data set less successful. Table 5. Percent correct on the training and validation sets Training Year Percent Correct Validation Year Percent Correct In Table 6, we have a cumulative percent of overall correctness, by week, of each model on the validation sets. To calculate this, each week was given a value of one if the model forecast matched the actual direction of movement of the S&P 500 in the following week. These values were summed from week one to each given week, then divided by the number of weeks to that point. The cumulative percent values that are above 0.5 are shown as shaded values. An interesting pattern we see in this table is that most years seems to be predominantly good or predominantly poor when using a model trained on the previous year. We see, in particular that the 2009 and 2010 validation set forecasts were both very poor from the beginning to the end. The years of 2005 through mid-2008 had reasonably good results up until the last few weeks of the session. That is, the models built were able to generalize on the new data. From mid-2008 to the end of 2010, the opposite is the case. These weeks all show that the previous year s trained model was not able to generalize to data it had not seen. The 2011 year was another time when the previous year s model had the ability to generalize on new data. In 2013 and 2014, the goodness of fit of the previous year s model is high is patches, but not overall. Finally, the 2015 year, up through April, again shows a good match to the previous year s model.

8 Table 6. Cumulative percent of correct forecasts in the validation year, values above 0.5 shaded. Year W w w w W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W

9 W W W W W W DISCUSSION AND CONCLUSIONS In this study, we used fourteen variables built from four base variables to forecast the direction of the S&P in the next period. The base variables included the S&P 500 past data, weekly mortgage rates, ten year notes, and Fed assets. Weekly values of all series were used. All models were built using the same set of input variables, and a new model was trained on each calendar year of data. Of interest in this study were three questions, the accuracy or fit of the models to their data, the variables identified by each model as being most important in the decision of forecast direction, and the accuracy of each trained model on the following year validation set. In terms of variables used by the models, each year picked its own relevant inputs and while there were some that appeared in the majority of models, notably the direction of the 30-year mortgage value and the skewness direction of the S&P 500 over 3 months, we see other variables moving in and out of relative importance from year to year. Accuracy of fit of the model on the training set was judged by the percent of time the model correctly identified next week s market direction on the training values. These are shown in Table 5. All models did well on this task, with correctness ranging from 78% to 98%. The models that did the least well were those trained on data during 2010 and In the middle of 2010, we saw a shift in the relationships between mortgage rates and 10 year notes, and the Fed assets. In the beginning of the year, both were above Fed assets. Then in mid-year, they dropped below Fed assets, and eventually, below the S&P as well. In 2014, we see a leveling off of the Fed assets with a downward spike in the S&P path (followed by a recovery). Notes and mortgage rates continued a slow decline with an upward spike in October, mirroring the movement of the S&P. The accuracy of each trained model on the following year s validation set data is given an overall summary view also in Table 5. Here we see a negative relationship between the percent correct on the training set and the percent correct on the validation set. The best years for forecasting used the models trained on 2010 and 2014, which were the poorest performing models on the training sets. But another way to view this is that models that do extremely well on their training set often fit that data so well that they cannot generalize as well to new data. The models that did generalize better, 2010 and 2014, both used fewer variables (4 and 5, respectively). Of the input variables, they had two in common, the direction of the 3-month skewness measure, and the direction the 30-year mortgage rates. The cumulative accuracy over each week of each year is shown in Table 6. This lets us see whether the models do well on the first few weeks of a year then deteriorate or do as well in later weeks as in earlier ones. Surprisingly, models that did well often remained good for most of the year. Values above 50% are shaded, and a number of these values are over 60%. We see that only two years (validation sets 2009 and 2010) did poorly for almost the entire period. All the other models had

10 patches where they were above 50%. But, in years where there are only patches where the fit is above 50%, those patches sometimes occur early and sometimes late in the year. We began with a set of variables that are traditionally related to the S&P 500 and show that their effect, while continuing to impact the direction the S&P moves, vary in a non-consistent way over this multi-year period. The C5.0 model is able to identify patterns very well within any particular year, but these patterns shift as the variable impact shifts. This makes the model forecasts on the following year much less accurate. REFERENCES Bates, D. (2000) Post-'87 crash fears in the S&P 500 futures option market, Journal of Econometrics, 94(1-2), doi: /s (99) Bowe, M. (2007) Discussion of an International Analysis of Earnings, Stock Prices and Bond Yields, Journal of Business Finance & Accounting, 34(3-4), Connolly, R., Stivers, C., & Sun, L. (2005), Stock market uncertainty and the stock bond return relation, Journal of Financial and Quantitative Analysis, 40, pp Dolvin, S. & Pyles, M. (2009) Daily Stock Returns: Momentum, Reversal, or Both, Financial Decisions, 21(2), Winter, Eastman, A.E., & Lucey, B. (2008) Skewness and asymmetry in futures returns and volumes, Applied Financial Economics, 18(10),pages Fleming, J., Kirby, C., & Ostdiek, B. (1998) Information and volatility linkages in the stock, bond, and money markets, Journal of Financial Economics, 49, pp Serfling, M., & Miljkovic, D. (2011) Time series analysis of the relationships among (macro) economic variables, the dividend yield and the price level of the S&P 500 Index, Applied Financial Economics, 21(15), Underwood, S. (2009) The cross-market information content of stock and bond order flow, Journal of Financial Markets, 12(2), Uryasev, S. (2011). Protecting Equity Investments: Options, Inverse ETFs, Hedge Funds, and AORDA Portfolios, American Optimal Decisions, Gainesville, FL. Xu, P., Han, Y., & Yang, J. (2012), U.S. Monetary Policy Surprises and Mortgage Rates, Real Estate Economics, 40(3),