A Comprehensive Look at The Empirical. Performance of Equity Premium Prediction

Transcription

1 RFS Advance Access published March 17, 2007 A Comprehensive Look at The Empirical Performance of Equity Premium Prediction Amit Goyal Emory University Goizueta Business School Ivo Welch Brown University Department of Economics NBER January 3, 2007 Abstract Our paper comprehensively reexamines the performance of variables that have been suggested by the academic literature to be good predictors of the equity premium. We find that by and large, these models have predicted poorly both in-sample and out-of-sample for thirty years now; these models seem unstable, as diagnosed by their out-of-sample predictions and other statistics; and these models would not have helped an investor with access only to available information to profitably time the market. Thanks to Malcolm Baker, Ray Ball, John Campbell, John Cochrane, Francis Diebold, Ravi Jagannathan, Owen Lamont, Sydney Ludvigson, Rajnish Mehra, Michael Roberts, Jay Shanken, Samuel Thompson, Jeff Wurgler, and Yihong Xia for comments, and Todd Clark for providing us with some critical McCracken values. We especially appreciate John Campbell and Sam Thompson for challenging our earlier drafts, and iterating mutually over working papers with opposite perspectives. amit_goyal@bus.emory.edu. ivo_welch@brown.edu. The Author Published by Oxford University Press on behalf of The Society for Financial Studies. All rights reserved. For permissions, please journals.permissions@oxfordjournals.org.

2 JEL Classification: G12, G14. 2

3 1 Introduction Attempts to predict stock market returns or the equity premium have a long tradition in finance. As early as 1920, Dow (1920) explored the role of dividend ratios. A typical specification regresses an independent lagged predictor on the stock market rate of return or, as we shall do, on the equity premium, Equity Premium(t) = γ 0 + γ 1 x(t 1) + ɛ(t). (1) γ 1 is interpreted as a measure of how significant x is in predicting the equity premium. The most prominent x variables explored in the literature are the dividend price ratio and dividend yield, the earnings price ratio and dividend-earnings (payout) ratio, various interest rates and spreads, the inflation rates, the book-to-market ratio, volatility, the investment-capital ratio, the consumption, wealth, and income ratio (CAY), and aggregate net or equity issuing activity. The literature is difficult to absorb. Different papers use different techniques, variables, and time periods. Results from papers that were written years ago may change when more recent data is used. Some papers contradict the findings of others. Still, most readers are left with the impression that prediction works though it is unclear exactly what works. The prevailing tone in the literature is perhaps best summarized by Lettau and Ludvigson (2001, p.842) It is now widely accepted that excess returns are predictable by variables such as dividend-price ratios, earnings-price ratios, dividend-earnings ratios, and an assortment of other financial indicators. There are also a healthy number of current papers which further cement this perspective; and a large theoretical and normative literature has developed that stipulates how investors should allocate their wealth as a function of the aforementioned variables. The goal of our own paper is to comprehensively reexamine the empirical evidence 1

4 as of early 2006, evaluating each variable using the same methods (mostly, but not only, in linear models), time-periods, and estimation frequencies. The evidence suggests that most models are unstable or even spurious. Most models are no longer significant even in-sample (IS), and the few models that still are usually fail simple regression diagnostics. Most models have performed poorly for over thirty years IS. For many models, any earlier apparent statistical significance was often based exclusively on years up to and especially on the years of the Oil Shock of Most models have poor out-of-sample (OOS) performance, but not in a way that merely suggests lower power than IS tests. They predict poorly late in the sample, not early in the sample. (For many variables, we have difficulty finding robust statistical significance even when they are examined only during their most favorable contiguous OOS sub-period.) Finally, the OOS performance is not only a useful model diagnostic for the IS regressions, but also interesting in itself for an investor who had sought to use these models for market-timing. Our evidence suggests that the models would not have helped such an investor. Therefore, although it is possible to search for, to occasionally stumble upon and then to defend some seemingly statistically significant models, we interpret our results to suggest that a healthy skepticism is appropriate when it comes to predicting the equity premium, at least as of early The models seem not robust. Our paper now proceeds as follows. We describe our data available at the RFS website in Section 2 and our tests in Section 3. Section 4 explores our base case predicting equity premia annually using OLS forecasts. In Sections 5 and 6, we predict equity premia on five-year and monthly horizons, the latter with special emphasis on the suggestions in Campbell and Thompson (2005). Section 7 tries earnings and dividend ratios with longer memory as independent variables, corrections for persistence in regressors, and encompassing model forecasts. Section 8 reviews earlier literature. Section 9 concludes. 2

5 2 Data Sources and Data Construction Our dependent variable is always the equity premium, i.e., the total rate of return on the stock market minus the prevailing short-term interest rate. Stock Returns: We use S&P 500 index returns from 1926 to 2005 from CRSP s monthend values. Stock returns are the continuously compounded returns on the S&P 500 index, including dividends. For yearly and longer data frequencies, we can go back as far as 1871, using data from Robert Shiller s website. For monthly frequency, we can only begin in the CRSP period, i.e., Risk-free Rate: The risk-free rate from 1920 to 2005 is the T-bill rate. Because there was no risk-free short-term debt prior to the 1920 s, we had to estimate it. Commercial paper rates for New York City are from the NBER s Macrohistory data base. These are available from 1871 to We estimated a regression from 1920 to 1971, which yielded T-bill Rate = Commercial Paper Rate, (2) with an R 2 of 95.7%. Therefore, we instrumented the risk-free rate from 1871 to 1919 with the predicted regression equation. The correlation for the period 1920 to 1971 between the equity premium computed using the actual T-bill rate and that computed using the predicted T-bill rate (using the commercial paper rate) is 99.8%. The equity premium had a mean (standard deviation) of 4.85% (17.79%) over the entire sample from 1872 to 2005; 6.04% (19.17%) from 1927 to 2005; and 4.03% (15.70%) from 1965 to Our first set of independent variables are primarily stock characteristics: Dividends: Dividends are twelve-month moving sums of dividends paid on the S&P 500 index. The data are from Robert Shiller s website from 1871 to

6 Dividends from 1971 to 2005 are from the S&P Corporation. The Dividend Price Ratio (d/p) is the difference between the log of dividends and the log of prices. The Dividend Yield (d/y) is the difference between the log of dividends and the log of lagged prices. (See, e.g., Ball (1978), Campbell (1987), Campbell and Shiller (1988a, 1988b), Campbell and Viceira (2002), Campbell and Yogo (2006), the survey in Cochrane (1997), Fama and French (1988), Hodrick (1992), Lewellen (2004), Menzly, Santos, and Veronesi (2004), Rozeff (1984), and Shiller (1984).) Earnings: Earnings are twelve-month moving sums of earnings on the S&P 500 index. The data are again from Robert Shiller s website from 1871 to June Earnings from June 2003 to December 2005 are our own estimates based on interpolation of quarterly earnings provided by the S&P Corporation. The Earnings Price Ratio (e/p) is the difference between the log of earnings and the log of prices. (We also consider variations, in which we explore multi-year moving averages of numerator or denominator, e.g., as in e 10 /p, which is the moving ten-year average of earnings divided by price.) The Dividend Payout Ratio (d/e) is the difference between the log of dividends and the log of earnings. (See, e.g., Campbell and Shiller (1988a, 1998) and Lamont (1998).) Stock Variance (svar): Stock Variance is computed as sum of squared daily returns on the S&P 500. G. William Schwert provided daily returns from 1871 to 1926; data from 1926 to 2005 are from CRSP. (See Guo (2006).) Cross-Sectional Premium (csp): The cross-sectional beta premium measures the relative valuations of high- and low-beta stocks and is proposed in Polk, Thompson, and Vuolteenaho (2006). The csp data are from Samuel Thompson from May 1937 to December Book Value: Book values from 1920 to 2005 are from Value Line s website, specifically their Long-Term Perspective Chart of the Dow Jones Industrial Average. The Book to Market Ratio (b/m) is the ratio of book value to market value for the Dow Jones Industrial Average. For the months from March to December, this is 4

7 computed by dividing book value at the end of the previous year by the price at the end of the current month. For the months of January and February, this is computed by dividing book value at the end of two years ago by the price at the end of the current month. (See, e.g, Kothari and Shanken (1997) and Pontiff and Schall (1998).) Corporate Issuing Activity: We entertain two measures of corporate issuing activity. Net Equity Expansion (ntis) is the ratio of twelve-month moving sums of net issues by NYSE listed stocks divided by the total end-of-year market capitalization of NYSE stocks. This dollar amount of net equity issuing activity (IPOs, SEOs, stock repurchases, less dividends) for NYSE listed stocks is computed from CRSP data as Net Issue t = Mcap t Mcap t 1 (1 + vwretx t ), (3) where Mcap is the total market capitalization, and vwretx is the value weighted return (excluding dividends) on the NYSE index. 1 These data are available from 1926 to ntis is closely related, but not identical, to a variable proposed in Boudoukh, Michaely, Richardson, and Roberts (2005). The second measure, Percent Equity Issuing (eqis), is the ratio of equity issuing activity as a fraction of total issuing activity. This is the variable proposed in Baker and Wurgler (2000). The authors provided us with the data, except for 2005, which we added ourselves. The first equity issuing measure is relative to aggregate market cap, while the second is relative to aggregate corporate issuing. Our next set of independent variables is interest-rate related: Treasury Bills (tbl): T-bill rates from 1920 to 1933 are the U.S. Yields On Short-Term United States Securities, Three-Six Month Treasury Notes and Certificates, Three Month Treasury series in the NBER Macrohistory data base. T-bill rates from 1934 to 2005 are the 3-Month Treasury Bill: Secondary Market Rate from the 1 This calculation implicitly assumes that the delisting return is 100 percent. Using the actual delisting return, where available, or ignoring delistings altogether, has no impact on our results. 5

8 economic research data base at the Federal Reserve Bank at St. Louis (FRED). (See, e.g., Campbell (1987) and Hodrick (1992).) Long Term Yield (lty): Our long-term government bond yield data from 1919 to 1925 is the U.S. Yield On Long-Term United States Bonds series in the NBER s Macrohistory data base. Yields from 1926 to 2005 are from Ibbotson s Stocks, Bonds, Bills and Inflation Yearbook, the same source that provided the Long Term Rate of Returns (ltr). The Term Spread (tms) is the difference between the long term yield on government bonds and the T-bill. (See, e.g., Campbell (1987) and Fama and French (1989).) Corporate Bond Returns: Long-term corporate bond returns from 1926 to 2005 are again from Ibbotson s Stocks, Bonds, Bills and Inflation Yearbook. Corporate Bond Yields on AAA and BAA-rated bonds from 1919 to 2005 are from FRED. The Default Yield Spread (dfy) is the difference between BAA and AAA-rated corporate bond yields. The Default Return Spread (dfr) is the difference between long-term corporate bond and long-term government bond returns. (See, e.g., Fama and French (1989) and Keim and Stambaugh (1986).) Inflation (infl): Inflation is the Consumer Price Index (All Urban Consumers) from 1919 to 2005 from the Bureau of Labor Statistics. Because inflation information is released only in the following month, we wait for one month before using it in our monthly regressions. (See, e.g., Campbell and Vuolteenaho (2004), Fama (1981), Fama and Schwert (1977), and Lintner (1975).) Like inflation, our next variable is also a common broad macroeconomic indicator. Investment to Capital Ratio (i/k): The investment to capital ratio is the ratio of aggregate (private nonresidential fixed) investment to aggregate capital for the whole economy. This is the variable proposed in Cochrane (1991). John Cochrane kindly provided us with updated data. 6

9 Of course, many papers explore multiple variables. For example, Ang and Bekaert (2003) explore both interest rate and dividend related variables. In addition to simple univariate prediction models, we also entertain two methods that rely on multiple variables (all and ms), and two models that are rolling in their independent variable construction (cay and ms). A Kitchen Sink Regression (all): This includes all the aforementioned variables. (It does not include cay, described below, partly due to limited data availability of cay.) Consumption, wealth, income ratio (cay): Lettau and Ludvigson (2001) estimate the following equation: k k c t = α+β a a t +β y y t + b a,i a t i + b y,i y t i +ɛ t, t = k+1,..., T k, (4) i= k i= k where c is the aggregate consumption, a is the aggregate wealth, and y is the aggregate income. Using estimated coefficients from the above equation provides cay ĉay t = c t ˆβ a a t ˆβ y y t, t = 1,..., T. Note that, unlike the estimation equation, the fitting equation does not use look-ahead data. Eight leads/lags are used in quarterly estimation (k = 8) while two lags are used in annual estimation (k = 2). (For further details, see Lettau and Ludvigson (2001).) Data for cay s construction are available from Martin Lettau s website at quarterly frequency from the second quarter of 1952 to the fourth quarter of Although annual data from 1948 to 2001 is also available from Martin Lettau s website, we reconstruct the data following their procedure as this allows us to expand the time-series from 1945 to 2005 (an addition of 7 observations). Because the Lettau-Ludvigson measure of cay is constructed using look-ahead (insample) estimation regression coefficients, we also created an equivalent measure that excludes advance knowledge from the estimation equation and thus uses only prevailing data. In other words, if the current time period is s, then we 7

10 estimated equation (4) using only the data up to s through k k c t = α+β s a a t+β s y y t+ b a,i a s t i+ b y,i y s t i+ɛ t, t = k+1,..., s k, (5) i= k i= k This measure is called caya ( ante ) to distinguish it from the traditional variable cayp constructed with look-ahead bias ( post ). The superscript on the betas indicates that these are rolling estimates, i.e., a set of coefficients used in the construction of one caya S measure in one period. A model selection approach, named ms. If there are K variables, we consider 2 K models essentially consisting of all possible combinations of variables. (As with the kitchen sink model, cay is not a part of the ms selection.) Every period, we select one of these models that gives the minimum cumulative prediction errors up to time t. This method is based on Rissanen (1986) and is recommended by Bossaerts and Hillion (1999). Essentially, this method uses our criterion of minimum OOS prediction errors to choose amongst competing models in each time period t. This is also similar in spirit to the use of a more conventional criterion (like R 2 ) in Pesaran and Timmerman (1995) (who do not entertain our NULL hypothesis). This selection model also shares a certain flavor with our encompassing tests in Section 7, where we seek to find an optimal rolling combination between each model and an unconditional historical equity premium average, and with the Bayesian model selection approach in Avramov (2002). The latter two models, cay and ms, are revised every period, which render insample regressions problematic. This is also why we did not include caya in the kitchen sink specification. 3 Empirical Procedure Our base regression coefficients are estimated using OLS, although statistical significance is always computed from bootstrapped F-statistics (taking correlation of 8

11 independent variables into account). OOS statistics: The OOS forecast uses only the data available up to the time at which the forecast is made. Let e N denote the vector of rolling OOS errors from the historical mean model and e A denote the vector of rolling OOS errors from the OLS model. Our OOS statistics are computed as R 2 = 1 MSE A MSE N, R 2 = R 2 (1 R 2 ) RMSE = MSE N MSE A, ( ) MSEN MSE A MSE-F = (T h + 1) MSE A ( ) T k T 1, (6), where h is the degree of overlap (h = 1 for no overlap). MSE-F is McCracken s (2004) F-statistic. It tests for equal MSE of the unconditional forecast and the conditional forecast (i.e., MSE = 0). 2 We generally do not report MSE-F statistics, but instead use their bootstrapped critical levels to provide statistical significance levels via stars in the tables. For our encompassing tests in Section 7, we compute ENC = T h + 1 T T t=1 (e 2 Nt e Nt e At ) MSE A, (7) which is proposed by Clark and McCracken (2001). They also show that the MSE-F and ENC statistics follow non-standard distributions when testing nested models, because the asymptotic difference in squared forecast errors is exactly 0 with 0 variance under the NULL, rendering the standard distributions asymptotically invalid. Because our models are nested, we could use asymptotic critical values for MSE tests provided by McCracken, and asymptotic critical values for ENC tests provided by 2 Our earlier drafts also entertained another performance metric, the mean absolute error difference MAE. The results were similar. These drafts also described another OOS-statistic, MSE-T = [ ( )] T T h + h (h 1)/T d/ŝe d, where d t = e Nt e At, and d = T 1 t d t = MSE N MSE A over the entire OOS period, and T is the total number of forecast observations. This is the Diebold and Mariano (1995) t-statistic modified by Harvey, Leybourne, and Newbold (1997). (We still use the latter as bounds in our plots, because we know the full distribution.) Again, the results were similar. We chose to use the MSE-F in this paper because Clark and McCracken (2001) find that MSE-F has higher power than MSE-T. 9

12 Clark and McCracken. However, because we use relatively small samples, because our independent variables are often highly serially correlated, and especially because we need critical values for our five-year overlapping observations (for which asymptotic critical values are not available), we obtain critical values from the bootstrap procedure described below. (The exceptions are that critical values for caya, cayp, and all models are not calculated using a bootstrap, and critical values for ms model are not calculated at all.) The NULL hypothesis is that the unconditional forecast is not inferior to the conditional forecast, so our critical values for OOS test are for a one-sided test (critical values of IS tests are, as usual, based on two-sided tests). 3 Bootstrap: Our bootstrap follows Mark (1995) and Kilian (1999) and imposes the NULL of no predictability for calculating the critical values. In other words, the data generating process is assumed to be y t+1 = α + u 1t+1 x t+1 = µ + ρ x t + u 2t+1. The bootstrap for calculating power assumes the data generating process is y t+1 = α + β x t + u 1t+1 x t+1 = µ + ρ x t + u 2t+1, where both β and ρ are estimated by OLS using the full sample of observations, with the residuals stored for sampling. We then generate 10,000 bootstrapped time series by drawing with replacement from the residuals. The initial observation preceding the sample of data used to estimate the models is selected by picking one date from the actual data at random. This bootstrap procedure not only preserves the autocorrelation structure of the predictor variable, thereby being valid under the 3 If the regression coefficient β is small (so that explanatory power is low or the in-sample R 2 is low), it may happen that our unconditional model outperforms on OOS because of estimation error in the rolling estimates of β. In this case, RMSE might be negative but still significant because these tests are ultimately tests of whether β is equal to zero. 10

13 Stambaugh (1999) specification, but also preserves the cross-correlation structure of the two residuals. 4 Statistical Power: Our paper entertains both IS and OOS tests. Inoue and Kilian (2004) show that the OOS tests used in this paper are less powerful than IS tests, even though their size properties are roughly the same. Similar critiques of the OOS tests in our paper have been noted by Cochrane (2005) and Campbell and Thompson (2005). We believe this is the wrong way to look at the issue of power for two reasons: 1. It is true that under a well-specified stable underlying model, an IS OLS estimator is more efficient. Therefore, a researcher who has complete confidence in her underlying model specification (but not the underlying model parameters) should indeed rely on IS tests to establish significance the alternative of OOS tests does have lower power. However, the point of any regression diagnostics, such as those for heteroskedasticity and autocorrelation, is always to subject otherwise seemingly successful regression models to a number of reasonable diagnostics when there is some model uncertainty. Relative to not running the diagnostic, by definition, any diagnostic that can reject the model at this stage sacrifices power if the specified underlying model is correct. In our forecasting regression context, OOS performance just happens to be one natural and especially useful diagnostic statistic. It can help determine whether a model is stable and well-specified, or changing over time, either suddenly or gradually. This also suggests why the simple power experiment performed in some of the aforementioned critiques of our own paper is wrong. It is unreasonable to propose a model if the IS performance is insignificant, regardless of its OOS performance. Reasonable (though not necessarily statistically significant) OOS performance is not a substitute, but a necessary complement for IS performance in order to establish the quality of the underlying model specification. The thought experiments and analyses in the critiques, which simply compare the power of 4 We do not bootstrap for cayp because it is calculated using ex-post data; for caya and ms because these variables change each period; and for all because of computational burden. 11

14 OOS tests to that of IS tests, especially under their assumption of a correctly specified stable model, is therefore incorrect. The correct power experiment instead should explore whether conditional on observed IS significance, OOS diagnostics are reasonably powerful. We later show that they are. Not reported in the tables, we also used the CUSUMQ test to test for model stability. Although this is a weak test, we can reject stability for all monthly models; and for all annual models except for ntis, i/k, and cayp, when we use data beginning in Thus, the CUSUMQ test sends the same message about the models as the findings that we shall report. 2. All of the OOS tests in our paper do not fail in the way the critics suggest. Low power OOS tests would produce relatively poor predictions early and relatively good predictions late in the sample. Instead, all of our models show the opposite behavior good OOS performance early, bad OOS performance late. A simple alternative OOS estimator, which downweights early OOS predictions relative to late OOS predictions, would have more power than our unweighted OOS prediction test. Such a modified estimator would both be more powerful and it would show that all models explored in our paper perform even worse. (We do not use it only to keep it simple and to avoid a cherry-picking-the-test critique.) Estimation Period: It is not clear how to choose the periods over which a regression model is estimated and subsequently evaluated. This is even more important for OOS tests. Although any choice is necessarily ad-hoc in the end, the criteria are clear. It is important to have enough initial data to get a reliable regression estimate at the start of evaluation period, and it is important to have an evaluation period that is long enough to be representative. We explore three time period specifications: the first begins OOS forecasts twenty years after data are available; the second begins OOS forecast in 1965 (or twenty years after data are available, whichever comes later); the third ignores all data prior to 1927 even in the estimation. 5 If a 5 We also tried estimating our models only with data after World-War II, as recommended by 12

15 variable does not have complete data, some of these time-specifications can overlap. Using three different periods reflects different tradeoffs between the desire to obtain statistical power and the desire to obtain results that remain relevant today. In our graphical analysis later, we also evaluate the rolling predictive performance of variables. This analysis helps us identify periods of superior or inferior performance and can be seen as invariance to the choice of the OOS evaluation period (though not the estimation period). 4 Annual Prediction Table 1: Annual Performance Table 1 shows the predictive performance of the forecasting models on annual forecasting horizons. Figures 1 and 2 graph the IS and OOS performance of variables in Table 1. For the IS regressions, the performance is the cumulative squared demeaned equity premium minus the cumulative squared regression residual. For the OOS regressions, this is the cumulative squared prediction errors of the prevailing mean minus the cumulative squared prediction error of the predictive variable from the linear historical regression. Whenever a line increases, the ALTERNATIVE predicted better; whenever it decreases, the NULL predicted better. The units in the graphs are not intuitive, but the time-series pattern allows diagnosis of years with good or bad performance. Indeed, the final SSE statistic in the OOS plot is sign-identical with the RMSE statistic in our tables. The standard error of all the observations in the graphs is based on translating MSE-T statistic into symmetric 95% confidence intervals based on the McCracken (2004) critical values; the tables differ in using the MSE-F statistic instead. The reader can easily adjust perspective to see how variations in starting or ending date would impact the conclusion by shifting the graph up or down (redrawing the y=0 horizontal zero line). Indeed, a horizontal line and the right-side scale Lewellen (2004). Some properties in some models change, especially when it comes to statistical significance and the importance of the Oil Shock for one variable, d/p. However, the overall conclusions of our paper remain. Figure 1 Figure 2 13

16 indicate the equivalent zero-point for the second time period specification, in which we begin forecasts in 1965 (this is marked Spec B Zero Val line). The plots have also vertically shifted the IS errors, so that the IS line begins at zero on the date of our first OOS prediction. The Oil Shock recession of 1973 to 1975, as identified by the NBER, is marked by a vertical (red) bar in the figures. 6 In addition to the figures and tables, we also summarize models performances in small in-text summary tables, which give the IS-R 2 and OOS-R 2 for two time periods: the most recent 30 years and the entire sample period. The R 2 for the subperiod is not the R 2 for a different model estimated only over the most recent three decades, but the residual fit for the overall model over the subset of data points (e.g., computed simply as 1-SSE/SST for the last 360 residuals). The most recent three decades after the Oil Shock can help shed light on whether a model is likely to still perform well nowadays. Generally, it is easiest to understand the data by looking first at the figures, then at the in-text table, and finally at the full table. A well-specified signal would inspire confidence in a potential investor if it had 1. both significant IS and reasonably good OOS performance over the entire sample period; 2. a generally upward drift (of course, an irregular one); 3. an upward drift which occurs not just in one short or unusual sample period say just the two years around the Oil Shock; 4. an upward drift that remains positive over the most recent several decades otherwise, even a reader taking the long view would have to be concerned with the possibility that the underlying model has drifted. There are also other diagnostics that stable models should pass (heteroskedasticity, residual autocorrelation, etc.), but we do not explore them in our paper. 6 The actual recession period was from November 1973 to March We treat both 1973 and 1975 as years of Oil Shock recession in annual prediction. 14

17 4.1 In-Sample Insignificant Models As already mentioned, if a model has no IS performance, its OOS performance is not interesting. However, because some of the IS insignificant models are so prominent, and because it helps to understand why they may have been considered successful forecasters in past papers, we still provide some basic statistics and graph their OOS performance. The most prominent such models are the following: Dividend Price Ratio: Figure 1 shows that there were four distinct periods for the d/p model, and this applies both to IS and OOS performance. d/p had mild underperformance from 1905 to WW-II, good performance from WW-II to 1975, neither good nor bad performance until the mid-1990s, and poor performance thereafter. The best sample period for d/p was from the mid 1930s to the mid 1980s. For the OOS, it was 1937 to 1984, although over half of the OOS performance was due to the Oil Shock. Moreover, the plot shows that the OOS performance of the d/p regression was consistently worse than the performance of its IS counterpart. The distance between the IS and OOS performance increased steadily until the Oil Shock. Over the most recent 30 years (1976 to 2005), d/p s performance is negative both IS and OOS. Over the entire period, d/p underperformed the prevailing mean OOS, too: d/p Recent All 30 Years Years IS R % 0.49% OOS R % 2.06% Dividend Yield: Figure 1 shows that the d/y model s IS patterns look broadly like those of d/p. However, its OOS pattern was much more volatile: d/y predicted equity premia well during the Great Depression (1930 to 1933), the period from World War II to 1958, the Oil Shock of , and the market decline of It had large prediction errors from 1958 to 1965 and from 1995 to 2000, and it had unremarkable performance in other years. The best OOS sample 15

18 period started around 1925 and ended either in 1957 or The Oil Shock did not play an important role for d/y. Over the most recent 30 years, d/y s performance is again negative IS and OOS. The full-sample OOS performance is also again negative: d/y Recent All 30 Years Years IS R % 0.91% OOS R % 1.93% Earnings Price Ratio: Figure 1 shows that e/p had inferior performance until WW-II, and superior performance from WW-II to the late 1970s. After the Oil Shock, it had generally non-descript performance (with the exception of the late 1990s and early 2000s). Its best sample period was 1943 to and 2004 were bad years for this model. Over the most recent 30 years, e/p s performance is again negative IS and OOS. The full-sample OOS performance is negative too. e/p Recent All 30 Years Years IS R % 1.08% OOS R % 1.78% Table 1 shows that these three price ratios are not statistically significant IS at the 90% level. However, some disagreement in the literature can be explained by differences in the estimation period. 7 7 For example, the final lines in Table 1 show that d/y and e/p had positive and statistically significant IS performance at the 90% level if all data prior to 1927 is ignored. Nevertheless, Table 1 also shows that the OOS-R 2 performance remains negative for both of these. Moreover, when the data begins in 1927 and the forecast begins in 1947 (another popular period choice), we find (Data Begins in 1927) e/p d/y (Forecast Begins in 1947) Recent All Recent All 30 Years Years 30 Years Years IS R % 3.20% 5.20% 2.71% OOS R % 3.41% 28.05% 16.65% Finally, and again not reported in the table, another choice of estimation period can also make a difference. The three price models lost statistical significance over the full sample only in the 1990s. This is not because the IS- RMSE has decreased further in the 1990 s, but because the prediction errors were more volatile, which raised the standard errors of point estimates. 16

19 Other Variables: The remaining plots in Figure 1 and the remaining IS insignificant models in Table 1 show that d/e, dfy, and infl essentially never had significantly positive OOS periods, and that svar had a huge drop in OOS performance from 1930 to Other variables (that are IS insignificant) often had good sample performance early on, ending somewhere between the Oil Shock and the mid-1980s, followed by poor performance over the most recent three decades. The plots also show that it was generally not just the late 1990s that invalidated them, unlike the case with the aforementioned price ratio models. In sum, twelve models had insignificant in-sample full-period performance and, not surprisingly, these models generally did not offer good OOS performance. 4.2 In-Sample Significant Models Five models were significant IS (b/m, i/k, ntis, eqis, and all) at least at the 10% two-sided level. Table 1 contains more details for these variables, such as the IS performance during the OOS period, and a power statistic. Together with the plots in Figure 2, this information helps the reader to judge the stability of the models whether poor OOS performance is driven by less accurately estimated parameters (pointing to lower power), and/or by the fact that the model fails IS and/or OOS during the OOS sample period (pointing to a spurious model). Book-market ratio: b/m is statistically significant at the 6% level IS. Figure 2 shows that it had excellent IS and OOS predictive performance right until the Oil Shock. Both its IS and OOS performance were poor from 1975 to 2000, and the recovery in was not enough to gain back the performance. Thus, the b/m model has negative performance over the most recent three decades, both IS and OOS. b/m Recent All 30 Years Years IS R % 3.20% OOS R % 1.72% 17

20 Over the entire sample period, the OOS performance is negative, too. The IS for OOS R 2 in Table 1 shows how dependent b/m s performance is on the first 20 years of the sample. The IS R 2 is 7.29% for the period. The comparable OOS R 2 even reaches 12.71%. As with other models, b/m s lack of OOS significance is not just a matter of low test power. Table 1 shows that in the OOS prediction beginning in 1941, under the simulation of a stable model, the OOS statistic came out statistically significantly positive in 67% 8 of our (stable-model) simulations in which the IS regression was significant. Not reported in the table, positive performance (significant or insignificant) occurred in 78% of our simulations. A performance as negative as the observed RMSE of 0.01 occurred in none of the simulations. Investment-capital ratio: i/k is statistically significant IS at the 5% level. Figure 2 shows that, like b/m, it performed well only in the first half of its sample, both IS and OOS. About half of its performance, both IS and OOS, occurs during the Oil Shock. Over the most recent 30 years, i/k has underperformed: i/k Recent All 30 Years Years IS R % 6.63% OOS R % 1.77% Corporate Issuing Activity: Recall that ntis measures equity issuing and repurchasing (plus dividends) relative to the price level; eqis measures equity issuing relative to debt issuing. Figure 2 shows that both variables had superior IS performance in the early 1930 s, a part of the sample that is not part of the OOS period. eqis continues good performance into the late 1930 s but gives back the extra gains immediately thereafter. In the OOS period, there is one stark difference between the two variables: eqis had superior performance during the Oil Shock, both IS and 8 The 42% applies to draws that were not statistically significant in-sample at the 90% level. It is the equivalent of the experiment conducted in some other papers. However, because OOS performance is relevant only when the IS performance is significant, this is the wrong measure of power. 18

21 OOS. It is this performance that makes eqis the only variable that had statistically significant OOS performance in the annual data. In other periods, neither variable had superior performance during the OOS period. Both variables underperformed over the most recent 30 years ntis eqis Recent All Recent All 30 Years Years 30 Years Years IS R % 8.15% 10.36% 9.15% OOS R % 5.07% 15.33% 2.04% The plot can also help explain dueling perspectives about eqis between Butler, Grullon, and Weston (2005) and Baker, Taliaferro, and Wurgler (2004). One part of their disagreement is whether eqis s performance is just random underperformance in sampled observations. Of course, some good years are expected to occur in any regression. Yet eqis s superior performance may not have been so random, because it [a] occurred in consecutive years, and [b] in response to the Oil Shock events that are often considered to have been exogenous, unforecastable, and unusual. Butler, Grullon, and Weston also end their data in 2002, while Baker, Taliaferro, and Wurgler refer to our earlier draft and to Rapach and Wohar (2006), which end in 2003 and 1999, respectively. Our figure shows that small variations in the final year choice can make a difference in whether eqis turns out significant or not. In any case, both papers have good points. We agree with Butler, Grullon, and Weston that eqis would not have been a profitable and reliable predictor for an external investor, especially over the most recent 30 years. But we also agree with Baker, Taliaferro, and Wurgler that conceptually, it is not the OOS performance, but the IS performance that matters in the sense in which Baker and Wurgler (2000) were proposing eqis not as a third-party predictor, but as documentary evidence of the fund-raising behavior of corporations. Corporations did repurchase profitably in the Great Depression and the Oil Shock era (though not in the bubble period collapse of ). 19

22 all The final model with IS significance is the kitchen sink regression. It had high IS significance, but exceptionally poor OOS performance. 4.3 Time-Changing Models caya and ms have no in-sample analogs, because the models themselves are constantly changing. Consumption-Wealth-Income: Lettau and Ludvigson (2001) construct their cay proxy assuming that agents have some ex-post information. The experiment their study calls OOS is unusual: their representative agent still retains knowledge of the model s full-sample CAY-construction coefficients. It is OOS only in that the agent does not have knowledge of the predictive coefficient and thus has to update it on a running basis. We call the Lettau and Ludvigson (2001) variable cayp. We also construct caya, which represents a more genuine OOS experiment, in which investors are not assumed to have advance knowledge of the cay construction estimation coefficients. Figure 2 shows that cayp had superior performance until the Oil Shock, and nondescript performance thereafter. It also benefited greatly from its performance during the Oil Shock itself. cay Recent All 30 Years Years some ex-post knowledge, cayp IS R % 15.72% some ex-post knowledge, cayp OOS R % 16.78% no advance knowledge, caya OOS R % 4.33% The full-sample cayp result confirms the findings in Lettau and Ludvigson (2001). cayp outperforms the benchmark OOS RMSE by 1.61% per annum. It is stable and its OOS performance is almost identical to its IS performance. In contrast to cayp, caya has had no superior OOS performance, either over the entire sample period or the most recent years. In fact, without advance knowledge, caya had the worst OOS R 2 performance among our single variable models. 20

23 Model Selection Finally, ms fails with a pattern similar to earlier variables good performance until 1976, bad performance thereafter. ms Recent All 30 Years Years IS R 2 OOS R % 22.50% Conclusion: There were a number of periods with sharp stock market changes, such as the Great Depression of (in which the S&P500 dropped from at the end of 1928 to 6.89 at the end of 1932) and the bubble period from (with its subsequent collapse). However, it is the Oil Shock recession of , in which the S&P500 dropped from in October 1973 to in September 1974 and its recovery back to in June 1975 that stands out. Many models depend on it for their apparent forecasting ability, often both IS and OOS. (And none performs well thereafter.) Still, we caution against overreading or underreading this evidence. In favor of discounting this period, the observed source of significance seems unusual, because the important years are consecutive observations during an unusual period. (They do not appear to be merely independent draws.) In favor of not discounting this period, we do not know how one would identify these special multi-year periods ahead of time, except through a model. Thus, good prediction during such a large shock should not be automatically discounted. More importantly and less ambiguously, no model seems to have performed well since that is, over the last thirty years. In sum, on an annual prediction basis, there is no single variable that meets all of our four suggested investment criteria from Page 14 (IS significance, OOS performance, reliance not just on some outliers, and good positive performance over the last three decades.) Most models fail on all four criteria. 21

24 5 Five-Yearly Prediction Table 2: Five-Yearly Frequency Some models may predict long-term returns better than short-term returns. Unfortunately, we do not have many years to explore 5-year predictions thoroughly, and there are difficult econometric issues arising from data overlap. Therefore, we only briefly describe some preliminary and perhaps naive findings. (See, e.g., Boudoukh, Richardson, and Whitelaw (2005) and Lamoureux and Zhou (1996) for more detailed treatments.) Table 2 repeats Table 1 with 5-year returns. As before, we bootstrap all critical significance levels. This is especially important here, because the observations are overlapping and the asymptotic critical values are not available. Table 2 shows that there are four models that are significant IS over the entire sample period: ntis, d/p, i/k, and all. ntis and i/k were also significant in the annual data (Table 1). Two more variables, d/y and tms, are IS significant if no data prior to 1927 is used. Dividend Price Ratio: d/p had negative performance OOS regardless of period. Term Spread: tms is significant IS only if the data begins in 1927 rather than An unreported plot shows that tms performed well from , poorly from , and then well again from Indeed, its better years occur in the OOS period, with an IS R 2 of 23.54% from This was sufficient to permit it to turn in a superior OOS RMSE performance of 2.77% per five-years a meaningful difference. On the negative side, tms has positive OOS performance only if forecasting begins in Using data and starting forecasts in 1947, the OOS RMSE and R 2 are negative. The Kitchen Sink: all again turned in exceptionally poor OOS performance. Model selection (ms) and caya again have no in-sample analogs. ms had the worst predictive performance observed in this paper. caya had good OOS performance of 2.50% per five-year period. Similarly, the investment-capital ratio, i/k, had both positive IS and OOS performance, and both over the most recent three decades as 22

25 well as over the full sample (where it was also statistically significant). i/k Recent All 30 Years Years IS R % 33.99% OOS R % 12.99% i/k s performance is driven by its ability to predict the 2000 crash. In 1997, it had already turned negative on its equity premium prediction, thus predicting the 2000 collapse, while the unconditional benchmark prediction continued with its 30% plus predictions: Forecast for Actual Forecast Forecast for Actual Forecast made in years EqPm Unc. i/k made in years EqPm Unc. i/k This model (and perhaps caya) seem promising. We hesitate to endorse them further only because our inference is based on a small number of observations, and because statistical significance with overlapping multi-year returns raises a set of issues that we can only tangentially address. We hope more data will allow researchers to explore these models in more detail. 6 Monthly Prediction and Campbell-Thompson Table 3 describes the performance of models predicting monthly equity premia. It also addresses a number of points brought up by Campbell and Thompson (2005), henceforth CT. We do not have dividend data prior to 1927, and thus no reliable equity premium data before then. This is why even our the estimation period begins only in

26 6.1 In-Sample Performance Table 3: Monthly and Campbell and Thompson Analysis Table 3 presents the performance of monthly predictions both IS and OOS. The first data column shows the IS performance when the predicted variable is logged (as in the rest of the paper). Eight out of eighteen models are in-sample significant at the 90% level, seven at the 95% level. Because CT use simple rather than log equity premia, the remaining data columns follow their convention. This generally improves the predictive power of most models, and the fourth column (by which rows are sorted) shows that three more models turn in statistically significant IS. 9 CT argue that a reasonable investor would not have used a models to forecast a negative equity premium. Therefore, they suggest truncation of such predictions at zero. In a sense, this injects caution into the models themselves, a point we agree with. Because there were high equity premium realizations especially in the 1980s and 1990s, a time when many models were bearish, this constraint can improve performance. Of course, it also transforms formerly linear models into non-linear models, which are generally not the subject of our paper. CT do not truncate predictions in their in-sample regressions, but there is no reason not to do so. Therefore, the fifth column shows a revised IS R 2 statistic. Some models now perform better, some perform worse. 6.2 Out-of-Sample Prediction Performance The remaining columns explore the OOS performance. The sixth column shows that without further manipulation, eqis is the only model with both superior IS (R 2 =0.82% and 0.80%) and OOS (R 2 = 0.14%) untruncated performance. The term-spread, tms, has OOS performance that is even better (R 2 = 0.22%), but it just misses statistical 9 Geert Bekaert pointed out to us that if returns are truly log-normal, part of their increased explanatory power could be due to the ability of these variables to forecast volatility. 24

27 significance IS at the 90% level. infl has marginally good OOS performance, but poor IS performance. All other models have negative IS or OOS untruncated R 2. The remaining columns show model performance when we implement the Campbell and Thompson (2005) suggestions. The seventh column describes the frequency of truncation of negative equity premium predictions. For example, d/y s equity premium predictions are truncated to zero in 54.2% of all months; csp s predictions are truncated in 44.7% of all months. Truncation is a very effective constraint. CT also suggest using the unconditional model if the theory offers one coefficient sign and the estimation comes up with the opposite sign. For some variables, such as the dividend ratios, this is easy. For other models, it is not clear what the appropriate sign of the coefficient would be. In any case, this matters little in our data set. The eighth column shows that the coefficient sign constraint matters only for dfr, and ltr (and mildly for d/e). None of these three models has IS performance high enough to make this worthwhile to explore further. The ninth and tenth columns, R 2 TU and RMSE TU, show the effect of the CT truncations on OOS prediction. For many models, the performance improves. Nevertheless, the OOS R 2 s remain generally much lower than their IS equivalents. Some models have positive RMSE but negative OOS R 2. This reflects the number of degrees of freedom: even though we have between 400 and 800 data months, the plain RMSE and R 2 are often so small that the R 2 turns negative. For example, even with over 400 months of data, the loss of three degrees of freedom is enough for cay3 to render a positive RMSE of (equivalent to an unreported unadjusted-r 2 of ) into a negative adjusted-r 2 of Even after these truncations, ten of the models that had negative plain OOS R 2 s still have negative CT OOS R 2 s. Among the eleven IS significant models, seven (cay3, ntis, e 10 /p, b/m, e/p, d/y, and dfy) have negative OOS R 2 performance even after the truncation. Three of the models (lty, ltr, and infl) that benefit from the OOS truncation are not close to statistical significance IS, and thus can be ignored. All in all, this leaves four models that are both OOS and IS positive and significant: 25