Improving Recession Probability Forecasts in the U.S. Economy

Transcription

1 Improving Recession Probability Forecasts in the U.S. Economy Munechika Katayama Louisiana State University This Draft: April 26, 2009 First Draft: July 2008 Abstract There are two margins to improve forecasting models of the U.S. recession probability: including additional variables and using a different functional form. Using out-of-sample and cross validation methods, I systematically compare the performance of various forecasting models that differ in terms of variables included and functional forms used. I find substantial gains from including additional variables, such as the S&P 500 and non-farm employment growth, together with the Treasury term spread. In addition, there is a room to further improve forecasting accuracy by utilizing a non-normal cumulative distribution function. I also examine this possibility by using the generalized Edgeworth expansion, which enables us to explore a wider set of skewness and excess kurtosis. We can obtains further gains by allowing for more flexibility in the functional form. Keywords: Recession Probability; Forecasting; Generalized Edgeworth Expansion. JEL Classification: C25, C53, E32, E37. I am indebted to James Hamilton for his encouragement and helpful comments. For helpful comments and suggestions, I thank Hiroaki Kaido and participants at the Ninth Annual Missouri Economics Conference and the UC Riverside Conference on Business Cycles. This paper is based on Chapter 3 of my doctoral dissertation at UCSD. 1

2 1 Introduction There are many attempts to forecast recessions in the U.S. economy. Information contained in forward-looking variables can be used for making predictions about the future state of the economy. Stock and Watson (1989) report that the slope of the Treasury yield curve provides useful information in constructing their leading indicator. Estrella and Mishkin (1998) examine the usefulness of financial variables in making predictions about future recession probabilities at various forecasting horizons. They find that the slope of yield curve is the best single predictor of future recession probabilities. 1 Other studies also utilize the predictive content of the term spread in order to forecast recession probabilities. 2 Although developments in the literature achieve some success in making predicting recession, there is a room to improve forecasting accuracy. To obtain further improvements, there are two margins that we can exploit. One is the question of selecting a set of predictor variables. The other is a choice of a functional form. Typical studies on forecasting recession probabilities utilize a simple binary response model, such as a probit or logit model. In this paper, I will explore the importance of the two margins in improving forecasting accuracy. Within the class of probit/logit models, there is favorable evidence for benefits from including additional information, other than the term spread. For example, Estrella and Mishkin (1998) find some evidence on usefulness of including an additional variable, such as a New York Stock Exchange index or GDP growth. Wright (2006) reports that including the level of the federal funds rate together with the term spread results in better forecasting performance than the term spread alone. 3 King, Levin, and Perli (2007) recently find superior predictive power of the 5-year corporate credit spread on AA-rated firms compared with the 10-year-3-month Treasury term spread over the Great Moderation period (since the mid 1980 s). Furthermore, they also report that once the corporate spread is augmented by the term spread, the forecasting performance dramatically improves by reducing false positive predictions of recession. There are also some attempts to extend a simple probit model in the recent literature. For ex- 1 Their finding is consistent with a simple rule of thumb that the Treasury yield curve inversions or negative term spreads are likely followed by recessions in subsequent periods and the term spread has been cited as a leading economic indicator. 2 Other studies include Dotsey (1998), Birchenhall, Jessen, Osborn, and Simpson (1999), Estrella, Rodrigues, and Schich (2003), Stock and Watson (2003), Clements and Galvão (2006), Rudebusch and Williams (2008). 3 He also finds some evidence favorable for controlling for a term premium proxy. 2

3 ample, Chauvet and Potter (2005) extend a probit model in a way that it allows for business-cycle dependent coefficients (i.e., multiple breaks) and/or autocorrelated errors. Kauppi and Saikkonen (2008) develop dynamic probit models that include lagged explanatory variables and lagged recessionary dummies. Dueker (2005) presents a framework (a Qual VAR) that enables us to treat a qualitative variable as endogenous in a typical VAR framework, so that we can obtain dynamic forecasts of the qualitative variable. Going beyond the typical univariate probit model with the term spread, such as allowing additional features in a forecasting model and/or including additional predictor variables, most likely would improve in-sample forecasting performance. However, at the same time, we also have to worry about a potential over-fitting problem and we want to make a forecasting model as parsimonious as possible. Hansen (2008) examines the pitfalls of relying on in-sample fit. He shows that too good in-sample fit (over-fitting) tends to be associated poor out-of-sample fit and that model selection based on in-sample fit is not reliable. In addition, he demonstrates that tendencies of yielding spurious results are much higher if we use in-sample evaluation and are more pronounced when we compare performance of a large number of alternative models. For example, Sephton (2001) applies non-linear and non-parametric methods (multivariate adaptive regression splines) to forecasting recession probability. While in-sample fit with this very flexible approach shows great success, he finds that out-of-sample predictions are relatively not helpful. In this paper, for these reasons, I will use out-of-sample and cross validation methods to examine the importance of the two margins that can contribute to improve forecasting performance. First, I systematically compare the forecasting performance of 6-month-ahead recession probability predictions in the U.S. economy. I look at all possible combinations of 32 variables up to trivariate models with 6 different functional forms. Results highlight the importance of variable selection. Finding the best combination of predictor variables greatly improve forecasting performance. Especially, among variables considered, the combination of the term spread, changes in the S&P 500 stock price index, and growth rate of non-farm employment is found to achieve the best forecasting accuracy. Furthermore, additional gains can be obtained by using non-normal cumulative distribution functions. These additional features help improve forecasting accuracies by amplifying positive signals during recessions and also by dampening false positive alarms during expansions. However, there is some mixed evidence on what features of functional form are necessary 3

4 in order to improve forecasting performance. Using known CDFs amounts to imposing restrictions on the combination of skewness and excess kurtosis. Thus, I proceed to allow for more flexibility in the functional form by utilizing the generalized Edgeworth expansion of Jarrow and Rudd (1982), which allows us to explore a wider space of skewness and excess kurtosis. By using the genenelized Edgeworth expansion, we can obtain further improvement in out-of-sample forecasting accuracy. Findings in this paper complements other developments in the literature mentioned above. We can easily include additional variables into other extensions to probit models and also the more flexible functional form can be incorporated into a dynamic model or it can include multiple breaks, in principle. The rest of the paper is organized as follows. Section 2 lays out basic framework of the study. I present basic binary response models for forecasting recession probability, including the discussion of variables to be used and the method of evaluating forecasting performance. Section 3 presents empirical results and performs robustness check. Section 4 introduces the generalized Edgeworth expansion and applies it to making predictions of the U.S. recession probability. Finally, Section 5 concludes. 2 Basic Framework 2.1 Recession Probability Forecasting Model Let y t represent an NBER recession binary variable, which equals 1 when the economy is in recession in month t and equals 0 in expansion. 4 Typical models of forecasting h-period-ahead recession probabilities using the information available at time t assume that Prob(y t+h = 1 x t ) = F (β x t ), (1) where F ( ) is a monotonically increasing function, whose range is the unit interval, β is a vector of coefficients associated with a vector of predictors x t = [1, x 1,t,, x k,t ], and k is the number of 4 It is important to mention that, in reality, the NBER s announcement often involves a significant time lag. This is one drawback of relying on the NBER s decision. We can employ alternative definitions of business cycles, such as more than two consecutive quarters of negative output growth as a recession. Alternatively, we can utilize an algorithm proposed by Chauvet and Hamilton (2006), which is motivated by the delay in the NBER s announcement, for nowcasting, in order to determine near-real-time values of y t. However, since it is the NBER s decisions that is most widely used in the academia as well as the public, I will use it to define the state of the U.S. economy. 4

5 variables included. It is commonly assumed that y t+h is a conditionally independent Bernoulli random variable, so that the likelihood function is given by: T [ L = F (β x t ) ] y t+h [ 1 F (β x t ) ] 1 y t+h. (2) t=1 In an empirical analysis, I will set h = 6 and focus on 6-month-ahead predictions. In this formulation, predicting recession probability involves two issues that are possibly related to each other. The first one is to choose a set of predictor variables, so that we can obtain useful information from data. A workhorse predictor in the literature is the Treasury term spread between 10-year and 3-month bonds, which is due to the finding in Estrella and Mishkin (1998). Given a choice of F ( ), finding a better combination of predictor variables obviously helps improve forecasting accuracy if those variables contain different information and jointly provide useful signals. Since there are no a priori variable selection procedures available, our approach is to try all possible combinations of variables in order to find a better combination of variables that are helpful and stable in forecasting recession probabilities. Although knowing the best single predictor is helpful, it is unlikely that we can obtain some insights about a better combination of variables by just looking at forecasting performance of single predictors because those variables that show relatively good forecasting performance tend to contain similar information. The second issue is how to translate signals into a probability measure between 0 and 1, which is related to a shape of F ( ). In order to guarantee that F ( ) is monotonically increasing and takes values between 0 and 1, we typically use a known cumulative distribution function (CDF). A popular choice is to use either the Standard Normal CDF (a probit model) or the Logistic CDF (a logit model), or some extensions to those (e.g., Chauvet and Potter, 2005; Kauppi and Saikkonen, 2008). However, the shape of F ( ) is not necessarily restricted to typical ones. In principle, the CDF of any continuous probability random variable will suffice. Differences in the shape of CDFs can be characterized in terms of skewness and excess kurtosis. It should be noted that it is not appropriate to use terms skewness and excess kurtosis here because we are not talking about characteristics of underlying statistical distributions, but just utilizing functional forms. However, for expositional simplicity, I will use those terms in describing 5

6 F (x) 1 F (x) x 0 x (a) Excess Kurtosis > 0 and Skewness = 0 (b) Excess Kurtosis > 0 and Skewness > 0 Figure 1: Role of Skewness and Excess Kurtosis the shape of a CDF. The consequence of allowing skewness and excess kurtosis is illustrated in Figure 1. Higher excess kurtosis makes a CDF steeper around the median of β x, as depicted in the left panel of Figure 1. On the other hand, allowing non-zero skewness makes a CDF asymmetric around F (β x) = 0.5, as illustrated in the right panel of Figure 1. With zero skewness, F (x) = 1 F ( x) for any x R. However, non-zero skewness implies F (x) 1 F ( x). 5 In order to understand what features of a CDF are helpful in improving forecasting accuracy, I will consider 6 different CDFs. Table 1 summarizes characteristics of the CDFs considered. In addition to the Standard Normal CDF and the Logistic CDF, I consider Student-t, Laplace, Gumbel, and Type III Generalized Extreme Value (GEV3). All location parameters and scale parameters, if applicable, are set to 0 and 1, respectively. These CDFs are chosen and configured to incorporate non-zero skewness and/or higher excess kurtosis that the Standard Normal CDF does not have. The first four CDFs have zero skewness and differ in terms of degree of excess kurtosis. The Logistic CDF has excess kurtosis of 1.2. The degrees of freedom parameter for Student-t is set to be 6.5, such that its excess kurtosis equals 2.4. The excess kurtosis of Laplace is 3. The last two CDFs also have positive skewness. For the Gumbel CDF, skewness is equal to and excess kurtosis is 2.4. The shape parameter of the Type III GEV, s, is set such that its excess kurtosis equals 3 (i.e., s = ). The associated skewness becomes Thus, for a given set of predictor variables, I will be able to infer the importance of allowing for excess kurtosis 5 In general, changing location and scale parameters does not affect forecasting results and they are fixed to avoid an identification problem. Changing the location parameter just changes the estimate of the constant term and changing the variance just results in different scaling of coefficients β. 6

7 Table 1: List of the Cumulative Distribution Functions Considered Type of CDF F (x) Skewness Excess Kurtosis 1 x ( ) u 2 Standard Normal exp du 0 0 2π 2 Logistic exp(x) 1 + exp(x) Student-t with ν = 6.5 Laplace x Γ( ν+1 2 ) νπγ( ν 2 ) ) (1 (ν+1) + u2 2 du ν 1 [1 + sgn(x) {1 exp ( x )}] Gumbel 1 exp { exp( x)} Type III GEV with s = { exp (1 + sx) 1/s} Note: All location parameters are set to be 0 and all scale parameters are set to be 1. ν represents degrees of freedom parameter for the Student-t distribution. Γ( ) is a gamma function. Excess kurtosis of Student-t is given by 6/(ν 4) for ν > 4. Skewness of the Gumbel distribution is 12 6ζ(3), where π 3 ζ( ) is a zeta function. s denotes a shape parameter of the generalized extreme value distribution. For the Type III GEV, skewness is given by Γ(1 3s)+3Γ(1 s)γ(1 2s) 2(Γ(1 s))3 (Γ(1 2s) (Γ(1 s)) 2 ) 3/2 Γ(1 4s) 4Γ(1 s)γ(1 3s)+6Γ(1 2s)(Γ(1 s)) 2 3(Γ(1 s)) 4 (Γ(1 2s) (Γ(1 s)) 2 ) 2. and excess kurtosis is give by in F ( ) by looking at results based on the first four CDFs. Furthermore, comparing the Gumbel with the Student-t or the Type III GEV with the Laplace enables us to obtain some insight on whether allowing positive skewness is helpful in improving forecasting performance Data Table 2 lists all 32 monthly variables considered in this paper. The sample period starts from January 1960 and ends at December It is chosen to maximize data availability and to include as many recession episodes as possible. As a result, the sample covers 8 post-war recessions. This is important in choosing a set of predictor variables that are robust. It is not so difficult to find a particular variable combination that can explain previous recessions ex post. However, there is no guarantee that a good predictor for a particular recession works well in predicting another one. The data set contains the term spreads, the credit spreads, various interest rates, employment 6 It is also possible to introduce negative skewness by using the Type III GEV. However, preliminary estimation indicated that forecasting performance based on the Type III GEV with negative skewness is inferior to others in general. So, I do not report their results. 7

8 Table 2: List of Variables Predictor Description Info. Lag Interest Rates 0 FF Federal Funds rate 0 3M 3-month Treasury Bill rate 0 5Y 5-year Treasury Bond rate 0 10Y 10-year Treasury Bond rate 0 AAA Moody s corporate bond yield, AAA 20 years or longer 0 AA Moody s corporate bond yield, AA 20 years or longer 0 A Moody s corporate bond yield, A 20 years or longer 0 Term Spreads TS10YFF 10Y-FF Treasury term spread 0 TS10Y3M 10Y-3M Treasury term spread 0 TS10Y5Y 10Y-5Y Treasury term spread 0 Credit Spreads CSAAA AAA - 10Y spread 0 CSAA AA - 10Y spread 0 CSA A - 10Y spread 0 Employment Data EMP Non-agricultural employment (log-differenced) 0 CEMP Civilian employment (log-differenced) 0 UICLAIM Initial unemployment insurance claims (log-differenced) 1 UNEMP Unemployment rate 0 UNEMPD Changes in unemployment rate 0 HOURS Average weekly hours in manufacturing (log-differenced) 0 Stock Price Indices DJ30 Dow Jones 30 average (% changes over 3 months) 0 SP500 S&P 500 stock price index (% changes over 3 months) 0 Monetary Aggregates M0 Monetary base (log-differenced) 1 M1 M1 (log-differenced) 1 M2 M2 (log-differenced) 2 Other Macroeconomic Variables CLI11 Composite leading indicators (11 series, 1987=100, log-differenced) 1 CPI CPI, all urban, all items (log-differenced) 1 EXP Consumer expectation ( = 100) 0 EXPD Changes in consumer expectation 0 HOUSE New private housing units authorized by building permits (log-differenced) 1 VENDOR Vendor performance (slower deliveries diffusion index, %) 0 INCOME Personal income less transfer payments (log-differenced) 2 IP Industrial production (log-differenced) 1 SALES Manufacturing & trade sales (log-differenced) 1 Note: Information lag is measured at the end of month. Strictly speaking, those employment data with zero information lag and vendor performance are not available at the end of the month. However, they will be available at the very beginning of the next month and there are virtually no considerable lags. Thus, they are categorized in the zero information lag variable. 8

9 data, stock price indices, monetary aggregates, and other macroeconomic variables. Most of those are investigated in the earlier studies or used in constructing a composite leading indicator. It also includes variables, to which the NBER Business Cycle Dating Committee pays particular attention in deciding business cycle peaks and troughs. Since Kane (2008) documents the usefulness of employment data in predicting occurrence of recessions, the data set includes employment-related variables as well. Recently, King, Levin, and Perli (2007) report that, in the period of the Great Moderation, the credit spread on AA-rated firms has particularly good forecasting performance. However, the corporate bond yields used in their study (maturity of 5-year and 10-year) do not cover the entire sample. Since there are not many recessionary periods, especially after the 1980 s, using shorter sample periods may have considerable effects on evaluating forecasting performance. Thus, I have decided not to include the credit spreads that King, Levin, and Perli (2007) have used. Instead, as a crude proxy, the data set include spreads between Moody s AAA-, AA-, or A- rated corporate bond yield (20 years or longer) and the 10-year Treasury bond yield. 7 When we focus on financial variables as predictors, we do not need to consider a gap between when an observation is made and when it is available for forecasting. However, as shown in Table 2, some series are not reported immediately and we need to take account of the information lag, in order to accurately assess forecasting models. In this paper, any variable z t represents the latest data on z available at month t, instead of an observation at month t. For example, industrial production (IP) has 1 month of the information lag. So, IP 2000:01 refers to the industrial production data on December Because the total number of models increases exponentially as the number of variables included (k) increases, I will restrict my attention to k 3. This will result in examining a total of 3 k=1 32! k!(32 k!) 6 = forecasting models. 2.3 Evaluating Forecasting Performance In order to evaluate various forecasting models, I will primarily focus on out-of-sample results. Hansen (2008) shows that in-sample and out-of-sample fits are negatively correlated, which implies 7 Since the 20-year Treasury bond yield data has discontinuity between January 1987 and September 1993, I use the 10-year T-bond rate, instead. Thus, precisely speaking, this credit spread is a combination of the true credit spread and the term spread. 9

10 that good in-sample performance is not a useful indicator of out-of-sample accuracy and that relying on in-sample fit is highly misleading. This over-fitting problem is particularly important, since the likelihood of obtaining spurious results is more pronounced when we search a large number of alternative models. For this reason, I will focus on recursive out-of-sample forecasting evaluation and I will use cross validation as a robustness check. For recursive (pseudo) out-of-sample forecasting exercises, the out-of-sample prediction starts from January 1989 and ends at the end of the full sample. The out-of-sample period covers the last three recessions in the U.S. economy. By adding observations one by one, I estimate a forecasting model again to produce a forecast for the next month. In reality, when we forecast future recession probability, we are not certain about the true state of the economy in recent months. It is because the decision of the NBER Business Cycle Dating Committee typically involves substantial time lag. To be realistic, I assume that forecasters do not know the true state of the economy for a year and assume that y t = y t 1 = = y t 11 = y t 12. In other words, a forecaster assumes that the economy is in the same state as a year ago. Following Clements and Galvão (2006), I will use three different measures for evaluating accuracy of out-of-sample recession probability predictions. The first measure is the probabilityanalogue of mean squared error, the quadratic probability score (QPS), which is commonly used in evaluating probability forecasts. The QPS is defined as QP S = 2 T T (ˆp t y t ) 2, (3) t=1 where ˆp t is the recession probability forecast month t. The QPS takes values between 0 and 2 and smaller value indicates more accurate forecasts. The second measure is the log probability score (LPS), which is defined as LP S = 1 T T { yt log( ˆp t ) + (1 y t ) log(1 ˆp t ) }. (4) t=1 The LPS ranges from 0 to + and a smaller value corresponds to more accurate predictions and penalizes larger mistakes more heavily than the QPS. 10

11 The last measure is the Kuipers Score (KS), which is given by KS = T t=1 y t1 [(ˆpt>0.5)] T t=1 y t T t=1 (1 y t)1 [(ˆpt>0.5)] T t=1 (1 y, (5) t) = hit rate false rate, (6) where 1 [ ] is an indicator function that equals 1 if its argument is true and 0 otherwise. The KS calculates the difference between the hit rate and the rate of false signals by using 50% probability of recession as a cutoff. The KS takes values between 1 and 1. A score of 1 corresponds to making perfect predictions. The KS evaluates the recession predictions from a slightly different aspect, compared with other two measures. Even when recession predictions never show strong indications (say, higher than 50% probability), it is possible to have seemingly good results based on the QPS and LPS. The KS discounts such weak predictions. In this sense, the KS captures the strength and accuracy of predictions by using the 50% probability cutoff. There is a potential problem of just relying on the recursive out-of-sample exercises described above, especially in the context of recession probability forecasting. Since there are not many recession episodes in the out-of-sample period, it is possible to select a forecasting model that has particularly good performance for the last three recessions, but not a next recession. In order to robustify the results, I will also carry out a cross-validation type exercise, called leaving 2-years out. The detailed procedures of the leaving 2-years out exercises are as follows. Let S = {(y t, x t h ) : t = 1,, T } denote a full sample and L τ = {(y t, x t h ) : t = τ 12,, τ + 12} represent a set of excluding observations. For each τ = 13,, T 12, (i) Take E τ = S \ L τ as a training sample. (ii) Estimate parameter values β τ based on E τ. (iii) Make a prediction for y τ by using β τ and x τ h and store it. (iv) Repeat steps (i) (iii). Then calculate the QPS, LPS, and KS based on {ŷ τ } T 12 τ=13. It is necessary to find a forecasting model whose forecasting performance is stable over time and is robust to different recessions. The recursive out-of-sample forecasting exercises seem to be 11

12 a more realistic setup. However, at the same time, it could be vulnerable to serially correlated errors. Given the conditional independence assumption, the leaving 2-years out exercise provides a completely valid evaluation and hopefully picks up a forecasting model whose performance is robust across different recession episodes. For this reason, it is very important to draw conclusions based on two different ways of evaluating forecasting performance. 3 Results 3.1 Univariate Probit Models First, we will start off by looking at forecasting performance within a class of univariate probit models of predicting 6-month-ahead recession probabilities (h = 6) as a benchmark. Table 3 summarizes the out-of-sample forecasting performance of univariate probit models. Overall, the out-of-sample forecasting performance evaluated by the QPS and LPS shows broadly similar patters for the rankings of predictor variables. While the QPS selects the composite leading indicator (CLI11) as the best predictor within univariate probit models, the LPS chooses the term spread between 10-year Treasury bond yield and the Federal Funds rate (TS10YFF) as the best predictor. Based on both the QPS and LPS, 3-month change in S&P 500 (SP500) appears to be one of best predictors. It is important to point out that these variables outperform the widely used recession probability predictor, the term spread between 10-year and 3-month Treasury yields (TS10Y3M). Given the forecasting horizon of 6 months, superiority of TS10YFF over the conventionally used TS10Y3M is true not only in the out-of-sample forecasts, but also in the entire sample (detailed in-sample results are not reported here). In contrast to better performance of these term spreads the credit spreads (CSA, CSAA, and CSAAA) have poor forecasting performance. Among other variables, CPI inflation (CPI) and University of Michigan s index of consumer expectation (EXP) have relatively good performance based on both the QPS and LPS. Some of employment related variables, such as changes in unemployment rate (UNEMPD) and growth rate of the initial unemployment insurance claims (UICLAIM), might contain useful information. However, other employment related variables, those variables that the NBER Business Cycle Dating Committee is paying attention to, and monetary aggregates have considerably poorer results. 12

13 Table 3: Variable Rankings with Univariate Probit Models QPS Ranking LPS Ranking KS Ranking CLI TS10YFF TS10YFF TS10YFF CLI TS10Y3M SP SP UNEMPD TS10Y3M TS10Y3M EXPD CPI EXP EMP EXP CPI IP UNEMPD UICLAIM SALES M UNEMPD UNEMP FF EXPD CEMP EXPD EMP HOURS UICLAIM IP M IP SALES A EMP UNEMP FF UNEMP CEMP HOUSE A HOURS AA SALES M AAA CEMP A VENDOR AA FF M AAA HOUSE M HOURS AA Y M AAA Y HOUSE VENDOR CSA Y M CSAA Y M CSAAA VENDOR INCOME M CSAAA Y CLI CSAA Y EXP CSA CSA CPI M CSAA UICLAIM INCOME M INCOME TS10Y5Y CSAAA TS10Y5Y M TS10Y5Y SP Although the rankings of out-of-sample forecasting accuracy based on the QPS and LPS are broadly consistent each other, some discrepancy results from the fact that the LPS penalizes larger mistakes more heavily. For instance, the levels of short-term interest rate, such as the Federal Funds rate (FF) and the 3-month Treasury bill yield (3M) are somewhat informative based on the QPS, the LPS suggests that they are relatively poor predictors. Figure 2 compares 6-month-ahead out-of-sample predictions that are selected as the best based on the QPS and LPS. The shaded areas indicate the NBER recession periods. Visual inspection of these picture suggests a potential pitfalls in relying on one particular measure of forecasting accuracy. Although predictions from CLI11 have the smallest errors in terms of the quadratic loss, 13

14 1 Univariate Probit with CLI Univariate Probit with TS10YFF Figure 2: Out-of-Sample Predictions from Univariate Probit Note: The shaded areas represent the NBER recessions. Horizontal axes measures recession probabilities. they are not useful forecasts at all in practice, since they appear to be just noisy signals and they do not show clear contrasts between recessions and expansions. On the other hand, out-of-sample predictions with TS10YFF have better distinctions about the state of the economy. Although overall performance is better than others, there are a couple of issues with predictions with TS10YFF. For the recession, the predictions made by the term spreads miss the timing of the recession and they do not have strong signals. This difficulty is also noted in other studies. See Stock and Watson (2003) for more detailed discussions on this issue. Furthermore, the magnitude of the recession signal is not strong enough, so that it is difficult to distinguish between a true signal and a false alarm. Peaks in the recession probability predictions never exceed 50%. For example, it 14

15 is extremely difficult in real time to distinguish these two humps during the 1990 s from those for the true signal for the 2001 recession. Overall pattern of this out-of-sample predictions is promising. However, it would be better if predictions exhibit stronger signals during true recessions. This gives us an incentive to pay attention to another measure of forecasting accuracy, the KS, which takes account of the accuracy and strength of predictions. The out-of-sample forecasting performance evaluated by using the KS gives us a completely different picture. Most predictor variables have zeros for the KS. In other words, they usually do not give us strong predictions about the occurrence of future recessions or correct predictions are largely offset by false predictions. In the worst case, we will get more false signals, which indicate more than 50% probability of a recession during an expansion, than correct ones. Such a situation happens to many of predictors that are considered to be good based on the QPS and LPS, such as CLI11, SP500, EXP, and CPI. 3.2 Other Univariate Models Now we will look at the performance of univariate models based on alternative CDFs, which allow positive skewness and/or excess kurtosis. Table 4 summarizes the ranking of univariate models based on out-of-sample fit. 8 As in the probit models, the QPS and LPS suggest that those models with CLI11, TS10YFF, SP500, and TS10Y3M have relatively good forecasting performance. For CLI11, while adding skewness marginally improves forecasting accuracy in the out-of-sample forecasting exercises compared to the probit model, allowing higher excess kurtosis alone deteriorates performance. Especially, the CLI11 with Laplace, which has the highest excess kurtosis with zero skewness is the worst among the models with CLI11. In fact, it is worse than the second-best predictor in probit models, TS10TFF. We can observe a similar pattern in those models with SP500. Within the class of univariate models with a term spread measure, Normal works better than alternative CDFs. In fact, according to the LPS, Normal with TS10YFF has the best result. There are some role played by using non-normal CDFs in improving forecasting accuracy. It is not easy to generalize the role of different functional forms and the benefit of allowing skewness 8 Due to space limitation, hereafter top 20 forecasting models are shown. All results are available from the author upon request. 15

16 Table 4: Top 20 Univariate Models QPS Ranking LPS Ranking KS Ranking CDF x 1 QPS CDF x 1 LPS CDF x 1 KS Gumbel CLI Normal TS10YFF Laplace TS10Y3M GEV3 CLI GEV3 CLI Laplace CLI Normal CLI Gumbel CLI Laplace EXP Logistic CLI Normal CLI Gumbel SP Student-t CLI GEV3 TS10YFF GEV3 SP Normal TS10YFF Logistic TS10YFF Laplace SP GEV3 TS10YFF Student-t CLI Normal TS10YFF Logistic TS10YFF Logistic CLI GEV3 TS10YFF Gumbel TS10YFF Student-t TS10YFF Logistic TS10YFF Student-t TS10YFF Laplace TS10YFF Gumbel TS10YFF Gumbel SP Laplace CLI Student-t TS10YFF Laplace CLI Gumbel TS10YFF Laplace TS10YFF Laplace TS10YFF Gumbel SP Normal TS10Y3M GEV3 SP GEV3 SP GEV3 TS10Y3M Normal SP Normal SP Gumbel TS10Y3M Student-t SP Logistic SP Logistic TS10Y3M Logistic SP Student-t SP Student-t TS10Y3M Laplace SP Laplace SP GEV3 EXP Normal TS10Y3M Gumbel TS10Y3M Gumbel EXP GEV3 TS10Y3M Normal TS10Y3M Laplace UNEMPD and/or excess kurtosis may be variables specific. However, we can see clear effects of using non- Normal CDFs if we look at the ranking based on the KS. Within the class of univariate models, most of models scores non-positive values for the KS. In other words, most of predictors do not make correct and strong predictions, in terms of the 50% probability cutoff. However, by using non-normal CDF we can improve the KS. There are 6 models with positive values for the KS. Especially, using Laplace is likely to contribute to producing stronger and more accurate signals, net of false alarms. Especially, CLI11, EXP and SP500 all have negative values for the KS in the probit models. Using non-normal CDF can alter bad performance into better predictions. 3.3 Bivariate Models Now we turn out attention to forecasting results based on bivariate models in order to see the importance of additional information. Table 5 shows the top 20 bivariate forecasting models based on the QPS, LPS, and KS. The combination of TS10YFF and SP500 is the best predictor variables based on the QPS and LPS. Both the QPS an LPS suggest that it is important to include a measure of term spread (either TS10YFF or TS10Y3M) and the relative importance of CLI11 decreases. Interestingly, there are 16

17 Table 5: Top 20 Bivariate Models QPS Ranking LPS Ranking KS Ranking CDF x1 x2 QPS CDF x1 x2 LPS CDF x1 x2 KS Gumbel TS10YFF SP Normal TS10YFF SP Logistic EXP UNEMP GEV3 TS10YFF SP Logistic TS10YFF SP Student-t EXP UNEMP Normal TS10YFF SP Student-t TS10YFF SP Laplace EXP UNEMP Student-t TS10YFF SP GEV3 TS10YFF SP Gumbel TS10Y3M SP Logistic TS10YFF SP Laplace TS10YFF SP GEV3 TS10Y3M SP Laplace TS10YFF SP GEV3 TS10YFF EMP Normal TS10Y3M SP Gumbel TS10Y3M SP Gumbel TS10YFF SP Logistic TS10Y3M SP GEV3 TS10Y3M SP Normal TS10YFF EMP Student-t TS10Y3M SP Normal TS10Y3M SP Logistic TS10YFF EMP Laplace TS10Y3M SP Logistic TS10Y3M SP Student-t TS10YFF EMP Normal TS10Y3M EMP Student-t TS10Y3M SP Gumbel TS10Y3M EXP GEV3 TS10Y3M EMP Laplace TS10Y3M SP Gumbel TS10YFF EMP Gumbel TS10Y3M EMP Gumbel CLI11 SP Gumbel TS10Y3M SP Laplace UICLAIM SP GEV3 CLI11 SP GEV3 TS10Y3M SP Gumbel EMP SP Laplace EXP UNEMP Normal TS10Y3M SP GEV3 EMP SP Normal TS10YFF EMP Laplace TS10YFF EMP Normal EMP SP GEV3 TS10YFF EMP GEV3 TS10Y3M EXP Student-t EMP SP Normal CLI11 SP Logistic TS10Y3M SP Logistic EMP SP Logistic TS10YFF EMP Student-t TS10Y3M SP Logistic CLI11 SALES Student-t TS10YFF EMP Gumbel CLI11 SP Student-t CLI11 SALES

18 some variables that have relatively poor performance in univariate models and produce good outof-sample forecasting accuracy, together with the term spread measure. According to the rankings based on the univariate models, SP500 and EMP are not a useful single predictor. Especially, EMP has relatively poor out-of-sample forecasting performance. However, all three measures of forecasting accuracy suggest that they are an important companion variable to other variables. In fact, that the best bivariate model is not a combination of the two best single predictors (CLI11 and TS10YFF). This suggests that the univariate ranking is not a helpful guide for choosing multiple predictors. The importance of SP500 together with a term spread measure is consistent with the finding of Estrella and Mishkin (1998). In our bivariate models, even without being combined with the term spread, some bivariate models that contain SP500 perform relatively well. However, it should be mentioned that King, Levin, and Perli (2007) do not find superiority of SP500 in conjunction with the term spread. Rather, they report that a combination of variables, which have better performance in univariate models, also perform better in bivariate models. 9 Based on the QPS, allowing positive skewness and excess kurtosis (Gumbel and GEV3) outperform the probit counterparts. However, introducing only excess kurtosis (Logistic, Student-t, and Laplace) worsens the QPS, compared with the probit. However, for the best bivariate models with TS10YFF and SP500, utilizing non-normal CDFs deteriorates forecasting accuracy, based on the LPS. In other words, it tends to some improvements in the sense of the quadratic loss, but at the same time, it is likely to produce larger mistakes. Especially, positive skewness makes the deterioration worse. Although forecasting performance generally improves by adding one more predictor, probably the biggest gain appears in the KS. In univariate models, only 6 models out of 192 have positive scores. In bivariate models, 294 models out of 3168 have at least positive values for the KS. Furthermore, the magnitude of the KS improves significantly. This means that by incorporating additional variables, we are likely to produce more accurate and stronger signals. Although the variable combination of TS10YFF and SP500 is the best based on the QPS and LPS, it does 9 This could be because of a couple of reasons. First, it could be attributed to the difference in periods used for the out-of-sample forecasting exercises and forecasting horizon. Second, it might be because of the fact that my data set does not include the credit spread measures that they use and perform quite well in their univariate models. Finally, it could be due to the difference in evaluating out-of-sample forecasting performance. They look at average out-of-sample predictions over two test periods, the 2001 recession and the post-2001 expansion. 18

19 not perform well in terms of the KS (KS = for Normal and KS = for Gumbel). Instead of using TS10YFF, the term spread with 3-month Treasury bill yields, together with SP500, gives us better results for the KS and provides consistently good performance across different forecasting accuracy measures. Relatively consistent performance can be also observed in the variable combination of TS10Y3M and EMP as well. 3.4 Trivariate Models and Overall Rankings Now we move on to the overall forecasting performance, including all of univariate, bivariate, and trivariate models with 6 different CDFs. Tables 6 lists the top 20 forecasting models out of forecasting models, based on different criteria and Figure 3 summarizes and compares overall performance of selected models. According to the out-of-sample forecasting performance, based on the QPS and the LPS, the Laplace with TS10YFF, SP500, and EMP is the best forecasting model. Regardless of the functional form used, this variable combination records outstanding out-of-sample performance. This results is consistent with the fact that in bivariate models, SP500 and EMP are found to be good companion variables to TS10YFF. Other models with TS10YFF, SP500, and VENDOR (vendor performance index), and those with TS10Y3M, SP500, and EMP also show relatively good forecasting performance based on the QPS and LPS. For the variable combination of TS10YFF, SP500, and EMP, there are some roles played by positive skewness and/or excess kurtosis in improving the forecasting accuracy. In terms of the QPS, allowing higher excess kurtosis without positive skewness is likely to outperform the probit counterpart. With the LPS, it is hard to see some clear patterns but there is non-trivial difference between the Laplace model and the probit model. However, it the term spread measure is replaced with TS10Y3M, the role of non-normal CDFs is completely flipped. For these models with TS10Y3M, SP500, and EMP, allowing additional features in the functional form deteriorates forecasting accuracy. In other words, the differences in the measures of the term spread (and interactions with other variables) affects the usefulness of the different functional form. Although there is favorable evidence that allowing excess kurtosis and skewness help improve forecasting accuracy, such a possibility can be variable specific. Out-of-sample forecasting accuracy evaluated by the KS suggests different ranking of models. 19

20 Table 6: Top 20 Models from Out-of-sample Forecasting Exercises QPS Ranking LPS Ranking CDF x 1 x 2 x 3 QPS CDF x 1 x 2 x 3 LPS Laplace TS10YFF SP500 EMP Laplace TS10YFF SP500 EMP Student-t TS10YFF SP500 EMP GEV3 TS10YFF SP500 EMP Logistic TS10YFF SP500 EMP Logistic TS10YFF SP500 EMP GEV3 TS10YFF SP500 EMP Normal TS10YFF SP500 EMP Normal TS10YFF SP500 EMP Student-t TS10YFF SP500 EMP Gumbel TS10YFF SP500 EMP Gumbel TS10Y3M SP500 EXP Gumbel TS10YFF SP500 VENDOR Gumbel TS10YFF SP500 EMP GEV3 TS10YFF SP500 VENDOR GEV3 TS10Y3M SP500 EXP Student-t TS10YFF SP500 VENDOR GEV3 TS10Y3M SP500 EMP Logistic TS10YFF SP500 VENDOR Normal TS10Y3M SP500 EMP Gumbel TS10YFF SP500 CEMP Gumbel TS10Y3M SP500 EMP Normal TS10Y3M SP500 EMP Normal TS10Y3M SP500 EXP Laplace TS10YFF SP500 VENDOR Logistic TS10Y3M SP500 EMP Gumbel TS10YFF SP500 TS10Y3M Student-t TS10Y3M SP500 EMP Gumbel FF SP500 5Y Gumbel TS10YFF SP500 EXP Normal TS10YFF SP500 VENDOR Student-t TS10YFF SP500 VENDOR GEV3 TS10YFF SP500 CEMP Logistic TS10YFF SP500 VENDOR Gumbel TS10YFF SP500 CLI GEV3 TS10YFF SP500 CEMP Gumbel TS10YFF SP500 UNEMPD Logistic TS10Y3M SP500 EXP Gumbel TS10YFF SP500 EXP Student-t TS10YFF SP500 CEMP KS Ranking CDF x 1 x 2 x 3 KS Gumbel TS10Y3M SP500 EMP GEV3 TS10Y3M SP500 EMP Logistic TS10Y3M SP500 EMP Student-t TS10Y3M SP500 EMP Laplace UNEMP EXP EMP Logistic UNEMP SP500 EMP Student-t UNEMP SP500 EMP Normal TS10Y3M SP500 EMP Laplace UNEMP SP500 EMP Gumbel UNEMP SP500 EMP GEV3 UNEMP SP500 EMP Laplace UNEMP EXP M GEV3 UNEMP EXP EMP Normal UNEMP EXP EMP Laplace TS10Y3M SP500 EMP Laplace UNEMP EXP UNEMPD Laplace UNEMP EXP IP Student-t UNEMP EXP EMP Logistic UNEMP EXP EMP Laplace UNEMP SP500 EXP

21 TS10YFF + SP500 + EMP TS10YFF + SP500 + VENDOR TS10Y3M + SP500 + EMP Others Gumbel 0.31 LPS GEV3 Laplace Student t Logistic Normal Logistic Normal GEV3 Student t Gumbel Laplace Student t Logistic Laplace Gumbel Normal GEV QPS Figure 3: Overall Performance of Selected Models in Out-of-sample Forecasting Exercises Note: Horizontal axis takes QPS and vertical axis measures the LPS. For both measures, the smaller the values are, the better the forecasting performance is. Thus, a southwest corner of the plot corresponds to the best forecasting model based on the QPS and the LPS. Top 20 models based on the QPS and LPS are plotted. 21

22 Table 7: Importance of the Two Margins Models QPS LPS KS Normal Laplace GEV3 Normal Laplace GEV3 Normal Laplace GEV3 Univariate * Bivariate * Trivariate * * Note: The univariate model includes TS10YFF. The bivariate model refers to models with TS10YFF and SP500 and the trivariate models add EMP to the bivariate models. Asterisks indicate the best model in each of univariate, bivariate, and trivariate models, based on different criteria. The best models according to the KS utilize information contained in TS10Y3M, SP500, and EMP. Especially, the one with Gumbel achieves the best result. For this combination of variables, positive skewness and excess kurtosis appear to be very helpful to improve the out-of-sample forecasting accuracy in terms of the KS. Good forecasting performance based on the QPS and LPS usually is not supported by the KS performance and vice versa. Except for the combination of TS10Y3M, SP500, and EMP, the variable combinations that are ranked relatively high in the KS ranking do not have good out-of-sample forecasting performance in terms of the QPS or LPS. In other words, making strong predictions during recessions involves risks of making positive false signals during expansions, which worsens the QPS and LPS. In this sense, the variable combination TS10Y3M, SP500, and EMP might be preferred since it has relatively stable performance across different measures of forecasting accuracy. 3.5 Discussion Forecasting performance based on all models indicates that the term spread measure (TS10YFF) is one of the most important variables, as widely documented in other studies. However, there are two margins to improve forecasting performance. One is to include additional predictors. By combining different information, we can increase accuracy of recession probability predictions. As demonstrated, a particular combination of variables greatly outperforms a typical univariate probit model with TS10YFF. Bivariate results suggest the importance of the stock price index (SP500) as a companion variable to the term spread measure. In addition, among the all combinations of variables considered in this paper, augmenting the term spread measure by SP500 and EMP results in the best out-of-sample forecasting performance. Another margin is to change a functional form of F ( ), in order to improve how to translate signals to a recession probability measure. 22

23 1 0.9 Univariate Probit Trivariate Probit Difference between Trivariate Laplace and Trivariate Probit Figure 4: Comparison of Out-of-Sample Predictions Note: Shaded areas indicate the NBER recession episodes. The top panel compares predictions from the univariate probit model with TS10YFF with those from the trivariate model with TS10YFF, SP500, and EMP. The bottom panel shows differences in predictions by changing the functional form. The bottom panel plots the difference between the trivariate Laplace and the trivariate probit. It is important to understand how these two margins contribute to better forecasting performance. Table 7 summarizes how forecasting performance improves by changing the two margins and illustrates different effects of the margins. We compare probit models with the Laplace and GEV3 counterparts. Provided that a forecaster knows which one is the best companion variable, including additional predictors has greater impact on improving all three measures of forecasting accuracy. However, it is possible to obtain non-negligible gains from changing the functional form. Although up to the bivariate models, probit specification works better than any other CDFs, changing the functional form has have bigger impacts, once EMP is included in addition to TS10YFF and SP