Estimation and Model Specification for Econometric Forecasting

Transcription

1 Manuel Sebastian Lukas PhD Thesis Estimation and Model Specification for Econometric Forecasting DEPARTMENT OF ECONOMICS AND BUSINESS AARHUS UNIVERSITY DENMARK

2 ESTIMATION AND MODEL SPECIFICATION FOR ECONOMETRIC FORECASTING By Manuel Sebastian Lukas A PhD thesis submitted to School of Business and Social Sciences, Aarhus University, in partial fulfilment of the requirements of the PhD degree in Economics and Business August 2014 CREATES Center for Research in Econometric Analysis of Time Series

3

4 PREFACE This dissertation was written in the period from September 2010 to August 2014 while I was enrolled as a PhD student at the Department of Economics and Business at Aarhus University. During my PhD studies I was affiliated with the Center for Research in Econometric Analysis of Time Series (CREATES), that is funded by the Danish National Research Foundation. I am grateful to the Department of Economics and Business and to CREATES for providing an inspiring, supportive, and friendly research environment, and for the financial support for attending conferences and courses. Parts of this dissertation were written during my research stay at the Rady School of Management at the University of California San Diego (UCSD) from August 2012 to January I thank Allan Timmermann for making this academically, professionally, and personally rewarding experience possible and I thank the Rady School of Management of the hospitality. I am grateful to the Aarhus University Research Foundation (AUFF) and the Department of Business and Social Science at Aarhus University for their financial support in connection for my stay at UCSD. I thank Jack Zhang for all the help and the hospitality during my stay in the United States, and for introducing me to the UCSD graduate student life. I am thankful to all people who have supported me in my research with their advice, comments, and suggestions. My main supervisor Bent Jesper Christensen and my co-supervisor Eric Hillebrand have supported me with guidance, expertise, and encouragement for both my independent research and our joint research projects. I wish to thank all fellow PhD students at Aarhus University for the excellent team spirit, both in academic and in (very) non-academic matters, which has made the past four years a great experiences. I especially wish to thank Rasmus, Heida, Kasper O., Andreas, and Anders L., who have accompanied me in the challenging and exciting transition from Master s to PhD student. During my PhD studies if have enjoyed many welcome breaks from research during coffee breaks, social events, and floorball matches with many of my colleagues, in particular Niels S., Juan Carlos, Anne F., Jonas E., Jonas M., Martin S., Niels H., Mark, Simon, Rune, Anders K., Laurent, Stine, Morten, and Christina. A big thanks goes to Johannes for sharing the LATEX template that is used for this dissertation. I am very grateful to CREATES, especially the Center Director Niels Haldrup and i

5 ii the Center Administrator Solveig Sørensen, for creating a great research environment and for the many interesting PhD courses that were organized by CREATES during my studies. I also wish to thank Niels Haldrup for allowing me to participate three times in the Econometric Game in Amsterdam for Team Aarhus University. I am indebted to my family and friends in Switzerland for their patience, their visits to Denmark, and their amazing hospitality on my visits back home. Last but not least, I am grateful to my girlfriend Tanja for supporting me during the busy and challenging time as PhD student. Manuel Sebastian Lukas Aarhus, August 2014

6 UPDATED PREFACE The predefence took place on September 30, The assessment committee consists of Asger Lunde, Aarhus University, Allan Timmermann, University of California, San Diego, and Christian Møller Dahl, University of Southern Denmark. I wish to thank the members of the committee for their detailed comments. After the predefence the dissertation has been revised to incorporate the changes required by the committee. Additionally, the committee has suggested improvements, some of which are incorporated in this revised version of the thesis. Manuel Sebastian Lukas Copenhagen, January 2015 iii

7

8 CONTENTS Summary vii 1 Bagging Weak Predictors Introduction Bagging Predictors Monte Carlo Simulations Application to CPI Inflation Forecasting Conclusion References Appendix Return Predictability, Model Uncertainty, and Robust Investment Introduction Investment and Confidence Sets Models and Data Empirical Results Conclusion References Frequency Dependence in the Risk-Return Relation Introduction The Empirical Risk-Return Relation Frequency Dependence in the Risk-Return Relation Frequency-Dependent Real-Time Forecasts Conclusion References v

9

10 SUMMARY This dissertation comprises three self-contained chapters with the theme of econometric forecasting as their common denominator. We analyze methods for parameter estimation and model specification of econometric models and apply these methods to macroeconomic and financial time series. Turning to econometric forecasting we shift the focus of econometric modeling from fitting all available data, testing for statistical significance, and testing for correct specification towards fitting future data, i.e., achieving good out-of-sample performance. Applying the classical econometric toolbox for parameter estimation and model specification is not always appropriate for forecasting because a statistically significant relation and good in-sample fit are insufficient to ensure satisfactory forecasting performance. It is therefore important to take into account the very aim of out-of-sample forecasting at the time when the model is estimated and specified. The three chapters in this dissertation each deal with some aspects of estimation and model specification for econometric forecasting with empirical applications to inflation rates, equity premia, and the risk-return relation. The first chapter, "Bagging Weak Predictors", is joint work with Eric Hillebrand. We propose a new bootstrap aggregation, bagging, predictor for situations where the predictive relation is weak, i.e., for situations in which predictors based on classical statistical methods fail to provide good forecasts because the estimation variance is larger than the bias effect from ignoring the relation. In the literature on econometric forecasting, it is often found that predictors suggested by economic theory do not lead to satisfactory forecasting results. Successful forecasting with such predictors requires prediction methods that reduce estimation variance. The bagging method of Breiman (1996) is based on bootstrap re-sampling and it can improve the properties of pre-test and other hard-threshold estimators by reducing the estimation variance. Standard bagging estimators are based on standard t-tests for statistical significance. A statistically significant relation is, however, not sufficient for successful out-ofsample forecasting. We therefore base our new bagging predictor on the in-sample test for predictive ability proposed by Clark and McCracken (2012). The null hypothesis of this test is that the inclusion or the exclusion of a predictor in a forecasting regression leads to equal forecasting performance. Thus, when the test is rejected, we know whether or not to include the predictor. By using the test of Clark and Mcvii

11 viii SUMMARY Cracken (2012), our predictor shrinks the regression coefficient estimate not to zero, but towards the null of the test which equates squared bias with estimation variance. We derive the asymptotic distribution in the asymptotic framework of Bühlmann and Yu (2002) and show that the predictor has a substantially lower the mean-squared error (MSE) compared to standard t-test bagging if a weak predictive relationship exists. Because the bootstrap re-sampling for bagging can be computationally heavy, we derive an asymptotic shrinkage representation for the predictor that simplifies computation of the estimator. Monte Carlo simulations show that our predictor works well in small samples. In the empirical application, we consider forecasting inflation using employment and industrial production in the spirit of the so-called Phillips Curve. This application fits our framework because inflation is notoriously hard to forecast from other macroeconomic variables. In the second chapter, "Return Predictability, Model Uncertainty, and Robust Investment", the model uncertainty in stock return prediction models is analyzed. Empirical evidence suggests that stock returns are not completely unpredictable, see, e.g., Lettau and Ludvigson (2010) for a comprehensive survey. Under stock return predictability, investment decisions are based on conditional expectations of stock returns. The choice of appropriate predictor variables is, however, subject to great uncertainty. In this chapter, we use the model confidence set approach of Hansen, Lunde, and Nason (2011) to quantify the uncertainty about expected utility from stock market investment, accounting for potential return predictability, for monthly data over the sample period 1966: :12 on the US stock market. We consider the popular data set of Welch and Goyal (2008), which contains standard predictor variables used in this literature. For the econometric analysis we take the perspective of a small investor with constant relative risk aversion (CRRA) utility and short-selling constraints. The model confidence set is then applied recursively and, for every month in the out-of-sample period, it identifies the set of models that contains the best model with a given confidence level. The empirical results show that the model confidence sets imply economically large and time-varying uncertainty about expected utility from investment. To analyze the economic importance of this model uncertainty we propose investment strategies that reduce the impact of model uncertainty. Reducing the model uncertainty with these strategies requires lower investment in stocks, but return predictability still leads to economic gains for the small investor. Thus, we conclude that although model uncertainty concerns reduce the share of wealth that investors wish to hold in stocks, it does not prevent them from benefiting from return predictability using econometric models. The third chapter, "Frequency Dependence in the Risk-Return Relation", is coauthored with Bent Jesper Christensen and considers a specification of the risk-return relation that allows for non-linearities in the form of frequency dependence. The risk-return relation is typically specified as a linear relation between stock returns and some measure of the conditional variance, motivated by the intertemporal capital

12 ix asset pricing model (ICAPM) of Merton (1973). Since the empirical analysis in Merton (1980), empirical estimation of the risk-return relation has attracted much attention in the literature. In this chapter we use the band spectral regression of Engle (1974) with the one-sided filtering approach of Ashley and Verbrugge (2008) to allow for frequency dependence in the risk-return relation, which is a feature that cannot be accommodated by a linear model. The combination of one-sided filtering and conditional variances constructed from lagged observations make our estimation approach robust to contemporaneous leverage and feedback effects. For daily returns and realized variances from high-frequency intra-daily data on the S&P 500 from 1995 to 2012 we strongly reject the null hypothesis of no frequency dependence. This finding is robust to changes in the conditional variance proxy. In particular, the rejection of the null hypothesis is strongest when we allow for lagged leverage effects in the conditional variance. Although the risk-return relation is positive on average over all frequencies, we find a large and statistically significant negative coefficient for periods of around one week. Subsample analysis reveals that the negative effect at these frequencies is not statistically significant before the financial crisis, but becomes very strong after July Accounting for the frequency dependence in the risk-return relation can improve the out-of-sample forecasting of stock returns after 2007, but only if the forecasting approach reduces in increased estimation variance from the additional parameters of the band spectral approach. References Ashley, R., Verbrugge, R. J., Frequency dependence in regression model coefficients: An alternative approach for modeling nonlinear dynamic relationships in time series. Econometric Reviews 28 (1-3), Breiman, L., Bagging predictors. Machine Learning 24, Bühlmann, P., Yu, B., Analyzing bagging. The Annals of Statistics 30 (4), Chernov, M., Gallant, R., Ghysels, E., Tauchen, G., Alternative models for stock price dynamics. Journal of Econometrics 116 (1), Clark, T. E., McCracken, M. W., In-sample tests of predictive ability: A new approach. Journal of Econometrics 170 (1), Corsi, F., A simple approximate long-memory model of realized volatility. Journal of Financial Econometrics 7 (2), Corsi, F., Reno, R., Discrete-time volatility forecasting with persistent leverage effect and the link with continuous-time volatility modeling. Journal of Business and Economic Statistics,

13 x SUMMARY Engle, R. F., Band spectrum regression. International Economic Review 15 (1), Gouriéroux, C., Monfort, A., Renault, E., Indirect inference. Journal of Applied Econometrics 8 (S1), S85 S118. Hansen, P. R., Lunde, A., Nason, J. M., The model confidence set. Econometrica 79 (2), Lettau, M., Ludvigson, S., Measuring and modeling variation in the risk- return tradeoff. In: Ait-Sahalia, Y., Hansen, L.-P. (Eds.), Handbook of Financial Econometrics. Vol. 1. Elsevier Science B.V., North Holland, Amsterdam, pp Merton, R. C., An intertemporal capital asset pricing model. Econometrica, Merton, R. C., On estimating the expected return on the market: An exploratory investigation. Journal of Financial Economics 8 (4), Welch, I., Goyal, A., A comprehensive look at the empirical performance of equity premium prediction. Review of Financial Studies 21 (4),

14 C H A P T E R 1 BAGGING WEAK PREDICTORS Manuel Lukas and Eric Hillebrand Aarhus University and CREATES Abstract Relations between economic variables can often not be exploited for forecasting, suggesting that predictors are weak in the sense that estimation uncertainty is larger than bias from ignoring the relation. In this chapter, we propose a novel bagging predictor designed for such weak predictor variables. The predictor is based on an in-sample test for predictive ability. Our predictor shrinks the OLS estimate not to zero, but towards the null of the test which equates squared bias with estimation variance. We derive the asymptotic distribution and show that the predictor can substantially lower the MSE compared to standard t-test bagging. An asymptotic shrinkage representation for the predictor is obtained that simplifies computation of the estimator. Monte Carlo simulations show that the predictor works well in small samples. In an empirical application we apply the new predictor to inflation forecasts. Keywords: Inflation forecasting, bootstrap aggregation, estimation uncertainty, weak predictors. 1

15 2 CHAPTER 1. BAGGING WEAK PREDICTORS 1.1 Introduction A frequent finding in pseudo out-of-sample forecasting exercises is that including predictor variables does not improve forecasting performance, even though the predictor variables are significant in in-sample regressions. For example, there is a large literature on forecast failure with economic predictor variables for forecasting inflation (see, e.g., Atkeson and Ohanian, 2001; Stock and Watson, 2009) and forecasting exchange rates (see, e.g., Meese and Rogoff, 1983; Cheung, Chinn, and Pascual, 2005). Including predictor variables suggested by economic theory, or selected by in-sample regressions, typically does not help to consistently out-perform simple time series models across different sample splits and model specifications. Forecasting failure can be attributed to estimation variance and parameter instability. In this chapter, we focus exclusively on the former. These two causes of forecast failure are, however, often interrelated in practice. If we are unwilling to specify the nature of instability, it is common practice to use a short rolling window for estimation to deal with parameter instability. While a short estimation window can better adapt to changing parameters, it increases estimation variance compared to using all data. In this sense, estimation variance can result from the attempt to accommodate parameter instability, such that our results are relevant for both kinds of forecast failure. This chapter is concerned with reducing estimation variance by bagging pre-test estimators when predictor variables have weak forecasting power. Modeling weak predictors in the framework of Clark and McCracken (2012) leads to a non-vanishing bias-variance trade-off. CM propose an in-sample test for predictive ability, i.e., a test of whether bias reduction or estimation variance will prevail when including a predictor variable. Based on this test, we propose a novel bagging estimator that is designed to work well for predictors with non-zero coefficient of known sign. Under the null of the CM-test, the parameter is not equal to zero, but equal to a value for which squared bias from omitting the predictor variable is equal to estimation variance. In our bagging scheme, we set the parameter equal to this value instead of zero whenever we fail to reject the null. For this, knowledge of the coefficient s sign is necessary. We derive the asymptotic distribution of the estimator and show that for a wide range of parameter values, asymptotic mean-squared error is superior to bagging a standard t-test. The improvements can be substantial and are not sensitive to the choice of the critical value, which is a remaining tuning parameter. We obtain forecast improvements if the data-generating parameter is small but non-zero. If the data-generating parameter is indeed zero, however, our estimator has a large bias and is therefore imprecise. Bootstrap aggregation, bagging, was proposed by Breiman (1996) as a method to improve forecast accuracy by smoothing instabilities from modeling strategies that involve hard-thresholding and pre-testing. With bagging, the modeling strategy is applied repeatedly to bootstrap samples of the data, and the final prediction is obtained by averaging over the predictions from the bootstrap samples. Bühlmann

16 1.1. INTRODUCTION 3 and Yu (2002) show theoretically how bagging reduces variance of predictions and can thus lead to improved accuracy. Stock and Watson (2012) derive a shrinkage representation for bagging a hard-threshold variable selection based on the t-statistic. This representation shows that standard t-test bagging is asymptotically equivalent to shrinking the unconstrained coefficient estimate to zero. The degree of shrinkage depends on the value of the t-statistic. Bagging is becoming a standard forecasting technique for economic and financial variables. Inoue and Kilian (2008) consider different bagging strategies for forecasting US inflation with many predictors, including bagging a factor model where factors are included if they are significant in a preliminary regression. They find that forecasting performance is similar to other forecasting methods such as shrinkage methods and forecast combination. Rapach and Strauss (2010) use bagging to forecast US unemployment changes with 30 predictors. They apply bagging to a pre-test strategy that uses individual t-statistics to select variables, and find that this delivers very competitive forecasts compared to forecast combinations of univariate benchmarks. Hillebrand and Medeiros (2010) apply bagging to lag selection for heterogeneous autoregressive models of realized volatility, and they find that this method leads to improvements in forecast accuracy. Our method requires a sign restriction in order to impose the null. We focus on a single predictor variable, because in this case, intuition and economic theory can be used to derive sign restrictions. For models with multiple correlated predictors, sign restrictions are harder to justify. In the literature, bagging has been applied for reducing variance from imposing sign restrictions on parameters. A hard-threshold estimator with sign restriction sets the estimate to zero if the sign restriction is violated. Gordon and Hall (2009) consider bagging the hard-threshold estimator and show analytically that bagging can reduce variance. Sign restrictions arise naturally in predicting the equity premium, see Campbell and Thompson (2008) for a hard-threshold, and Pettenuzzo, Timmermann, and Valkanov (2013) for a Bayesian approach. Hillebrand, Lee, and Medeiros (2013) analyze the bias-variance trade-off from bagging positive constraints on coefficients and the equity premium forecast itself, and they find empirically that bagging helps improving the forecasting performance. The remainder of the chapter is organized as follows. In Section 1.2, the bagging estimator for weak predictors is presented and asymptotic properties are analyzed. Monte Carlo results for small samples are presented in Section 1.3. In Section 1.4, the estimator is applied to CPI inflation forecasting using the unemployment rate and industrial production as predictors. Concluding remarks are given in Section 1.5.

17 4 CHAPTER 1. BAGGING WEAK PREDICTORS 1.2 Bagging Predictors Let y be the target variable we wish to forecast h-steps ahead, for example consumer price inflation. The variables x is a potential predictor variable that can be used to forecast the target variable y. Let T be the sample size. At time t, we forecast y t+h,t using the scalar variable x t as predictor and a model estimated on the available data. In our framework we consider the simple regression relation y t+h,t = µ + β T x t + u t+h, (1.1) that is used to obtain h-steps ahead forecasts of the variable y, and where β T is a coefficient that depends on the sample size to reflect a weak predictive relation. The focus of our analysis is estimation of the coefficient β T of the predictor variable x. We start with the following assumptions regarding the unrestricted leastsquares estimate of the coefficient, ˆβ T, and the estimator of its asymptotic variance, ˆσ 2,T. To reduce notational clutter we suppress the dependence of the asymptotic variance on the fixed forecast horizon h. Assumption 1.1 T 1/2 ( ˆβ T β T ) d N (0,σ 2 ), (1.2) and let ˆσ 2,T > 0 be a consistent estimator of σ2 <, i.e., ˆσ2,T σ2 0. Given the asymptotic variance from Assumption 1.1, we analyze weak predictors by considering the following parameterization, β T = T 1/2 bσ, (1.3) where we assume that the sign of b is known. Without loss of generality, we assume that b is strictly positive, i.e., si g n(b) = 1. For a given sample of length T and a given forecast horizon h, we start with considering two forecasting models, the unrestricted model (UR) that includes the predictor variable x t and the restricted model (RE) that contains only an intercept. Let ˆµ RE R T and ( ˆµU T, ˆβ T ) be the OLS parameter estimates from the restricted model and the unrestricted model, respectively. The forecasts for y t+h,t from the unrestricted and restricted models are denoted and ŷ U R t+h,t = ˆµU R T + ˆβ T x t, (1.4) ŷ RE t+h,t = ˆµRE T, (1.5) respectively. In practice, we are often not certain whether to include the weak predictor x t in the forecast model or not, i.e., whether RE or UR yields more accurate forecasts. In p

18 1.2. BAGGING PREDICTORS 5 such a situation, it is common to use a pre-test estimator. Typically, the t-statistic ˆτ T = T 1/2 ˆβT ˆσ 1,T is used to decide whether or not to include the predictor variable. Let I(.) denote the indicator function that takes value 1 if the argument is true and 0 otherwise. The one-sided pre-test estimator is ˆβ PT T = ˆβ T I( ˆτ T > c), (1.6) for some critical value c, for example 1.64 for a one-sided test at the 5% level. We focus on one-sided testing because we assumed that the sign of β is known. The hard-threshold indicator function involved in the pre-test estimator introduces estimation uncertainty, and it is not well designed to improve forecasting performance. Bootstrap aggregation (bagging) can be used to smooth the hard-threshold and thereby improve forecasting performance (see Bühlmann and Yu, 2002; Breiman, 1996). The bagging version of the pre-test estimator is defined as ˆβ BG T = 1 B B b=1 ˆβ b I( ˆτ b > c), (1.7) where ˆβ b and ˆτ are calculated from bootstrap samples, and B is the number of b bootstrap replications. The bagging estimator and the underlying t-statistic pre-test estimator are based on a test for β = 0. We use the estimated value of the coefficient, ˆβT, if this null hypothesis can be rejected at some pre-specified significance level, e.g., 5%. However, this test does not directly address the actual question of the model selection decision, i.e., whether or not the coefficient can be estimated accurately enough to be useful for forecasting for the given sample size. Rather, it is a test for whether the coefficient is zero or not. Clark and McCracken (2012) (CM henceforth) propose an asymptotic in-sample test for predictive ability for weak predictors to test whether estimation uncertainty outweighs the predictive power of a predictor in terms of mean-squared error. The null hypothesis equates asymptotic estimation variance and squared bias. In terms of squared bias and variance of the estimator of the coefficient β T, this null hypothesis becomes H 0,C M : lim T E[(β T ) 2 ] = lim T E[( ˆβ T β T ) 2 ]. (1.8) T T Under Assumption 1.1 and the parameterization (1.3), we have that the null hypothesis H 0,C M is true for b 2 σ 2 = σ2. Thus the null hypothesis is true for b = 1 as we have assumed that b is positive. Looking at the distribution of the t-test statistic under the null hypothesis H 0,C M, Assumption 1.1, and using Equation (1.3), we get T 1/2 ˆβT ˆσ 1,T = T 1/2 ( ˆβ T β T ) ˆσ 1,T + T 1/2 T 1/2 bσ ˆσ 1 d,t N (0,1) + 1. (1.9) The distribution under the null is a non-central distribution. This non-central asymptotic distribution is used to obtain critical values c for the t-statistic under the hypothesis H 0,C M following Clark and McCracken (2012).

19 6 CHAPTER 1. BAGGING WEAK PREDICTORS The asymptotic distribution is non-central, because under the null hypothesis the coefficient is not zero. The critical values c are different than for the standard significance test and depend on the sign of b (see Clark and McCracken, 2012, for details). More importantly, imposing the null hypothesis of the CM-test is not achieved by setting β = 0. Therefore we cannot set β = 0 if the CM-test does not reject the null hypothesis. Instead, we impose this null hypothesis, which can be achieved by setting the coefficient to an estimate of the asymptotic variance, β 0,C M = var[ ˆβ T ] = T 1 ˆσ 2,T = T 1/2 ˆσ,T. (1.10) Note that we utilized the sign restriction on b to identify the sign of β 0,C M under the null. This results in the following pre-test estimator based on the CM-test, which we call CMPT (Clark-McCracken Pre-Test). ˆβ C MPT T = ˆβ T I( ˆτ T > c) + T 1/2 ˆσ,T I( ˆτ T c), (1.11) where, for the same confidence level, the critical value c is different from the critical value c used in the standard pre-test estimator (1.6), because the asymptotic distributions of the test statistics differ. The bagging version of the CMPT estimator (1.11), henceforth called CMBG, is defined as ˆβ C MBG T = 1 B B b=1 [ ˆβ b I( ˆτ b > c) + T 1/2 ˆσ,T I( ˆτ b c) ]. (1.12) The first term in the sum is exactly the standard bagging estimator, except for the different critical values. The critical values for C MBG come from the normal distribution N (1,1), while critical values for standard bagging come from the standard normal distribution. The second term in the sum of Equation (1.12) stems from the cases where the null is not rejected for bootstrap replication b. Note that we do not re-estimate the variance under the null, ˆσ 2,T, for every bootstrap sample. The main reason to apply bagging are hard-thresholds, which are not involved in the estimation of ˆσ 2,T, such that there is no obvious reason for bagging the variance estimator Asymptotic Distribution and Mean-Squared Error We have proposed an estimator that is based on the CM-test and better reflects our goal of improving forecast accuracy rather than testing statistical significance. In this section, we derive the asymptotic properties of this estimator to see if, and for which parameter configurations, this estimator indeed improves the asymptotic meansquared error (AMSE). The asymptotic distribution for bagging estimators has been analyzed for bagging t-tests by Bühlmann and Yu (2002), and for sign restrictions by Gordon and Hall (2009). The following assumption on the bootstrapped least-squares estimator ˆβ T is needed for the analysis of the bagging estimators.

20 1.2. BAGGING PREDICTORS 7 Assumption 1.2 (Bootstrap consistency) sup P [T 1/2 ( ˆβ T ˆβ T ) v] Φ(v/σ ) = o p (1), (1.13) v R where P is the bootstrap probability measure. In Assumption 1.2 we assume that the bootstrap distribution converges to the asymptotic distribution of the CLT in Assumption 1.1. Under Assumption 1.2, with a local-to-zero coefficient given by model (1.3), Bühlmann and Yu (2002) derive the asymptotic distribution for two-sided versions of the pre-test and the bagging estimators. The one-sided versions considered in this chapter follow immediately as special cases. Let φ(.) denote the pdf and Φ(.) the cdf of a standard normal variable. Proposition 1.1 (Special case of Bühlmann and Yu (2002), Proposition 2.2) Under Assumption 1.1, and model (1.3) T 1/2 ˆσ 1 PT,T ˆβ T d (Z + b)i(z + b > c), (1.14) and, with additionally Assumption 1.2, T 1/2 ˆσ 1 BG,T ˆβ T d (Z + b)φ(z + b c) + φ(z + b c), (1.15) where Z is a standard normal random variable. The proposition follows immediately from Bühlmann and Yu (2002). The asymptotic distributions depend on the predictor strength b and the critical value c. For the pre-test estimator, the indicator function enters the asymptotic distribution. The distribution of the bagging estimator, on the other hand, contains smooth functions of b and c. Bühlmann and Yu (2002) show how this can reduce the variance of the estimator substantially for certain values of b and c. We adapt this proposition to derive the asymptotic distributions of the estimators CMPT, given by Equation (1.11), and CMBG, given by Equation (1.12). Proposition 1.2 Under Assumption 1.1, and model (1.3) T 1/2 ˆσ 1 C MPT d,t ˆβ T (Z + b)i(z + b > c) + I(Z + b c), (1.16) and, with additionally Assumption 1.2, T 1/2 ˆσ 1 C MBG d,t ˆβ T (Z + b)φ(z + b c) + φ(z + b c) + 1 Φ(Z + b c), (1.17) where Z is a standard normal variable. The proof of the proposition is given in the appendix. The asymptotic distributions are similar to those of the pre-test and bagging estimators (BG and PT), but

21 8 CHAPTER 1. BAGGING WEAK PREDICTORS involve extra terms due to the different null hypothesis. For CMPT, the extra term is simply an indicator function, and for CMBG it involves the standard normal cdf Φ( ). Figures 1.1 and 1.2 show asymptotic mean-squared error, asymptotic bias, asymptotic squared bias, and asymptotic variance of the pre-test and bagging estimators for test levels 5% and 1%, respectively. Note that the t-test and the CM-test use different critical values, c and c. The results for the two different significance levels, 5% and 1%, are qualitatively identical. The effect of choosing a lower significance level is that the critical values increase, and the effects from pre-testing become more pronounced. For the asymptotic mean-squared error (AMSE), we get the usual picture for PT and PTBG (see Bühlmann and Yu, 2002). Bagging improves the AMSE compared to pre-testing for a wide range of values of b, except at the extremes. CMBG compares similarly to CMPT, but shifted towards the right compared to BG and PT. When looking at any given value b, there are striking differences between the estimators based on the CM-test and the ones based on the t-test. Both CMPT and CMBG do not perform well for b close to zero, but the AMSE decreases as b increases, before starting to slightly increase again. For values of b from around 0.5 to 3, CMBG performs better than BG. For values larger than 3 the estimators PT, BG, and CMBG perform similarly and get closer as b increases. Thus, the region where CMBG does not perform well are values of b below 0.5. The asymptotic biases for CMPT and CMBG are largest at b = 0. For all estimators, the bias can be both positive or negative, depending on b. Bagging can reduce bias compared to the corresponding pre-test estimation, in particular in the region where the pre-test estimator has the largest bias. CMPT and CMBG have very low variance for b close to zero, because the CM-test almost never rejects for these parameters. However, as the null hypothesis is not close to the true b in this region, CMPT and CMBG are therefore very biased. As b increases slightly, CMBG has the lowest asymptotic variance for b up to around 3. The asymptotic results show that imposing a different null hypothesis dramatically changes the characteristics of the estimators. The estimator based on the CM-test is not intended to work for b very close to zero. In this case, the standard pre-test estimator has much better properties. For larger b, the CM-based estimators give substantially better forecasting results. These results highlight that the CM-based estimators will be useful for relations where the coefficient is expected to be strictly positive or strictly negative, but too small to exploit with an unrestricted coefficient estimator Asymptotic Shrinkage Representation Stock and Watson (2012) provide an asymptotic shrinkage representation of the BG estimator. This representation, henceforth called BGA, is given by ˆβ BG A T = ˆβ [ ] T 1 Φ(c ˆτ T ) + ˆτ 1 T φ(c ˆτ T ) (1.18)

22 1.2. BAGGING PREDICTORS 9 AMSE Abias PT BG CMPT CMBG b b Squared Abias Avar b b Figure 1.1. Comparison of asymptotic mean-squared error (AMSE), asymptotic bias (Abias), asymptotic square bias (Abias square), and asymptotic variance (Avar) as a function of b for 5% significance level.

23 10 CHAPTER 1. BAGGING WEAK PREDICTORS AMSE Abias PT BG CMPT CMBG b b Squared Abias Avar b b Figure 1.2. Comparison of asymptotic mean-squared error (AMSE), asymptotic bias (Abias), asymptotic square bias (Abias square), and asymptotic variance (Avar) as a function of b for 1% significance level.

24 1.2. BAGGING PREDICTORS BGA CMBGA β^ Figure 1.3. Shrinkage estimators BGA and CMBGA (y-axis) for a given value of the unrestricted parameter estimate ˆβ (x-axis) for σ = 0.2 and 5% level. Dotted line is 45 line. ˆβ BG A T and Stock and Watson (2012, Theorem 2) show under general conditions that T = + o P (1). This allows computation without bootstrap simulation. While bootstrapping can improve test properties, bagging can improve forecasts even without actual resampling. There is no reason to suspect that the estimator based on the asymptotic distribution will be inferior to the standard bagging estimator. Therefore, we consider a version of the bagging estimators that samples from the asymptotic, rather than the empirical, distribution of ˆβ T. We can find closed-form solutions for estimators that do not require bootstrap simulations. The asymptotic version of CMBG is henceforth referred to as CMBGA. Proposition 1.3 (Asymptotic Shrinkage representation) Apply CMBG with the asymptotic distribution of ˆβ T under Assumption 1.2, then ˆβ C MBG A T = ˆβ [ ] T 1 Φ( c ˆτ T ) + ˆτ 1 T φ( c ˆτ T ) + ˆτ 1 T Φ( c ˆτ T ). (1.19) The proof of the proposition is given in the appendix. The representation is very similar to BGA in Equation (1.18), with an extra term for the contribution for the null C MBG A of the CM-test. Note that we can express ˆβ as the OLS estimator ˆβ T T multiplied BG A by a function that depends on the data only through the t-statistic ˆτ T, just like ˆβ. T Figure 1.3 plots BGA and CMBGA against the OLS estimate ˆβ T. The vertical deviation from the 45 line indicates the degree and direction of shrinkage applied by the estimator to the OLS estimate ˆβ T. This reveals the main difference between BGA and CMBGA. Rather than shrinking towards zero, CMBGA shrinks towards ˆσ,T, ˆβ BG

25 12 CHAPTER 1. BAGGING WEAK PREDICTORS which makes a substantial difference for b close to 0. For larger ˆβ T, the CMBGA, and thus CMBG, shrink more heavily downwards than BGA. 1.3 Monte Carlo Simulations The asymptotic analysis suggests that our modified bagging estimator can yield significant improvements in MSE for the estimation of β. This section uses Monte Carlo simulations to investigate the performance for the prediction of y t+h,t in small samples using the estimators presented above. In our linear model (1.1), lower MSE for estimation of β can be expected to translate directly into lower MSE for prediction of y t+h,t. For the Monte Carlo simulations, we generate data from the following model that is designed to resemble the empirical application of inflation forecasting: y t+h,t = µ + β T x t + u t+h u t+h = ɛ t+h + θ 1 ɛ t+h θ h 1 ɛ t+1 x t = φx t 1 + v t ɛ t N (0,σ 2 ɛ ) v t N (0,σ 2 v ). (1.20) We allow for serially correlated errors in the form of an MA(h-1) model. The choice of AR(1) for x t is guided by the model for the monthly unemployment change series selected by AIC. As we vary the sample size, the predictor variable x t modeled as a weak predictor with coefficient β T = T 1/2 bσ. We consider values b {0,0.5,1,2,4}. For b = 1, we are indifferent between estimating β unrestrictedly and no using the predictor variable. For higher (lower) values of b, including the predictor variable should improve (deteriorate) the forecasting performance. Table 1.1 presents an overview of all these methods. We are interested in the small-sample properties and consider sample sizes T {25,50,200}. Furthermore, we set µ = 0.1 and φ = 0.66, which we take from the our empirical example, i.e., monthly changes in unemployment. Additionally, we consider φ = 0.9 to investigate the behavior for more persistent processes. Finally, we consider the forecast horizons h = 1 and h = 6. The MA coefficients are set to θ i = 0.4 i for 1 i h 1, and 0 otherwise. The critical values are taken from the respective asymptotic distribution of both tests for significance levels 5% and 1%. We run 10,000 Monte Carlo simulations and use 299 bootstrap replications for bagging. Columns 2 through 9 of Tables show the MSE for the different estimators listed in Table 1.1. The last two columns show the rejection frequencies for the t-test and CM-test. The MSE is reported in excess of var[u t+h ], which does not depend on the forecasting model, such that the true model with known parameters will have MSE of zero.

26 1.3. MONTE CARLO SIMULATIONS 13 Table 1.1. Forecasting methods for Monte Carlo and empirical application Name Method Formula for forecast ŷ t+h,t RE Restricted Model ˆµ RE T UR Unestricted Model ˆµ U T R T x t PT Pre-Test t-test ˆµ U R T + I( ˆτ T > c) ˆβ T x t BG Bagging t-test ˆµ U T R + B 1 B ˆβ b=1 b I( ˆτ b > c)x t BGA Asymptotic BG ˆµ U T R + ˆβ [ ] T 1 Φ(c ˆτ T ) + ˆτ 1 T φ(c ˆτ T ) x t ( ) CMPT Pre-Test CM-test ˆµ U T R + ˆβT I( ˆτ T > c) + T 1/2 ˆσ,T I( ˆτ T c) x t CMBG Bagging CM-test ˆµ U R T + ( 1B B b=1 ˆβ b I( ˆτ b > c) + T 1/2 ˆσ,T I( ˆτ b c) ) x t ( CMBGA Asymptotic CMBG ˆµ U T R + ˆβ T [1 Φ( c ˆτ T ) + ˆτ 1 T φ( c ˆτ T ) + ˆτ 1 T Φ( c ˆτ T )] ) x t Note: ˆµ T and ˆβ T are the OLS estimates that depend on the forecast horizon. For different values of b, we get the overall patterns expected from the asymptotic results for all parameter configurations, sample sizes T, persistence parameters φ, and forecast horizons h. For b = 0 the restricted model is correct. Forecast errors of the restricted model stem only from mean estimation. The CM-based methods perform worst, as the null hypothesis b = 1 is incorrect, and the CM-test rejects very infrequently. The null of the t-test-based pre-test estimator is correct and is imposed whenever the test fails to reject, which happens frequently under all parameter configurations. This allows PT and its bagging version to achieve a lower MSE than the unrestricted model. For b = 0.5, the predictor is still so weak that the unrestricted model always performs best. The difference between using t-tests and CM-tests is not as large as it is for b = 0. Setting b = 1 imposes that the unrestricted and restricted methods asymptotically have the same MSE for estimation of β. For T = 25, however, the restricted model has substantially lower MSE than the unrestricted model for the prediction of y t+h,t. The difference disappears as the sample size grows. The rejection frequency for the CM-tests is fairly close to the nominal size for h = 1. For h = 6 the test is over-sized in small samples. Despite these small sample issues of the test, the CM-based estimators work well when b = 1 even for T = 25 with φ = 0.66 in Tables 1.2 and 1.4. For φ = 0.9, shown in Tables 1.3 and 1.5, CM-test and t-test-based estimators perform very similarly for T = 25. For b = 2, the CM-based method is able to improve the MSE, even though the null hypothesis is not precisely true. The magnitude of the improvement depends on the persistence parameter φ, critical value, and sample size. For b = 4 the coefficient is large enough such that the unrestricted model dominates. All other models except RE provide very similar performance. Both the CM test and the standard significance

27 14 CHAPTER 1. BAGGING WEAK PREDICTORS test reject very frequently, such that the different values of the coefficient under the two null hypotheses are less important. Our Monte Carlo simulations confirm that the asymptotic properties of the coefficient estimators carry over to the small sample behavior of the estimators and the resulting forecasting performance for the target variable. The bagging version of the CM-test can be expected to perform well when bias is not too small relative to the estimation uncertainty, i.e., b is not close to zero. If bias is much smaller than estimation uncertainty, then methods that shrink towards zero dominate. Our estimators will work well if the predictor is weak but the coefficient is large enough that excluding the predictor induces a substantial bias.

28 1.3. MONTE CARLO SIMULATIONS 15 Table 1.2. Monte Carlo Results for φ = 0.66 and c 0.95 MSE Rejection % RE UR PT PTBG PTBGA CMPT CMBG CMBGA t-test CM-test Panel 1: b = 0 h=1 T = T = T = h=6 T = T = T = Panel 2: b = 0.5 h=1 T = T = T = h=6 T = T = T = Panel 3: b = 1 h=1 T = T = T = h=6 T = T = T = Panel 4: b = 2 h=1 T = T = T = h=6 T = T = T = Panel 5: b = 4 h=1 T = T = T = h=6 T = T = T = Notes: MSE calculated in excess of var[u t+h ], and multiplied by 100.

29 16 CHAPTER 1. BAGGING WEAK PREDICTORS Table 1.3. Monte Carlo Results for φ = 0.9 and c 0.95 MSE Rejection % RE UR PT PTBG PTBGA CMPT CMBG CMBGA t-test CM-test Panel 1: b = 0 h=1 T = T = T = h=6 T = T = T = Panel 2: b = 0.5 h=1 T = T = T = h=6 T = T = T = Panel 3: b = 1 h=1 T = T = T = h=6 T = T = T = Panel 4: b = 2 h=1 T = T = T = h=6 T = T = T = Panel 5: b = 4 h=1 T = T = T = h=6 T = T = T = Notes: MSE calculated in excess of var[u t+h ], and multiplied by 100.

30 1.3. MONTE CARLO SIMULATIONS 17 Table 1.4. Monte Carlo Results for φ = 0.66 and c 0.99 MSE Rejection % RE UR PT PTBG PTBGA CMPT CMBG CMBGA t-test CM-test Panel 1: b = 0 h=1 T = T = T = h=6 T = T = T = Panel 2: b = 0.5 h=1 T = T = T = h=6 T = T = T = Panel 3: b = 1 h=1 T = T = T = h=6 T = T = T = Panel 4: b = 2 h=1 T = T = T = h=6 T = T = T = Panel 5: b = 4 h=1 T = T = T = h=6 T = T = T = Notes: MSE calculated in excess of var[u t+h ], and multiplied by 100.