Comparing one-day-ahead Value-at-Risk forecasts with daily and intraday data in times of volatile volatility

Transcription

1 Comparing one-day-ahead Value-at-Risk forecasts with daily and intraday data in times of volatile volatility Steffen Günther a, Jörg Laitenberger b We compare the performance of Value-at-Risk (VaR) forecasting models, using volatility estimates based on squared daily returns and on realized variance (RV), calculated with intraday high-frequency returns. Intuitively we would expect the models using the entire set of intraday information to possess a higher predictive ability, but we confirm for data from nine stock indices and four exchange rates earlier results that a simple GARCH model combined with an appropriate density function with fat tails gives VaR forecasts as good as those from a comparable model using RV as volatility predictor, in which we assess performance by means of four different backtesting procedures. Restricting the analysis to those days with large changes in volatility, the RV-based models markedly outperform the forecasts based on daily returns. We conclude that using RV for forecasting purposes is especially well suited in periods of large volatility changes. Keywords: Realized Volatility; Value at Risk; Volatility forecasting JEL-class.: C52; C53; G17 a Steffen Günther, School of Economics & Business, Martin-Luther University Halle-Wittenberg, Große Steinstr. 73, D Halle, Germany; ; steffen.guenther@wiwi.uni-halle.de; b Jörg Laitenberger, School of Economics & Business, Martin-Luther University Halle-Wittenberg, Große Steinstr. 73, D Halle, Germany; joerg.laitenberger@wiwi.uni-halle.de

2 1. Introduction Value at Risk (VaR) is the most widely used measure of risk for controlling the probability of large negative returns. A forecast of the future VaR, which is simply a quantile of the distribution of returns, is usually done by specifying a distribution function and a scale parameter. 1 Typically the latter is an assessment of the conditional variance of the distribution. Modelling and forecasting volatility has thus been one of the main areas of research in financial econometrics. Depending on the type of data at hand, at least two methods can be used. The first one is the well known family of (G)ARCH models, that make use of squared daily returns, usually computed from closing prices, that has been one of the main workhorses of financial econometrics for at least 30 years. The second method makes use of intraday returns, the most prominent example of which is the realized variance (RV), introduced by Taylor and Xu (1997) and Andersen and Bollerslev (1998). RV is defined as the sum of the intraday squared returns, sampled at some fixed sampling frequency. Under the assumption that the logarithmic security prices follow an Itô process, in the limit when the number of intraday price observations increases, RV is equal to the quadratic variation of the process. Due to microstructure noise, there are some issues with the sampling frequency, such that RV is only a noisy measure of the true volatility. 2 Andersen et al. (2001) have pointed out, that one of the main merits of RV is that it is a consistent, model-free estimate of the true volatility. Among other things, RV is supposed to provide a relatively precise and quickly adapting estimate of current volatility, because it exploits valuable intraday information. 3 There is by now an extensive literature on what type of model is best suited for the prediction of future VaR s. 4 For the (G)ARCH type models several suggestions have been made, how to account for asymmetries and long-memory effects. In the case of intraday data, several alterna- 1 An exception is Engle and Manganelli (2004) where the future VaR is directly predicted using former VaR realizations. 2 See Bandi and Russell (2008) and Zhang et al. (2012) for discussions on how to deal with this problem. Hansen and Lunde (2005b) discusses how to cope with non trading hours. 3 Andersen et al. (2003), p See Nieto and Ruiz (2016) for a recent survey. 2

3 tives to RV have been proposed 5 and extensions incorporating asymmetries and other statistical observations have been discussed. Results frequently depend on the data used, and the way the forecast is measured. For the prediction of the daily variability of stocks, RV seems to be a very good choice. 6 On the other hand, when it comes to volatility prediction for option pricing or for risk management, results are mixed. Giot and Laurent (2004) find no advantage in the use RVmodels compared to ARCH type models when VaR is predicted, and Angelidis and Degiannakis (2008) even find evidence in favor of models based on daily returns when the pricing of options is targeted. This is somewhat of a puzzle, since the intraday data incorporate all information that is in the daily returns, and much more on top of this. Therefore we would expect that a model taking account of more information cannot perform worse than the models with reduced data. In order to cast a bit of light onto this issue, we analyse in this article the predictive ability of models based on RV and daily returns in specific time periods. More precisely, we conjecture that RV should have particularly good predictive power in situations where the volatility is undergoing large and fast changes. Because the squared daily return is only a very noisy measure of actual volatility, the (G)ARCH models include the actual return with a relatively small weight only. In contrast to this RV is a very precise measure, and therefore the RV models give it a heavy weight. In the case of a sudden outburst of volatility this leads to a rather sluggish incorporation of the actual events in the case of daily data, while RV is taking account of the new situation almost immediately. The drawback seems to be, that the RV models overreact in situations where the volatility changes are only transient, such that the RV models tend to underperform the models based on daily returns in times of calm. 7 To our knowledge, this is the first study that correlates volatility forecasts with past volatility changes. The only study that addresses a similar question 5 Alternatives to RV, also making use of intraday information, are the realized range (Martens and Van Dijk (2007)) or the realized power (Ghysels et al. (2006)), among others. See Fuertes et al. (2009) for an evaluation of their forecasting ability. 6 Koopman et al. (2005). 7 Because most times are (fortunately) rather calm times, the latter effect is statistically not significant, because the few instances of missed forecasts disappear in the vast majority of hits. 3

4 seems to be Fuertes et al. (2009). They report evidence on the forecasting value of RV during periods of specific market conditions, namely up- versus down-market days and low- versus high-volume days. We perform this analysis with respect to the prediction of VaR for the confidence levels 5% and 1%, and we only use rather simple econometric models in order to focus the attention on the choice of data. For both daily returns and RV we use one basic GARCH model, and in order to prevent the results to be driven solely by the simple structure of the basic model, we also incorporate one more sophisticated model for each data set, namely the GJR-GARCH model, 8 that takes asymmetries into account, for the daily returns, and the HAR model, 9 that captures long-range serial dependence in volatility, in the case of RV. Concerning the distribution of returns, again we are not aspiring to be at the front line of actual research. 10 We restrict ourselves to the normal distribution, the skewed Student distribution and the filtered historical simulation (FHS). Although well known to be a bad choice, we include the normal distribution as a benchmark. The skewed Student distribution and the FHS have both been shown to deliver good results. 11 We assess the performance of our forecasting models with four different backtesting procedures, namely the Unconditional Coverage test proposed by Kupiec (1995), the Independence test of Christoffersen (1998), the Dynamic Quantile test of Engle and Manganelli (2004) and a duration-based test proposed by Christoffersen and Pelletier (2004). 12 We first evaluate the models with respect to the entire time series. The parameters of the models are estimated in-sample within a rolling window of 500 days. Subsequently an out-ofsample forecast of next day s VaR is computed. Next we compare the accuracy of the forecasted VaRs. As indicated above, we confirm the findings of Giot and Laurent (2004), that there is no advantage in using the RV models. Both the models using RV and daily returns make accurate 8 Glosten et al. (1993). 9 Corsi (2009). 10 See Kuester et al. (2006) and Braione and Scholtes (2016) for a comprehensive review on the choice of the distribution function. 11 See Giot and Laurent (2004) and Louzis et al. (2014). 12 Berkowitz et al. (2011) report the results of an assessment of the power of these tests applied to desk-level data. 4

5 predictions with the skewed Student distribution and the FHS. And both type of models perform badly with the normal distribution. Next we investigate whether there is a concentration of exceptions after days of large changes in volatility. We measure volatility changes as changes in RV on two consecutive days and as a large difference in two consecutive squared daily return. While there is no clustering of breaks after large differences in squared returns for either type of model, the number of outliers increases significantly for the models based on daily returns, when volatility changes are measured with changes in RV and when the threshold for the identification of days of large volatility changes is increased, while the RV models still predict an accurate number of exceptions. This paper is organized as follows. First, the dataset and the methods for forecasting variance and Value at Risk are outlined in Section 2. Section 3 presents the results when the entire dataset is included. Section 4 describes our main findings, namely that the RV models significantly outperform models based on daily squared returns after days of large volatility changes. Section 5 summarizes the paper. 2. Dataset and VaR forecasting methods 2.1. Data We use the data of nine stock market indices and of four exchange rates: the French CAC40, German DAX, American Dow Jones Industrial Average (in the following DJI), S&P 500, Russell 2000 (RUT), NASDAQ Composite Index (NAS), European Euro Stoxx 50 (ES 50), British FTSE 100 and Japanese Nikkei 225 (Nik), and the exchange rates between the US-Dollar and the Euro, the Japanese Yen, the Swiss Franc and the British Pound. For the stock indices, the period investigated is January 3, 2000 to August 17, 2015, For the exchange rates the data starts on January 2, 1996 and finishes on March 27, All data is from the Oxford-Man Institute s realized library. 13 We use the Realized Variance estimator with a 5-minute grid of intraday 13 To be more precise, it is version 0.1 for the exchange rates and version 0.2 for the stock data, obtained from Heber et al. (2009). 5

6 prices to reduce the impact of microstructure noise. In the case of stock indices a 1-minute subsampling grid to increase the information is used to estimate the variance. 14 For the exchange rates only a version with a 30-second subsampling is provided. To account for the overnight-gap we apply the scaling estimator introduced by Hansen and Lunde (2005b). We simply scaled up trading hours the chosen Realized Variance estimator of the trading hours so that RV t = δrvt with δ = m t=1 rt 2 / m trading hours t=1 RVt. The daily returns are computed using the daily closing prices. The realized volatility RV t at date t is defined as the sum of all squared 5-minutes returns of date t: T RV t = rt,j 2 j=1 where r t,j = ln(p t,j ) ln(p t,j 1 ) is the j-th 5-minute return on day t. Andersen et al. (2003) have shown that RV t is a consistent and unbiased estimator of the daily variance, if the 5-minute returns are uncorrelated Models We consider two-steps VaR forecasting models. 15 That is we assume the following model of returns r t = µ t + σ t z t where µ t and σ t are the conditional mean and standard deviation of the returns, and z t is iid distributed with mean zero and variance one. For the one-day returns at hand we can safely assume µ t = 0. We are therefore left with the task of finding a suitable distribution for z t and a forecast of the conditional volatility σ t. For the distribution of z t we will use three well known approaches. The first two approaches assume a parametric functional form of the distribution. In this case z t i.i.d.d(0, 1, ξ) with D(0, 1, ξ) equal to any desired standardized distribution 14 See Zhang et al. (2012) for more details on subsampling and averaging. 15 In the sense of Nieto and Ruiz (2016). 6

7 with possibly a set of further parameters ξ. For D we first use the normal distribution to check the stylized fact that the distribution of daily returns divided by the realized standard deviation is approximately Gaussian, and second the skewed Student s t distribution, in order to take account of heavy tails and gain/loss asymmetry. The parameters of the t-distribution are estimated by fitting the last 500 daily observations and updated with a rolling window. The third approach uses the Filtered Historical Simulation (FHS), that is the z t are drawn out of the standardized empirical realizations of the last 500 returns, where every past return is scaled by the current volatility that prevailed that day. The forecast of the conditional volatility σ t is the more challenging problem. We use two classes of models. The first two models are models based on the squared daily returns, r 2, and the second two models use RV Models based on r 2 For the r 2 -based volatility forecasts we implement a standard GARCH(1,1) model 16, where the conditional variance is modelled as σ t+1 = ω + αr 2 t + βσ 2 t (1) and the GJR-GARCH(1,1) of Glosten et al. (1993) defined by σ t+1 = ω + αr 2 t + γl t r 2 t + βσ 2 t (2) where 1, if r t < 0 L t = 0, if r t Bollerslev (1986). 7

8 The GJR-GARCH contains an additional term to consider the leverage effect. In light of the results of Hansen and Lunde (2005a), that there is no significant improvement in applying higher order models, we restrict our analysis to the simplest GARCH(1,1) specification. Giot and Laurent (2004) use an APARCH(1,1) model, that is slightly more general that the GJR-GARCH Models based on RV We first also implement a simple GARCH(1,1) model, where RV t is inserted in lieu of r 2 t in the regression equation: σ t+1 = ω + αrv t + βσ 2 t. (3) Shephard and Sheppard (2010) have investigated a whole family of models of this class, that they labelled as High Frequency Based Volatility (HEAVY) models. In general, the HEAVY models can be based on additional Realized Measures (RM). Engle and Gallo (2006) have analysed a joint model with RV t and rt 2 both included (and the daily range) in (3), but they found the coefficient of the rt 2 -term to be small. We will refer to model (3) as GARCH-RV in order to emphasize the analogies to (1). The second variance forecasting model based on realized variance is the HAR-RV model by Corsi (2009) defined by RV t+1 = c + βrv t + β (w) RV (w) t + β (m) RV (m) t. (4) Here, RV (w) t and RV (m) t are the weekly respectively the monthly moving averages of Realized Variance. With the assumption of five trading days a week, RV (w) t is calculated as RV (w) t = (RV t 4 + RV t 3 + RV t 2 + RV t 1 + RV t )/5. The monthly value is calculated analogously but with the assumption of 21 trading days. This linear model can be estimated by OLS. With reference to the stylized fact that the logarithmic Realized Variance is approximately normally 8

9 distributed, we perform a log-transformation to (4) to increase the performance of parameter estimation. So for OLS we use ln(rv t+1 ) = c + βln(rv t ) + β (w) ln(rv (w) t ) + β (m) ln(rv (m) t ) + ɛ t+1 (5) with ɛ t+1 i.i.d.n (0, σ 2 ɛ ). Undoing these transformation for forecasting leads to ( RV t+1 t = exp c + βln(rv t ) + β (w) ln(rv (w) t ) + β (m) ln(rv (m) ) t ) exp(σ 2 ɛ /2). (6) 2.3. Evaluation procedure All in all we implement four different variance forecasting models combined with three approaches for the distribution of conditional returns, so that we investigate twelve model combinations for each of the nine stock market indices and the four exchange rates. We aim at predicting the Value-at-Risk (VaR) at the confidence levels 1 α with α=5% and α=1%. The VaR at the confidence level 1 α is the α-quantile of the distribution of next period s returns: VaR t,α = sup{r P (r t r I t 1 ) VaR t,α } where I t 1 is information up to time t 1. Correspondingly VaR t,α = F 1 t (α) with F 1 t being the inverse distribution function of the conditional distribution of returns. We use a rolling window of 500 days 17 to determine the in-sample parameters of the models. Subsequently we perform an out-of-sample forecast of the one day ahead VaR. Our calculations are summarized as follows: Step 1: Set t = 0 as the first day we want to calculate VaR for tomorrow. Step 2: Use the data of r 2 t and RV t from t = t 499 to t = t to In most empirical VaR-relevant papers it is used a window size of at least The decision for a certain window is a trade off between: more data for better parameter estimation vs. less data to capture more relevant market conditions. We decided to give more weight to the latter. 9

10 1.... calculate δ and scale up Realized Variance in the same time estimate the parameters of the four variance forecasting models by QMLE for (1), (2) and (3) and by OLS for (5) and carry out in-sample forecasts for the same time and an out-of-sample forecast for t use the in-sample forecasts to standardize the returns of the same time which is necessary for FHS and to get a standardized distribution D(0, 1, ξ) estimate the parameter set ξ of the skewed Student s t distribution by MLE. Step 3: Calculate V ar(α) for t + 1 with the out-of-sample variance forecast and either the inverse of D or the empirical quantile of the standardized returns to the levels α = 5% and α = 1%. Step 4: Set t = t + 1 and return to Step 2 if you have not reached the final date. 3. Forecasting performance with respect to the whole dataset 3.1. The failure rate For the stock index data we obtain for every model more than out-of-sample forecasts. For the exchange rates we have a little more than forecasts. To assess the performance of the predicted VaRs we use four different backtesting procedures. First we use Kupiec (1995) s Unconditional Coverage test. We count the number of days, when r t+1 < V ar(α) t+1. The empirical failure rate ˆα is then the proportion of exceptional days to the total number of forecasts. A model is accurate when the empirical failure rate ˆα is equal to the expected failure rate α. We apply Kupiec (1995) s likelihood ratio test to verify the null hypothesis H 0 : α = α against H 1 : α α. Table 1 shows the results of the Kupiec-test for the exchange rates and six out of nine stock indices. Neither the use of GARCH and GJR-GARCH nor the use of Realized Variance forecasting models leads to suitable VaR calculations, when the returns are assumed to be normally 10

11 Table 1: P-values from Kupiec s likelihood ratio test. assumption/method normal skewed-t FHS α 5% 1% 5% 1% 5% 1% DAX HAR-RV *** 0.000*** ** ** GARCH(1,1)-RV *** 0.000*** * * GARCH(1,1) *** 0.000*** ** GJR-GARCH(1,1) *** 0.000*** * Dow Jones Industrial Average HAR-RV ** *** GARCH(1,1)-RV * *** GARCH(1,1) *** GJR-GARCH(1,1) *** Nikkei 225 HAR-RV ** *** * GARCH(1,1)-RV ** *** GARCH(1,1) ** GJR-GARCH(1,1) *** Euro Stoxx 50 HAR-RV ** *** GARCH(1,1)-RV *** * GARCH(1,1) *** GJR-GARCH(1,1) ** 0.000*** * S&P 500 HAR-RV ** *** GARCH(1,1)-RV ** *** GARCH(1,1) *** GJR-GARCH(1,1) * *** FTSE 100 HAR-RV *** *** ** ** * GARCH(1,1)-RV *** *** GARCH(1,1) * *** GJR-GARCH(1,1) *** *** ** USD/EUR HAR-RV *** ** GARCH(1,1)-RV *** * GARCH(1,1) ** GJR-GARCH(1,1) ** USD/JPY HAR-RV *** GARCH(1,1)-RV *** GARCH(1,1) *** GJR-GARCH(1,1) *** USD/CHF HAR-RV *** *** *** ** GARCH(1,1)-RV *** *** *** * GARCH(1,1) ** * ** GJR-GARCH(1,1) * ** USD/GBP HAR-RV *** GARCH(1,1)-RV *** GARCH(1,1) *** GJR-GARCH(1,1) *** *, ** and *** indicates that the null hypothesis H 0 : α = α is rejected at a 10%, 5% and 1% significance level. 11

12 distributed. This is especially the case at the confidence level α = 1%, where the model is rejected for 45 out of the 52 computed p-values. Applying a 10% level of significance, 18 no model combination is able to generate a correct failure rate.for the VaR at α = 5%, results improve and for the NASDAQ, USD/EUR, USD/JPY and USD/GBP the normal assumption seems to be applicable, but the overall picture shows that the other models perform much better. With that knowledge in mind, we will not include VaR calculations with normal assumption in the remaining study. For the models with the FHS or a skewed Student s t distribution as the distribution of returns most p-values indicate that the null hypothesis, and as a consequence the model combination, cannot be rejected, even at a 10% significance level. For the exchange rate data, the FHS noticeably outperforms the skewed Student s t distribution. All in all the FHS has the fewest rejections (not a single one at a 1% significance level, just one at 5% and five out of 104 at a 10% significance level) and, thus, could be declared as the best distributional assumption for VaR calculation within the framework of this paper. Our main interest concerns the differences in performance between models based on r 2 and those based on RV. For the stock indices, a very unexpected finding is that the GARCH(1,1) is performing amazingly well. We detect a single rejection of H 0 : ˆα = α for the german DAX combined with the skewed Student s t distribution and α = 5% at a significance level of 5%. That means that in 35 of the 36 examined models the combination of GARCH(1,1) and either the FHS or the assumption of a skewed Student s t distribution performs very well in the case of stock indices. For the exchange rates, the results are inconclusive. The rejection of a model depends much more on the conditional distribution of returns than on the variance forecast. Quite remarkable is that there is no rejection in the case of the FHS, irrespective of the chosen variance forecast. 18 Failing to reject an incorrect model (Type II error) could be costly especially for financial institutions, therefore a higher significance level of 5% or even 10% seems to be appropriate. 12

13 Next we apply the Independence test of Christoffersen (1998). A perfect VaR-forecasting model should feature a sequence of violations 1, if r t < V ar t I t = 0, otherwise, (7) that are independent and identically distributed. Christoffersen (1998) proposes to test, whether the sequence forms a first-order Markov process with transition probabilities, that are independent of the latest observation. As a third test, we perform the Dynamic Quantile test of Engle and Manganelli (2004). This test is generalization of the previous Independence test. For a given number of lagged observations and observed quantiles, the autocorrelation of a transformed hit sequence is calculated. Under the null hypothesis of a perfect VaR model, these autocorrelations must vanish. Finally we apply the Duration-based test of Christoffersen and Pelletier (2004). Christoffersen and Pelletier (2004) observe that a well specified Var-model implies, that the duration between two violations of the hit sequence should have an exponential distribution with a mean of α. The test proceeds by testing if the observed frequency of durations is consistent with a flat hazard rate. As mentioned before, we favor higher significance levels. With a significance level of 10%, the HAR-RV model is rejected in ten instances in the last four columns of Table 1, the GARCH- RV six times, the GARCH(1,1) three times and the GJR-GARCH also six times. Our first result is therefore, that for the simple forecasting models used in this study, the choice of conditional distribution is much more important than the variance forecast. Particularly the FHS is an excellent choice. There is no rejection by using the combination of the FHS and either the GARCH(1,1) or the GJR-GARCH(1,1). All in all, we cannot confirm our expectation that Realized Variance-based VaR predictions are more reliable than r 2 -based ones. 13

14 Table 2: P-values for the Dynamic Quantile Test. assumption/method normal skewed-t FHS α 5% 1% 5% 1% 5% 1% DAX HAR-RV *** *** * GARCH(1,1)-RV *** *** * * GARCH(1,1) *** *** ** GJR-GARCH(1,1) ** *** * Dow Jones Industrial Average HAR-RV *** *** ** * GARCH(1,1)-RV ** *** ** GARCH(1,1) *** *** ** ** GJR-GARCH(1,1) *** Nikkei 225 HAR-RV ** *** * GARCH(1,1)-RV ** ** ** GARCH(1,1) *** *** * *** * GJR-GARCH(1,1) * ** Euro Stoxx 50 HAR-RV *** GARCH(1,1)-RV *** GARCH(1,1) *** GJR-GARCH(1,1) * *** S&P 500 HAR-RV *** *** ** GARCH(1,1)-RV ** *** ** GARCH(1,1) *** GJR-GARCH(1,1) * *** * FTSE 100 HAR-RV ** *** *** GARCH(1,1)-RV ** *** *** ** GARCH(1,1) *** ** GJR-GARCH(1,1) * *** USD/EUR HAR-RV GARCH(1,1)-RV GARCH(1,1) ** * GJR-GARCH(1,1) USD/JPY HAR-RV *** GARCH(1,1)-RV *** GARCH(1,1) *** GJR-GARCH(1,1) *** USD/CHF HAR-RV ** *** ** GARCH(1,1)-RV ** *** ** GARCH(1,1) * * GJR-GARCH(1,1) ** *** USD/GBP HAR-RV ** GARCH(1,1)-RV ** GARCH(1,1) ** *** * GJR-GARCH(1,1) * ** * *, ** and *** indicates that the null hypothesis H 0 : α = α is rejected at a 10%, 5% and 1% significance level. 14

15 Table 3: P-values for the Duration-Based Independence Test. assumption/method normal skewed-t FHS α 5% 1% 5% 1% 5% 1% DAX HAR-RV GARCH(1,1)-RV ** GARCH(1,1) GJR-GARCH(1,1) *** *** *** Dow Jones Industrial Average HAR-RV GARCH(1,1)-RV GARCH(1,1) GJR-GARCH(1,1) * * Nikkei 225 HAR-RV * GARCH(1,1)-RV GARCH(1,1) GJR-GARCH(1,1) Euro Stoxx 50 HAR-RV GARCH(1,1)-RV GARCH(1,1) GJR-GARCH(1,1) *** ** 0.019** * ** ** S&P 500 HAR-RV GARCH(1,1)-RV * ** * *** GARCH(1,1) GJR-GARCH(1,1) ** *** *** FTSE 100 HAR-RV GARCH(1,1)-RV GARCH(1,1) GJR-GARCH(1,1) ** ** ** USD/EUR HAR-RV GARCH(1,1)-RV GARCH(1,1) * * * * * GJR-GARCH(1,1) * USD/JPY HAR-RV GARCH(1,1)-RV GARCH(1,1) * GJR-GARCH(1,1) USD/CHF HAR-RV * GARCH(1,1)-RV ** GARCH(1,1) * ** GJR-GARCH(1,1) *** * *** USD/GBP HAR-RV GARCH(1,1)-RV *** GARCH(1,1) ** GJR-GARCH(1,1) *, ** and *** indicates that the null hypothesis H 0 : α = α is rejected at a 10%, 5% and 1% significance level. 15

16 3.2. The weight of the recent past The results in the last section are puzzling. Why are the RV-based forecasts not much better than the r 2 -based forecasts, given that they comprise so much more information? When looking at the coefficients of regression of the different models, a remarkable result can be seen. Table 2 presents the averages of all parameters estimated for the different variance forecasting models. For this purpose the HAR-RV parameters are estimated anew by applying OLS to formula (4) instead of (5) in order to demonstrate the influence of the explanatory variables as a factor. The underlying intention of this procedure is a better comparison with the other models. Table 4: Mean Parameter Estimation. HAR-RV GARCH-RV GARCH GJR-GARCH β β (w) β (m) α β α β α γ β DAX DJIA NIKKEI ES S&P Russell CAC NASDAQ FTSE USD/EUR USD/JPY USD/CHF USD/GBP The main findings are: If today s level of variance is measured by RV then the influence of this measurement for tomorrow s forecast is much higher than the influence of r 2 in their respective models. Comparing the α-values in the GARCH and the GARCH-RV columns, we can see that the coefficients in the GARCH model, relying on r 2, are all below 10%, while the respective coefficients in the GARCH-RV model are between 18% and 45%. A similar picture is revealed for the coefficients of the GJR-GARCH, where the sum of the α- and the γ-coefficients all lie below 20%. From this it is obvious that the variance forecasting models relying on r 2, GARCH and GJR-GARCH, react more sluggish to changes in market variance than the RV -based models GARCH-RV and HAR-RV. The latter, however, have the ability to pick up changes in variance quickly and carry them over to tomorrow s prediction. 16

17 4. Forecasting performance in times of large changes in volatility Given the results of the last section we expect the r 2 -based models to manifest a higher risk for an exception when market variance suddenly increases sharply. To identify the relevant days we define a large change in volatility as a large difference between two adjacent days of either the Realized Variance or the squared return: RV t = RV t RV t 1 (8) r 2 t = r 2 t r 2 t 1. (9) We use absolute changes and not relative ones, because the variance level in t 1 could be very small so that a huge relative change is possible although the absolute variance level is still small. To clarify that imagine daily returns of 0.1% in t 1 and -1.5% in t. The variance in t, measured by r 2, is 225 times higher than the day before. If you consider returns of 1% in t 1 and -8% in t the variance only increases by the factor 64 whereas the second example is of particular relevance for our investigation. Notice that we only consider positive changes. For the next step we select those days where RV t or rt 2 exceed a certain quantile q. These are the days following a large increase in the volatility measure. The larger q is, the more extreme will be the shift in variance, and the smaller is the number of days considered. Next we count the number of VaR exceptions in the subset of days for every forecasting model. Let Q(q) be the fraction of empirical exceptions in proportion to the entire number of exceptions: Q(q) = Ẽ q (1 q)e, (10) where Ẽq is the quantity of exceptions depending on the chosen quantile q and E is the number of expected exceptions, when exceptions occur independently of volatility changes. The function Q(q) describes how many of the observed outliers are concentrated on days with large 17

18 volatility changes. Q(q) > 1 implies that the subset of days possesses more exceptions than the average day. A well performing VaR forecasting model should feature a function Q(q) that is approximately one, because the risk for an exception should always be α, independent of the prevailing market situation. Q-values that strongly differ from one (for certain chosen quantiles) indicate that there is unused information by means of which more suitable VaR models could be constructed Graphical inspection In the following we use the graphs of Q(q) to visualize our results. Figure 1 presents the function Q(q), whereby the changes in variance are measured by RV. The four subplots represent the four different variance forecasting models. All calculations are performed with respect to a distribution function generated with the FHS and the VaR is computed for a confidence level of α=5%. For reasons of clarity merely three graphs are grouped together at a time which represent the plotted results for the DAX, the Dow Jones Industrial Average and the Nikkei, respectively. However, all the plots for the remaining data are in the appendix. Honestly it does not look much spectacular (it s worth waiting for Figure 2) but particularly when looking at the GARCH-subplot, it can be seen that the quotients of all three markets are greater than one starting from quantiles of about 50%. This is approximately the point where differences in variance become noticeable. Looking at the GJR-GARCH in the same figure, similar results can be seen for the NIKKEI. The upper two subplots, that visualize the GARCH-RV and the HAR-RV within the mentioned framework, differ. While the quotients of the DAX fluctuate about one, they are clearly below one for the Dow Jones and the NIKKEI when considering higher quantiles. In Figure 2 VaR levels are reduced from α=5% to α=1%. The resulting graphs are not as smooth as compared to Figure 1 owing to the fact that the relevant data volume for calculating Q(q) is smaller. The generated results, however, are much more impressive. While Q(q) is 18

19 Figure 1: Quotients - VaR method: FHS, α=5%, Variance change: RV. mostly about or below one when considering the GARCH-RV and HAR-RV models, it is distinctly greater than one for the r 2 -based variance models, when looking at higher quantiles and thus on days following large positive changes in variance. Finally, figure 3 and 4 present the results when volatility changes are measured by r 2. The rest of the framework remains unchanged: same approach, same levels, same markets. There are no great differences in accuracy between the models, whether daily data or intraday data is employed. The function Q(q) remains about one up to quantiles of 50%, and falls below one for higher values. This indicates that all models predict too few exceptions on days following large shifts in returns. 19

20 Figure 2: Quotients - VaR method: FHS, α=1%, Variance change: RV Is the result statistically significant? From graphical inspection we conjecture, that there is a significant difference between the forecasting performance of the models based on daily squared returns r 2 and those based on Realized Variance RV on days following large changes in volatility, measured as differences in the RV values. Second we also posit that the probability of a VaR deviation is larger on those days for the models based on r 2. On the other hand when variance change is measured by r 2, the risk for an exception is lower for high quantiles. This pertains to all model combinations. We are going to inspect these conjectures by testing if there is a significant trend in the functions Q(q). We apply the Mann-Kendall (MK) test for monotonic trends. A simple alternative would be linear regression analysis to indicate a significant slope but the advantages for the MK-test are that there is no need for normal distributed residuals and the trend can be other than linear. 20

21 Figure 3: Quotients - VaR method: FHS, α=5%, Variance change: r 2. The latter is of particular importance on grounds that Q tends not to be linear as shown in the preceding figures. The null hypothesis H 0 of this test is that there is no monotonic trend. It is a two-sided test so that H 0 has to be rejected when there is either a downward or an upward trend. We focus on increasing risks when the threshold q increases. We check the results of the relevant test statistic S, that is defined as S = n 1 n k=1 j=k+1 sgn(x j x k ) (11) 21

22 Figure 4: Quotients - VaR method: FHS, α=1%, Variance change: r 2. where 1, if (x j x k ) > 0 sgn(x j x k ) = 0, if (x j x k ) = 0. 1, if (x j x k ) < 0 The observed variable x is the function Q(q) and n = 90, given by the quantiles from 1% to 90%. Positive values for the test statistic S are regarded as indicators for an upward trend. For such instances, p-values are computed in order to identify statistical significance. The results are presented in Table 3. A "+" indicates that the test statistics S has a positive value and "+*" indicates that this signal is distinct so that a significant positive monotonic trend, with a significance level of 1%, is ascertained. 22

23 Table 5: Results of MK-test for significant upward trends in Q. assumption/method skewed-t FHS α 5% 1% 5% 1% DAX HAR-RV + GARCH(1,1)-RV + +* + GARCH(1,1) +* +* +* +* GJR-GARCH(1,1) +* +* + +* Dow Jones Industrial Average HAR-RV GARCH(1,1)-RV GARCH(1,1) +* +* +* +* GJR-GARCH(1,1) +* +* +* Nikkei 225 HAR-RV + +* GARCH(1,1)-RV + + GARCH(1,1) +* +* +* +* GJR-GARCH(1,1) +* +* +* +* Euro Stoxx 50 HAR-RV GARCH(1,1)-RV GARCH(1,1) +* +* +* +* GJR-GARCH(1,1) +* S&P 500 HAR-RV GARCH(1,1)-RV GARCH(1,1) +* +* +* +* GJR-GARCH(1,1) +* +* +* +* FTSE100 HAR-RV + +* + GARCH(1,1)-RV +* +* GARCH(1,1) +* +* +* +* GJR-GARCH(1,1) +* +* +* +* USD/EUR HAR-RV GARCH(1,1)-RV GARCH(1,1) +* +* GJR-GARCH(1,1) +* +* USD/JPY HAR-RV GARCH(1,1)-RV GARCH(1,1) +* +* GJR-GARCH(1,1) + +* USD/CHF HAR-RV GARCH(1,1)-RV GARCH(1,1) +* +* GJR-GARCH(1,1) +* +* USD/GBP HAR-RV +* + +* GARCH(1,1)-RV +* +* +* GARCH(1,1) +* +* +* +* GJR-GARCH(1,1) +* +* +* +* + indicates that the test statistic S is positive. +* indicates a significant (level=1%) upward trend. 23

24 For every variance model we computed 52 p-values: two VaR approaches with two different levels on 13 datasets (nine stock market indices and 4 exchange rates). For the stock indices the results are rather unambiguous. For the GARCH model all 36 tests and for the GJR-GARCH 29 of 36 tests indicate a significant upward trend. In contrast for the RV-based models only 4 models display a trend, 3 of which are significant. The result for the exchange rate data is a bit less clearcut. But we still have a significant trend in almost half of the r 2 -based models, and no trend in the RV-based models, in the cases of the Euro, the Yen and the Swiss Franc. The British Pound is somehow a puzzle, since it features a significant trend in almost all models, except for the HAR-RV in the case of the VaR at the confidence level 1%. So we conclude: Yes, indeed. There is a significant upward trend in VaR outliers for models based on daily squared returns, on days after large volatility changes. This trend disappears, when models based on Realized Variance are used. Overall we can safely assert, that RV-based VaR forecasts are more reliable than those computed with the help of daily squared returns in times of volatile volatility. 5. Conclusion and directions for future research Don t trust the first impression. If one is merely considering the accuracy of VaR estimates computed by employing r 2 -based variance forecasting models as opposed to the VaR estimates as calculated by the usage of RV -based variance forecasting models, no significant differences are observable. However, backtesting the accuracy of the VaR forecasts only verifies whether a defined level of exceptions is realized over a certain time horizon. The question that is not answered is whether the risk of a shortfall is correctly assessed day by day. This paper shows that there is at least one measurement ( RV ) that reveals differences in VaR prediction, that a simple test failed to indicate. Our empirical results are as follows: The risk for a VaR outlier increases after days of large RV -changes when a forecasting model based on daily squared returns is used. So ignoring intraday data can lead to poor forecasts. 24

25 The relevance of the results reported in this paper are limited by the simple forecasting models that we have used. Further research should clarify whether the same observations emerge in models that make a more sophisticated, and maybe better use of the information contained in daily data. Obviously volatility shifts are not the only event that might drive future volatility forecasts. Thus there is a vast universe of potential indicators that could be subjected to a similar analysis. 19 Finally, in a companion paper, we are presenting and discussing a family of volatility models incorporating the asymmetric reaction to volatility shifts. A. p-values from Kupiec likelihood ratio test Table 6: Remaining p-values from Kupiec s likelihood ratio test. assumption/method normal skewed-t FHS α 5% 1% 5% 1% 5% 1% Russell 2000 HAR-RV ** GARCH-RV GARCH(1,1) * ** GJR-GARCH(1,1) ** CAC 40 HAR-RV *** 0.000*** * * GARCH-RV ** *** * GARCH(1,1) ** *** GJR-GARCH(1,1) ** *** * NASDAQ HAR-RV * GARCH-RV GARCH(1,1) *** GJR-GARCH(1,1) *** *, ** and *** indicates that the null hypothesis H 0 : α = α is rejected at a 10%, 5% and 1% significance level. 19 Fuertes et al. (2009) relate volatility forecast to market conditions, namely days of up- and down-markets and lowand high trading volume. 25

26 References ANDERSEN, T. G. AND BOLLERSLEV, T Answering the skeptics: Yes, standard volatility models do provide accurate forecasts. International economic review 39: ANDERSEN, T. G., BOLLERSLEV, T., DIEBOLD, F. X., AND EBENS, H The distribution of realized stock return volatility. Journal of financial economics 61: ANDERSEN, T. G., BOLLERSLEV, T., DIEBOLD, F. X., AND LABYS, P Modeling and forecasting realized volatility. Econometrica 71: ANGELIDIS, T. AND DEGIANNAKIS, S Volatility forecasting: Intra-day versus inter-day models. Journal of International Financial Markets, Institutions and Money 18: BANDI, F. M. AND RUSSELL, J. R Microstructure noise, realized variance, and optimal sampling. The Review of Economic Studies 75: BERKOWITZ, J., CHRISTOFFERSEN, P., AND PELLETIER, D Evaluating value-at-risk models with desk-level data. Management Science 57: BOLLERSLEV, T Generalized autoregressive conditional heteroskedasticity. Journal of econometrics 31: BRAIONE, M. AND SCHOLTES, N. K Forecasting value-at-risk under different distributional assumptions. Econometrics 4:3. CHRISTOFFERSEN, P. AND PELLETIER, D Backtesting value-at-risk: A duratin-based approach. Journal of Financial Econometrics 2: CHRISTOFFERSEN, P. F Evaluating interval forecasts. International economic review pp CORSI, F A simple approximate long-memory model of realized volatility. Journal of Financial Econometrics 7:

27 ENGLE, R. F. AND GALLO, G. M A multiple indicators model for volatility using intra-daily data. Journal of Econometrics 131:3 27. ENGLE, R. F. AND MANGANELLI, S Caviar: Conditional autoregressive value at risk by regression quantiles. Journal of Business & Economic Statistics 22: FUERTES, A.-M., IZZELDIN, M., AND KALOTYCHOU, E On forecasting daily stock volatility: The role of intraday information and market conditions. International Journal of Forecasting 25: GHYSELS, E., SANTA-CLARA, P., AND VALKANOV, R Predicting volatility: getting the most out of return data sampled at different frequencies. Journal of Econometrics 131: GIOT, P. AND LAURENT, S Modelling daily value-at-risk using realized volatility and arch type models. Journal of Empirical Finance 11: GLOSTEN, L. R., JAGANNATHAN, R., AND RUNKLE, D. E On the relation between the expected value and the volatility of the nominal excess return on stocks. The journal of finance 48: HANSEN, P. R. AND LUNDE, A. 2005a. A forecast comparison of volatility models: does anything beat a garch (1, 1)? Journal of applied econometrics 20: HANSEN, P. R. AND LUNDE, A. 2005b. A realized variance for the whole day based on intermittent high-frequency data. Journal of Financial Econometrics 3: HEBER, G., LUNDE, A., SHEPHARD, N., AND SHEPPARD, K Oxford-man instituteï 1 2 s realized library, version 0.2. KOOPMAN, S. J., JUNGBACKER, B., AND HOL, E Forecasting daily variability of the s&p 100 stock index using historical, realised and implied volatility measurements. Journal of Empirical Finance 12:

28 KUESTER, K., MITTNIK, S., AND PAOLELLA, M. S Value-at-risk prediction: A comparison of alternative strategies. Journal of Financial Econometrics 4: KUPIEC, P. H Techniques for verifying the accuracy of risk measurement models. The Journal of Derivatives 3. LOUZIS, D. P., XANTHOPOULOS-SISINIS, S., AND REFENES, A. P Realized volatility models and alternative value-at-risk prediction strategies. Economic Modelling 40: MARTENS, M. AND VAN DIJK, D Measuring volatility with the realized range. Journal of Econometrics 138: NIETO, M. R. AND RUIZ, E Frontiers in var forecasting and backtesting. International Journal of Forecasting 32: SHEPHARD, N. AND SHEPPARD, K Realising the future: forecasting with highfrequency-based volatility (heavy) models. Journal of Applied Econometrics 25: TAYLOR, S. J. AND XU, X The incremental volatility information in one million foreign exchange quotations. Journal of Empirical Finance 4: ZHANG, L., MYKLAND, P. A., AND AÏT-SAHALIA, Y A tale of two time scales. Journal of the American Statistical Association 100: