The hidden dangers of historical simulation q

Transcription

1 Journal of Banking & Finance 30 (2006) The hidden dangers of historical simulation q Matthew Pritsker * The Federal Reserve Board, Mail Stop 91, Washington, DC 20551, United States Available online 14 June 2005 Abstract Many large financial institutions compute the Value-at-Risk (VaR) of their trading portfolios using historical simulation based methods, but the methodsõ properties are not well understood. This paper theoretically and empirically examines the historical simulation method, a variant of historical simulation introduced by Boudoukh et al. [Boudoukh, J., Richardson, M., Whitelaw, R., The best of both worlds, Risk 11(May) 64 67] (BRW), and the filtered historical simulation method (FHS) of Barone-Adesi et al. [Barone-Adesi, G., Bourgoin F., Giannopoulos, K., DonÕt look back. Risk 11(August) ; Barone-Adesi, G., Giannopoulos K., Vosper L., VaR without correlations for nonlinear portfolios. Journal of Futures Markets 19(April) ]. The historical simulation and BRW methods are both under-responsive to changes in conditional risk; and respond to changes in risk in an asymmetric fashion: measured risk increases when the portfolio experiences large losses, but not when it earns large gains. The FHS method is promising, but its risk estimates are variable in small samples, and its assumption that correlations are constant is violated in large samples. Additional refinements are needed to account for time-varying correlations; and to choose the appropriate length of the historical sample period. Ó 2005 Elsevier B.V. All rights reserved. q Board of Governors of the Federal Reserve System. The views in this paper represent those of the author and not necessarily those of the Federal Reserve Board or other members of its staff. The author thanks Daniel Davidson for valuable research assistance. * Tel.: / ; fax: address: mpritsker@frb.gov /$ - see front matter Ó 2005 Elsevier B.V. All rights reserved. doi: /j.jbankfin

2 562 M. Pritsker / Journal of Banking & Finance 30 (2006) JEL classification: C15; C52; G21; G28 Keywords: Value-at-Risk; Backtest; Historical simulation 1. Introduction A portfolioõs Value-at-Risk, or VaR, is the most that the portfolio is likely to lose over a given time horizon except in a small percentage of circumstances. For example, if a portfolio is expected to lose no more than $10,000,000 over the next day, except in 1% of circumstances, then its VaR at the 1% confidence level, over a one-day VaR horizon is $10,000,000. Because of VaRÕs simplicity, it is used as a basis for Capital Regulation: the 1996 Market Risk Amendment to the Basel Accord sets market risk capital requirements for banksõ and broker dealersõ based on the 10-day 1% VaR of their trading portfolios; and it allows 10-day 1% VaR to be measured as a multiple of one-day 1% VaR. Computing VaR is complicated because it depends on the joint distribution of all of the instruments in a portfolio; therefore, three simplifying steps are usually employed. First the dimension of the problem is reduced by modeling the change in the value of the instruments as depending on a small set of risk factors f. Second, the relationship between f and the value of instruments is approximated where necessary. 1 Finally, a simplifying assumption about the distribution of f is required. The errors in VaR estimation depend on the reasonableness of the simplifying assumptions. The purpose of this paper is to study the reasonableness of the historical simulation distribution assumption. Historical simulation methods model the distribution of f nonparametrically based on its past history. Because many large banks currently use or plan to use historical simulation methods, it is important that practitioners and regulators understand the properties of historical simulation. This paper studies historical simulation, as well as two variants: the BRW method, introduced by Boudoukh et al. (1998), and the filtered historical simulation (FHS) method, introduced by Barone-Adesi et al. (1998) and Barone-Adesi et al. (1999). 2 The empirical performance of historical simulation has been examined by Hendricks (1996), and Beder (1995) among others; and the empirical performance of FHS has been studied by Barone-Adesi et al. (2001) and Barone-Adesi and Giannopoulos (2002). The analysis here departs from related work in two ways. First, I analyze historical simulation from a theoretical and empirical perspective. The theory provides new insights on these methods. Second, the earlier empirical analysis studied the methodsõ performance with real data. A disadvantage of using real data is that true VaR is not known. As a result, the VaR methodsõ ability to track true 1 See Pritsker (1997). 2 Variations on the FHS method include Hull and White (1998) and McNeil and Frey (2000). The Hull White and FHS methods are identical when the VaR time horizon is one period. McNeil and Frey combines the FHS approach with extreme value theory.

3 M. Pritsker / Journal of Banking & Finance 30 (2006) VaR can only be measured indirectly, and it is difficult to quantify the errors in the VaR estimates. By contrast, I analyze the properties of historical simulation VaR estimates when it is applied to artificial data that was simulated from realistic time series models that were fit to real data. Because true VaR is known in the artificial data, it is possible to observe the VaR errors and directly study their properties. To focus solely on errors in distributional assumptions, I abstract from other sources of error in risk estimates by only examining simple spot positions in underlying stock indices or exchange rates. Before presenting the formal results, it is useful to present an example that illustrates the problems with historical simulation. VaR distributional assumptions are often judged in practice by whether the VaR measures provide the correct conditional and unconditional coverage for risk (Christoffersen, 1998; Diebold et al., 1998; Berkowitz, 2001). Correct unconditional coverage implies the portfolioõs losses exceed k% VaR measures k% of the time in very large samples. A VaR measure which achieves correct unconditional coverage is correct on-average. Correct conditional coverage is more stringent: it requires the measure to be correct on every day. A VaR measure is unlikely to provide exactly correct conditional coverage, but, one would hope that VaR estimates increase when risk appears to have clearly increased. To study if they do, I examined how three VaR measures respond to the stock market crash of October 19, The crash seems indicative of a general increase in equity risk; this should be reflected in how the VaR measures respond. All three VaR measures use a one-day holding period, a 1% confidence level, and all use a sample size of N = 250 days. The first VaR measure is historical simulation. This method computes VaR from the empirical cumulative distribution function (CDF) of the profit and loss (P&L) that todayõs portfolio would have earned if it was held on each of N days in the recent past. One disadvantage of historical simulation is that it assigns an equal probability weight of 1/N to each days return, which is equivalent to assuming returns are independently and identically distributed (i.i.d.) through time. This is unrealistic because return volatility is time-varying, and periods of high and low volatility cluster together (Bollerslev, 1986). When returns are not i.i.d., placing more weight on recent returns might better represent the risk of todayõs portfolio. This can be accomplished by choosing a small N, or by down-weighting past returns. The BRW method chooses the latter approach by assigning probability weights that sum to 1, but decay geometrically at rate k. For example, the three most recent returns receive weights w(1), k * w(1), and k 2 * w(1). VaR is calculated from the empirical CDF of the reweighted returns. Fig. 1 illustrates the response of VaR estimates to the crash for a portfolio which is long the S&P 500. The VaR methods are historical simulation, and BRW with k = 0.99, and k = The choice of N and the decay factors match those used by BRW. VaR estimates in the figure are presented as negative numbers because they represent amounts of loss in portfolio value. A larger VaR amount means that the amount of loss associated with the VaR estimate has increased. The principal result of the analysis is that the historical simulation VaR estimate has almost no response to the crash (Fig. 1 Panel A). To understand why, note that the historical simulation 1% VaR estimate corresponds to the third lowest return in a

4 564 M. Pritsker / Journal of Banking & Finance 30 (2006) Fig. 1. One percent VaR measures for long equity portfolio in October The figure tracks the response of VaR estimates (solid boxes) to daily returns (clear circles) during the period surrounding the October 19, 1987 market crash. 250 day rolling sample. After the crash, the second lowest return before the crash became the new VaR estimate. Because the second and third lowest returns are close in magnitude, the crash had little impact on the VaR estimate for historical simulation. By contrast, both BRW VaR measures respond strongly, rising in magnitude to the size of the crash itself (Fig. 1, Panels B and C). This occurs because they assign more than 1% probability to the crash. Unfortunately, the BRW method does not perform as well as this example suggests. To see why, consider a portfolio which is short the S&P 500. Risk estimates for long and short portfolios should respond similarly to the crash, but the BRW VaR estimates do not increase for the short portfolio until a few days after the crash when the portfolio lost money as the market recovered (Fig. 2, Panels B and C). The BRW method failed to see the increase in risk for the short-portfolio because the BRW and historical simulation methods only estimate risk from the lower tail of the P&L distribution. Because the crash was in the upper tail of the short port- Fig. 2. One percent VaR measures for short equity portfolio in October The figure tracks the response of VaR estimates (solid boxes) to daily returns (clear circles) during the period surrounding the October 19, 1987 market crash.

5 M. Pritsker / Journal of Banking & Finance 30 (2006) Fig. 3. Five percent VaR Measures for long equity portfolio in October The figure tracks the response of VaR estimates (solid boxes) to daily returns (clear circles) during the period surrounding the October 19, 1987 market crash. folioõs P&L distribution, it was ignored. This means that the BRW and historical simulation methods have the disturbing property that estimated risk never increases after large gains. The sluggish adjustment of the BRW and historical simulation methods to changes in risk at the 1% level are much worse at the 5% level. The strongest evidence for the problem is losses for the long portfolio exceeded 5% VaR on 25% of the days in October, which is far too often for correct conditional coverage (Fig. 3). Similar result hold for the short portfolio (not shown). 3 The historical simulation distributional assumption will affect the performance of other risk measures that rely on it. I illustrate this point later, by examining how it affects estimates of conditional Value-at-Risk (CVaR), which measures expected losses when losses exceed VaR. 4 The remainder of the paper studies historical simulation in more detail. Sections 2 and 3 explore the properties of the historical simulation and BRW methods from a theoretical and empirical viewpoint. Section 3.3 studies the ability of standard backtesting-based diagnostics to detect the errors in historical simulation methods. An alternative diagnostic approach is also proposed. Section 4 examines filtered historical simulation. Section 5 concludes. 2. Theoretical properties of BRW methods This section derives general properties of BRW methods. The results also apply to historical simulation because it is a BRW method. The simplest way to implement BRWÕs approach without using their precise method is to construct a history of N 3 Some results are listed as not shown in order to shorten the paper. The not shown results are available in Pritsker (2001). 4 Unlike VaR, CVaR is a coherent risk measure and hence inherits the desirable properties of coherent risk measures (see Artzner et al., 1999; Rockafellar and Uryasev, 2002).

6 566 M. Pritsker / Journal of Banking & Finance 30 (2006) hypothetical returns that the portfolio would have earned if held for each of the previous N days, r t 1,...,r t N, and then use these returns and geometrically declining weights to construct the empirical cumulative distribution function (CDF) of r at time t: Gðx; t; NÞ ¼ XN i¼1 1 frt i 6xgw t i. Because the empirical cumulative distribution function (unless smoothed) is discrete, the BRW solution for VaR at the C % confidence level will typically be sandwiched between 2 returns that have cumulative distributions that are slightly less than and greater than C. These returns correspond to BRW u and BRW o VaR estimates that slightly under and overstate VaR respectively: BRW u ðtjk; N; CÞ ¼ infðr 2fr t 1 ;...; r t N gjgðr; t; NÞ P CÞ; BRW o ðtjk; N; CÞ ¼ supðr 2fr t 1 ;...; r t N gjgðr; t; NÞ 6 CÞ. Stated in terms of portfolio losses, BRW u (tjk,n,c) is the largest of the N loss observations (loss = return) for which the probability of experiencing a greater loss is at least equal to C; and BRW o (tjk,n,c) is the smallest loss for which the probability of experiencing a greater loss is less than C. BRWÕs VaR estimator smoothly interpolates between the BRW u and BRW o estimates; numerical comparisons show that it behaves similarly to the BRW u estimator (not shown). Because it is easier to prove results for the BRW u estimator, I focus on its properties in all of the analysis and empirical results. The question of interest is how do BRW risk estimates respond to changes in risk. To begin to answer this question, the next proposition derives a necessary condition for the BRW u risk estimates to increase. Proposition 1. If r t <BRW u (t,k,n) then BRW u (t + 1,k,N) 6 BRW u (t, k,n). Proof. See the appendix. h The proposition shows if losses at time t do not exceed VaR at time t, then the VaR estimate for time t + 1 will be no greater than it was at time t. Therefore, losses exceeding VaR at time t is a necessary condition for the BRW u risk estimate to increase; the finding that the crash did not increase BRW risk estimates for the short portfolio is an extreme version of this result. An implication of the proposition is that if the C % VaR estimate for time t is correct, then with probability 1 C, the risk estimate will not be revised upward tomorrow when using a BRW method. This result is very general; it does not depend on the process generating returns. Whether this frequency of upward risk estimate revision is problematic depends on the stochastic process for returns. The rest of the paper studies single asset portfolios for which returns follow GARCH(1, 1) processes. GARCH(1, 1) processes are very tractable and they are first-order reasonable models for the heteroskedasticity of

7 M. Pritsker / Journal of Banking & Finance 30 (2006) many financial time series. 5 I will assume that each assets daily returns have mean 0, and follow a GARCH(1,1) process 6 : p r t ¼ ffiffiffiffi h t ut ; ð1þ h t ¼ a 0 þ a 1 r 2 t 1 þ b 1h t 1. ð2þ The regularity conditions for the GARCH(1,1) process are that u t is independently, identically, continuously distributed, with mean 0, and variance 1; its distribution is symmetric around 0; and its CDF is denoted F(Æ). The parameters a 0, a 1, and b 1 are all assumed to be greater than zero; and a 1 + b 1 < 1. The latter conditions ensure that the GARCH process has a long run finite unconditional variance, h, which is equal to a 0 /(1 a 1 b 1 ). For the Garch(1, 1) processes, it is straightforward to work out the probability that a VaR estimate should increase tomorrow given that it is conditionally correct today. The answer has a simple form when h t is at its long run mean. Proposition 2. When returns follow a GARCH(1, 1) process as in Eqs. (1) and (2) and h t is at its long run mean, then ProbðVaR tþ1 > VaR t Þ¼2Fð 1Þ. If u t is normally distributed then ProbðVaR tþ1 > VaR t Þ¼2 Uð 1Þ 0:3173; where U(Æ) is the standard normal CDF. Proof. See the appendix. h Propositions 1 and 2 taken together imply if a 1% VaR estimate is correct and near its long-run average value, then it should increase the next day 32% of the time (when u t is distributed normally); but it will only be revised up 1% of the time using the BRW method. This result is stated formally below. Corollary 1. Assume that (1) returns follow a GARCH(1,1) process as in Eqs. (1) and (2), (2) h t is at its long run mean, (3) VaR is estimated by the BRW or historical simulation method, and (4) the VaR estimate for confidence level c at time t is correct. Then, the conditional probability that VaR will increase over the next period but the VaR estimate will not is given by 2 * F( 1) c, which is equal to 2 * U( 1) c when u t is normally distributed. 5 The Skewed Student Asymmetric Power ARCH (Skewed Student APARCH) model is probably better than the GARCH(1, 1) model for fitting the conditional heteroskedasticity of exchange rates (Mittnik and Paolella, 2000) or equity indices (Giot and Laurent, 2003), but it is less tractable. 6 For readers who are unfamiliar with GARCH models, h t is the conditional variance of returns at time t. Eq. (2) shows that when returns at time t 1 are large in magnitude (high r 2 t 1 ) then this increases the variance of the return at time t, causing periods of high volatility to cluster together.

8 568 M. Pritsker / Journal of Banking & Finance 30 (2006) The quantitative importance of failing to detect increases in risk depends on the likelihood that very large increases are missed. When returns follow GARCH(1,1) processes, the likelihood of not detecting VaR increase of various sizes is provided below. Proposition 3. Under the assumptions of Corollary 1, the probability that VaR at time t þ 1 is at least x% greater than at time t, but the increase is not detected at time t þ 1 is given by 8 qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi >< 2F 1 þ x2 þ2x a 1 c; 0 < x < kða 1 ; cþ; ProbðDVaR > x%; no detectþ ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi >: F 1 þ x2 þ2x a 1 ; x P kða 1 ; cþ; qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð3þ where kða 1 ; cþ ¼ 1þ 1 a 1 þ a 1 ½F 1 ðcþš 2. Proof. See the appendix. h The economic significance of Proposition 3 depends on the GARCH(1, 1) parameter a 1. The next section estimates the parameters and then analyzes economic significance. 3. Empirical analysis of BRW methods 3.1. GARCH estimates and theoretical predictions The theory in the previous section predicts that the historical simulation and BRW methods will be slow in adjusting to changes in conditional risk. To further quantify the theoretical predictions, I fit GARCH(1, 1) models with Gaussian innovations to the log daily returns of the exchange rates of 10 currencies versus the US dollar, and then used the fitted parameters, reported in Panel A of Table 1, to study the probability of not detecting increases in VaR of various sizes. 7 When conditional variance is near its long-run mean, the only parameter that affects the detection probabilities is a 1 ; calculations were performed for the high, low, and mean estimated values of a 1 (0.05, 0.20, and ), when return innovations are Gaussian as well as when they are distributed as a Student-t with 10 degrees of freedom that has been normalized to have variance The returns are for January 2, 1973, to November 6, 1997, for the exchange rate of the US dollar against the British pound, the Belgian franc, the Canadian dollar, the French franc, the Deutsche mark, the Yen, the Dutch guilder, the Swedish kronor, the Swiss franc, and the Italian lira. The parameter estimates for the French franc and Italian lira, do not satisfy the restriction that a 1 + b 1 < 1. Therefore, some of the theoretical results are not strictly correct for these two exchange rates, but they would be correct for processes with slightly smaller values of b 1. 8 This degrees of freedom figure is within the range of results for GARCH(1,1) processes that have been fit to foreign exchange rates (see Bollerslev, 1987; Baillie and Bollerslev, 1989).

9 M. Pritsker / Journal of Banking & Finance 30 (2006) Table 1 GARCH(1, 1) parameters for US dollar denominated exchange rates Currency A. Whole sample B. First half of sample a 0 a 1 b 1 a 0 a 1 b 1 British pound ( ) ( ) ( ) (0.0043) (0.0069) (0.0061) Belgian franc ( ) ( ) ( ) (0.0078) (0.0082) (0.0071) Canadian dollar ( ) ( ) ( ) (0.0017) (0.0097) (0.0099) French franc ( ) ( ) ( ) (0.0037) (0.0077) (0.0054) Deutsche mark ( ) ( ) ( ) (0.0114) (0.0074) (0.0086) Japanese yen ( ) ( ) ( ) (0.0031) (0.0035) (0.0032) Netherlands guilder ( ) ( ) ( ) (0.0042) (0.0047) (0.0033) Swedish kronor ( ) ( ) ( ) (0.0124) (0.0117) (0.0110) Swiss franc ( ) ( ) ( ) (0.0119) (0.0088) (0.0088) Italian lira ( ) ( ) ( ) (0.0006) (0.0066) (0.0042) Notes: The table provides GARCH(1, 1) parameter estimates with standard errors in parenthesis for US dollar exchange rates against the listed currencies. Details are contained in Sections 2 and 3 of the text. The main result is that when return innovations are Gaussian, there is a 31% probability that increases in VaR will not be detected. Many of the undetected increases are small, but there is a reasonable probability that large increases will also not be detected, for example, for the high value of a 1, there is a 4% chance that VaR could increase more than 25% within a day, but not be detected; for the low value of a 1, there is a 4% chance that VaR will increase 7% within a day without being detected (Fig. 4). Moments of the conditional distribution of undetected increases in VaR are contained in Table 2; the Table shows, for example, the expected undetected 1-day increase in VaR for the British pound is about 5-1/2% with a standard deviation of the same amount. When return innovations are instead distributed t(10), the probability that a rise in VaR will go undetected declines slightly, but the expected size of the undetected rise increases. It is important to stress that Proposition 3, Table 2, and Fig. 4 quantify the probability that a VaR increase of a given size will not be detected on the day that it occurs. It is possible that VaR could increase for many days in a row without being detected, causing VaR errors to accumulate and grow through time, but, the theoretical results do not quantify how large the VaR errors can become. The next subsection uses the simulations to address this question.

10 570 M. Pritsker / Journal of Banking & Finance 30 (2006) Fig. 4. Probability VaR increases will go undetected. Notes: The figure plots the probability that various percentage increases in 1-day 1% Value-at-Risk will not be detected using the historical simulation and BRW methodologies when exchange rate returns follow GARCH(1, 1) processes with the low, high, and mean values of a 1 from Panel A of Table 1, and when the return innovations are distributed Nð0; 1Þ (Panel A) or standardized Student-t with 10 degrees of freedom (Panel B). Results are presented for a 1 = (solid), (long dashes), and (short dashes). All curves were computed conditional on the event that before true VaR changes, the VaR estimates are correct, and the variance of returns are at their long run averages. Additional details are provided in Section 2 of the text Simulated performance of BRW methods This section simulates the performance of the BRW u method using 200 years of daily exchange rate data that were generated using the parameters from Panel A of Table 1. 9 Although the theoretical predictions in Corollary 1 and Proposition 3 are made under restrictive conditions, for a one-day VaR horizon, the theoretical predictions (Table 2) closely match the simulated probability that increases in VaR will not be detected, and the conditional distribution of the undetected increases (Table 3). The simulations also show that about 95% of the time that VaR increases, the BRW u risk estimates do not increase. Because of this sluggish adjustment, changes in BRW and historical simulation VaR estimates have a low correlation with daily changes in true VaR (Table 4). Additionally, the VaR estimates are not very accurate: The average root mean squared error (RMSE) across the different currencies is approximately 25% of true VaR. The VaR estimates do eventually change in response to changes in true VaR; therefore the correlation of the VaR estimates and true VaR is higher than the correlation of their changes (Table 4). Nevertheless, because of the slow adjustment, the VaR errors build-up through time, and sometimes become quite large. This point is illustrated for the British pound over a 2-year period when the return innovations are distributed t(10). The results are dramatic: there are five episodes in which true VaR exceeds estimated VaR for periods of time exceeding 0.1 years. During some spells the amounts by which true VaR exceeds estimated VaR can build up to become 9 Simulation results are not presented for the Italian lira because for its estimated GARCH parameters, its conditional volatility process was explosive.

11 M. Pritsker / Journal of Banking & Finance 30 (2006) Table 2 Theoretical distribution of undetected percentage increases in VaR Currency Mean Std Skewness A. Gaussian innovations British pound Belgian franc Canadian dollar French franc Deutsche mark Japanese yen Netherlands guilder Swedish kronor Swiss franc Italian lira B. Standardized t 10 innovations British pound Belgian franc Canadian dollar French franc Deutsche mark Japanese yen Netherlands guilder Swedish kronor Swiss franc Italian lira Notes: For spot foreign exchange positions in the listed currencies against the US dollar, the table presents theoretical results on the mean, standard deviation, and skewness (Skewness = E[(x E(x))/Std(x)] 3 )of increases in 1% one-day VaR conditional on the increases not being detected when using the historical simulation or BRW methods. Exchange rate returns follow the GARCH(1, 1) processes in Panel A of Table 1. The calculations condition on the previous VaR estimates being correct, and the variance of returns being at their long run averages. as large as 60% (Fig. 5). For completeness, I repeated this analysis for Gaussian return innovations and for historical simulation estimates of 1% CVaR, that were based on the average of the three lowest returns in a sample size of 300. The results were similar to the earlier analysis (not shown). 10 Given the poor performance of the BRW and historical simulation methods, it is important to understand whether the methods that regulators and risk practitioners use to detect errors in VaR methods are capable of detecting the errors in historical simulation. This is briefly examined in the next section Can backtesting detect the problems? How should a risk manager evaluate the adequacy of his risk model? Ideally, he should do so by directly comparing the modelõs risk estimates against true risk, and 10 A more efficient method to compute CVaR is fit the tails of the distribution using extreme value theory and then use the fitted tails to estimate CVaR.

12 572 M. Pritsker / Journal of Banking & Finance 30 (2006) Table 3 Simulated distribution of undetected percentage increases in VaR Currency P(Var Inc, Not Det) P(Not DetjVar Inc) Mean Std Skew A. BRW with k = 0.97 British pound Belgian franc Canadian dollar French franc Deutsche mark Japanese yen Netherlands guilder Swedish kronor Swiss franc B. BRW with k = 0.99 British pound Belgian franc Canadian dollar French franc Deutsche mark Japanese yen Netherlands guilder Swedish kronor Swiss franc C. Historical simulation British pound Belgian franc Canadian dollar French franc Deutsche mark Japanese yen Netherlands guilder Swedish kronor Swiss franc Notes: For three VaR methods, and a spot position in each exchange rate, the table presents the probability that VaR increases and is not detected, the probability that VaR increases are not detected, and the mean, standard deviation, and skewness of undetected increases in VaR. then decide whether the errors are acceptable. Because true risk is unobservable, the direct approach is infeasible; instead risk models are usually assessed indirectly by testing the forecasting properties of the risk estimates. Many indirect tests of VaR models are referred to as backtests because they are based on the history of the risk modelõs VaR exceedances. A k% VaR exceedance is a Bernoulli random variable that takes the value 1 on days that losses exceed k% VaR, and 0 otherwise. If a VaR model has correct unconditional coverage, then k% VaR exceedances should occur k% of the time. If a VaR model has correct conditional coverage, past exceedances should not help forecast future exceedances. Therefore, the autocorrelation function of the VaR exceedances should be equal to 0 at all lags. Correct conditional coverage

13 M. Pritsker / Journal of Banking & Finance 30 (2006) Table 4 Simulated properties of BRW and Hist Sim VaR estimators Currency % Exceed RMSE % RMSE Corr w/var Corr w/dvar A. BRW with k = 0.97 British pound Belgian franc Canadian dollar French franc Deutsche mark Japanese yen Netherlands guilder Swedish kronor Swiss franc B. BRW with k = 0.99 British pound Belgian franc Canadian dollar French franc Deutsche mark Japanese yen Netherlands guilder Swedish kronor Swiss franc C. Historical simulation British pound Belgian franc Canadian dollar French franc Deutsche mark Japanese yen Netherlands guilder Swedish kronor Swiss franc Notes: For three VaR methods, when foreign exchange returns are generated as in Table 1 the table presents results on the frequency with which losses exceed VaR estimates, the VaR estimates root mean squared error, percentage root mean squared error, and correlation of the levels and changes of VaR with true VaR. implies the autocorrelation function of the history of VaR errors (VaR Estimate True VaR) should be 0 as well. The Market Risk Amendment to the Basle Capital Accord requires firms to backtest their 1%, 1-day VaR estimates for market risk using a 1-year return history. In this section, I analyze whether such backtests can identify the inadequacies of the historical simulation and BRW methods when returns are generated by the GARCH(1,1) processes in Table 1. The main finding is that the backtests have too little power to identify the deficiencies. In unconditional backtests losses exceed 1% VaR only 1.5% of the time (Tables 3 and 4). When using 1-year of data, this is too small a number of exceedances to reject correct unconditional coverage, and is in

14 574 M. Pritsker / Journal of Banking & Finance 30 (2006) Fig. 5. True and historical simulation estimates of 1% VaR for British pound. When the British pound is simulated using the estimated GARCH(1, 1) parameters from Panel A of Table 1 with standardized t(10) errors, the figure plots 2 years of the simulated time series of true (dashed line) and estimated (solid line) 1% VaR for a one-day time horizon. accord with earlier findings on the low power of unconditional backtests (Kupiec, 1995). 11 Conditional coverage tests were only examined for the BRW and historical simulation methods for 200 years of simulated data for the British pound. For all methods, the first six autocorrelation of the VaR exceedances are no greater than 0.05, and drop to 0 thereafter (Fig. 6). More than three years of data are required to reject correct conditional coverage based on these autocorrelations. 12 This is roughly in line with Christoffersen and Pelletier (2004), who find, in tests based on the expected duration between VaR exceedances, that a 2-year and 3-year backtests have less than a 50% and 70% chance of detecting the errors in historical simulation. 13 Although backtests have low power, the autocorrelation function of the VaR errors shows that conditional coverage is poor: the 50th order (2-1/2 month) autocorrelation for the best of the methods (BRW with k = 0.97) is 0.1 (not shown), and for the worst method, historical simulation, it is an astonishing 50% (Fig. 6). This con- 11 When exceedances occur 1.5% of the time and are expected to occur 1% of the time, then the normal approximation to the binomial distribution shows that more than 2-1/2 years of daily data are required to reject the null of correct unconditional coverage in a one-sided 95% test. 12 For details, see Pritsker (2001). 13 Despite the low power of these tests, Berkowitz and OÕBrien (2002) found positive first-order autocorrelation of VaR exceedances for 2 of 6 banks they examined.

15 M. Pritsker / Journal of Banking & Finance 30 (2006) Fig. 6. Autocorrelation of VaR errors and VaR exceedances for historical simulation estimates of VaR for British pound. The figure presents daily autocorrelations for the errors in 1-day historical simulation VaR estimates, and for the corresponding VaR exceedances. Results are presented for 1% VaR errors (solid), 5% VaR errors (dashes), 1% VaR exceedances (short dashes), and 5% VaR exceedances (dots). firms that the VaR errors of these methods decay slowly through time; and can accumulate to become large. The low power of standard backtests shows that enhanced backtests (Berkowitz, 2001; Christoffersen and Pelletier, 2004) as well as other approaches for testing risk models are needed. This paper uses a simulation-based approach that I refer to as parametric bootstrapping (PB). Its purpose is to test the validity of the simplifying assumptions and short-cuts that are used in risk modeling. Historical simulation can be viewed as a shortcut for formally modeling the conditional volatility of asset returns. To test the shortcut, a first-order reasonable model, that does not assume the short-cut is true, is fit to real data. The first-order model is then treated as the true model; data is simulated from it, and risk estimates from the true model are directly compared with risk estimates that are made when applying the shortcut. This approach to testing closely resembles the ideal approach that is described in the introduction to this subsection. If the first-order model is realistic, poor performance of the short-cut in the simulations is strong evidence against its accuracy in a real risk modeling situation. Because the PB approach makes the errors from using a shortcut transparent, it has been and should be applied to test other shortcuts and simplifying assumptions that are used in risk management (Pritsker, 1997). This concludes the discussion of the historical simulation and BRW methods. Both methods model assets returns nonparametrically, but at the cost of inappropriately adjusting to changes in conditional volatility. The next section discusses filtered historical simulation, which blends the nonparametric properties of historical simulation, with parametric modeling of conditional volatility.

16 576 M. Pritsker / Journal of Banking & Finance 30 (2006) Filtered historical simulation Filtered historical simulation is a Monte Carlo based approach that combines parametric modeling of risk factor volatility with nonparametric modeling of the factor innovations. It is best explained by illustrating its use in computing 10-day, 1% VaR. To begin, consider a simple portfolio that depends on a single risk factor r, and assume that r follows a GARCH(1, 1) process as in Eqs. (1) and (2); also assume that the processõs parameters, the function F(Æ), and h t +1, have all been estimated, and are known at time t. Given these estimates, 10-day return paths for r can be generated by (1) drawing t +1 from its distribution F(Æ); (2) applying t +1 and h t +1 in Eq. (1) to generate r t +1 ; (3) applying h t +1 and r t +1 in (2) to generate h t +2. Repeatedly iterating on these steps generates the entire 10-day return path. When many 10-day return paths have been generated, their associated empirical CDF can be used to compute 10-day 1% VaR or other risk measures. To implement the FHS method, the parameters of the GARCH(1, 1) process are estimated using quasi-maximum likelihood estimation (QMLE). The QMLE approach treats as if it is normally distributed. Under appropriate regularity conditions even if is not normally distributed, QMLE recovers the parameters and the conditional volatilities h t at each time t (Bollerslev and Wooldridge, 1992). Substituting r t and h t in Eq. (1) recovers estimates of the past realizations of. Because is i.i.d., draws from F(Æ) are made by drawing from its empirical distribution. FHS extends naturally to multiple risk factors: GARCH(1, 1) models are estimated separately for each risk factor; and draws from the empirical distribution of are made by randomly drawing a past date, and using that dateõs realization of the vector (Barone-Adesi et al., 1999). The main advantages of this implementation are that the volatility models are simple, and the correlation matrix of does not need to estimated. The FHS approach has also been extended to allow for more complicated volatility models (Audrino and Barone-Adesi, 2005); and in a univariate setting it has been extended to allow for modeling the tails of using extreme value theory (McNeil and Frey, 2000) Empirical analysis of filtered historical simulation An important assumption in the FHS methodology is that the vector is i.i.d. through time, which implies its correlation matrix is constant. This assumption could be unreasonable in long time-series. To study this issue, I fit GARCH(1, 1) models for the exchange rates in Section 3, using return data from January 1973 to June 1986 (Table 1, Panel B). The resulting time series of estimated was split in half, and I used FisherÕs z transformation to test the null of constant correlations across 14 Engle and Manganelli (2004) present an alternative modeling approach in which the quantiles of portfolio value follow an autoregressive process with non-normal return innovations. Its disadvantages relative to FHS are that the parameters need to be re-estimated if the portfolio changes; additionally, the approach makes risk attribution difficult because it models portfolio P&L, but not the individual factors.

17 M. Pritsker / Journal of Banking & Finance 30 (2006) subsamples. 15 Although the tests for each correlation coefficient are not independent, the null hypothesis was overwhelming rejected for 86 out of 90 of the correlation coefficients, and many correlation differences were economically significant (not shown). One possible solution is to implement FHS with smaller samples for. To investigate this possibility, I repeated the correlation comparison using data from adjacent 1-year intervals, and found that while there were fewer rejections, even over a 2-year period correlations are not constant (not shown). These results suggest a short historical sample of (small N) is required to achieve constant correlations; however, this presents a tradeoff because reducing N creates errors in the empirical distribution of that could reduce the accuracy of the risk estimates. To study the role of N in accuracy, while abstracting from other sources of error, I used FHS to estimate 10-day 1% VaR and CVaR for a spot position in the British pound while assuming its GARCH parameters, todayõs conditional volatility, and the history of the last N Õs are known. Results are presented for six different distributions of and for N that ranged from 2 years to 10 years of daily observations. For each specification, I computed 1000 VaR and CVaR risk estimates that were each based on 10, day sample paths. To compute true risk, was drawn from its true distribution and 1 million sample paths were used. The main results from the analysis is that FHS estimates of VaR and CVaR are highly variable, and positively skewed. The variability and skewness is especially acute when returns are fat tailed and N is small. For example, when return innovations have very fat tails (3 degrees of freedom), and N = 500, a 95% confidence interval shows that VaR could be understated by as much as 32% or overstated by as much as 95% (Table 5); similarly CVaR could be understated by as much as 44% or overstated by as much as 83% (Table 6). As intuition for these results, note that for small N FHS tend to either under- or over-sample the extremes of the distribution of. For example, if N = 500, the probability that the history of will not contain a draw from the upper or lower 0.2% tail of its return distribution is or about 13%. This is problematic because on a 10 day sample path, the probability that at least one draw should come from the 0.2% tails is , or 4%. Since these 4% of sample paths contain extreme draws that are important for 1% VaR, their omission is likely to lead to lower VaR estimates. On the other hand, over-sampling is also problematic. For example, the probability that 500 historical realizations of will contain at least one draw in the 0.05% tail is about 22%. If such a draw exists, the FHS method will assign it a probability weight of 0.2% and will hence oversample from it, leading to very large estimates of risk when oversampling occurs. Before closing, I would like to draw a contrast with the FHS results from Pritsker (2001). The earlier analysis emphasized the median downward bias of FHS estimates of VaR in small samples. I now believe the more important issues are nonconstant 15 Following Kendall and Stuart (1963), let n be the number of subsample observations, let r and q be the estimated and true correlation, and define z and n by z ¼ 0.5 log 1þr 1þq 1 r, and n ¼ 0.5 log 1 q. z n is q 1 approximately normal with mean 2ðn 1Þ and variance n 3.Ifz 1 and z 2 are independent subsample estimates of z, then under the null of constant correlation 0.5(n 3)(z 1 z 2 ) 2 is asymptotically v 2 (1).

18 578 M. Pritsker / Journal of Banking & Finance 30 (2006) Table 5 Distribution of VaR errors for filtered historical simulation N df S.D. Mean Median 2.5% 25% 75% 97.5% Notes: The table reports the distribution of errors in 10-day 1% FHS VaR estimates as a percentage of true VaR. N is the length in days of the return history that is used to compute VaR. The return innovations are distributed as a standardized Student-t with df degrees of freedom. The case df = 1, the corresponds to the standard normal distribution. correlations and methods to reduce FHS estimator variability. Addressing these issues remains a topic for future FHS research. 5. Summary and conclusion This paper studied the properties of risk estimates that use the historical simulation method, and a variant that was introduced by Boudoukh, Richardson, and Whitelaw (BRW). Risk estimates using both methods respond sluggishly to changes in conditional volatility, and respond to large price moves asymmetrically: risk estimates increase after large losses, but not after large gains. Because of these deficiencies, errors in risk estimates accumulate through time and sometimes become very large. Despite the large errors, traditional backtests have little power to detect them. This paper shows that a better approach to evaluate the risk methods is to use a parametric bootstrapping approach as is done in this paper.

19 M. Pritsker / Journal of Banking & Finance 30 (2006) Table 6 Distribution of CVaR errors for filtered historical simulation N df S.D. Mean Median 2.5% 25% 75% 7.5% Notes: The table reports the distribution of errors in 10-day 1% FHS CVaR estimates as a percentage of true CVaR. N is the length in days of the return history that is used to compute CVaR. The return innovations are distributed as a standardized Student-t with df degrees of freedom. The case df = 1, the corresponds to the standard normal distribution. This paper also studied filtered historical simulation (FHS). The FHS method is promising because it blends parametric modeling of conditional volatility with nonparametric modeling of innovations to the risk factors. This paper shows that additional research for choosing the length of the methodõs historical sample period is needed because too long a sample violates the methodõs assumption of constant conditional correlation, while too short a sample reduces the accuracy of the methodõs nonparametric elements. Appendix A A.1. Proof of Proposition 1 When the VaR estimate using the BRW method is estimated for returns during time period t + 1, the return at time t N is dropped from the sample, the return

20 580 M. Pritsker / Journal of Banking & Finance 30 (2006) at time t receives weight 1 kn, and the weight on all other returns are k times their 1 k earlier values. Define, r(c) ={r t i, i =1,...,NjG(r t 1 ;t,n) 6 C}. To verify the proposition, it suffices to examine how much probability weight the VaR estimate at time t + 1 places on those returns for which r > BRW u (t,k,n). There are two cases to consider: Case 1. r t N 62 r(c). In this case, since r t 62 r(c) by assumption, then G( BRW u (t,k,n); t +1,k,N) =kg( BRW u (t,k,n) 6 C. Therefore, BRW u ðt þ 1; k; NÞ 6 BRW u ðt; k; NÞ. Case 2. r t N 2 r(c). In this case, since r t 62 r(c) by assumption, then G( BRW u (t,k,n);t +1,k,N) <k G( BRW u (t,k,n)<c. Therefore, BRW u ðt þ 1; k; NÞ 6 BRW u ðt; k; NÞ. A.2. Proof of Proposition 2 The long run mean of h t, denoted a h, is equal to 0 1 a 1 b 1. Therefore, when h t ¼ h, h t +1 > h t if and only if a 0 þ a 1 r 2 t þ b 1 h t > h t ; a 0 þ a 1 h t u 2 t þ b 1 h t > h t ; a 0 þ 1 b 1 < u 2 t a 1 h t a ; 1 a 1 þ b 1 1 þ 1 b 1 < u 2 t a 1 a ; 1 1 < u 2 t. Finally, the result follows because: Probð1 < u 2 t Þ¼Probðu t < 1ÞþProbðu t > 1Þ ¼2F ð 1Þ. When u t is standard normal then F(.) = U(Æ). h A.3. Proof of Proposition 3 Let A(x) denote the event that VaR tþ1 VaR t > 1 þ x; let B denote the event that VaR increases and is not detected; and let C(x) = A(x) \ B. It suffices to compute the probability of C(x) for all x > 0 to complete the proof. sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi! a 0 þ a 1 r Prob½AðxÞŠ ¼ Prob 2 t þ b 1 h t > 1 þ x ða1þ ¼ Prob h t sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi! a 0 þ a 1 h t u 2 t þ b 1 h t > 1 þ x h t. ða2þ