P2.T5. Market Risk Measurement & Management Jorion, Value-at Risk: The New Benchmark for Managing Financial Risk, 3 rd Edition Bionic Turtle FRM Study Notes By David Harper, CFA FRM CIPM www.bionicturtle.com
Jorion, Chapter 6: Backtesting VaR DEFINE BACKTESTING AND EXCEPTIONS AND EXPLAIN THE IMPORTANCE OF BACKTESTING VAR MODELS.... 3 EXPLAIN THE SIGNIFICANT DIFFICULTIES IN BACKTESTING A VAR MODEL.... 4 DESCRIBE THE PROCESS OF MODEL VERIFICATION BASED ON EXCEPTIONS OR FAILURE RATES. 5 2
Jorion, Chapter 6: Backtesting VaR Define backtesting and exceptions and explain the importance of backtesting VaR models. Explain the significant difficulties in backtesting a VaR model. Describe the process of model verification based on exceptions or failure rates. Define and identify type I and type II errors. Explain why it is necessary to consider conditional coverage in the backtesting framework. Describe the Basel rules for backtesting. Define backtesting and exceptions and explain the importance of backtesting VaR models. Model validation is the process of asking: is this a model adequate? Validation tools include: Backtesting Stress testing Independent review and oversight Backtesting is the verification that actual losses are reasonably consistent with projected losses. Backtesting compares the history of VaR forecasts to actual (realized) portfolio returns. According to Jorion, backtesting: Gives a reality check on calibration Central to Basel Committee s ground-breaking decision to allow internal VaR models for capital requirements Under the Basel II internal models approach (IMA) to Market Risk, banks must backtest their VaR model (in addition to stress-testing); i.e., the green/yellow/red traffic light We can backtest a VAR model with relative ease. When the VaR model is perfectly calibrated, the number of observations falling outside VAR should be in line with the confidence level. The number of exceedences is also known as the number of exceptions. With too many exceptions, the model underestimates risk. This is a major problem because too little capital may be allocated to risk-taking units; penalties also may be imposed by the regulator. Too few exceptions are also a problem because they lead to excess, or inefficient, allocation of capital across units. The number of loss observations (e.g., daily losses) that exceed the VaR is called the number of exceedences or exceptions. For example, if the VaR model is perfectly calibrated: A 95% daily VaR should be exceeded about 13 days per year (5% * 252 = 12.6) A 99% daily VaR should be exceeded about 8 days per three years (3 * 252 *1% = 7.6) 3
Jorion: Backtesting is a formal statistical framework that consists of verifying that actual losses are in line with projected losses. This involves systematically comparing the history of VAR forecasts with their associated portfolio returns. These procedures, sometimes called reality checks, are essential for VAR users and risk managers, who need to check that their VAR forecasts are well calibrated. If not, the models should be reexamined for faulty assumptions, wrong parameters, or inaccurate modeling. This process also provides ideas for improvement and as a result should be an integral part of all VAR systems. Backtesting is also central to the Basel Committee's ground-breaking decision to allow internal VAR models for capital requirements. It is unlikely the Basel Committee would have done so without the discipline of a rigorous backtesting mechanism. Otherwise, banks may have an incentive to understate their risk. This is why the backtesting framework should be designed to maximize the probability of catching banks that willfully understate their risk. On the other hand, the system also should avoid unduly penalizing banks whose VAR is exceeded simply because of bad luck. This delicate choice is at the heart of statistical decision procedures for backtesting. Explain the significant difficulties in backtesting a VaR model. There are at least two difficulties in backtesting a VaR model: Backtesting remains a statistical accept/reject decision with two errors (Type I and Type II). Consequently, backtesting cannot tell us with 100% confidence whether our model is good or bad. Our decision to deem the model good or bad is itself a probabilistic (less than certain) evaluation. An actual portfolio is contaminated by (dynamic) changes in portfolio composition (i.e., trades and fees), but the VaR assumes a static portfolio o Contamination minimized in short horizons o Risk manager should track both the actual portfolio return and the hypothetical return o Sometimes a cleaned-return approximation is used: actual return minus (-) fees/commissions/net income 4
Describe the process of model verification based on exceptions or failure rates. We verify model by recording the failure rate. Under null hypothesis that model is correctly calibrated (Null H0: correct model), number of exceptions (x) follows a binomial probability distribution ( ) = (1 ) Jorion s backtest: illustrating a 99.0% VaR model backtest with binomial distribution The assumptions are simple: The backtest (aka, estimation) window is one year with 250 trading days; T = 250 The bank employed a 99.0% confident value at risk (VaR) model; p = 0.01 The backtest entails analyzing the results of an actual (realized) series of results. Because each daily outcome either exceeded the VaR or did not, the historical window of observations can be characterized with a binomial distribution (although please notice we implicitly assume independence). In the case of a correct model (below left; p = 1%), the probability of a Type I error is 10.8% (i.e., 1 89.2%). In the case of an incorrect model (below right; p = 3%), the probability of a Type II error is 12.8%. p 0.01 p 0.03 T 250 T 250 =BINOMDIST() =BINOMDIST() pmf CDF pmf CDF 0 8.1% 8.1% 0 0.0% 0.0% 1 20.5% 28.6% 1 0.4% 0.4% 2 25.7% 54.3% 2 1.5% 1.9% 3 21.5% 75.8% 3 3.8% 5.7% 4 13.4% 89.2% 4 7.2% 12.8% 5 6.7% 95.9% 5 10.9% 23.7% 6 2.7% 98.6% 6 13.8% 37.5% 7 1.0% 99.6% 7 14.9% 52.4% 8 0.3% 99.9% 8 14.0% 66.3% 9 0.1% 100.0% 9 11.6% 77.9% 10 0.0% 100.0% 10 8.6% 86.6% The left-hand panel above characterizes the distribution of a correct 99.0% model (p = 0.01). Over 250 trading days, we should not be surprised to observe, for example, three exceedences because the probability of this outcome is fully 21.5%. On the other hand, the probability is only 0.0806% that a correct 99.0% VaR model would produce 9 exceedences (rounded to 0.1% above). 5
The same distributions above are plotted below: 30% 25% 20% 15% 10% 5% 0% Correct model (p=0.01) 0 1 2 3 4 5 6 7 8 9 10 Under the dubious assumption of indepenence (recall the binomial assumes i.i.d.), the binomial model can be used to test whether the number of exceptions is acceptably small. If the number of observations is large, we can approximate this binomial with the normal distribution. That is due to the central limit theorem, in the case; although the binomial already tends to approximate the normal as the sample size increases. Normal approximation of the binomial distribution (which characterizes the backtest) 30% 25% 20% 15% 10% 5% 0% Jorion provides the shortcut based on the normal approximation: Incorrect model (p=3%) 0 1 2 3 4 5 6 7 8 9 10 = And (1 ) (0,1) (x) is the number of observed exceedences; e.g., VaR was exceeded (x) times over the backtest window (pt) is the ex ante expected number of exceedences; e.g., we expect a 95.0% VaR to be exceeded on (0.05*T) days over a (T) day horizon because that is the mean of the binomial distribution (x pt) is the distance from our observation (sample mean) to the hypothesized mean, if the model is correct SQRT[p*(1-p)*T] is the standard deviation, or standard error, which standardizes the distance, such that (z) is approximately a standard normal variable 6
In Jorion s boxed J.P. Morgan Example (Box 6-1), the VaR confidence level is 95.0% and the sample consists of 252 trading days. Consequently, the cutoff is given by (please note the normal deviate = 1.96 because this is a two-tailed significance test): Cutoff = (1-95% confidence) * 252 + 1.96 * SQRT(95% * 5% * 252) = 19.4 Inputs: JP Morgan VaR 95% 15 0.6937 51.2% Days (T) 252 16 0.9827 67.4% 17 1.2718 79.7% 18 1.5608 88.1% 19 1.8498 93.6% 20 2.1389 96.8% 21 2.4279 98.5% 22 2.7169 99.3% 23 3.0060 99.7% X Cutoff: 19.4 1.9600 95.0% 7