The efficiency of Anderson-Darling test with limited sample size: an application to Backtesting Counterparty Credit Risk internal model

Transcription

1 The efficiecy of Aderso-Darlig test with limited sample size: a applicatio to Backtestig Couterparty Credit Risk iteral model Matteo Formeti 1,2, Luca Spadafora 1,3, Marcello Terraeo 1, ad Fabio Rampoi 1 arxiv: v1 [q-fi.rm] 18 May UiCredit S.p.A., Piazza Gae Auleti 3, Mila, Italy 2 Uiversità Carlo Cattaeo - LIUC, Scuola di Ecoomia e Maagemet, C.so Matteotti, Castellaza (VA) 3 Uiversità Cattolica del Sacro Cuore, Faculty of Mathematical, Physical ad Natural Scieces, via dei Musei, Brescia May 19, 2015 (Pre-Prit Versio) Abstract This work presets a theoretical ad empirical evaluatio of Aderso-Darlig test whe the sample size is limited. The test ca be applied i order to backtest the risk factors dyamics i the cotext of Couterparty Credit Risk modellig. We show the limits of such test whe backtestig the distributios of a iterest rate model over log time horizos ad we propose a modified versio of the test that is able to detect more efficietly a uderestimatio of the model s volatility. Fially we provide a empirical applicatio. JEL: C19. C22 Keywords: Aderso-Darlig Test, Backtestig, Couterparty Credit Risk. The views, thoughts ad opiios expressed i this paper are those of the authors i their idividual capacity ad should ot be attributed to UiCredit S.p.A. or to the authors as represetatives or employees of UiCredit S.p.A. Corrispodig Author, mformeti@liuc.it

2 Itroductio Backtestig is defied as "the quatitative compariso of a model s forecasts agaist realized values" (BCBS, [7]). I Couterparty Credit Risk (CCR), the model forecasts regard the estimates of iterest rates, credit spreads, equity or commodity values, that are the uderlyig risk factors drivig the mark-to-market of OTC derivatives, up to the logest maturity of the cotracts. As remarked by BCBS [7], baks choose their ow best ad appropriate method to aggregate, ad the validate, the overall quality of the model forecasts. This ca be doe through a sythetic value, such as the outcome of a statistical test, havig the goal to detect weakess of model forecasts. We remark that such forecasts are computed up to the logest maturity ad directly affect the exposure towards a couterparty. So backtestig is oe of the istrumets through which Risk Maagemet assesses the forecast of couterparty exposure, ad ideed the bak s risk weighted asset value. A failure i that test advocates a model chage such as a differet model parametrizatio or eve a chage i model assumptios (e.g. log-ormal or t-distributio). Baks, havig a CCR iteral model, compute backtestig: (i) o risk-factors level with the aim to validate the properties of the stochastic process used to simulate iterest rates, credit spreads, forex ad equities; (ii) o trades level, such as plai vailla or exotic optios aimig at validate the sigle deal exposure; (iii) o couterparty level to validate the soudess of the estimated exposures. We remark that risk maagers are iterested i backtestig all the forecast distributio shape for a give risk factor, i order to detect a uderestimatio of risk (i.e., variace) that may lead to a uderestimatio of the Regulatory Capital measure (RWA) or the maagerial measures (Expected Positive Exposure, Potetial Future Exposure). I this paper we study from a empirical ad theoretical poit of view the statistical properties of the Aderso-Darlig (AD) test ([1], [2]) used to backtest risk factors, i the particular case of limited sample size. I fact, as AD test has bee widely used i empirical literature, from biology to sociology, due to its well kow statistical properties for large sample (see [10] ad [11]), its robustess whe sample is limited has ever bee studied, to the best of our kowledge, i the case of CCR modelig. O the other had, regulatio asks baks to verify the modelig choice through a backtestig program (CRR art. 293-b), to use at least three years of historical data for model estimatio purposes (CRR art ) ad to have procedures i place to idetify ad cotrol the risks for couterparties where the exposure rises beyod the oe-year horizo (CRR, art.289-6). With this respect, we observe that from a purely statistical poit of view, the use of overlappig time widows to verify model performace caot be cosidered a sigificat improvemet for what cocer the reductio of statistical ucertaity of the forecasted variables. I fact, it ca be show that i our cotext, if the radom variables are i.i.d., the additioal iformatio icluded i a statistical estimator based o overlappig time widows, compared with the oe related to a o-overlappig estimator is ot eough to reduce sigificatly the related statistical ucertaity. As a cosequece, a backtestig methodology based o overlappig time widows would face similar statistical issues related to the small sample size as the oes related to a o-overlappig methodology. Furthermore, the limited sample size is also a uavoidable coditio liked to the limited legth of available market data history (e.g. USD/EUR started i 1999). For these reasos, we focus o the AD statistical properties maily whe the backtestig dataset is made by 5-10 observatios. 1

3 It is importat to remark that for risk maagemet purposes, ad i particular for CCR backtestig validatio, test should be able to detect a volatility uderestimatio that is more dagerous from a risk maagemet poit of view tha volatility overestimatio. This meas that test rejectio power should be higher whe the forecasted distributios has a smaller variace tha the empirical oe. I additio, this property should also hold i case of limited sample size i order to deal with practical situatios where the dataset is typically small. For these reasos, we propose a modified versio of AD test that may help risk maager to detect easily the uderestimatio of volatility. We highlight that uiformity tests, such as the AD, Kolmogorov-Smirov (KS) [17], Jarque-Brera [13] or Cramer-vo Mises (CM) [5], help to statistically validate the model forecastig values because, at each backtestig date, we ca map the realized value to the correspodig p-value of the forecasted distributio. I particular, for a give risk factor (e.g. iterest rate, foreig exchage, commodity) r, backtestig date t ad time horizo s, the p- value F (r) correspodig to the realizatio r(t + s) is computed accordig to the followig algorithm: F (r) = 1 N+2 for r(t + s) < ˆr (1) F (r) = i+1 N+2 for ˆr (i) < r(t + s) < ˆr (i+1) F (r) = N+1 N+2 for r(t + s) > ˆr (N) where ˆr (i) represets the i forecasted value, ad N the total umber of forecasted values. As a cosequece, the collectio of the ordered mapped values should be uiformly distributed if the model is perfectly matchig the realized values. A example of the applicatio of this tests i the Couterparty Credit Risk backtestig framework ca be foud i [3]. The structure of the paper is as follows. I sectio 1 we briefly summarize the AD test ad its reduced efficiecy whe the sample size is limited. Sectio 2 proposes a modified versio of AD test i order to detect faster a uderestimatio of the volatility. The i sectio 3 we compute a umerical exercise that use real data to geerate fictitious time series i order to compare the AD test, ad our modified versio, with respect to KS test. Last, i sectio 4 we show as a case study the backtestig of the Black-Karasiski model applied to the Euribor 6 Moth. Sectio 5 summarizes our results. 1 Aderso-Darlig Test Aderso ad Darlig [1] desiged a statistical test i order to determie whether a give sequece of radom variables X = {x 1,..., x } comes from a theoretical cumulative distributio fuctio (CDF) F (x). The ull hypothesis H 0 is that the data follow F (x), so this test should be used to prove that the data do ot follow F (x), give a cofidece level. The AD test is based o the estimatio of the followig radom variable: W 2 = + [F (x) F (x)] ) 2 df (x) (1) ( F (x)(1 F (x)) 2

4 where F (x) is the target CDF ad F (x) is the empirical distributio derived from the data. The umerator of Eq. (1) represets the distace of the theoretical distributio from the empirical oe, while the deomiator represets the variace of the empirical estimatio of F (x) whe the cetral limit theorem holds, i.e. whe is large eough. I other words, Eq. (1) represets the average of the squared errors betwee the two distributios (theoretical ad empirical) weighted by the implicit ucertaity due to the estimatio method of the empirical CDF (order statistics). As the CDF of a radom variable is always distributed uiformly betwee zero ad oe (i.e. F (x) U(0, 1)), W 2 is a fuctio of uiformly distributed radom variables whe H 0 ad the cetral limit theorem hold. I particular, it does ot deped o the distributio F (x). I this cotext, we observe that whe the variace of the uiform distributio (F (x)(1 F (x))/) is close to zero, i.e. for rare evets, F (x) 0 or F (x) 1, the squared error is magified by the small deomiator; i this sese we cosider the AD test as more sesitive with respect to the tails of the distributio. O the other had, we poit out that, if the umber of observatios used to perform the AD test is low, the deomiator of the itegral i Eq. 1 is large, i.e. a large variace is associated to the differece betwee the theoretical ad the empirical distributio; as a cosequece also a large differece betwee the distributios would fall iside the variace amplitude ad the AD test will ot be able to reject H 0 as the ucertaity i the measuremet will be too large to brig to ay coclusio. So, i order to reject H 0, the differece betwee the theoretical CDF ad the empirical oe has to be larger tha their statistical ucertaity. Eq. (1) ca be also expressed as: W 2 = k=1 2k 1 l(f (x k)) + l(1 F (x +1 k )) (2) where x i X are the empirical data ordered from the smallest to the largest values ad is the size of the sample (i.e. umber of backtestig dates). The empirical distributio of W 2 was estimated by AD ([1]) ad we report the percetiles of W 2 distributio i Tab. (1) with the first lie idicatig the upper tail probabilities ad the secod lie represetig the correspodig percetiles. Probability Percetile Table 1: Upper tail percetiles for Aderso-Darlig W 2 test. 1.1 Aalytical Result: the Efficiecy of AD Test The AD test ca be used for CCR backtestig purposes as a tool to verify whether the model forecasted distributios are comparable with the empirical oe, for a give cofidece level. I this cotext, the ull hypothesis shall be that the two distributios are equal. So the test gives positive outcome whe the ull hypothesis is ot rejected, meaig the model distributio is ot sufficietly differet from the empirical oe to cosider the model as wrog. Give our backtestig approach, a large ucertaity i the empirical CDF estimatio is due to the small umber of observatios. This egatively affects the rejectio rate of our 3

5 test acceptig distributios just because of the limited sample size. As a cosequece, we questio about the itroductio of a efficiet measure i such test i order to icrease its accuracy whe the sample size is small. I geeral, a measure is cosidered efficiet ad faithful if its ucertaity is much smaller tha its expected value 1 ; i our case the statistical ucertaity is related to the empirical estimatio of the CDF, while the expected value is the theoretical CDF for each poit of the empirical distributio. Ufortuately the simple variace estimatio for each poit of the distributio is ot eough as the AD measure requires a sum over all the probabilities F (x i ) for {i = 1,..., }, ad oe has to cosider also the whole covariace matrix of the order statistics. So at each sigle poit the expected value of our empirical estimatio is give by F (x i ) = p i, ad the covariace structure is give by p mi(i,j) p i p j [16]. We ca defie the coefficiet of variatio (CoV) or relative stadard deviatio, for each poit of the distributio as: c = σ µ (3) = 1 = / mi (p, q) pqdpdq 1 1/ pdp (4) ( 1 1) 2 ( ) ( ) (5) 2 2 = where σ ad µ represet the stadard deviatio ad the expected value of the sum of over 1 estimated probabilities o a rage betwee 1/ ad 1, i.e. 1/ F (x)df (x). I Fig. 1 we show the decay of the coefficiet of variatio as a fuctio of the umber of observatios. I order to obtai a CoV below 10%, = 50 observatios are required. The CoV idicator gives importat iformatio about the performace of the AD test whe the sample size is small, ad it ca be used as a warig level whe AD test is applied. O the other had, we remark that CoV is derived assumig the Cetral Limit Theorem (CLT) holds, whereas this hypothesis does ot hold for very small sample sizes. Therefore oe should cosider additioal correctios to Eq. 3 that we do ot take ito cosideratio for this work. (6) (7) 2 Asymmetric Extesio of AD Test I order to detect the uderestimatio of the actual volatility whe the umber of observatios is small, we derive a extesio of the AD test that ca be used for risk maagemet purposes. The mai idea for AD test extesio relies o the observatio that, whe the sample size is small, it is easier to reject the ull hypothesis whe the empirical variace of the distributio is larger tha the forecasted oe; o the cotrary, whe the empirical variace is smaller tha the forecasted oe, may observatios fall iside the theoretical distributio, so it is more 1 The approach is ot cosistet if the Expected value is equal to zero 4

6 Figure 1: Decay of the coefficiet of variatio as a fuctio of the umber of observatios difficult to reject the H 0 hypothesis (this fact is further discussed with umerical examples i the ext sectios). Startig from this observatio, oe could magify this asymmetric effect defiig a AD-Asym fuctio that itroduces a more proouced o liear behaviour whe the differeces betwee the theoretical ad the empirical CDF are large, as i the case of variace uderestimatio. We stress that the term asymmetric refers to the differet behavior of the AD test whe the variace of the forecasted distributio is over/uder estimated ad ot to the aalytical form of W 2. We geeralized Eq. (1) as: W 2 Asym = + [F (x) F (x)] 2β ) β df (x) (8) ( F (x)(1 F (x)) where β 1 is the parameter that cotrols the amplitude of the asymmetric effect; obviously whe β = 1 we recover Eq. (1). I this paper we focus our attetio o the special case β = 2. I this way, the small variace amplitude due to the small is compesated by the β expoet. I order to apply this ew asymmetric formulatio of the AD-test i practical situatios, we have to: estimate the itegral i the Eq. (8); obtai the distributio of the W 2 radom variables assumig that H 0 is true; compare the empirical value of W 2 with the theoretical distributio obtaied at the previous poit ad decide if H 0 should be rejected at a give cofidece level. Oce a good estimatio of Eq. (8) is obtaied, the secod ad the third steps ca be overcome by a umerical simulatio of the W 2 r.v. ad cosiderig the obtaied CDF. O the cotrary, some care is required for the itegral estimatio give the small sample size. I particular, followig the lies i [1], we cosider a sample {x 1,..., x } observatios ad we defie x 0 = 0 5

7 ad u = F (x) so W 2 Asym = = k=1 uk u k 1 ( k 1 k=1 ( k 1 1) 4 2 u 1 1 = γ + k=1 ( ) u k 1 4 ( ) (1 u) u 2 du (9) α 1 (k) u k ) 4 1 u + 2 ( k 1 2 ( ) ( k k 1 ) ( ) k 1 3 log(u) ) 3 u k log(1 u) + u (10) u k 1 + α 2 (k)log(u k ) + α 3(k) u k 1 + α 4(k)log(1 u k ) (11) where u k = F (x k ) ad γ, α 1 (k), α 2 (k), α 3 (k), α 4 (k) are fuctios reported i the Appedix. Eq. (11) gives a ew AD-Asym idicator that measures the differeces betwee empirical ad theoretical CDF, emphasisig outliers discrepacies. I the followig sectios, we refer to Eq. (11) as AD-Asym idicator i order to compare its performaces with respect to the stadard AD test. 2.1 Testig Aderso-Darlig i limited sample size I this sectio we compare the AD test ad our proposed modificatio (Eq. 11) with the aim to obtai some isight o the AD-Asym performaces whe the umber of observatios is small. We remark that for risk maagemet purposes, as i the CCR case, the mai goal is to reject more efficietly the forecasted (theoretical) distributios havig a smaller variace tha the actual (empirical) oe (i.e. volatility uderestimatio). We perform our umerical simulatios assumig differet theoretical probability desity fuctios (PDFs). I particular, we compare a Gaussia distributio N(0, 1), assumed to be our theoretical PDF, with other Gaussia distributios havig same mea but differet variaces. We estimate for a fixed umber of observatios the rejectio rate of the AD test as a fuctio of the real stadard deviatio cosidered to geerate the sample set. We expect that, for large eough sample size, AD test rejects the differet distributios with a rejectio rate equal to 1 i 100% of the cases, with the oly exceptio whe the real variace is equal to the theoretical oe, i.e. equal to 1. I the latter case, the rejectio rate depeds o the cofidece level required for the test, i our case set equal to 5%. I Fig. 2 we report the results of our aalysis for small sample size. As expected, the rejectio rate is higher for larger stadard deviatios but the overall performace is ot high. For example, we obtai a rejectio rate aroud 50% whe cosiderig 5 observatios ad the stadard deviatio is twice larger tha the empirical oe. The asymmetry betwee smaller ad larger stadard deviatios becomes less ad less evidet as the umber of observatios icreases: whe = 100 the rejectio rate becomes symmetric. I particular, the AD test performs better where we are more iterested i (the right part of Fig. 2-a), rejectig the most extreme cases whe there is volatility uderestimatio. I fact, the AD test has a higher rejectio rate whe the distributio of real data has the same mea of the forecasted distributio, but a larger variace. The reaso is, as already discussed i the previous sectio, that a realized PDF with a larger variace has a large probability 6

8 (a) AD-SS. (b) AD-Asym. Figure 2: Rejectio rate as a fuctio of the stadard deviatio (a) Sample size with 5 observatios. (b) Sample size with 20 observatios. Figure 3: Rejectio rate as a fuctio of the stadard deviatio to geerate outliers (with respect to the forecasted PDF) eve with few observatios. O the cotrary, if oe cosiders the realized PDF with a lower variace tha the forecasted oe, all the observatios will fall iside the forecasted distributio ad it is more difficult to coclude if the two distributios are differet. We replicate the same aalysis comparig the AD test with our AD-Asym versio usig differet sample size. Fig. 2-b shows that we have a similar rejectio rate whe volatility is overestimated (σ < 1) ad a higher rejectio rate whe volatility is uderestimated. Therefore, we coclude that if we eed to avoid volatility uderestimatio the AD-Asym test performs better especially i limited sample. Fig. 3-a plots the results whe the sample size is made of five observatios: the AD-Asym double rejects the ull hypothesis whe the volatility of the forecasted distributio is half the realized (e.g., AD-Asym rejects 40% istead of 22% whe σ=1.5); the result is similar whe observatios are twety as show i Fig. 3-b. I order to exted our result, We compare the rejectio rate of differet statistical tests aimig at idetifyig if a sample, that is always limited i our case, belogs to a give theoretical distributio. I particular, we cosidered five differet statistical tests: Stadard symmetric AD test, as described i the previous sectios (SS) AD test with tail sesitive idicatio, (AD-Asym) Cramer-Vo Mises (CM) 7

9 (a) Tests assumig volatility overestimatio. (b) Tests assumig volatility uderestimatio. (c) Tests assumig mea overestimatio ad the correct volatility. (d) Tests assumig mea overestimatio ad volatility uderestimatio. Figure 4: Rejectio rate of differet statistical tests Kolmogorov-Smirov (KS) Fig. 4 shows the results of our aalyses as a fuctio of ; i particular, i each figure, the rejectio rate of differet statistical tests is compared assumig a Gaussia N(0, 1) theoretical distributio ad, empirical Gaussia distributios with differet meas ad stadard deviatios. As expected, AD-Asym test shows the overall better performaces (higher rejectio rate) tha the other tests, whe the volatility is uderestimated. As it is a well-kow fact that the distributios of fiacial returs frequetly show fat tails behavior, we aalysed the performaces of differet statistical tests cosiderig a Gaussia N(0,1) theoretical distributio ad a empirical t-studet distributio with variace equal to oe ad ν degrees of freedom. I Figs. 5 we show the results of our aalyses. The overall result is that AD-Asym performaces are higher tha the oes related to the AD test, although we kow such test are strogly iflueced by the degrees of freedom cosidered. This fact ca be explaied cosiderig the fat-tails ature of the t-studet distributio that implies a higher risk if compared to a Gaussia distributio with the same variace. For this reaso, the use of a Gaussia distributio would imply a uderestimatio of the risk that is better idetified by AD-Asym test, as already spotted out i the previous sectio. I additio, we observe that the covergece of the rejectio rate to 100% is ot reached whe = 150, o the cotrary to the Gaussia case. The reaso for this behaviour ca be foud if oe cosiders that, as the first two momets (mea ad the variace) of the theoretical ad the empirical distributio are exactly met, oe eed to estimate the higher momets i order to observe differeces betwee the PDFs. Give that for a fixed umber of observatios the ucertaity icreases with the momet order, it is reasoable to expect that the umber of observatios 8

10 eeded to obtai the covergece must be larger tha the oe obtaied with Gaussia PDFs (see sectio 1.1). (a) t-studet with ν = 2.8 degrees of freedom. (b) t-studet with ν = 3 degrees of freedom. (c) t-studet with ν = 3.5 degrees of freedom. Figure 5: Rejectio rate for AD test (blue lie with diamods) ad AD-Asym test (red lie wi dots) assumig a Gaussia N(0,1) theoretical distributio ad a t-studet distributio 3 Numerical Exercise Oce uderstood the mai effect from a theoretical poit of view, we test the AD i a realistic case. We cosider the Euribor 6-moth iterest rate havig historical time series sice 1999 ad, as a cosequece, it has five/six observatios i the backtestig sample, if we are usig three years of historical data for estimatio purposes ad a forecastig horizo of two years. However, i order to address the property of AD test i limited sample size but havig the possibility to exted our aalysis to higher sample size, we detred the iterest rate time series usig the Hodrick-Prescott [12] filter applied to the five days time series up to March I this way, we were able to obtai more robust calibratio results of iterest rate model. We the estimate a Black-Karasiski [9] iterest rate model o the cycle compoet of the log-filtered time series i order to geerate, usig that estimated parameters, fictitious time series with the desired sample to backtest. We use the Black-Karasiski short-rate model because of its aalytical tractability ad for the added beefit that rates caot become egative. I fact, the Black-Karasiski models the logarithm of the cyclical compoet of the iterest rate y(t) = exp(r(t)) usig the Orstei-Uhlebeck stochastic process: dy t = k ( α y t ) dt + σdwt (12) 9

11 where k captures the speed of the y t log-cycle toward its log equilibrium value α (the mea level), σ is the volatility of the process ad W t is the browia motio. We estimate the followig parameters usig a momet matchig approximatio formula method o the overall dataset: Θ [ α = ; k = ; σ = ] The momet matchig method guaratees the level of mea reversio is positive, which is cosistet with the values of the risk factor. I particular, the estimatio method is based o a two-steps formulas where the mea reversio level is firstly estimated ad the plugged i the mea reversio rate: α = 1 y(t k ) (13) k=1 k = k=1( y(tk ) y(t k 1 ) ) 2 2 ( k=1 y(tk ) a ) 2 (14) I Fig.6 we plot the Euribor 6 Moths, the filtered cyclical ad the tred compoet. We Figure 6: Iterest rate Euribor 6 moths filtered out with Hodrick-Prescott ad simulatios usig the estimated parameters [ α, k, σ] simulate iterest rates scearios usig the estimated Θ parameters ad a costat umber of backtested observatios, i order to verify the statistical properties of the tests at differet time horizos. I the Fig. 7 we show the results of the AD ad KS rejectio rate usig at every simulatio a backtestig sample with 5 observatios. All simulatios are performed 10 thousad times usig the same parameters. The rejectio rate is plotted agaist the ratio betwee the average stadard deviatio of the simulated process (usig Θ parameter) ad 10

12 (a) Time Horizo: 1 Week. (b) Time Horizo: 1 Moth. (c) Time Horizo: 1 Year. (d) Time Horizo: 2 Years. Figure 7: AD vs KS rejectio rate usig 5 observatios i the backtestig sample the stadard deviatio at every backtestig dates (calibrated cosiderig a time widow of three-years ad keepig k costat 2 ): = σ sim / σ bkt where the variace of the process is estimated as described i Appedix B. 3 As a cosequece, a positive implies a uderestimatio of the volatility of the backtestig sub-samples, therefore we expect a higher rejectio rate of both tests, while a egative implies the backtestig volatility is higher tha the oe that geerated the fictitious time series, so we expect a lower or eve a zero rejectio rate. The umerical exercise seems to cofirm the theoretical aalysis show i Figs. 1-2 where the AD test has a higher rejectio rate compared to KS whe sample size is low. 4 We cotrol these results usig 10, 25 ad 100 observatios obtaiig similar results. Last, we otice that the volatility estimated usig three years is 2 The variace of the process, as described i Appedix B, depeds o the mea reversio rate k so i order to avoid a misaligmets betwee the volatility used to simulate the process (Θ) ad the oes used for backtestig purposes, we opt to take a costat k, with a overall effect that we verified is egligible for our purposes. 3 The variace of the Black-Karasiski process depeds o the time horizo, so we expect such differece o the x-axis as show i Figures 7 4 The extreme boudaries of Fig. 7 have few observatios so we discard it from our aalysis. 11

13 Figure 8: Distributio of = σ sim / σ bkt o average lower tha the oe used for simulatig the fictitious real iterest rate. This is cofirmed i Fig. 8 where we plot the histogram of the distributio of. 4 A empirical applicatio of Aderso-Darlig test I this sectio, we show a example of a risk factor backtestig applyig the AD ad KS test o iterest rate forecasted values obtaied usig the Black-Karasiski [9] short-rate model. We computed this exercise to show a real test case of forecastig log time horizo ad, at the same time, to verify the performace of AD test i real case. We still remark that we are iterested i backtestig mark to future distributio at differet horizo sice regulatory measuremets ad maagerial exposures are derived from the etire distributio of iterest rates products. For simplicity reaso, we still use the Black-Karasiski as give by equatio (12) applied to the Euribor 6-Moth time series where parameters are calibrated usig 3 years of historical data ad the momet matchig method show i equatio (13). We slightly modify the short rate model (12), i order to perform tests with alterative model set-up, as follow d y t = k ( α(t) y t ) dt + γσdwt (15) where the parameter γ is a adjustmet to the volatility value. Fig. 9 plots the Euribor 6-Moths ad the forecasted distributio at two years whe settig γ = 1 5. We remark that the backtestig time widow covered the iterest rate regime switch occurred durig the Lehma crisis, ad i geeral iterest rate models would have serious problems to correctly forecast the crisis ad the followig period. Fig. 9 shows that: (i) oly five backtestig dates are available for testig the model due to the log forecast horizos; (ii) the empirical data ofte fall i the extreme tail of the forecasted distributio; (iii) oly by icreasig model volatility or adjustig the mea oe could capture the realised values. Table 2 reports the simulated rates ad the correspodig real rates at the five backtestig dates. Table 3 reports the results of the AD, AD-Asym ad KS tests for differet level of volatility assumig a 5% 5 We use 3000 simulatios at each backtestig dates. 12

14 Figure 9: Euribor 6-Moths: forecastig distributio at 2 years time horizo Date 24 dec dec dec dec dec 12 Value Miimum Average Maximum Table 2: Euribor 6-Moth ad statistics forecasted values at two years time horizo level of cofidece 6 : all tests reject the hypothesis that the model is correct (see the bold row ad colum). I order to compare the performace of AD, AD-Asym ad KS test we re-parametrized the model by icreasig volatility. I particular, we icrease γ from oe up to three times the estimated volatility at each backtestig date ad we observe that, as expected, tests accept the ull hypothesis whe the forecasted distributio is very large. O the cotrary the AD-Asym is less reactive to such adjustmets compared to AD ad KS. We cosider it as a empirical proof of coservativeess of our test. Last, we check the robustess of those results with differet term of the curve (i.e, 1 year, 5 year ad 10 year term) obtaiig similar results. Test/γ AD Reject Reject Reject Accept Accept AD-Asym Reject Reject Reject Reject Accept KS Reject Reject Reject Accept Accept Table 3: AD, AD-Asym ad KS results for Euribor 6-Moth forecasted values at 2 year time horizo. 6 A test rejects the ull hypothesis whe the p-value is lower tha the cofidece level. 13

15 5 Coclusio The Europea CCR/CRD IV requires to perform a i-depth aalysis of model s forecasts used to compute the Couterparty Credit Risk exposures. As a cosequece, baks eed to perform a backtestig program which results are very importat for the assessmet of model weakesses that impact o couterparty exposures ad risk weighted assets. O the same time, baks face a commo situatio whe forecastig risk factors at log time horizos but satisfyig the requiremet of three years data for model calibratio: a very limited sample dataset. With this respect, from a purely statistical poit of view, the use of overlappig time widows to verify model performace caot be cosidered a sigificat improvemet for what cocer the reductio of statistical ucertaity of the forecasted variables. As a cosequece, a proper approach for the CCR model backtestig should be developed takig ito cosideratio the strog costrais give by the limited statistical power of small dataset. With this respect it worths to observe that, i geeral, the statistical test should be able to detect easily the model s volatility uderestimatio, that is more dagerous from a risk maagemet poit of view, tha volatility overestimatio. For these reasos, i this work we focused o the Aderso-Darlig (AD) test with the goal to uderstad whe test is able to detect a volatility uderestimatio, ad we derive a extesio of the AD test that ca be used for risk maagemet purposes. The mai idea for such extesio relies o the observatio that AD test rejects more the case where the real variace of the distributio is larger tha the forecasted oe, but the rejectio rate is i ay way low due to the ucertaity related to the small sample size. As a cosequece, we magify this asymmetric effect itroducig a more proouced o liear behavior whe the differeces betwee the forecasted ad the real distributio are large whe the model s volatility is uderestimated. This meas that test rejectio power will be higher whe the forecasted distributios has a smaller variace tha the real oe. Ad this property should also hold i case of limited sample size i order to be coservative. We verified the property of our modified test i the limited sample size case showig that it has a overall better rejectio performace i compariso to the stadard uiform test whe the forecastig distributio is wrog. We check this result comparig it with the AD test ad other uiform test, such as the Kolmogorov-Smirov test, usig a umerical example that forecasts the iterest rate Euribor 6-Moth values at two year time horizo with a Black-Karasiski short rate model. We the use such model to backtest the real iterest rate durig the time widow that lead us to reject the chose model. 14

16 A Fuctios Defiitio for AD-Asymmetric Test I the followig, we report the fuctios defied i Eq. (9): γ = u + ( ) u 1 ( ( ) 3 ( ) log ( u + 1) ) ( 1) 3 log (u ) ( 1) (16) u α 1 (k) = 4k3 6k 2 + 4k 1 4 (17) α 2 (k) = 2 6k2 6k + 2 4k 3 + 6k 2 4k (18) α 3 (k) = ( + k)4 + ( k + 1) 4 4 (19) α 4 (k) = k 2 + 6k 2 + 4k 3 6k 2 + 4k 1 4 (20) B Variace of the Black-Karasiski process Var[X t ] = E[(X t ) 2 ] E[(X t )] 2 (21) E[(X t ) 2 ] = exp {2 l(x 0 )e kt + 2a(1 e kt ) + σ2 } k (1 e 2kt ) E[(X t )] = exp {l(x 0 )e kt + a(1 e kt ) + σ2 } 4k (1 e 2kt ) 15

17 Refereces [1] Aderso, T.W., Darlig, D.A., (1954), A Test of Goodess-of-Fit. Joural of the America Statistical Associatio 49: [2] Aderso, T.W., (2010), Aderso-Darlig Tests of Goodess-of-Fit, mimeo Stadford Uiversity [3] Afuso, F., Karyampasz, D., Nawrothx, A., (2014), A Soud Basel III compliat framework for backtestig Credit Exposure Models, Workig Paper [4] Capital Requiremets Directive, (Jue, ), Directive ad a Regulatio (EU) No 648/2012 [5] Aderso, T.W. (1962), O the Distributio of the Two-Sample Cramer-vo Mises Criterio, The Aals of Mathematical Statistics, 33 (3), pp [6] Basel Committee o Bakig Supervisio, (1996), Supervisory Framework For The Use of Backtestig i Cojuctio With The Iteral Models Approach to Market Risk Capital Requiremets [7] Basel Committee o Bakig Supervisio, (2010), Soud practices for backtestig couterparty credit risk models. Basel Committee o Bakig Supervisio [8] Basel Committee o Bakig Supervisio, (2011), A global regulatory framework for more resiliet baks ad bakig systems [9] Black, F., Karasiski, P. (1991), Bod ad Optio pricig whe Short rates are Logormal, Fiacial Aalysts Joural, pp. 52ï 59 [10] Daiel, W. W., (1990), Applied oparametric statistics, Secod Editio, Duxbury Thomso Learig, Pacific Grove, CA. [11] Gibbos, J. D., Chakraborti, S., (2003), Noparametric Statistical Iferece, Marcel Dekker, Ic., New York [12] Hodrick, R., Prescott, E. C. (1997), Postwar U.S. Busiess Cycles: A Empirical Ivestigatio, Joural of Moey, Credit, ad Bakig 29 (1): 1Ð16 [13] Jarque, C., Bera, A.K., (1980), Efficiet tests for ormality, homoscedasticity ad serial idepedece of regressio residuals, Ecoomics Letters, 6 (3), pp [14] Rahma, M., Pearso, L.M., Heie, H.C., A Modified Aderso-Darlig Test for Uiformity, Bulleti Malaysia Mathemathical Scieces Society (2), 29(1), pg (2006) [15] Rahma, M. Chakrobartty, S., (2004), Tests for uiformity: a comparative study, J. Korea Data & Iformatio Sci. Soc. 15(1), pp [16] Shervish M., (1997), Theory of Statistics, Spriger Series i Statistics [17] Smirov N.V., (1948), Tables for estimatig the goodess of fit of empirical distributios, Aals of Mathematical Statistics 19, pp

18 [18] Stephes, M.A. (1986), Tests Based o EDF Statistics, I D Agostio, R.B. ad Stephes, M.A. Goodess-of-Fit Techiques, New York [19] Watso, G.S., (1961), Goodess-of-fit tests o a circle, Biometrika 48, pp