DEPARTMENT OF ECONOMICS AND FINANCE COLLEGE OF BUSINESS AND ECONOMICS UNIVERSITY OF CANTERBURY CHRISTCHURCH, NEW ZEALAND

Transcription

1 DEPARTMENT OF ECONOMICS AND FINANCE COLLEGE OF BUSINESS AND ECONOMICS UNIVERSITY OF CANTERBURY CHRISTCHURCH, NEW ZEALAND Model Selecion and Tesing of Condiional and Sochasic Volailiy Models Massimiliano Caporin and Michael McAleer WORKING PAPER No. 58/00 Deparmen of Economics and Finance College of Business and Economics Universiy of Canerbury Privae Bag 4800, Chrischurch New Zealand

2 Model Selecion and Tesing of Condiional and Sochasic Volailiy Models* Massimiliano Caporin Deparmen of Economics and Managemen Marco Fanno Universiy of Padova Ialy Michael McAleer Economeric Insiue Erasmus School of Economics Erasmus Universiy Roerdam and Timbergen Insiue The Neherlands and Insiue of Economic Research Kyoo Universiy Japan Sepember 00 * The second auhor wishes o acknowledge he financial suppor of he Ausralian Research Council, Naional Science Council, Taiwan, and he Japan Sociey for he Promoion of Science. This paper is o appear in Handbook on Financial Engineering and Economerics: Volailiy Models and Their Applicaions, L. Bauwens, C. Hafner and S. Lauren (eds.), Wiley, New York.

3 Absrac This paper focuses on he selecion and comparison of alernaive non-nesed volailiy models. We review he radiional in-sample mehods commonly applied in he volailiy framework, namely diagnosic checking procedures, informaion crieria, and condiions for he exisence of momens and asympoic heory, as well as he ou-of-sample model selecion approaches, such as mean squared error and Model Confidence Se approaches. The paper develops some innovaive loss funcions which are based on Value-a-Risk forecass. Finally, we presen an empirical applicaion based on simple univariae volailiy models, namely GARCH, GJR, EGARCH, and Sochasic Volailiy ha are widely used o capure asymmery and leverage. Keywords: Volailiy model selecion, volailiy model comparison, non-nesed models, model confidence se, Value-a-Risk forecass, asymmery, leverage. JEL: C, C, C5, C58.

4 . Inroducion Model selecion and model comparison, especially of he condiional mean or firs momen of a given random variable, has been widely considered in he sciences and social sciences for an exended period. The relevance and imporance of such a opic comes from he recognized fac ha he rue daa generaing processes are generally unknown. As a resul, several approaches have been proposed o verify if a given model is able o replicae or capure he empirical feaures observed on sample daa (he realizaion of he daa generaing process), and o check if here is a preference across alernaive models ha migh be considered given he sample daa and he purposes of he analysis. In his paper we focus on model comparison and selecion in a specific framework, namely univariae volailiy models for financial ime series. From he seminal work of Engle (98) and Bollerslev (986), GARCH models have become a very popular ool in empirical finance. They have been generalized in several ways (see, for example, Bollerslev e al. (99, 994) and McAleer (005)). A companion family of models is ha of sochasic volailiy (SV), inroduced by Taylor (98, 986), and exended in several direcions (see, for example, Ghysels e al. (996) and Asai e al. (006)). Tradiional mehods for model selecion and comparison could easily be exended and applied wihin specific families of models (for insance, wihin GARCH or wihin SV specificaions). However, some model classes, or some specific models wihin a given model class, may be non-nesed, hereby requiring appropriae approaches or novel echniques for he model selecion sep. In he following, we will consider separaely he comparison of alernaive specificaions insample, hereby resoring o nesed and non-nesed model comparison, diagnosic checking, and ou-of-sample model comparison based on he forecass of given models. The discussion herein is based on univariae models ha are capable of capuring financial ime series asymmery and/or leverage, bu he resuls presened can be generalized o oher model classes a he univariae level. The mehods can also be generalized o he mulivariae level, following Paon and Sheppard (009) and Caporin and McAleer (009, 00). The remainder of he paper proceeds as follow. In subsecion., we inroduce he models o be used. Secion discusses he model selecion and esing mehods, disinguishing beween in-sample and ou-of-sample approaches. Secion 3 includes an empirical example on a se of sock marke indices. Finally, Secion 4 gives some concluding commens. 3

5 . Model specificaions In his paper we illusrae some approaches o model selecion and comparison making use of simple and well-known univariae volailiy models. We consider he radiional GARCH(,), is exension o capure asymmery, he GJR(,) model of Glosen e al. (00), Exponenial GARCH(,) (EGARCH), and he Auoregressive SV() (also known as SV) specificaions. We choose hese hree models for wo simple reasons, namely hey are non-nesed and can capure asymmery and leverage (wih he obvious exclusion of GARCH(,), which is a benchmark model). In order o simplify model evaluaion and comparison, we assume in he following ha he analyzed reurn series, r, has been filered from is mean, so ha we can focus on a zeromean series, ε, ha display condiional heeroskedasiciy, ε = σ z. Furhermore, he uni variance innovaion,, is a sandardized residual. z If he condiional variances, σ, follow a GARCH(,) model, he following equaion represens heir law of moion: σ = ω+ αε + βσ, () where ω 0, α 0, β 0, and α + β are sufficien condiions o guaranee posiive condiional variances for all observaions. If we inroduce asymmery o GARCH(,), we obain he asymmeric or hreshold model of Glosen e al. (993), GJR(,): ( ) σ ω αε γε I ε 0 βσ = + + < + () Where he parameer γ capures asymmery, and ( ) akes he value when ε < 0, and 0 oherwise. I ε < is an indicaor funcion, which 0 A clarificaion is needed here in order o avoid a common misconcepion beween asymmery and leverage: (i) asymmery is a feaure ha is inended o capure he empirical regulariy ha posiive and negaive shocks of equal magniude have differen impacs on volailiy; (ii) 4

6 leverage is inended o capure he possibiliy ha negaive shocks increase volailiy while posiive shocks decrease volailiy. As a maer of model design, few condiional volailiy models allow for leverage effecs. For example, GARCH is symmeric and hence has no leverage. Despie commens o he conrary in various economeric sofware packages (for insance, EViews and Malab), he GJR model (also known as Threshold GARCH, or TGARCH) may be asymmeric, bu i is unlikely o have leverage, as he ARCH effec mus be negaive, which is conrary o virually every empirical finding in he financial economerics lieraure. The hird model we consider is he EGARCH(,), where he condiional variance equaion is defined in erm of log-variances: ln ( ) z z ln ( σ = ω+ α + γ + β σ ). (3) Noe ha he coefficiens need no o be posiive, while β < o avoid explosive variance paerns. In equaion (3), he parameers α and γ influence he presence of asymmery and leverage: if z 0, he shock s impac on condiional variances is ( α + γ) z, while if z < 0, he impac is ( γ α) z. As a resul, if γ = 0 he model is symmeric, and hence canno have leverage. We have asymmery (wih a larger impac on volailiy of negaive shocks as compared wih posiive shocks of similar magniude) if γ < 0, and leverage (whereby negaive shocks increase volailiy and posiive shocks decrease volailiy) if γ < 0 and γ < α < - γ. The EGARCH model can have leverage, bu wo resricions on he parameers α and γ mus be saisfied (some economeric sofware manuals sae incorrecly ha leverage arises hrough a consrain on a single parameer, namely γ ). Finally, we consider he sochasic volailiy model, which assumes ha he innovaion erm follows: ε = exp h z (4) where is a uni variance innovaion and he condiional variance exp h is driven by he z following dynamic equaion for : h ( ) 5

7 h+ φ0 φh = + +η (5) where he parameers are no required o be posiive, φ < o avoid explosive paerns, and he innovaion erm, η, has variance σ η. As shown Yu (005), he SV model displays a leverage effec if he wo innovaion erms, η and z, are negaively correlaed, while asymmery may be included following, for insance, he approaches of Danielsson (994), So e al. (00), and Asai and McAleer (005).. Model Selecion and Tesing Selecion of he bes or he mos appropriae model may be based on in-sample or ou-ofsample crieria, or boh. In he following, we will address hese wo approaches separaely. Such a choice derives from purely illusraive purposes, and should no be inerpreed as a preference for one of he wo mehods. Indeed, idenificaion of an opimal model would seem o require an opimal balance beween hese wo approaches. In empirical applicaions we search for models ha capure he feaures of he analyzed daa, and ha provide accurae ou-of-sample forecass. Boh elemens may no be presen over all models and hus, in empirical sudies, a rade-off will likely exis. This possible inconsisency may be resolved in par by evaluaing he purpose of an empirical exercise. Srucural analysis may have greaer emphasis on in-sample fi, while forecasing exercises will necessarily concenrae on ou-of-sample oucomes. Neverheless, boh aspecs need o be considered, as does he role of research experise.. In-sample comparisons This paper examines condiional volailiy (GARCH) models and sochasic volailiy (SV) processes. We focus on alernaive model specificaions ha belong o he same family (eiher GARCH or SV). If he models we compare have known mahemaical and asympoic properies (such as sric saionariy of he underlying random process, and consisency and asympoic normaliy of he esimaors), we may compare hem by checking if he condiions ensuring he exisence of momens or asympoic properies are saisfied. In principle, models 6

8 where hese condiions are no saisfied, or do no even exis, should be discarded. In pracice, his is ypically no he case. For insance, log-momen condiions ensuring sric saionariy and ergodiciy of GARCH models are repored in Nelson (990) and Bougerol and Picard (99), among ohers. These condiions are also sufficien for consisency and asympoic normaliy of quasi-maximum likelihood esimaors (QMLE) (for example, see Elie and Jeanheau (995), and Boussama (000)). Sronger bu simpler momen condiions for ergodiciy, saionariy, consisency and asympoic normaliy of he QMLE, have been provided in Ling and McAleer (00a, b) and McAleer e al. (007). In pracice, log-momen condiions are generally difficul o verify, especially for mulivariae processes, while momen condiions may be considered as a useful diagnosic check. Noably, well wrien sofware should implemen hese condiions (which are generally represened as non-linear parameric resricions) wihin he esimaion sep, hereby implicily checking hem. As an example, consider he GJR model of equaion (). In his case, he saionariy and ergodiciy condiion, under he assumpion ha shocks follow a symmeric densiy, is given as α + 0.5γ + β <, while he condiion for he exisence of he fourh-order momen is β + αβ + 3α + βγ + 3αγ + 0.5γ <. The log-momen condiion is given as E ln ( αz ) + γz I( z < 0) + β < 0, bu i could be difficul o verify as i requires he evaluaion of he expecaion of a funcion of an unknown random variance and of unknown coefficiens. From a differen viewpoin, we may compare models wih respec o he feaures hey are supposed o be capuring. For example, we may prefer volailiy models wih asymmery o specificaions characerized by a symmeric news impac curve. Model preference based on model flexibiliy should obviously be mached wih he saisical significance of esimaed parameers associaed wih a paricular feaure. For insance, referring o he GJR model, i can capure asymmery hough no leverage, and hence is more flexible han he symmeric GARCH(,) specificaion. However, GJR should be preferred empirically o GARCH if he esimaed asymmery coefficien, γ, is saisically significan. Similarly, if we consider he SV model wih leverage (his model can capure leverage, and hence asymmery), replacing (5) wih (see Danielsson, 994): h φ0 φh δε δ ε η = (6) Then ess of he coefficiens δ and δ could be associaed wih he significance of boh he size and sign of shocks. 7

9 Tess of significance associaed wih single parameers or of model feaures are linked o diagnosic procedures based on he likelihood funcion. In fac, model comparison could also consider esing nesed and/or non-nesed models. In general, when we compare models belonging o he same family (such as wihin GARCH or SV), hese are ypically nesed comparisons. Therefore, he validiy of parameric resricions could be evaluaed by significance ess or, more appropriaely, by Likelihood Raio (LR) or Lagrange Mulipliers (LM) ess. In order o presen some simple examples, he GJR(,) model ness he simple GARCH(,) model under a zero resricion on he parameer driving he asymmery; APARCH ness GARCH which is obained fixing he power coefficien o ; SV model wih asymmery ness he simpler SV model under a zero parameric resricion similar o ha of GJR. In hese cases, assuming correc specificaion of he model (paricularly of he innovaion densiy), LM and LR ess have he sandard asympoic properies, and he LM saisic can be evaluaed when he analyic score is available (see Fiorenini e al. (996) for an example). For a comparison of models belonging o separae (or non-nesed) families of hypoheses, such as GARCH versus SV, or EGARCH versus GARCH, non-nesed ess are required. Ling and McAleer (000) and McAleer e al. (007) propose simple procedures o compare GARCH and GJR models agains he EGARCH model. Denoe by ˆ σ he esimaes of ime variance obained from a GJR model, and consider he following EGARCH specificaion: ( v ) ( ) ( ˆ ω α η v γη β δ σ ) ln = ln + ln (7) where v are he EGARCH condiional variances and η are he EGARCH sandardized residuals. The es of he EGARCH null hypohesis agains he GJR alernaive corresponds o esing δ = 0. Similarly, he es wih GJR as he null involves a es of δ = 0 in he auxiliary regression: ( 0) σ = ω+ αε + γε ε < + βσ + δ (8) ˆ I v where vˆ is he esimae of he ime variance obained from an EGARCH model. The corresponding ess for GARCH agains EGARCH can be obained as special cases of hose given above. 8

10 A differen es for GARCH agains EGARCH was proposed in Lee and Brorsen (997). The auhors suggesed a es based on he likelihood of wo compeing non-nesed models, based on he procedures developed in Cox (96, 96). The Cox es compares wo parameric models by evaluaing he difference beween maximum likelihood values as a deviaion from is expecaion. Lee and Bronsen (997) evaluae he es saisic by using Mone Carlo mehods. However, i is no clear if he condiions underlying he Cox es are saisfied. In paricular, he wo likelihoods should belong o separae families, ha is, for a given parameer choice, he null hypohesis canno be arbirarily closely approximaed by he alernaive. A furher aspec ha may affec he validiy of he es of he EGARCH model as he null hypohesis is ha he saisical properies of EGARCH are as ye no known. The approach of Cox (96, 96) is also closely relaed o he comparison mehods oulined in Kim e a. (998), who sugges a likelihood raio es for non-nesed models by obaining he sampling disribuion of he es saisic hrough Mone Carlo mehods. In his case, he esed non-nesed models are GARCH and Sochasic Volailiy, making he Cox es more appropriae. The procedure oulined in Kim e al. (998) involves he following seps: () Esimae he GARCH and SV model parameers and evaluae he corresponding ; ˆ L x; ˆ θ, respecively, where he likelihoods, denoed by LSV ( x θ SV ) and GARCH ( GARCH ) circumflex denoes esimaed parameers, evaluae he likelihood raio saisics: ( 늿 θ ) ( θ ) LR = log L x; log L x; SV, GARCH SV SV GARCH GARCH ( 늿 θ ) ( θ ) LRGARCH, SV = log LGARCH x; GARCH log log LSV x; SV, where he firs model represens he null hypohesis, and he SV densiy is evaluaed by simulaion mehods, following he procedure in Kim e al. (998); () Simulae M pahs under he null, esimae boh models on each pah, and evaluae he M likelihood raio saisics; (3) Tes he null hypohesis using a Mone Carlo es, deermining he p-value of he empirical likelihood raio saisics under he simulaed densiy of he LR es saisic. Noe ha, he LR es saisic is no consrained o be posiive as he wo models are nonnesed. Moreover, by reversing he null and alernaive hypoheses, he es oucomes may lead o rejecion or non-rejecion of boh models as he respecive null hypoheses. Clearly, he procedure oulined in Kim e al. (998) derives an approximae LR saisic densiy, and is also influenced by he fac ha he rue parameers are no known. To sae he obvious, his es is compuaionally inensive. 9

11 Kobayashi and Shi (005) propose a closely relaed es for EGARCH agains SV. Their approach differs from he previous mehod as hey modify he SV model. In fac, hey consider he following SV parameerizaion: ε = σzexp h z (9) h ε ε = η (0) φ0 φh α β ση exp( h ) exp( h ) z 0 ρ D, η 0 ρ () The model of Kobayashi and Shi (005) is a slighly modified version of he model in Danielsson (994), where he volailiy equaion includes a dependence on boh he sign and size of he sandardized innovaions. Noably, he model includes boh asymmery and leverage as he parameers need no be posiive. In he conex of he slighly modified SV model, EGARCH is associaed wih he parameric resricion, σ = 0. Kobayashi and Shi (005) propose a Lagrange Muliplier (LM) es for η he null hypohesis σ η = 0 (EGARCH) agains an alernaive of posiive variance for he volailiy equaion. The LM es has an advanage ha only EGARCH needs o be esimaed. The Mone Carlo simulaions repored o verify he size and power of he es show ha he LM es for EGARCH agains SV has good size and reasonable power (bu he resuls would seem o be heavily dependen on he values of he parameers). In addiion o hypohesis esing approaches, informaion crieria may also be considered o compare models by using heir likelihood penalized by a funcion of he number of parameers and number of sample observaions. These mehods allow a comparison of models where he condiional variances depend on observable quaniies, such as GARCH and EGARCH, bu canno be applied o compare GARCH and SV as he likelihood funcion for SV models differs from ha of condiional variance specificaions (see Kim e al. (998) for an example of he evaluaion of he SV likelihood by simulaion mehods). Alernaive models of variances and volailiy may also be compared hrough heir abiliy o capure he heeroskedasiciy inheren in financial ime series. The mos common approach for diagnosic checking is he Ljung-Box es saisic applied o he squared sandardized residuals, wih he preferred model as he one ha permis greaer whiening of he residuals. Furhermore, disribuional hypoheses could also be considered in order o evaluae which 0

12 densiy is closer o he analyzed daa. Sandard ess such as he Jarque-Bera normaliy es, or he more general Kolmogorov-Smirnov, may be considered in his conex. In-sample comparisons, and he subsequen choice of he bes model, may be opimal for srucural analysis, bu i does no guaranee an opimal choice for ou-of-sample forecasing. In his case, he lieraure provides a number of alernaive approaches for model comparison. In he following secion, we presen some ha are ailored for comparing condiional variance models.. Ou-of-sample comparisons A comparison of SV and GARCH models ou-of-sample may follow wo differen approaches: a direc comparison of variance forecass, or an indirec comparison of variance models hrough he possible uses of he corresponding variance forecass. This dichoomy follows from Paon and Sheppard (009), who presen a number of alernaive heoreical approaches.... Direc model evaluaion Wihin he direc comparison, alernaive models are conrased by ess direcly based on variance forecass. Denoe by ˆ σ j, he ime variance forecas of model j, and by σ he rue and unknown variance a ime. For each model we may evaluae, over a given forecas horizon, a se of sandard quaniies. Two well known examples are he Mean Absolue Error (MAE) and Mean Squared Error (MSE): MAE m j () m ( j) = σ ˆ σ, = MSE m ( j) ( σ ˆ σ j, ). (3) = = m

13 Given hese quaniies for each model, he preferred model will ypically have lower values of boh MAD and MSE, meaning lower deviaions from he rue variance. A closely relaed comparison mehod is he use of Mincer-Zarnowiz (969) regressions, where he variance forecass are used as explanaory variables for he rue variance: σ = α + βσˆ + ε. (4) j, In his alernaive framework, opimal models should have α = 0 and β =, wih a higher value of R. Therefore, models providing appropriae or similar coefficien values in (4) could be ranked by means of R values. However, wo problems arise in boh he Mincer-Zarnowiz-ype regressions and in he use of MSE or MAE: (i) he rue variance is no known; and (ii) ranking models on he basis of one or more saisical indicaors is no necessarily a formal saisical es. Wih respec o he firs issue, unbiased esimaes of he rue variance could be recovered by realized volailiy esimaors (see Barndorff-Nielsen and Shephard (00a,b) and Barndorff- Nielsen e al. (008), among ohers). When high-frequency daa are no available, he rue variance could be approximaed by he squared de-meaned reurn observed a ime, a he cos of a large noise componen. Neverheless, in he case of he Mincer-Zarnowiz regressions, Meddahi (00) shows ha he rankings based on R are consisen o he inclusion of noise in he proxy used for σ. Model equivalence could be esed more formally, for insance, by he approach proposed by Diebold and Mariano (995), and generalized by Paon (00). We may compare models by using ess based on loss funcion differenials, whereas MAE and MSE could be considered as specific loss funcions. As shown in Paon (00), he use of proxies for he underlying rue volailiy induces disorions in he model ranking for some loss funcions. Paon (00) proves ha wo loss funcions are robus o noisy volailiy proxies, and allows an unbiased model ordering. These loss funcions are he MSE and QLIKE, as given below: m MSE ( j) = ( h m ˆ σ j, ) (5) =

14 QLIKE ( j) m h = ln ˆ σ j, + m (6) = ˆ σ j, where h is a proxy for he rue unobserved volailiy σ. Alernaive models for σ can be compared by ess of equal predicive abiliy, which are associaed wih he null hypohesis of he expeced null loss funcion differenial: ( ) ( ) = ( ) ( ) = ( ) H0 : E MSE j E MSE i E MSE j MSE i E lfmse, j, i = 0 (7) where we may wrie a similar expression for QLIKE, and he expecaion is evaluaed using he sample counerpars repored in (5) and (6). Building on he resuls in Diebold and Mariano (996), he es saisic is given as: MSE ( i, j) τ = LF MSE Var lf ( j, i) ( j, i) MSE, (8) where ( ) LF j i lf j i h h m m (, ) = ( ) ( 늿 ) (,, = σ, σ, ) lf, ( j, i) MSE MSE MSE j i m = m = and Var is a heeroskedasiciy and auocorrelaion consisen variance esimaor (wih idenical equivalence relaions available for he QLIKE loss funcion). The es saisic is asympoically disribued as a sandardized normal, which allows a simple evaluaion of he null hypohesis. In fac, he es is equivalen o a significance es of he inercep in a regression of he loss funcion differenials lf ( ) MSE,, j i over a consan, and is hus readily available in all compuer sofware packages ha implemen robus linear regression mehods. A relevan limiaion of he comparisons based on Diebold-Mariano ype ess is ha hey represen pairwise comparisons, so ha i is no possible o exclude a priori he possibiliy of having differen model rankings associaed wih differen robus loss funcions. The lieraure conains several approaches ha have aemped o resolve his issue, such as he Realiy Check of Whie (000), he Superior Predicive Abiliy es of Hansen (005), and he Model Confidence Se (MCS) of Hansen e al. (005). 3

15 We sugges he use of he Model Confidence Se as his mehod provides a confidence se of saisically equivalen models. The approach developed in Hansen e al. (005) consiues a esing framework for he null hypohesis of equivalence across models, which is described by mean of loss funcions. By referring o he MSE loss funcion (similar quaniies can be obained for he QLIKE loss funcion), and assuming ha he se M conains a number of differen models used o produce forecass in a given ou-of-sample range, he null hypohesis of MCS is given as: ( ) H0 : E lfmse, j, i = 0, i > j, i, j M (9) The null hypohesis can be esed by means of wo es saisics proposed in Hansen e al. (005), namely: R = max j, i ( j, i) ( j, i) LFMSE M (0) Var lf MSE, SQ = ji, M, j> i LFMSE ( j, i ) Var lf MSE, ( j, i). () ( ) Boh ess saisics are based on a boosrap esimae of he variance, Var lf MSE, j, i. As he disribuion is non-sandard, he rejecion region is deermined using boosrap p-values under he null hypohesis. If he null of equal predicive abiliy across all models is rejeced, he wors performing model is excluded from he se M. Such a model is idenified using: j = arg max j M LFMSE( j, i) Var LFMSE( j, i ) i i M,i j M,i j / () where he variance is again deermined hrough boosrap echniques. The equal predicive abiliy of he remaining models should also be esed, hereby ieraing he evaluaion of he es saisics in (0) and (), and he idenificaion of he wors performing model in (). 4

16 The procedure sops when he null hypohesis of equal predicive abiliy of he models sill included in he se is no rejeced. Subsequenly, he MCS mehod provides a se of saisically equivalen models wih respec o a given loss funcion. I should be noed ha he opimal model se could conain a single model.... Indirec model evaluaion Indirec evaluaion mehods consider he uses of alernaive variance forecass. For insance, condiional variances could be used o price derivaives, or o define he marke risk exposure of a porfolio. The lieraure has recenly addressed he opic, focusing mainly on mulivariae models (for example, see Caporin and McAleer (00), Clemens e al. (009), Paon and Sheppard (009), and Lauren e al. (009), among ohers). A he univariae level, he approaches are much more widespread and have generally focused on specific applicaions. Many sudies deal wih he evaluaion of alernaive GARCH specificaions wihin a Valuea-Risk (VaR) framework (for example, see Caporin (008), Berkowiz (00), and Lopez (999, 00)). Considerable empirical research has focused on ess for he evaluaion of VaR forecass. These are used o deermine if a model is more appropriae wih respec o compeiors in deermining he fuure expeced risk of a financial insrumen (such as a financial porfolio). In his framework, consider a variable displaying heeroskedasiciy, possibly characerized by a ime-varying mean, and wih an unspecified condiional densiy (wih addiional parameers conained in he vecor θ): x I f ( x ; μ, σ, θ). (3) The one-day VaR for x is defined as: ( α ) ( VaR x ; + f x ˆ ; E I, E I + + +, ) dx α = μ σ θ (4) + where he ime-varying mean and variance are replaced by heir condiional expecaions, he addiional parameers are esimaed, and α is he VaR confidence level. Under normaliy, he 5

17 α = μ +Φ + + α σ + VaR has a simpler expression, namely ( ; ) ( ) VaR x E I E I, where Φ ( α ) is he α -quanile of he sandardized normal. Thus, VaR depends on he models used o capure he mean and variance dynamics. The evaluaion of alernaive mean and variance specificaions by using VaR could follow wo approaches: (i) es if he VaR ou-of-sample forecass saisfy he condiion E I( x < VaR( x ; α )) = α, ha is, if he expeced number of VaR violaions (namely, where reurns are lower han he forecas VaR) is equal o he VaR confidence level; (ii) compare models by means of loss funcions. Tess include he radiional mehod of Kupiec (995) which, as shown in Lopez (999, 00) and Caporin (008), have limied power in discriminaing across alernaive variance specificaions. Thus, loss funcions should be preferred, making an indirec comparison of GARCH and SV models very similar o he direc comparison. In he following, we provide an inerpreaion of VaR model comparisons by means of he Model Confidence Se which, o he bes of our knowledge, would seem o be novel. Loss funcions based on VaR forecass have been proposed, for insance, by Lopez (999) and Caporin (008). We sugges he following: )) i) IF = I ( x < VaR( x; α ; (5) ( ) ii) PIF = I ( x < VaR ( x; α) ) + ( x VaR ( x; α) ) ; (6) iii) AD x VaR( x; α ) g( x ) = ; (7) ( ) ( ) iv) ( ; α ) SD = x VaR x g x ; (8) ASD = AD + λsd ; (9) v) p RL = max VaR x; ; VaR x j; α ). (30) 60 j= 60 vi) ( α ) ( In he previous lis, he firs funcion (he indicaor loss funcion, IF) idenifies excepions, while he second penalizes excepions by using he squared deviaion beween realized reurns and VaR (penalized indicaor funcion PIF). The hird and fourh loss funcions could be read as firs-order and second-order losses, respecively, beween VaR and realized reurns (Absolue Deviaion, AD, and Squared Deviaion, SD, loss funcions, respecively). 6

18 They boh depend on g( x ), a funcion of he observed variable, x, ha focuses he loss funcions, for insance, only on negaive reurns g( x ) I( x 0) ( ) ( ) ( ; ) = <, on VaR violaions g x = I x < VaR x α or, finally, on he enire reurns pah (if se equal o ). In he fifh loss funcion, we combine he previous wo, adding a parameer, λ, o modify he weigh of a componen (which can be used o increase or decrease he impac of squared deviaions). Noe ha he fourh loss funcion is equivalen o he second if ( ) = I( x < 0) VaR ( x ; α ) is always negaive. Finally, he las loss funcion is also known as Regulaory Loss, and depends on a penaly erm, p, which is calibraed over he number of excepions in he las 50 days (3 up o 4 excepions, 3.4, 3.5, 3.65, 3.75, 3.85 for 5, 6, 7, 8, and 9 excepions, respecively, and 4 for more han 9 excepions). One sriking advanage of hese loss funcions is ha hey are no based on he rue volailiy, bu sill depend on he volailiy forecass. Thus, hey could be used wihin a MCS framework o compare alernaive models, wihou suffering from he problems associaed wih he replacemen of he rue variance by a noisy proxy. These mehods could also be used in he mulivariae framework and be applied o porfolios, in which he included asse variances follow a heeroskedasic densiy. g x and 3. Empirical Example In his secion we presen an empirical comparison of he mehods discussed above. Daily sock marke oal reurn indices, as repored in Table, are examined for We consider he large cap sock marke indices of France (CAC40), Germany (DAX), Swizerland (SMI), Hong Kong (HS), and USA (S&P500). Reurns are compued from index levels as r = 00 ln ( I) ln ( I ). For each series, we repor he descripive saisics and sample period, which differ across marke indices as holidays have been removed from he daa on a single series basis, and hese are no common over he counries considered. For each reurn series, we fi four specific models, namely GARCH(,), GJR(,), EGARCH(,), and SVOL(). The models are esimaed on a rolling basis, using a window of 000 observaions, and under normaliy. The four models are hen used o produce onesep-ahead variance forecass, from January 004. The models are compared using some of he mehods described in he previous secion. In paricular, we consider he Ling and McAleer (000) es for comparing GJR and GARCH agains EGARCH, he Likelihood Raio es in comparing GARCH agains GJR, he Diebold- Mariano es using he MSE and QLIKE loss funcions across all model pairs, and he Model Confidence Se approach using he MSE and QLIKE loss funcions, he loss funcions in 7

19 (6)-(30) wih hree VaR levels (%, 5% and 0%), and he loss funcion in (3) wih he % VaR level. For he ASD loss funcion, we se λ=. Furhermore, in order o verify he sabiliy of resuls over ime, we compare he models over differen ou-of-sample periods, and we consider annual comparisons from 004 o 009 (ha is, for 5 differen years). We sar wih he in-sample comparison of models using he Ling and McAleer (000) and LR ess. As we esimaed he models over a rolling sample of 000 observaions, we have a se of around 500 esimaes of all models (for esimaion samples ending from 3 December 003 o 30 December 009). The number of esimaes is no equal across all series as hese differ wih respec o naional holidays. Table repors he percenage of rejecions of he null hypohesis a he 5% confidence level over he enire se of esimaes available for each series. Table highlighs ha GJR(,) is always preferred o is GARCH(,) counerpar for he CAC40, DAX, SMI, and S&P500 indices, while only for he HS index does GJR no improve in-sample over GARCH in 3% of cases. A differen picure emerges when we compare non-nesed models, namely GARCH and GJR agains EGARCH. We use he Ling and McAleer (000) es and consider four possible comparisons, modifying he null and alernaive models accordingly. The Ling and McAleer (000) es adds he fied variances under he alernaive o he auxiliary regression equaion for he condiional variance equaion under he null. A significan coefficien of he added variable provides evidence agains he null model. The resuls for DAX, SMI and S&P500 are quie similar in ha here is a large fracion of rejecions when he null model is he GARCH and GJR specificaion, and a small fracion of rejecions when he null model is EGARCH. Therefore, EGARCH is he preferred condiional volailiy model. This finding is no surprising as EGARCH is more flexible han GARCH and GJR, can exhibi asymmery and leverage, and here are no resricions on he parameers of he model. However, for CAC40 and HS, he resuls do no suppor a paricular model, eiher suggesing ha any alernaive model is an improvemen over he null (CAC40) in a large fracion of cases, or ha no model can improve he null (HS) (again in a large percenage of cases). In order o shed some ligh on his resul, we recompued Table over wo subsamples, and , and he oucome is repored in Table 3. We do no repor he LR es as he oucomes are sable across he wo subsamples, wih he excepion of he HS index (for his index, he rejecion frequency is higher in he second subsample). Table 3 suggess EGARCH is opimal for he S&P500 index over he period , while asymmery is significan for he period , ha is, he GARCH esimaes clearly improve when we include he EGARCH variances, here is lile o choose beween GJR and EGARCH. For SMI, he empirical resuls are conrary o he above. GARCH is clearly rejeced for , bu here is no clear preference beween GJR and EGARCH. In , EGARCH 8

20 performs beer as compared wih GJR and GARCH. The resuls for he DAX and CAC40 indices are similar o hose of SMI for , while for here is no clear preference across he alernaive models for DAX. The resuls for CAC40 sugges a mild preference for GJR. Finally, for he HS index, he evidence suggess a small percenage of rejecions of he null hypohesis, alluding o he fac ha mos models provide very similar condiional variance paerns. Moving o he ou-of-sample comparison, we sar from he Diebold-Mariano es oucomes (direc evaluaion mehod) using he MSE and QLIKE loss funcions. In order o evaluae model performance across differen marke phases, we consider separaely each ou-ofsample year. Table 4 repors some salien empirical findings (he full se of empirical resuls is available from he auhors upon reques). Focusing on he MSE loss funcion, all empirical models seem very similar for all sock marke indices, wih he null hypohesis of zero loss funcion differenial being rejeced only in few cases. When we consider he QLIKE loss funcion, he null hypohesis is rejeced more frequenly, wih he finding seemingly independen of he sample used for model evaluaion (he resuls are similar for 006 and 008, wo years wih very differen volailiy and reurns). In his case, here are some differences across he sock marke indices, bu he oucomes sugges a preference for GJR over GARCH, and of GJR and EGARCH over SVOL. Furhermore, GJR and EGARCH are generally equivalen. Alhough some preference ordering across models may appear in some cases, he limiaion of he Diebold-Mariano es is ha i only considers pairwise comparisons. As suggesed in Secion, he Model Confidence Se mehod overcomes his resricive comparison. A number of ables collecs he resuls over he enire se of loss funcions, and over he ouof-sample years and sock marke indices. Tables 5-9 repor he Model Confidence Se resuls based on he R saisic in () for seleced ou-of-sample periods. The resuls for he saisic SQ are equivalen and are no repored (he enire se of resuls is available from he auhors upon reques). For each sock marke index, we evaluae he four alernaive models by using he MSE and QLIKE loss funcions, as well as he loss funcions defined in (6)-(30). If we consider he S&P500 index (Table 5), he resuls differ across he ou-of-sample evaluaion periods. In 004, all models are equivalen as hey are all included in he confidence se independenly of he loss funcion used for heir evaluaion. For 006, some differences appear across he loss funcions. For MSE and QLIKE, he opimal model is GJR; IF and PIF exclude, in mos cases, SVOL from he confidence se; AD, SD, and ASD sugges ha he opimal models are GJR and SVOL; finally, RL prefers he GARCH and EGARCH specificaions. In summary, here is no a clear preference for a specific model. Model 9

21 preference depends on he loss funcion under consideraion, and on he sample period used for model evaluaion. The las finding may be inerpreed as confirmaion of he in-sample and direc model comparison oucomes, which did no provide a clear indicaion of a single model. This inerpreaion is corroboraed by he 008 resuls for he S&P500 index: MSE considers all models as equivalen; QLIKE prefers GJR; IF, PIF and RL indicae a preference for GARCH and GJR; while AD, SD and ASD sugges ha he opimal models are EGARCH and SVOL. Similar paerns are observed for he oher sock marke indices in ha all models are equivalen under some specific ou-of-sample periods, and wih model preferences changing wih respec o he loss funcion used. Some behaviour is, however, common. When he Model Confidence Se includes fewer models han hose ha are available, he saisically equivalen models generally differ beween he IF-PIF-RL loss funcions and he AD-SD- ASD loss funcions. The former indicae a preference for GARCH and GJR, while he laer end o suppor EGARCH and SVOL. Thus, i seems ha he second se of loss funcions has a preference for more flexible models. Such behaviour may depend on he srucure of he loss funcions hemselves: AD and SD (and hence also ASD) monior he enire evoluion of condiional variances wihou focusing on he excepions or wihou penalizing VaR wih respec o pas violaions. The ineviable conclusion o be drawn is ha when we give a large relevance o volailiy spikes, mos models appear relevan, and simple specificaions may perform as well as heir more flexible counerpars. If we consider he evoluion over ime of he condiional volailiy, hen more flexible models are o preferred. 4. Concluding Remarks In his paper we reviewed some exising mehods for model selecion and esing of nonnesed univariae volailiy models. We firs considered in-sample mehods, such as nesed and non-nesed hypohesis esing, and diagnosic checking procedures (such as Ljung-Box and disribuional hypoheses). We hen focused on ou-of-sample approaches based on model forecas evaluaion. Saring from he radiional mean squared error and mean absolue error crieria, we considered more general loss funcions based on Value-a-Risk forecass, compared by means of he Model Confidence Se approach. Finally, we presened an empirical example using he less common approaches for model comparison, namely nonnesed hypohesis esing and VaR-based loss funcions. The paper was based on simple univariae specificaions focusing on volailiy asymmery and leverage. The proposed loss funcion approaches can easily be used on he forecass produced by oher univariae specificaions, as well as mulivariae models. 0

22 Table : Sample Saisics of Index Reurns Sock Marke Number Sandard Mean Index of observaions Deviaion Min Max Asym. Kur. CAC DAX SMI HS S&P Table : Rejecion Percenages of he Null Hypohesis in he Full Sample Tes Null model Alernaive Marke Index CAC40 DAX SMI HS S&P500 LR es GARCH GJR 00.00% 00.00% 00.00% 68.36% 99.80% Ling-McAleer GARCH EGARCH 70.44% 65.5% 84.79% 6.4% 83.44% Ling-McAleer GJR EGARCH 44.40% 59.4% 5.4% 9.98% 65.53% Ling-McAleer EGARCH GARCH 63.60% 36.% 3.30% 39.05% 3.58% Ling-McAleer EGARCH GJR 55.66%.68%.84% 39.78% 39.47% Noe: The null hypohesis of he LR es is associaed wih a preference for he GARCH model agains he GJR. For he Ling and McAleer (000) es, he alernaive model column denoes he model whose variances are used as addiional explanaory variables in he dynamics governing he variances as given by he null model. The rejecion of he null hypohesis is associaed wih a non-significan coefficien and signals a preference for he null model over he alernaive one. Table 3: Rejecion Percenages of he Null Hypohesis in Two Subsamples Tes Null model Alernaive Marke Index CAC40 DAX SMI HS S&P o 006 Ling-McAleer GARCH EGARCH 47.99% 43.5% 75.39%.5% 75.63% Ling-McAleer GJR EGARCH 0.89% 34.79% 3.55% 8.0% 6.46% Ling-McAleer EGARCH GARCH 63.8% 3.95% 0.53% 4.36% 0.86% Ling-McAleer EGARCH GJR 73.67% 8.74%.50% 43.30% 3.5% 007 o 009 Ling-McAleer GARCH EGARCH 9.95% 87.66% 94.0% 30.5% 88.08% Ling-McAleer GJR EGARCH 78.07% 83.99% 90.78%.98% 67.93% Ling-McAleer EGARCH GARCH 63.3% 58.53% 36.0% 35.80% 53.43% Ling-McAleer EGARCH GJR 37.47% 34.78% 33.0% 36.33% 64.49% Noe: In he Ling and McAleer (000) es, he alernaive model column denoes he model whose variances are used as addiional explanaory variables in he dynamics governing he variances as given by he null model. The rejecion of he null hypohesis is associaed wih a non-significan coefficien and signals a preference for he null model over he alernaive one.

23 Table 4: Diebold-Mariano Tes Saisics for Seleced Years Index GJR EGARCH SVOL EGARCH SVOL SVOL GARCH GARCH GARCH GJR GJR EGARCH MSE loss funcion ou-of-sample period: 004 CAC DAX SMI HS S&P QLIKE loss funcion ou-of-sample period: 004 CAC DAX SMI HS S&P MSE loss funcion ou-of-sample period: 006 CAC DAX SMI HS S&P QLIKE loss funcion ou-of-sample period: 006 CAC DAX SMI HS S&P MSE loss funcion ou-of-sample period: 008 CAC DAX SMI HS S&P QLIKE loss funcion ou-of-sample period: 008 CAC DAX SMI HS S&P Noe: The es evaluaes he null of zero expeced difference beween he loss funcion of he firs row model minus he loss funcion of he second row model. The es saisic is disribued as a sandardized normal. Significan values (5% confidence level) indicae a preference for he firs row model (if negaive - in bold) or for he second row model (if posiive - in ialics underlined).

24 MSE QLIKE Table 5: S&P500 Model Confidence Se IF PIF AD SD ASD % 5% 0% % 5% 0% % 5% 0% % 5% 0% % 5% 0% Ou-of-sample period: 004 GARCH GJR EGARCH SVOL Ou-of-sample period: 006 GARCH GJR EGARCH SVOL Ou-of-sample period: 008 GARCH GJR EGARCH SVOL Noe: The able repors he Model Confidence Se over differen loss funcions and periods. Bold values denoe he models ha are included a he 0% confidence level in he confidence se (hese models are saisically equivalen if compared using he loss funcion repored in he firs and second rows). The loss funcions names correspond o hose in (6)-(7) and (6)-(3), he second row repors he Value-a-Risk confidence level (when needed). RL 3

25 MSE QLIKE Table 6: CAC40 Model Confidence Se IF PIF AD SD ASD % 5% 0% % 5% 0% % 5% 0% % 5% 0% % 5% 0% Ou-of-sample period: 004 GARCH GJR EGARCH SVOL Ou-of-sample period: 006 GARCH GJR EGARCH SVOL Ou-of-sample period: 008 GARCH GJR EGARCH SVOL Noe: The able repors he Model Confidence Se over differen loss funcions and periods. Bold values denoe he models ha are included a he 0% confidence level in he confidence se (hese models are saisically equivalen if compared using he loss funcion repored in he firs and second rows). The loss funcions names correspond o hose in (6)-(7) and (6)-(3), he second row repors he Value-a-Risk confidence level (when needed). RL 4

26 MSE QLIKE Table 7: DAX Model Confidence Se IF PIF AD SD ASD % 5% 0% % 5% 0% % 5% 0% % 5% 0% % 5% 0% Ou-of-sample period: 004 GARCH GJR EGARCH SVOL Ou-of-sample period: 006 GARCH GJR EGARCH SVOL Ou-of-sample period: 008 GARCH GJR EGARCH SVOL Noe: The able repors he Model Confidence Se over differen loss funcions and periods. Bold values denoe he models ha are included a he 0% confidence level in he confidence se (hese models are saisically equivalen if compared using he loss funcion repored in he firs and second rows). The loss funcions names correspond o hose in (6)-(7) and (6)-(3), he second row repors he Value-a-Risk confidence level (when needed). RL 5

27 MSE QLIKE Table 8: SMI Model Confidence Se IF PIF AD SD ASD % 5% 0% % 5% 0% % 5% 0% % 5% 0% % 5% 0% Ou-of-sample period: 004 GARCH GJR EGARCH SVOL Ou-of-sample period: 006 GARCH GJR EGARCH SVOL Ou-of-sample period: 008 GARCH GJR EGARCH SVOL Noe: The able repors he Model Confidence Se over differen loss funcions and periods. Bold values denoe he models ha are included a he 0% confidence level in he confidence se (hese models are saisically equivalen if compared using he loss funcion repored in he firs and second rows). The loss funcions names correspond o hose in (6)-(7) and (6)-(3), he second row repors he Value-a-Risk confidence level (when needed). RL 6

28 MSE QLIKE Table 9: HS Model Confidence Se IF PIF AD SD ASD % 5% 0% % 5% 0% % 5% 0% % 5% 0% % 5% 0% Ou-of-sample period: 004 GARCH GJR EGARCH SVOL Ou-of-sample period: 006 GARCH GJR EGARCH SVOL Ou-of-sample period: 008 GARCH GJR EGARCH SVOL Noe: The able repors he Model Confidence Se over differen loss funcions and periods. Bold values denoe he models ha are included a he 0% confidence level in he confidence se (hese models are saisically equivalen if compared using he loss funcion repored in he firs and second rows). The loss funcions names correspond o hose in (6)-(7) and (6)-(3), he second row repors he Value-a-Risk confidence level (when needed). RL 7