Model risk adjusted risk forecasting

Transcription

1 Model risk adjusted risk forecasting Fernanda Maria Müller a,, Marcelo Brutti Righi a a Business School, Federal University of Rio Grande do Sul, Washington Luiz, 855, Porto Alegre, Brazil, zip Abstract This work presents an approach to adjusting financial risk forecasting by model risk associated with the econometric models applied. Our proposal is obtained by a convex combination of forecasts of a coherent risk measure penalized by a model risk measure. Based on Limitedness, axioms of coherence are preserved. In order to obtain a model risk optimum adjustment, we present an approach that minimizes the expected value of the sum between costs from risk overestimation and underestimation. Additionally, we suggested an axiomatic framework for model risk measures belonging to worst-case approach. Based on this framework, we analyze model risk measures introduced in the literature and we recommend other procedures to quantify model risk. To show the practical usefulness of our proposal we realized an empirical analysis. Our findings indicate the best performance of model risk adjusted risk forecasting. Keywords: Model risk; Model risk measures; Risk measures; Adjusted risk forecasting; Risk management. 1. Introduction The increasing complexity of financial products has revealed a greater need for financial institutions to use statistical model outputs to assess the risks that they are exposed to. Most common method quantifies the risk, being linked to variance, Value at Risk (VaR) Corresponding author addresses: fernanda.muller@ufrgs.br (Fernanda Maria Müller), marcelo.righi@ufrgs.br (Marcelo Brutti Righi) Preprint submitted to XVIII EBFin April 3, 2018

2 and Expected Shortfall (ES), by means of the empirical data distribution. Other methods often employed are parametric, such as GARCH (Generalized Autoregressive Conditional Heteroskedasticity) models, and semi-parametric, for instance, Quantile Regression and Filtered Historical Simulation (FHS) techniques. The reliability of the results depends on the method and model used. In general, risk measures are very sensitive to a choice model. Two distinct models, or a model with two different parametrizations, will generate different values. The uncertainty concerning the choice model or method leads to a risk referred to in the literature as model risk. In finance, this risk is mainly related to uncertainty regarding the data generating process. However, different definitions are used to define it. For some authors, model risk refers to the risk of using an incorrect model (see Hull and Suo (2002) and Barrieu and Ravanelli (2015)). For others, this definition is more specific and it is related to the different sources of model risk (see Derman (1996)). In our study, similar to Federal Reserve (2011), model risk is related to losses arising from the use of an incorrect or inadequate model and from uncertainty present in the estimation process. A similar definition is also employed by Boucher et al. (2014). The research to reduce model risk is mainly concentrated in two streams: (i) Model averaging techniques, such as Bayesian model averaging (BMA), which performed a weighted average of models according to some priori probability, as accomplished by Liu and Maheu (2009) and Chua et al. (2013); and (ii) Computational or analytical adjustments, as fulfilled, for instance, in Alexander and Sarabia (2012), Boucher and Maillet (2013), and Gourieroux and Zakoïan (2013). Although good statistical results are identified, the quantification of model risk through these streams are not direct. After recent revisions to the Basel II Market Risk Framework, which require that financial institutions quantify their model risk and manage it like other types of risk (see Basel Committee on Banking Supervision (2009) and Federal Reserve (2011)), the attention of the model risk literature has focused on procedures of quantifying. Despite the progress made, these procedures are not consolidated at the same level as when quantifying market risk, and the theoretical discussion of their properties is 2

3 incipient. Theoretical discussions gained a boost, in the risk management literature, after the pioneering work of Artzner et al. (1999), and they are the first step to identifying validity and a practical sense of risk measures in financial trouble. The main approach used in literature to measure model risk is the worst-case 1, which takes the worst model over all models. Based on this approach, various procedures are derived, including the difference between the upper and the lower functional bounds of estimation models, and the difference between upper bound and a reference model, given a set of candidate models. The measures belonging to this approach are applied in a set of estimates or forecasts and have similar characteristics to deviation measures, such as Non-negativity and Translation Insensitivity. These properties classify them as a metric for model risk and not a monetary risk measure, once they do not respect Monotonicity and Translation Invariance axioms (see Artzner et al. (1999)). Thus, through their use, it is not possible directly to determine the capital needed to cover that risk, which is a main concern of the regulatory agencies and researchers under a model risk context (see Basel Committee on Banking Supervision (2009) and Kellner and Rösch (2016)). As an alternative to quantifying the monetary impact of model risk, some authors argue for incorporating model risk, obtained by these procedures, into risk measures used in the financial industry. However, this alternative may compromise the theoretical properties of each component, especially the fundamental Monotonicity axiom. This contradicts the natural intuition of economic capital regulation, i.e. a position with highest loss has the highest risk. To work around this deficiency, and to get a model risk adjusted risk estimate, we used the theoretical results presented by Righi (2017a) for limited risk measures. This set of the measures combine risk and deviation measures, ρ + D. Based on the proposed Limitedness axiom, the author proves that this set of measures is a sub-class of coherent risk measures. Unlike the author, our risk estimate is obtained by the convex combination of forecasts of a 1 It is also possible to apply risk measures directly in a loss or error function, from some estimation or forecasting procedure. More details are found in Detering and Packham (2016). 3

4 coherent risk measure 2 ρ λ, and a model risk measure belong to worst-case approach 3 MR. Our proposal provides a more solid protection once it increases the value of weighted risk forecasting by model risk associated with the econometric models applied. From a regulator point of view, the penalty function provides sufficient capital to avoid the costs of unexpected losses arising from the uncertainty present in the estimation process and referring to the choice model. Additionally, we presented an approach that lets us incorporate the model risk amount that minimizes the expectation of sum between costs from risk overestimation and underestimation. Similar to Righi (2017b), these costs are measured by non - negative variables, where G represents costs from risk overestimation and L from risk underestimation. As a complementary objective, we formalized an axiomatic framework for model risk measures based on deviation measures. In our study, a model risk measure refers to a functional that transforms model risk in a positive real number, and fulfills Non-negativity, Translation Insensitivity, Positive Homogeneity, and Sub-additivity axioms. We considered only the measures based on the worst-case approach, by virtue of their theoretical properties, and because they encompass most of the proposed procedures in the literature. Therefore, we consider proposed measures. Adaptations or extensions are cited where convenient. We also present adjustments and other procedures were introduced too. To the best of our knowledge, our paper is the first to provide an alternative for adjusting the risk forecasting by model risk, which preserves the financial interpretation. In the studies of Alexander and Sarabia (2012), Boucher and Maillet (2013), and Boucher et al. (2014) an alternative to adjusting the VaR estimates using a maximum entropy criterion, and 2 Consider a random result of a position X. M = {M 0, M 1,, M k } := {ˆρ 0 (X), ˆρ 1 (X),, ˆρ k (X)} is a non-empty set of estimates or forecasts of a functional of interest in finance, where k is the number of estimates without M 0, which refers to an estimate of a reference model. n = k + 1 is the total number of estimates, including M 0. Thus, M R n. Given that ˆρ i (X),, ˆρ n (X) is a collection of estimates of a coherent risk measure, their convex combination, i.e. ρ λ (M) = n i=0 λ im i := n i=0 λ i ˆρ(X i ), where n i=0 λ i = 1, λ i 0, i = 0, 1,, n, being λ 0 the weight given to the reference model is a coherent risk measure. Throughout the study, we are going to refer to ρ λ as weighted risk forecasting. 3 We emphasized that a model risk adjusted individual risk forecast is a particular case of the proposed approach. In this case, given a set of forecasts M, an individual forecast has λ = 1 and the other λ = 0. For more details, see the previous footnote. 4

5 backtesting frameworks is introduced. These authors focus on adjusting estimates of an individual risk measure, i.e. VaR, and not a class of measures. Also, there is no formalization of the theoretical properties of the adjusted risk estimate. Our work resembles the study of Kerkhof et al. (2010). This research combines VaR and ES forecasting, with a model risk measure belonging to worst-case approach. However, in their case there is no guarantee that the theoretical properties are maintained because the adjustment does not respect Limitedness. This axiom is necessary so that the composition maintains the properties of the individual components, as demonstrated by Righi (2017a). Unlike previous works, our study employs two distinct approaches used in model risk management, model risk measures, which are applied to quantify model risk, and model averaging techniques, which are used to reduce it. Since the seminal article of Bates and Granger (1969), several studies have documented that the use of forecasts combinations can provide more accurate results, compared to forecasts of a single procedure. As examples of studies exploring the advantages of model averaging techniques, we mention Pesaran et al. (2009), and Wang et al. (2016). As a second main contribution, we extend our approach to achieving the value of model risk that minimizes costs related to risk overestimation and underestimation. This approach is closely related to the problem of capital determination/requirement. From the regulatory point of view, there is a greater concern about reducing losses from risk underestimation and, consequently, capital requirement underestimation. However, from internal risk management, there is also a concern on reducing the opportunity costs originating from risk overestimation. The main approaches introduced in the model risk literature are mainly focused on the regulatory perspective, i.e. shorten cost risk underestimation (see, for instance, Boucher and Maillet (2013)). Although risk underestimation, commonly, lead to more dangerous outcomes, it is important for the institution to minimize both costs. Our proposal is inspired in the study of Righi (2017b). Unlike our work, the purpose of the author is to obtain a robust risk measurement at both costs, instead of a model risk optimum adjustment. Furthermore, this approach, when applied to a set of models accepted by regulatory author- 5

6 ities for internal modelling, allows for the reduction of the problem of regulatory arbitrage 4 and model misspecification investigated by Kellner and Rösch (2016). In addition, our paper is the first to formalize an axiomatic framework for model risk measures based on deviation measures. This framework will give greater practical support to the use of model risk measures in financial problems. Previous studies, such as Cont (2006), Kerkhof et al. (2010) and Barrieu and Scandolo (2015), only discuss the axioms of the functional used to get model risk measures (VaR and ES). When computing model risk, the measure loses the theoretical properties of individual components. Our framework has similarity with the axioms of generalized deviations measures proposed by Rockafellar et al. (2006) and extended by Pflug (2006), and Rockafellar et al. (2008), and also explored by Furman et al. (2017) for Gini-type measures of risk and variability. Additionally, our study formalizes a general approach to quantify model risk, which can be applied in different contexts to those used in this work. In the study of Barrieu and Scandolo (2015) a first attempt is made in this direction. In another way, in our work we explore the procedures presented in the literature, we introduce model risk measures to capture unexplored characteristics, and we discuss the theoretical and practical characteristics of them. Model risk measures introduced allow for capturing model risk in extreme events and asymmetrically. In some areas, such as the financial markets, model risk underestimation has a distinct impact on its overestimation. An example is the different costs from risk underestimation and overestimation. As a complementary contribution, our work presents an updated literature on the subject, which can be used as a guide in this research topic in finance. This study also provides theoretical support to investors and managers, for examining the main tools that are being proposed to measure this type of risk. To show the practical utility of our proposed approach, we have realized an empirical 4 Regulatory arbitrage, in the sense used, refers to two institutions that have the same portfolio and use different internal models, approved by the regulator, and so quantify different amounts of capital requirement. As they keep the same portfolio, they are required to hold the same or at least almost the same amount of regulatory capital. 6

7 illustration using NASDAQ log-returns. Forecasting performance was analyzed using loss functions, which allows for a determination of what risk procedure results in the best prediction. General details about loss functions are given in Gneiting (2011). Previous studies, such as, Kerkhof et al. (2010), use backtesting tools to compare competing predictions. However, backtesting allows for an evaluation of the quality of the forecasts in the sense of the individual measure, instead of allowing for the comparison of different risk estimation procedures. The remainder of this paper is structured as follows: Section 2 presents a background regarding risk and deviation measures, and Section 3 introduces the proposed approach. Section 4 describes an illustrative example with real finance data exposing how the proposed approach behaves in practical terms in relation to unadjusted risk forecasting. Section 5 presents the final considerations of this study. 2. Background Consider a random result X of an asset or portfolio (X 0 is a gain, and X < 0 is a loss), defined in the space of random variables X := X (Ω,F,P), with distribution function F X and inverse F 1 X. We define X+ = max(x,0), X = max( X,0), inf X and sup X as the infimum and supremum values of X, and E[X] as its expected value. A functional ρ : X R is a coherent risk measure if it satisfies the following axioms: Monotonicity: if X Y, then ρ(x) ρ(y ), X, Y X. Translation Invariance: ρ(x + C) = ρ(x) C, X X, C R. Positive Homogeneity: ρ(0) = 0, and ρ(λx) = λρ(x), X X, and λ R +. Sub-additivity: ρ(x + Y ) ρ(x) + ρ(y ), X, Y X. Additionally, a coherent risk measure fulfills the following axiom: Limitedness: ρ(x) inf X = sup X, X X. 7

8 The first axiom, Monotonicity, indicates that, if the loss of a financial position is greater in all situations, then its expected risk is always greater. By second axiom, if adding a certain gain C to a position X, it is expected that the risk of this position decreases by the same amount. The third axiom indicates that the risk of a position increases proportionally with its size. Sub-additivity informs that the risk of two combined assets (portfolio) is less than or equal to the sum of individual risks of the portfolio assets. This axiom refers to the principle of diversification. Sub-additivity and Positive Homogeneity together are known as Sub-linearity, and they imply Convexity. A risk measure is monetary if it satisfies Monotonicity and Translation Invariance. When a risk measure meets Monotonicity, Translation Invariance, Positive Homogeneity, and Sub-additivity is a coherent risk measure in the sense of Artzner et al. (1999). If a risk measure fulfills Monotonicity, Translation Invariance and Convexity is a convex risk measure in the sense of Föllmer and Schied (2002) and Frittelli and Gianin (2005). As showed by Righi (2017a), a coherent (convex) risk measure always respects Limitedness. This axiom ensures that the risk of a position is never greater than the maximum loss. Examples of coherent risk measures include Expected Loss (EL), ES and Expectile Value at Risk (EVaR), which are defined by: (i) EL(X) = E[X], (ii) ES α (X) = E[X X F 1 X (α)], (iii) EVaR α (X) = arg min θ E[ α 1 X θ (X θ) 2 ], where 1 a is an indicator function with value 1 if a is true and 0 otherwise, and α (0, 1) is significance level. The negative sign of risk measures is used to indicate a monetary loss. Focusing on coherent risk measures, we considered EL, which computes the expected value (mean) of a loss. We also presented the ES, which represents the expected value of losses that exceed α - quantile of interest, i.e. VaR losses 5. The next measure is EVaR, which 5 VaR can be defined by: VaR α (X) = inf{x : F X (x) α} = F 1 (α). This measure represents the 8 X

9 can be interpreted as the amount of money that needs to be added to a position to have a sufficiently high ratio between losses and gains. As exposed by Bellini et al. (2014), EVaR is a coherent risk measure for α 0.5. A functional D : X R + is a deviation measure, which can fulfill the following axioms: Non-negativity: D(X) = 0, X constant, and D(X) > 0, X non-constant, where X X. Translation Insensitivity: D(X + C) = D(X), X X, and C R. Positive Homogeneity: D(0) = 0, and D(λX) = λd(x), X X, and λ R +. Sub-additivity: D(X + Y ) D(X) + D(Y ), X, Y X. Additionally, a deviation measure can respect the following axiom: Lower Range Dominance: D(X) E[X] inf X, X, Y X. The first axiom ensures that a non-constant position has non-negative deviation. Translation Insensitivity indicates that the deviation in relation to expected value does not change if a constant value is added. The third axiom, Positive Homogeneity, alone implies that D(0) = 0, and together with Translation Insensitivity implies that D(C) = 0, C R. Lower Range Dominance restricts the value of the measure to a range between expectation and the minimum value. In the basic sense, a deviation measure fulfills Non-negativity and Translation Insensitivity. Whenever D respects Translation Insensitivity, Non-negativity, Sub-additivity, and Positive Homogeneity is a generalized deviation measure in the sense of Rockafellar et al. (2006), and when D satisfies Non-negativity, Translation Insensitivity and Convexity axioms, this belongs to a class of convex deviation measures, discussed by Pflug (2006). maximum loss that is expected for a given significance level and period. Despite its popularity, VaR is not a coherent risk measure once it does not respect Sub-additivity. 9

10 The most common generalized and convex deviation measures in financial literature are defined by: (i) D 1 (X) = sup X inf X, (ii) D 2 (X) = sup X E[X], (iii) D 3 (X) = E[X] inf X, (iv) D 4,p (X) = (E[ X E[X] p ]) 1 p, (v) D 5,p (X) = (E[((X E[X]) ) p ]) 1 p, (vi) D 6,p (X) = (E[((X E[X]) + ) p ]) 1 p, (iv) D 7 (X) = ES α (X E[X]), where p = 1 and 2, and ES α is Expected Shortfall. The first deviation measure, D 1 (X), refers to full range, which computes the distance between infimum and supremum values of X. D 2 (X) quantifies the distance of supremum value of X in relation to its expected value, and D 3 (X) the distance of expected value of X in relation its infimum value. D 4,p (X) is know as p - Deviation. This measure analyzes average distance of X from its expected value. When p = 1 one obtains the total relative distance, and p = 2 gives greater relevance to greater distances between point observations of X and E[X]. D 5,p (X) and D 6,p (X) refers to p - Semi Deviation. These measures analyse, respectively, the average distance of X above and below from its expected value. The last measure is Conditional Value at Risk Deviation, which quantifies average deviation below the α - quantile. 3. Proposed approach In our case, instead of considering a random result X, we consider M, which is a nonempty set of estimates or forecasts of a functional of interest in finance that needs to be estimated (typically, the expectation or a risk measure). M := {M 0, M 1,, M k }, where 10

11 M 0 is an estimate or forecast of a functional using a reference model, and k is the number of estimates without M 0. The total number of estimates is represented by n = 1 + k. Thus, M R n. We define M min = min(m) := min(m 0, M 1,, M k ), M max = max(m) := max(m 0, M 1,, M k ), and M λ = E[M] := 1 n i=0 n [M i], where M i is the estimate i. Furthermore, (M) = max( M, 0), (M) + = max(m, 0), and M 1 M 2 = min(m 1,M 2 ). The choice of which set of models should be used to get M can be made by considering measures that represent scenarios inside some distance from M 0, including Bregman distance, as performed by Breuer and Csiszár (2016); or by models that cannot be rejected on the basis of misspecification tests employing the available past data, such as, backtestings, as executed by Kellner and Rösch (2016). Model risk measure MR : R n R + is a functional that summarizes model risk in a positive real number, and meets Non-negativity, Translation Insensitivity, Positive Homogeneity, and Sub-additivity axioms. The procedures introduced in the literature usually comply with Non-negativity and Translation Insensitivity. Additionally, Positive Homogeneity and Subadditivity axioms are important in the model risk context. Sub-additivity corroborates with earlier studies that show model averaging techniques, as for instance BMA, handling satisfactorily with model risk. Positive Homogeneity is interesting because, together with Subadditivity, this can be replaced by Convexity, and multiplication by a scalar is an important property for a functional. In general, the model risk measures proposed in the literature, which belong to worst-case approach, have similar structure to the deviation measures described in the previous section. The main differences are that MR are applied to M instead of X, and these procedures use a reference model in place of expected value. For instance, D j,p, j = 1, 2, 3, 4 coincide with the model risk measures introduced, respectively, by Cont (2006), Kerkhof et al. (2010), Breuer and Csiszár (2016) and Krajcovicova et al. (2017). Generally, other metrics of the literature (to quantify model risk) are constructed by the ratio between these measures or by the ratio of one of them and a variable to standardize the measurement value, such as M 0, M λ, M max, 11

12 and M min. We refer to, as examples, the procedures of Barrieu and Scandolo (2015), Bernard and Vanduffel (2015) and Kellner and Rösch (2016). Other adaptations are also made. See, for instance, Danielsson et al. (2016). However, these procedures do not fulfill the axioms of a model risk measure. In order to capture the characteristics of model risk that are not explored so far, model risk in extreme events and asymmetrically, we suggested to use model risk measures with a similar structure to the following deviations D j,p j = 5, 6, 7. We named model risk measures described by MR j,p, j = 1,,7 and p = 1, 2, where p is considered only in the measures that have p. Furthermore, to mitigate the criticisms made to the use of the reference model, we recommend changing M 0 by M λ. The main criticism refers to the reference model. This model is known only in the simulation environment, and its choice is subject to preferences and to the empirical knowledge of the agent. These may distort the assessment of model risk. We emphasize that, when using M λ in place of M 0, the properties are not affected 6. These model risk measures are referred to as MRj,p, j = 1,,7 and p = 1, 2. Thus, given that ρ λ : R n R respects coherence axioms, and MR : R n R + is a model risk measure, our model risk adjusted risk forecasting is obtained as follows: ρ MR (M) = ρ λ (M) + βmr(m), 0 β 1, (3.1) where ρ λ (M) = n i=0 λ im i := λ i ˆρ(X i ), n i=0 λ i = 1, and λ i 0, i = 0,, n, is obtained by a collection of forecasts of a coherent risk measure ˆρ(X i ), weighted by λ i. β is the proportion of model risk included. For simplicity, we named ρ MR as adjusted risk forecasting. For MR can be used model risk measures described, as well as other versions that comply with their theoretical properties. Procedures that do not fulfill the desired axioms can compromise the theoretical properties of ρ MR and its financial interpretation. ρ MR fulfills coherence axioms 6 The proof will be omitted by similarity of model risk measures with deviation measures known in the literature. See Rockafellar et al. (2006) for more details. 12

13 if and only if it complies Limitedness. This axiom is satisfied when { ρ λ (M) M min β β r := inf MR(M) } : M R n, MR(M) > 0, (3.2) where ρ λ (M) = ρ λ (M). When ρ λ + MR respects coherence axioms, MR fulfills lower range dominated. Proof is found in Righi (2017a). The optimal proportion for the model risk included in adjustment, which minimizes the expectation of sum between costs from risk overestimation and underestimation, can be obtained by β, which is defined by: β (M) := β G,L(M) = arg min β [0,β r] E[((X y)+ )G + ((X y) )L], (3.3) where y = (ρ λ (M) + βmr(m)), 0 β β r, G and L are non-negative random variables that indicate, respectively, costs from risk overestimation and underestimation, being G L. (X y) + indicates a circumstance in which the result of position X is better that the result of y. This difference must be invested at a rate G. (X y), representing a situation wherein the capital reserve is not enough to cover losses. This amount must be obtained at a rate of L. 4. Illustrative Example 4.1. Methodological procedures To illustrate the proposed approach, we provide an empirical analysis, with NASDAQ log-returns from January 2010 to December 2017, totaling 2,083 daily observations. In the estimation process, we use a window of 1,000 days. Thus, we have an out-sample period composed by 1,083 daily points. To compute ρ MR, our choices for MR are MR j,p and 13

14 MRj,p, where j = 1,, 6 and 7 p = 1, 2. For ρ we used elicitable 8 and co-elicitable coherent risk measures, which include EL, ES, and EVaR. EL and EVaR possess the property of elicitability and ES, although it is not elicitable, can in practice be jointly elicited with VaR. For details see Acerbi and Szekely (2014). This property is important in risk management because it allows for determining among competitive models what results in the best forecast. VaR is not used because it is not a limited risk measure. However, we indirectly consider it to get a scoring function of ES. We compute risk measures using AR(p) (auto-regressive) - GARCH(q,s) model 9, which can be described in the following manner: X t = µ t + p φ i X t i + ɛ t, i=1 ɛ t = σ t z t, z t i.i.d. F (0,1), q s σt 2 = ω + a j ɛ 2 t j + b k σt k, 2 (4.1) j=1 k=1 where t = 1,, T is period, X t is the return, µ t represents the expectancy, φ is the autoregressive component, ɛ t is the innovation in expectation, z t is a white noise process with distribution F, σt 2 is the conditional variance, and ω, a and b are parameters of the GARCH model. For F we assume normal (norm), skewed normal (snorm), Student-t (std), skewed Student-t (sstd), generalized error (ged), skewed generalized error (sged), normal inverse Gaussian (nig), and Johnson SU (jsu) distributions. Thus, eight models were used to compute 7 In our illustration, model risk measures are computed nonparametrically. We did not include MR 7 and MR7 through the difficulty by estimating ES with historical simulation in small samples. We emphasized that other approaches can be used to obtain the data distribution in small samples, such as Spline functions. However, our intention here is only to illustrate the proposed approach and to demonstrate its practical usefulness in financial problems, mainly in the capital determination problem. 8 A functional is named elicitable when it is the minimizer of some score function. For ρ MR, the elicitability is maintained because, under Translation Invariance, ρ λ (M) + MR(M) can be interpreted as ρ λ (M ), where M = M MR(M), being MR(M) an adjustment in the initial value of the set of estimates. 9 In the risk management literature, AR(p)-GARCH(q,s) is a common model employed to compute risk measures. We refer the studies of Weiß (2013), Righi and Ceretta (2015), and Müller and Righi (2018) for a comparison of alternative models that can be considered, in univariate and multivariate context, for financial risk forecasting. Although our empirical analysis is univariate, extensions to the multivariate case can be utilized. We chose the univariate estimation for simplicity and for being typically used in illustrative examples in the model risk measures research. 14

15 risk measures 10. The parameters are estimated through the Quasi-Maximum Likelihood. The model used is AR(1)-GARCH(1,1). We chose the number of lags to include equations of conditional mean and variance through the Akaike information criterion (AIC). Similarly to Kerkhof et al. (2010) and Krajcovicova et al. (2017), we utilized as M 0 the model that follows a normal distribution, i.e. GARCH model with Normal innovations. The values of α used are for ES and for EVaR 11. At each step, one-step-ahead forecast is obtained 12 of EL, ES and EVaR. In the same step, is computed ρ MR, as Equation (3.1). ρ λ is obtained by means of equally weighted risk measures. Different weights can be considered without compromising the original s theoretical properties, given that n i=0 λ i = 1 and λ i 0. Here, our intention is only to illustrate the proposed approach. The values of β are obtained at each step by β r 1. To evaluate forecast accuracy we compute, for each risk measure, loss functions (L), which for EL, ES, and EVaR are defined by: L EL (X, y) = E[(X y) 2 ], L ES (X, y, z) = E[(I α)z IX + e y (y z + I α (z X)) ey + 1 log(1 α)], L EVaR (X, y) = E[α((X y) + ) 2 + (1 α)((x y) ) 2 ], (4.2) where y is a point forecast, being y = ρ, y = ρ λ and y = ρ MR, and z is a point forecast obtained using Value at Risk 13, so ρ = VaR α, being z = ρ, z = ρ λ and z = ρ MR, I is an indicator function that assumes value equal to 1 if X < z, and 0 otherwise, and E[X] = 1 T T t=1 X t. Additionally, average value of forecasts (Mean), of β (β) and of model 10 In the studies that employ model averaging techniques it is common to use a small number of models. Hoogerheide et al. (2010) and Wang et al. (2016), for example, used four and eight models, respectively, in their analysis. 11 For ES, α equal to 0.025, is recommended by the Basel Committee on Banking Supervision (see Basel Committee on Banking Supervision (2013)), and for EVaR, the value equal to is closely comparable. These values the α are also employed in the study of Bellini and Di Bernardino (2017). 12 We used one-step-ahead forecast because it is usually employed in empirical studies. 13 Value at Risk is estimated using α =

16 risk (MR) are described, as well as the ratio of model risk in relation to weighted risk forecasting (Prop.), i.e. MR(M) ρ λ (M). In this analysis, we compared results obtained by ρmr with ρ, and ρ λ. ρ and ρ λ are referred as unadjusted risk measures. We also performed an empirical analysis with NASDAQ log-returns to compare the quality of the adjusted forecasting obtained with and without β. For costs of risk overestimation G and underestimation L, we used daily yield rates of the U.S. Treasury Bill with a maturity of three months and the U.S. Dollar based Overnight London Interbank Offered Rate (LIBOR) 14. These rates reflect a risk-free investment with liquidity, where the surplus over capital requirement can be invested, and a rate for loans, when capital requirement is not enough, respectively. The results are compared using loss functions described in (4.2). Additionally, we display average values of forecasts and of β, and the percentage variation of loss function and β value (% ) of financial risk forecasting obtained without and with β Results We expose, in Figure 1, log-returns evolution of the index over the observation period. As seen, log-returns display periods of calm and greater instability, which is a common feature in financial returns. The returns to the right of the dotted line refer to out-of-sample period. In the Figures 2, 3 and 4, we provide the evolution of model risk (black line) and weighted risk forecasting (gray line), computed for EL, ES, and EVaR for out-of-sample period, and in Table 1, we described the results of the performance of adjusted and unadjusted risk forecasting. As observed, model risk varies over period, and their value becomes greater in periods that coincide with the highest instability of returns. Similar behavior is identified in the study of Danielsson et al. (2016). It is also observed that each model risk measure provides a distinct assessment of model risk. For ES and EVaR, we noticed more similar behavior in the results of MR. A possible reason for this is that both measures are tail risk measures, although they have conceptual differences. Among model risk measures, MR 1, 14 These assets are commonly used in the literature and were also used in the study of Righi (2017b). 16

17 together with MR 3 and MR3 for EL, MR 2 and MR2 for ES, and MR 2 and MR3 for EVaR, computed the highest values of model risk. In contrast, the lowest values of model risk are quantified by MR 5,1 and MR 5,2 for EL, and by MR 6,1 and MR 6,2 for ES and EVaR. < Insert Figures 1-4 and Table 1> The results obtained by MR 1 allow us to conclude that there is a large amplitude between the minimum and maximum values of M. This dispersion is greater for EL forecasting. The large dispersion between estimates can lead to serious problems to institution. For example, when considering an individual forecast, the risk manager may decide on different amounts of capital requirement, which may not be sufficient to absorb losses from unexpected impacts. Or the manager can determine a high amount of capital requirement that could be applied to other investments. Another problem is an unbalanced regulatory environment, once the individual risk measures determine different amounts of regulatory capital. In addition to financial losses, an inappropriate model used for risk management can result in poor business and strategic decision-making, or damage the reputation of the institution (Federal Reserve, 2011). These problems can be mitigated by our proposal because it allows us to obtain more robust risk forecasting. With regard to performance, it is observed that adjusted risk forecasting tends to have a lower loss function, i.e. more accurate predictions. In the context of market risk, Righi and Ceretta (2016), Furman et al. (2017) and Righi and Borenstein (2017) discussed, from theoretical and practical dimensions, the advantages of a more complete analysis when using risk and deviation measures. However, in their studies, the aim is to obtain a composition between risk and deviation measures, in place of a model risk adjustment. We observed that the quality of forecasts of adjusted risk forecasting depends on the choice of model risk measure. Aggressive model risk measures, such as MR 1, have a tendency of overestimating the true model risk and, so, ρ MR. For instance, as observed in Table 1, for EL, ρ MR 1 present the worst loss function, followed by ρ MR 3 and ρ MR 3. With regard to this, MR3, MR 3, along with MR 1, are the measures that quantify the highest model risk. In contrast, for 17

18 EL, adjusted risk forecasting with model risk measures that quantify lowest model risk, generally, ρ MR 5,1 and ρ MR 5,2, offers the better performance. With regard to EL, ρ λ has lowest loss function. Considering ES and EVaR, adjusted risk forecasting have the best performance, being the lower loss function showed, respectively, by ρ MR 5,1 and ρ MR 3. For both risk measures, it is also observed that MR with lower values, in general, result in adjustments with better quality in forecast. In addition, we noticed that model risk measures proposed in this research are the most suitable to be considered in the adjustment and they act better than unadjusted risk forecasting. For example, for ES, loss function of ρ MR 5,1 is approximately 1.05% and 2.55% lower than loss functions of ES estimated by GARCH norm and GARCH std, respectively, and 0.315% of ρ λ. While this difference is small, the impact on capital determination in the financial industry can be very large. Consider a bank that invests one billion dollars in equities. When computing the capital requirement, for 1 day, using ρ MR 5,1 and GARCH std, we have a difference in regulatory capital of almost six million 15. The amount of capital required, when employing unadjusted risk forecasting, generates an opportunity cost because an unnecessary portion of capital will be allocated for investment security. These results are maintained over different forecast windows and estimation, as well as for different assets and reference models. When analyzing β, we note that, in most cases, its value is equal to 1. In what regards MR 1, β is for EL, for ES and for EVaR. Concerning MR 2, β is 1, and for EL, ES, and EVaR, respectively. MR 2 is the procedure discussed by Kerkhof et al. (2010). In their research, it is suggested that obtaining total market risk is achieved by the sum of nominal market risk (obtained by VaR and ES) and model risk. However, when adjusting to ES forecasting by the total value of MR 2, its financial 15 Capital requirement/determination is computed directly from point estimate of a market risk measure multiplied by portfolio value. The average value of ρ MR5,1 is , which results in a capital determination of $ 25,681,703.20, for a portfolio value of one billion dollars. In another way, as the average value of ES obtained by GARCH std is , this measure results in a capital determination of $ 31,491, The difference corresponds to $ 5,809,

19 interpretation is compromised. On the case, ES value is not suitable for the computation of required levels of capital reserves. Thus, we realize that β value has an important role in the optimal determination capital. Its value indicates whether the adjusted forecasting respects Limitedness. Moreover, when β is greater than necessary it will increase costs from risk overestimation, or when its value is less than necessary it will not be enough to cover costs from risk underestimation. For institutions, it is important to reduce both costs to a minimum because they can lead to irreversible loss. Based on this perspective, we present in (3.3) an approach that allows us to obtain β value that minimizes the expectation of sum between costs from risk overestimation and underestimation. Results are described in Table (2). < Insert Table 2 > The results of the Table (2) indicate that, for EL, there is a reduction in loss function value. This difference becomes greater mainly for adjusted risk forecasting by model risk measures that quantify the highest model risk value, such as, MR 1, MR 3 and MR3. For these measures, the reduction of model risk included in the adjustment is about 11%. This is observed when analyzing the percentage variation between the average value of β and β. Regarding ES and EVaR, we noted that β value is very small, approximately between 5 and 7. Besides, results indicate a more parsimonious capital determination to reduce the costs mentioned. To find the model risk optimum amount that should be included in the adjustment, i.e. to get β, the manager or regulator must have correctly defined their cost rates. In periods of greater instability, which result in a higher model risk, the difference between G and L will be larger, once that risk underestimation is much more punitive than its overestimation. These results are maintained when analyzing different scenarios (window estimation and forecasting, and other assets). 19

20 5. Final Considerations In this paper, we presented an approach to adjusting financial risk forecasting for the model risk associated with the econometric models in use. Our proposal is compounded by a convex combination of forecasts of a coherent risk measure and a model risk measure, ρ λ + MR. Based on Limitedness, axioms of coherence and financial interpretations are preserved. We also proposed an alternative to determine the model risk optimum amount that should be included in adjustment, in order to reduce the costs from risk overestimation and underestimation. Additionally, we suggested an axiomatic framework for model risk measures. In our study, a model risk measure is a functional belonging to worst-case approach that respects Non-negativity, Translation Insensitivity, Positive Homogeneity, and Sub-additivity axioms. To show the practical usefulness of the proposed approach, we performed an empirical analysis with NASDAQ log-returns. The forecast accuracy was analyzed using loss functions. Our results indicate a better performance of adjusted risk forecasting. We also observed that model risk measures suggested in this study, which quantify asymmetrically model risks, in general, result in forecasts with lower loss function. Additionally, we realize that the model risk adjustment obtained by reducing costs from risk overestimation and underestimation results in more parsimonious forecasting. Thus, in general, it can be observed that adjusted risk forecasting allows, more accurately, outputs for financial decisions. In addition to the contexts explored in this study, our proposal can be applied to other financial problems. For future research, we suggest the generalization of our approach to multivariate cases and its use in other empirical analyses. Acknowledgements We gratefully acknowledge the partial financial support from Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) and Fundação de Amparo a Pesquisa do Estado do Rio Grande do Sul (FAPERGS), Brazil. 20

21 References Acerbi, C., Szekely, B., Back-testing expected shortfall. Risk, 76. Alexander, C., Sarabia, J. M., Quantile uncertainty and value-at-risk model risk. Risk Analysis 32 (8), Artzner, P., Delbaen, F., Eber, J.-M., Heath, D., Coherent measures of risk. Mathematical Finance 9 (3), Barrieu, P., Ravanelli, C., Robust capital requirements with model risk. Economic Notes 44 (1), Barrieu, P., Scandolo, G., Assessing financial model risk. European Journal of Operational Research 242 (2), Basel Committee on Banking Supervision, Revisions to the Basel II market risk framework. Bank for International Settlements. URL Basel Committee on Banking Supervision, Fundamental review of the trading book: A revised market risk framework. Consultative Document, October. URL Bates, J. M., Granger, C. W., The combination of forecasts. Operational Research Quarterly, Bellini, F., Di Bernardino, E., Risk management with expectiles. The European Journal of Finance 23 (6), Bellini, F., Klar, B., Müller, A., Gianin, E. R., Generalized quantiles as risk measures. Insurance: Mathematics and Economics 54,

22 Bernard, C., Vanduffel, S., A new approach to assessing model risk in high dimensions. Journal of Banking & Finance 58, Boucher, C. M., Danélsson, J., Kouontchou, P. S., Maillet, B. B., Risk models-at-risk. Journal of Banking & Finance 44, Boucher, C. M., Maillet, B. B., Learning by failing: a simple VaR buffer. Financial Markets, Institutions & Instruments 22 (2), Breuer, T., Csiszár, I., Measuring distribution model risk. Mathematical Finance 26 (2), Chua, C. L., Suardi, S., Tsiaplias, S., Predicting short-term interest rates using bayesian model averaging: Evidence from weekly and high frequency data. International Journal of Forecasting 29 (3), Cont, R., Model uncertainty and its impact on the pricing of derivative instruments. Mathematical finance 16 (3), Danielsson, J., James, K. R., Valenzuela, M., Zer, I., Model risk of risk models. Journal of Financial Stability 23, Derman, E., Model risk. Quantitative Strategies Research Notes, Goldman Sachs, New York, NY. Detering, N., Packham, N., Model risk of contingent claims. Quantitative Finance 16 (9), Federal Reserve, Supervisory guidance on model risk management. Board of Governors of the Federal Reserve System, Office of the Comptroller of the Currency, URL Föllmer, H., Schied, A., Convex measures of risk and trading constraints. Finance and stochastics 6 (4),

23 Frittelli, M., Gianin, E. R., Law invariant convex risk measures. In: Advances in mathematical economics. Springer, pp Furman, E., Wang, R., Zitikis, R., Gini-type measures of risk and variability: Gini shortfall, capital allocations, and heavy-tailed risks. Journal of Banking & Finance 83, Gneiting, T., Making and evaluating point forecasts. Journal of the American Statistical Association 106 (494), Gourieroux, C., Zakoïan, J.-M., Estimation-adjusted VaR. Econometric Theory 29 (4), Hoogerheide, L., Kleijn, R., Ravazzolo, F., Van Dijk, H. K., Verbeek, M., Forecast accuracy and economic gains from bayesian model averaging using time-varying weights. Journal of Forecasting 29 (1-2), Hull, J., Suo, W., A methodology for assessing model risk and its application to the implied volatility function model. Journal of Financial and Quantitative Analysis 37 (02), Kellner, R., Rösch, D., Quantifying market risk with value-at-risk or expected shortfall? consequences for capital requirements and model risk. Journal of Economic Dynamics and Control 68, Kerkhof, J., Melenberg, B., Schumacher, H., Model risk and capital reserves. Journal of Banking & Finance 34 (1), Krajcovicova, Z., Perez-Velasco, P. P., Vázquez Cendón, C., A novel approach to quantification of model risk for practitionners. Working Paper, URL 23

24 Liu, C., Maheu, J. M., Forecasting realized volatility: a bayesian model-averaging approach. Journal of Applied Econometrics 24 (5), Müller, F. M., Righi, M. B., Numerical comparison of multivariate models to forecasting risk measures. Risk Management 20 (1), Pesaran, M. H., Schleicher, C., Zaffaroni, P., Model averaging in risk management with an application to futures markets. Journal of Empirical Finance 16 (2), Pflug, G. C., Subdifferential representations of risk measures. Mathematical programming 108 (2-3), Righi, M. B., 2017a. A composition between risk and deviation measures. Working Paper. URL Righi, M. B., 2017b. A robust approach for minimization of risk measurement errors. Working Paper. URL arxiv.org/pdf/ pdf Righi, M. B., Borenstein, D., A simulation comparison of risk measures for portfolio optimization. Finance Research Letters 24, Righi, M. B., Ceretta, P. S., A comparison of expected shortfall estimation models. Journal of Economics and Business 78, Righi, M. B., Ceretta, P. S., Shortfall deviation risk: an alternative for risk measurement. Journal of Risk 19 (2), Rockafellar, R. T., Uryasev, S., Zabarankin, M., Generalized deviations in risk analysis. Finance and Stochastics 10 (1), Rockafellar, R. T., Uryasev, S., Zabarankin, M., Risk tuning with generalized linear regression. Mathematics of Operations Research 33 (3),

25 Wang, Y., Ma, F., Wei, Y., Wu, C., Forecasting realized volatility in a changing world: A dynamic model averaging approach. Journal of Banking & Finance 64, Weiß, G. N., Copula-GARCH versus dynamic conditional correlation: an empirical study on VaR and ES forecasting accuracy. Review of Quantitative Finance and Accounting 41 (2),

26 Log returns Figure 1: NASDAQ log-returns for the period between January 2010 to December The returns to the right of the dotted line are from the out-of-sample period. 26

27 MR6,1 (M) MR6,2(M) MR5,2 (M) MR6,1(M) MR5,1 (M) MR5,2(M) MR4,2 (M) MR5,1(M) MR4,1 (M) MR4,2(M) MR3 (M) MR4,1(M) MR2 (M) MR3(M) MR1(M) MR2(M) MR6,2 (M) Figure 2: Weighted EL forecasting (gray line) and model risk measures (black line) for NASDAQ log-returns, during the period from November of 2013 to December of Note: This figure shows the evolution of weighted EL (Expected Loss) forecasting (gray line) (ρ λ ) and of model risk. ρ λ is computed by equally weighted risk forecasting. ρ is quantified using an AR(1)-GARCH(1,1) model with normal (norm), skewed normal (snorm), Student-t (std), skewed Student-t (sstd), generalized error (ged), skewed generalized error (sged), normal inverse Gaussian (nig), and Johnson SU (jsu) distributions. MRj,p, and MR j,p j = 1,, 6 p = 1,2 are, respectively, model risk measures, and their adjusted version using M λ instead of M0. M0 is GARCH model with normal distribution. In the estimation process, the estimation window is 1,000 days and the out-of-sample period is 1,083 days. 27