Validation of Nasdaq Clearing Models

Transcription

1 Model Validation Validation of Nasdaq Clearing Models Summary of findings swissquant Group Kuttelgasse 7 CH-8001 Zürich Classification: Public Distribution: swissquant Group, Nasdaq Clearing October 20, 2016

2 CONTENTS Contents 1 Introduction 1 2 swissquant s Model Validation Framework 2 3 Nasdaq Clearing Policies 3 4 OMS II Model 4 5 CFM Model 6 6 SPAN Model 8 7 Conclusion 10 i

3 1 Introduction Nasdaq Clearing provides clearing services as central counterparty (CCP) in accordance with European Regulation. Following regulatory requirements, Nasdaq Clearing requested the swissquant Group to conduct its annual model validation of its margining models and of its internal policies. The purpose of the validation is to ensure the theoretical and empirical soundness of the margining models used by Nasdaq Clearing and the appropriateness of its internal policies. Moreover, Nasdaq Clearing have defined special validation areas such as model changes, or extension of a model to other markets, which have been thoroughly investigated by the swissquant Group. The purpose of this document is to summarize the main findings of the model validation on Nasdaq Clearing internal policies, on the OMS II model for equity futures and options, on the CFM model for fixed income products and on the SPAN model for commodities and power. The special validation areas consist of the model for the World Basket of Futures, which is a new product of Nasdaq Clearing within the OMS II model and the Gemini project, which is an extension of the current SPAN model for gas and electricity to other European Markets than Nordics and Germany. The model validation has been conducted following the structure recommended by the Model Validation Policy of Nasdaq Clearing. As input to the validation, the swissquant Validation team has received the Margin Methodology guides and Model Instructions for all three models. Moreover, it has received all the internal policy documents, which define the general principles valid accross all margining models at Nasdaq Clearing. Additionally, numerical data consisting of backtesting data, sensitivities analyses and stress testing data have been provided to support the model validation. For the special validation areas, a complete set of input data has been provided for the World Basket of Futures such that the model could be replicated. For the Gemini project a set of electricity prices from Nordic countries, Germany and France have been provided such that a comparison of the characteristics of the new markets could be assessed in order to confirm the appropriateness of applying the SPAN model for the new markets. 1

4 2 swissquant s Model Validation Framework The swissquant Group follows an internal standard when conducting Model Validation for CCPs, while taking into account the structure defined by the Model Validation Policy specific to each CCP. In particular, all the findings of the model validation have been subjected to a rigorous examination by the swissquant Methodology Board, so are the final validation documents. This is to ensure the highest quality standards of the work given in all the documents. The process involves a round table discussion of the theoretical and empirical analysis presented by the Model Validation team to senior quantitative analysts at swissquant Group. For outline purposes, swissquant uses a parsimonious traffic light grading scheme in this document. Details of the applied grading criteria are summarised in Table 1. Colour Code Grade Criteria Sign-off Tentative Sign-off Rejection ˆ Methodology is judged to be appropriate and adequate for the intended purpose and in line with the relevant regulatory requirements and Nasdaq Clearing s policies. ˆ Methodology is judged to be generally appropriate and adequate for the intended purpose; ˆ However, certain limitations which are of immaterial impact to the results are revealed during the model validation process; ˆ Identified issues can easily be rectified within a reasonable period of time; ˆ Future analysis and monitoring need to be conducted during the next model validation projects. ˆ Methodology is judged to be inappropriate for the intended purpose; ˆ Serious concern on certain elements of the approach and/or critical remedial actions required; ˆ Concerns must be addressed as soon as possible. Table 1: Summary of the grading criteria 2

5 3 Nasdaq Clearing Policies The summary of the findings regarding the Nasdaq Clearing Policies are listed in Table 2 using the traffic light grading scheme. Validation Area Grade Recommended Actions & Comments Back Testing Policy The back testing policy is well structured and documented. It follows common best practices among CCP. However, it should be stated that the Model level backtesting must consist of out-of-sample backtests, which are performed on hypothetical portfolios (fixed portfolio, long-short, options strategies, etc). Those clean backtests test the statistical performances of the margin model. Additionally, the historical data horizon for the back testing ( backtesting period ) and the look-back period should not be dependent from each other, since they are conceptually unrelated. Clearing Risk Policy The clearing risk policy is well structured and documented. It follows common best practices among CCP. Margin Parameter Policy The margin parameter policy is well structured and documented. It follows common best practices among CCP. However, it is stated in the parameter estimation procedure that seasonality of the data and presence of repeated data patterns should be taken into account. Several commodities products cleared by Nasdaq Clearing have clear seasonal behavior, however it is not explicitly modelled in the margin. The only action is to have a minimum lookback window of 1 year in order to include all seasonal levels in the calibration procedure. Model Validation Policy The model validation policy is well structured and documented. It follows common best practices among CCP. However, in the backtesting section, it should be stated that both clean and dirty backtest (called entreprise level backtest at Nasdaq Clearing) must be performed. The clean backtest on out-of-sample data is very important for assessing the statistical performance of the margin model. Sensitivity Testing and Analysis Policy The sensitivity testing and analysis policy is well structured and documented. It follows common best practices among CCP. Stress Testing Policy The stress testing policy is well structured and documented. It follows common best practices among CCP. However, in the definition of historical extreme events, it should be also mentioned that large upwards movement (e.g. market rebound) are also very important in the context of CCP stress testing, since Clearing Members may have large short portfolios. Instructions for the Member Risk Committee Instructions for the Clearing Risk Committee The instructions for the member risk committee policy is well structured and documented. It follows common best practices among CCP. The instructions for the clearing risk committee policy is well structured and documented. It follows common best practices among CCP. Table 2: Summary of model validation findings for the policies 3

6 4 OMS II Model The summary of the findings regarding the OMS II model are listed in table 3 using the traffic light grading scheme. Validation Area Grade Recommended Actions & Comments General Framework of the Model OMS II model is a scenario-based model. Its strengths consist in its simplicity and robustness. However: ˆ it does not model the volatility dynamically, therefore the model equally reacts whether a margin break has happened recently or further into the lookback period; Nasdaq prefers stability of the margins over adaptability; ˆ the recent trend in the industry, partially driven by regulation, seems to point towards VaR-type model; Model Documentation Numerical Inputs for backtesting Backtest results Correlation between Underlyings The model instructions document and the margins guide document give a detailed description of the model. The first focuses more on the technical aspects, while the latter gives many practical examples. In the model instructions some formulas and model parameters need to be updated to reflect the last changes in the usage of the model. A further potential improvement would be to include a description of the fallback rules, i.e. how missing observations are handled (e.g. for very illiquid underlyings, long-dated futures, etc.). This topic is currently not addressed in the documents. The time series of the computed initial margins and the actual P&L were made available from 2006 for a large number of single stocks and indexes: ˆ 10 years of data is a long enough period to perform statistically significant backtests. ˆ however, regarding the portfolios composition, it s important for next year validation to check the performance of the model on portfolios with multiple assets and with options. The margins computed by the model are conservative. Compared to the chosen confidence level, backtests with too few violations are more frequent than those with too many. One of the main features of equity market is the correlation structure existing between underlyings. The model can capture correlation and netting effects through the use of the window method. However this feature is not activated in the production system which assumes no correlation between instruments. This choice is highly conservative, at the expense of margins efficiency. Table 3: Summary of model validation findings for the OMS II model 4

7 Validation Area Grade Recommended Actions & Comments Asymmetry of Returns Another stylized fact of equity market is the asymmetry of returns. OMS II model does not account for returns asymmetry, as only absolute returns are processed. This is again a conservative choice as the worst returns, without distinction between positive and negative ones, are used to compute the risk intervals. Implied Volatility Shift Implied volatility risk is captured in the construction of the risk arrays, which include upward, unchanged and downward-shifted volatility. Evaluation is performed across scenarios using the correct option pricing formula. However, a constant shift is applied regardless of the time-to-maturity and the moneyness of the option. Nasdaq defines the size of the shift by looking at the options where it has the largest exposure (short-term options). The approach is simple and robust, but its performance is unclear for more complex strategies (e.g. spread of 2 options, delta-vega hedged options). The choice of including a bid-ask spread ensures some buffer for this multi-options portfolio. Implied Volatility Surface Implied volatility surface for liquid options is provided by a large and independent data provider with a dense grid of strikes and all the traded matuirities. This is a good modelling practice. For illiquid options, the skew is ignored and the implied volatility is set: ˆ equal to the average of ATM options for given underlying and maturity: this is not a conservative choice for OTM options that usually show an higher implied volatility; ˆ equal to a constant value (usually 40%): this is a quite crude assumption but exposure to these options is very limited. Special Focus (Futures Basket) Time series of index futures in local currency along with the FX rates in SEK were provided for more than 10 years. Besides, some baskets compositions were indicated by Nasdaq and some others were defined by swissquant. We have backtested these baskets and performed a sensitivity analysis on the main parameters. We have also conducted an analysis on the FX risk. We have not found any criticalities. Table 3: Summary of model validation findings for the OMS II model 5

8 5 CFM Model The summary of the findings regarding the CFM model are listed in table 4 using the traffic light grading scheme. Validation Area Grade Recommended Actions & Comments General Framework of the Model The CFM model is a scenario-based model. Its strengths consist in its simplicity and robustness. It employs the current best practices to deal with fixed income instruments, namely: ˆ Bootstrapping to construct yield curves; ˆ Multi-curve approach for pricing; ˆ PCA for modelling the curves and margining. However: ˆ it does not model the volatility dynamically, therefore the model equally reacts whether a margin break has happened recently or further into the lookback period; Nasdaq prefers stability of the margins over adaptability; ˆ the recent trend in the industry, partially driven by regulation, seems to point towards VaR-type model. Model Documentation Numerical Inputs for backtesting Backtest results The model instructions document and the margins guide document give a detailed description of the model. The first focuses more on the technical aspects, while the latter gives many practical examples. A further potential improvement would be to include a description of the fallback rules, i.e. how missing observations are handled (e.g. for very illiquid underlyings, long-dated futures, etc.). This topic is currently not addressed in the documents. Time series of the computed initial margins and the actual P&L were made available from 2013 only for 6 single instrument portfolios: ˆ 4 years of data is a quite short period to perform a statistically significant backtest, especially with 5 lead days; for next year validation longer time series should be provided; ˆ moreover, regarding the composition of the portfolios, it s important for next year validation to check the performance of the model on more instruments and on portfolios with multiple assets (the exposure to options is almost insignificant). The margins computed by the model are conservative. Compared to the chosen confidence level, backtests with too few violations are more frequent than those with too many. Table 4: Summary of model validation findings for the CFM model 6

9 Validation Area Grade Recommended Actions & Comments Correlation between Instruments The model is designed to capture: ˆ Intra-curve correlation (within the same curve), which is a built-in feature of the model due to PCA; however the risk associated to PCs beyond the third is neglected; ˆ Inter-curve correlation (between curves), which is handled through the 3D-window method; however, the latter might be cumbersome to use and the complexity increases with the number of risk groups involved in a portfolio. Principal Components Analysis Margin Model Parameters The main risk factors involved in the fixed income markets are the yield curves. Yield curves are usually wellmodelled by the first three principal components, but this could no longer be true in case of complex market dynamics. Nasdaq monitors the explanatory power of these 3 principal components on a daily basis and the system is ready to incorporate additional components should the explanatory power deteriorate. An idiosyncratic factor might be included to capture PCs beyond the third one. Margin model parameters are clearly described in the documents and the sensitivity to the main parameters is constantly monitored by Nasdaq. For next year validation, it would be interesting to test the impact of changing the priority tree structure involved in the 3D-window method. Table 4: Summary of model validation findings for the CFM model 7

10 6 SPAN Model The summary of the findings regarding the SPAN model are listed in table 5 using the traffic light grading scheme. Validation Area Grade Recommended Actions & Comments General Framework of the Model The SPAN model is a well known scenario-based model. Its strengths consist in its simplicity and diffusion among market participants/clearing houses. However: ˆ it does not model the volatility dynamically, therefore the model equally reacts whether a margin break has happened recently or further into the lookback period; Nasdaq prefers stability of the margins over adaptability; ˆ the recent trend in the industry, partially driven by regulation, seems to point towards VaR-type model. Model Documentation Numerical Inputs for backtesting Backtest results Volatility Curve The model instructions document and the margins guide document give a detailed description of the model. The first focuses more on the technical aspects, while the latter gives many practical examples. A further potential improvement would be to include a description of the fallback rules, i.e. how missing observations are handled (e.g. for very illiquid underlyings, long-dated futures, etc.). This topic is currently not addressed in the documents. Time series of the computed initial margins and the actual P&L were made available from 2014 for a large number of single instrument linear portfolio: ˆ 3 years of data are a quite short period to perform a statistical significant backtest; for next year validation longer time series should be provided; ˆ regarding the composition of the portfolios, it s important for next year validation to check the performance of the model on portfolios with multiple assets and with options. The margins computed by the model are conservative. Compared to the chosen confidence level, backtests with too few violations are more frequent than those with too many. One of the main features of commodity markets is the time-to-maturity effect on returns of futures and forward. The volatility curve synthetically captures this effect in a conservative way. Table 5: Summary of model validation findings for the SPAN model 8

11 Validation Area Grade Recommended Actions & Comments Correlation between Instruments Nasdaq allows for netting between contracts written on the same commodity and between some commodities. Netting is allowed on a conservative base. However: ˆ the netting procedure makes the model much more complex to communicate; ˆ the inter- and intra-commodity netting are done indipendently of each other (the first is based on time buckets, the second on tiers); this can be sub-optimal from a margin efficiency perspective; ˆ the number of parameters grows significantly. Estimation Period Exogenous Parameters Options Special Focus (Gemini Project) Regarding the estimation period, we suggest using at least 2 years (now the minimum is set to 1 year, but some commodities already use 2 years); for instance for power markets winter is usually the most volatile period of the year but suppose the last winter was extremely calm due to higher than expected temperature; in this case it is important to have at least a second winter in the sample. Moreover using one year implies having only 90 days from the most volatile period. Apart from the three main model parameters (confidence level, liquidation period and estimation period) and those estimated within the model, there are few parameters that are set exogenously by the user (e.g. volatility floor). Most of them are used to make the model more conservative, but they also reduce the transparency of the model. Implied volatility risk is captured in the construction of the risk arrays, which include upward and downward shift of the volatility. Evaluation is performed across scenarios using the correct option pricing formula. Moreover different shifts are used depending on the time-to-expiration of the option. Although the moneyness of the options is not considered and a common shift is applied to options in the same bucket, the model should be able to handle single positions in options. For more complex strategy (e.g. spread of 2 options, delta-vega hedge options), the performance is unclear. Given the relative small exposure of options, we don t see this area as critical. For electricity market, data from the Nordic, German and French market were provided. These data include futures with different maturities and delivery periods. We tested whether the data referring to the French market differ significantly from the other two markets which were already cleaned by Nasdaq. Although the data available were quite limited, we have not found significant differences and we don t see any criticality in applying the current model to the new markets. In terms of new types of contract covered (monthly deferred vs deferred settlement futures), we also don t see any criticality. As time-series of P&L and margins become available, a proper backtest should be conducted. Table 5: Summary of model validation findings for the SPAN model 9

12 7 Conclusion The swissquant Model Validation Team has performed the validation of Nasdaq Clearing margining models following the Model Validation policy of Nasdaq Clearing. The input to the validation included documentation in the form of Model Instructions and Margin Methodology guides documents. Additionally numerical data such as timeseries data, backtesting results, sensitivity analyses and stress testing results have been provided for the model validation. Additional documentation and numerical data have been provided for the special validation areas, e.g. World Basket of Futures and Gemini project. In general, the theoretical framework of all three margining models (OMS II, CFM, SPAN) is good. The models are mostly scenario-based models with parameters calibrated using risk-based models (e.g. extreme value theory). The margining models are using common statistical methods amoung CCPs, such as Principal Component Analysis (PCA), empirical distribution from historical returns, etc. The margining models are highly dependent on the parameters and therefore Nasdaq Clearing monitors the performance of the model via backtesting and sensitivity testing on a daily basis to ensure that a re-calibration of parameters can happen quickly if needed. Moreover, when approximations are defined to simplify the margining models, the conservative approach is always selected. In particular, the modelling of options implied volatilities could be more sophisticated, however the chosen approximations tend to be on the conservative side. The model documentations have been reviewed and are generally good. The combination of the Model Instructions document and the Margin Methodology guide are sufficient to understand how the models work. A potential improvement to the documentation would be to include a description on the methods handling missing prices for very illiquid underlyings or long-dated futures (also called fallback rules ). Regarding the numerical input for backtesting, improvements should be made for the next years model validations. Ideally, the backtesting should be performed on out-of-sample data with a backtesting period of 5 to 10 years. Moreover, the backtests should be performed on an extensive set of hypothetical portfolios including long portfolio, long-short portfolio and options strategies (e.g. delta hedged portfolios). Therefore, a recommendation is to state clearly in the backtesting policy under model level backtesting that the above-mentioned procedure should be followed in the limit of what is possible with the current software implementation. For the OMS II model, the swissquant Model Validation team has received backtesting data for more than 10 years backtesting period, however no options portfolio were provided. A recommendation for next year validation is to focus on portfolio consisting of strategies on equity and index options. For the CFM model, only 4 years of backtesting period and only a limited set of portfolio have been provided. A recommendation for next year validation is to increase the backtesting period and to include portfolios on multiple assets. For the SPAN model, only 3 years of backtesting period and only a limited set of portfolio have been provided. A recommendation for next year validation is to increase the backtesting period and to include portfolio with multiple assets, especially with options. The backtesting results of the given portfolios for all three models have shown that the margining models always behave in a conservative way. Therefore, from a regulatory perspective, the three margining models fulfil their role of protecting the CCP in case of a clearing member s default. 10

13 For the special validations areas, the margining models for World Basket of Futures, within the OMS II model, and the Gemini project, within the SPAN model, have been thoroughly investigated. Several statistical analysis have been performed and the conclusion is that the models are appropriate for the new purposes. Therefore, no critical point have been raised on those special validation areas. 11

14