PRE CONFERENCE WORKSHOP 3

Transcription

1 PRE CONFERENCE WORKSHOP 3 Stress testing operational risk for capital planning and capital adequacy PART 2: Monday, March 18th, 2013, New York Presenter: Alexander Cavallo, NORTHERN TRUST 1

2 Disclaimer The views and opinions expressed during this session are those of the speakers and do not necessarily reflect the views and opinions of their current or previous employers. All examples of analysis presented during this session are only examples. Any assumptions made within the analysis are those of the presenter. 2

3 9:00 10:30 EMPIRICAL MODELS FOR STRESS TESTING OPERATIONAL RISK Defining the appropriate objective Shortcomings of ad hoc approaches Developing a coherent stress testing methodology Potential approaches/methodologies for stress testing operational risk Real world challenges encountered in developing operational risk stress testing models Alexander Cavallo, Senior Vice President, Operational Risk and Risk Analytics, NORTHERN TRUST 3

4 Session 3 Agenda 9:00 10:30 Introduction to stress testing operational risk Presenters: Alexander Cavallo, Risk Specialist, Operational Risk and Risk Analytics, NORTHERN TRUST Mark Weber, Manager, Model Risk Management, NORTHERN TRUST Karl R. Chernak, Director, Operational Risk, RBS CITIZENS FINANCIAL GROUP 11:00 12:30 Empirical models for stress testing operational risk Lead Presenter: Alexander Cavallo, Risk Specialist, Operational Risk and Risk Analytics, NORTHERN TRUST 1:30 3:00 Model Validation Issues for Operational Risk Projection Models in CCAR/CapPR Lead Presenter: Mark Weber, Manager, Model Risk Management, NORTHERN TRUST 3:30 4:30 Deriving Economic Value from Stress Testing Lead Presenter: Karl R. Chernak, Director, Operational Risk, RBS CITIZENS FINANCIAL GROUP 4:30 5:00 Closing Session: Panel Discussion/Q&A Presenters: Alexander Cavallo, Risk Specialist, Operational Risk and Risk Analytics, NORTHERN TRUST Mark Weber, Manager, Model Risk Management, NORTHERN TRUST Karl R. Chernak, Director, Operational Risk, RBS CITIZENS FINANCIAL GROUP 4

5 Defining the appropriate objective: Empirical support for the impacts of macroeconomic factors in the model(s) An effective stress testing framework and model should 1. Make use of well designed scenarios tailored to the banking organization s business and risks 2. Present well documented assumptions 3. Apply sound methodologies to assess potential impact on the banking organization s financial condition 4. Incorporate high quality data and appropriate assumptions about the performance of the institution under stress to ensure that the outputs are credible and can be used to support decision making In this section of the workshop we focus on the quantitative models that translate a given macroeconomic scenario into operational losses For the time being, we assume that bank scenarios are appropriately created with sufficient documentation to support key parameters and assumptions 5

6 Defining the appropriate objective: Empirical support for the impacts of macroeconomic factors in the model(s) Quantitative Models Quantitative models that map the exogenous shocks to the risk exposures and define impacts that are further propagated through the firm Modeling Consideration Theoretical support for stress testing models Empirical support for stress testing models Operational Risk Most LDA implementations have no risk factors and assume that all outcomes are purely random (a random number of events occur with random severities) Few published studies to rely upon for model building suggestions Credit Risk, Market Risk, Counterparty Risk, etc. Macroeconomic factors are key risk drivers of parameters in these risk models Large academic and practitioner literature for most important risk parameters Incorporating macroeconomic factors to operational risk models is much more challenging that for other risk categories 6

7 The Loss Distribution Approach is most commonly used to evaluate operational risk exposures Most institutions use the Loss Distribution Approach (LDA) to estimate operational risk capital requirements The LDA looks at sub segments of historical loss data that are assumed to be sufficiently well behaved to permit statistical modeling (units of measure or UOMs) The aggregate annual loss distribution is decomposed into frequency and severity components Frequency and severity parameters are estimated separately Frequency and severity are recombined to estimate the distribution of total losses occurring in one year Illustration of the Loss Distribution Approach Unit of Measure Loss Data Frequency Distribution # of loss events per year Monte Carlo Simulation Severity Distribution Aggregate Annual Loss Distribution Annual losses ($) $ value of loss event 7

8 Interpreting the aggregate loss distribution Aggregate Loss Distribution (ALD) Mode (37% in ALD) 1 2 The purple curve represents the aggregate loss distribution Because the loss distribution is right skewed, the Mode and Median are less than the Mean (Expected Operational Loss) Distribution of Aggregate Loss 2 Median Operational Loss (50% in ALD) Expected Operational Loss (80% in ALD) 1 Aggregate Annual Loss (Log scale)

9 Expected operational loss is the central focus in stress testing For stress testing purposes, the bank needs to make a projection of operational losses ($) for each period in the projection horizon for each macroeconomic scenario A key point in stress testing is to evaluate the potential cash flow impacts of macroeconomic stress, to assess whether the institution has sufficient capital to withstand stressed losses This corresponds to the concept of expected operational loss (EOL) in the context of operational risk capital modeling and can be computed in several ways depending upon whether macroeconomic factors are assumed to affect loss frequency, loss severity, or both where is loss frequency given macroeconomic factors Z is average loss severity given macroeconomic factors Z 9

10 Two categories of empirical models may be considered for stressing expected operational loss where is loss frequency given macroeconomic factors Z is average loss severity given macroeconomic factors Z 1. Models for the impact of macroeconomic factors on loss frequency 2. Models for the impact of macroeconomic factors on loss severity Before discussing empirical models for the three potential options, we first discuss the problems of some simple ad hoc approaches 10

11 Distribution of Aggregate Loss Ad hoc Approach # 1: Using a higher percentile for estimating EOL In the past, some institutions have produced stressed EOL estimates by simply selecting a higher percentile of the aggregate loss distributions Aggregate Loss Distribution (ALD) Mode (37% in ALD) Median Operational Loss (50% in ALD) Expected Operational Loss (80% in ALD) 90th %ile in ALD 95th %ile in ALD 99th %ile in ALD Aggregate Annual Loss (Log scale) Regulatory feedback has been very negative There is no empirical basis to support selecting any particular percentile over another (no way to determine how much stress is enough) The assumption that the baseline frequency and severity parameters persist during stress is unsupported 11

12 Distribution of Aggregate Loss Ad hoc Approach # 2: Increasing key parameters for EOL without empirical support In the past, some institutions have produced stressed EOL estimates by simply modifying baseline estimates of frequency or severity parameters (for example, use baseline frequency + 2 SD) Aggregate Loss Distribution (ALD) Mode (37% in ALD) Median Operational Loss (50% in ALD) Expected Operational Loss (80% in ALD) "Stressed" ALD (baseline freq. + 2 SD) Aggregate Annual Loss (Log scale) Regulatory feedback has been very negative There is no empirical basis to support this type of adjustment (no way to determine how much stress is enough) 12

13 Distribution of Aggregate Loss A coherent stress testing methodology builds an empirical basis for changing the risk profile The empirical support for changing the risk profile in the stressed scenario can be based upon a model, historical analysis, or judgment informed by model or historical results Aggregate Loss Distribution (ALD) Mode (37% in ALD) Median Operational Loss (50% in ALD) Expected Operational Loss (80% in ALD) "Stressed" ALD (When macro factors Z=Z*) Aggregate Annual Loss (Log scale) Regulatory feedback has generally been positive Regulators appear to understand that for some risk classes, a satisfactory model may not be attainable In such cases, sufficient justification for the change in risk profile may need to be developed from other information sources 13

14 Incorporating macroeconomic stress via loss frequency Count regression models are widely used to estimate models that account for the impact of macroeconomic factors on operational loss frequency In count regression models, observed loss frequency in period t (N t ) is assumed to depend on explanatory variables for that period (Z t ) and an unknown parameter vector ( For stress testing applications, the explanatory variables (Z t ) may include relevant macroeconomic factors along with other variables thought to influence loss frequency (e.g. seasonal effects, structural shifts, etc.) Because loss frequency is an integer number, practitioners typically assume loss frequency has a Poisson or Negative Binomial distribution The parameters of the model are commonly estimated with Maximum Likelihood Estimation or a similar numerical algorithm yielding parameter estimates 14

15 Incorporating macroeconomic stress via loss frequency After estimating the count regression model, projections of loss frequency,, for a specific set of hypothetical macroeconomic conditions,, are straightforward to calculate This projection approach is a credible, data based mechanism for projecting the number of operational losses that might occur under a variety of different hypothetical macroeconomic conditions A different frequency projection will be estimated for each different combination of explanatory variables that is assumed for the hypothetical projection The statistical foundation for estimating count regression models can be found in any number of intermediate to advanced statistics or econometrics textbooks Judgmental overrides of projections may be needed when the model results are highly implausible from a business perspective overrides are likely to be closely scrutinized by regulatory supervisors 15

16 Incorporating macroeconomic stress via loss frequency CAUTION: The validity of these loss frequency projections relies upon the validity of the underlying statistical model and its assumptions Frequency projections may be biased if the model assumptions fail some care should be exercised to assess whether The assumed distribution is incorrect A relevant explanatory variable is omitted from the model The unobserved determinants of loss frequency (the error terms) are correlated from one period to another The observed historical data is not a reasonably representative sample Another critical assumption of the projection approach is that the relationship between the explanatory variables and loss frequency remains constant over the projection horizon NOTE: This assumption is a very strong assumption and may not be reasonable for projecting loss frequency when the hypothetical macroeconomic conditions are well outside the range observed in the historical data sample that was used for model fitting 16

17 Incorporating macroeconomic stress via loss severity: Simple approaches Some institutions use regression analysis of average loss severity by time period to estimate models that incorporate macroeconomic factors into the estimation of EOL Similar to count regression models, basic linear regression models assume that observed average loss severity (Y t ), depends on explanatory variables (Z t ), an unknown parameter vector ( and random error (u t ) For stress testing applications, the explanatory variables (Z t ) may include relevant macroeconomic factors along with other variables thought to influence average loss severity The average severity model is generally sometimes assumed to be a simple linear model such as, or a log linear model like, (which ensures that average severity is strictly positive) The parameter estimates of the model,, can be estimated with many different estimators 17

18 Incorporating macroeconomic stress via loss severity: Simple approaches After estimating the regression model, projections of average severity,, for a specific set of hypothetical macroeconomic conditions,, are straightforward to calculate or This projection approach is a credible, data based mechanism for projecting the number of average severity of operational losses that might occur under a variety of different hypothetical macroeconomic conditions A different average severity projection will be estimated for each different combination of explanatory variables that is assumed for the hypothetical projection As noted previously, judgmental overrides of projections may be needed when the model results are highly implausible from a business perspective overrides are likely to be closely scrutinized by regulatory supervisors 18

19 Incorporating macroeconomic stress via loss severity: Simple approaches CAUTION: The validity of these average severity projections relies upon the validity of the underlying statistical model and its assumptions Average severity projections may be biased if the model assumptions fail some care should be exercised to assess whether The assumed functional form for the model is incorrect A relevant explanatory variable is omitted from the model The unobserved determinants of average severity (the error terms) are correlated from one period to another The observed historical data is not a reasonably representative sample As was the case for frequency projections under stressed conditions, the same comments apply for average severity projections under stressed conditions The assumption of a stable relationship between outcome and explanatory variables is critical The same concerns about extrapolation of the estimated relationship far out of sample apply 19

20 Incorporating macroeconomic stress via loss severity: Simple approaches Note: Empirical analysis of average values for heavy tailed distributions can be complicated by the dominance of the extremes The mean is a non robust estimator The presence of a small number of dominantly extreme value can obliterate the information about central tendencies contained in all the other data points The impact of this non robustness is attenuated in larger data samples, but the mean in any finite sample can be made arbitrarily large or small by a small number of sufficiently extreme data values This problem can be addressed in a number of ways Robust statistics of central tendency such as the median could be used in place of the mean The mean can be estimated using robust approaches to mitigate the impact of the extremes Such solutions to the problem of extremes create other challenges Total observed loss may not be well approximated The regulatory position on acceptable treatment of such extremes is unclear 20

21 Incorporating macroeconomic stress via loss severity: Parametric approaches In principle, it is possible to estimate models for the severity of losses that incorporate explanatory variables to change the parameters of the assumed severity distribution Conditional on the observed values of the explanatory variables pertinent to time period t (Z t ), individual loss severity (X it ) is assumed to follow a parametric distribution F with location parameter t and scale parameter t For stress testing applications, the explanatory variables (Z t ) may include relevant macroeconomic factors from time period t along with other variables thought to influence the shape of the severity distribution where and are functions that ensure the location and scale parameters remain in the admissible range of values for distribution F This model can be estimated by Maximum Likelihood Estimation to recover the estimates of parameters 1 and 2 21

22 Incorporating macroeconomic stress via loss severity: Parametric approaches Projections of period t average loss severity can be obtained following steps similar to what has already been discussed 1. Project the severity parameters for period t given the specific hypothetical macroeconomic conditions, Using the projected values of the severity parameters, a projection of average loss severity,, can be made where f is the probability density for distribution F The same caveats and limitations previously discussed apply for this methodology Average severity projections may be biased if the model assumptions do not hold The assumption of a stable relationship between outcome and explanatory variables is critical and extrapolation of the estimated relationship far out of sample may be inappropriate 22

23 Challenges encountered: Linking loss data to time periods Incorporating macroeconomic factors to operational losses requires linking time period specific explanatory variables to individual or aggregated operational losses The sequence of dates involved in an operational loss event is 1. An operational event occurs (potential operational loss) 2. Operational event is discovered (financial impact may be unknown) 3. Financial impact is taken in accounting entries when the financial impact of the event is measurable (meaning a monetary amount can be determined with reasonable certainty or is reasonably estimable) As a result, many operational loss events Remain undiscovered for quite some time Lack sufficient certainty of financial impact to warrant accounting recognition until some time after discovery Have financial impacts that change over time as more is learned about the event 23

24 Challenges encountered: Linking loss data to time periods Currently, there does not appear to be a dominant industry practice regarding the use of occurrence date, discovery date, or settlement date for operational risk modeling and stress testing Empirical patterns and stress test findings are likely to be very different depending upon which date is used for linking macroeconomic factors and analyzing loss frequency In the end, there is typically insufficient information to distinguish macroeconomic impacts associated with conditions at event occurrence, at loss discovery, and at financial recognition Moreover, there are plausible theoretical explanations by which conditions at each of the different dates impact loss occurrence and/or severity These considerations may explain the apparent difficulty in finding macroeconomic risk drivers for operational risk 24

25 Challenges encountered: Time series issues Regression based approaches for stress testing require that the underlying data be suitable for empirical modeling There is a wide literature documenting the spurious relationships that can be obtained when fundamental properties of time series data are not properly considered A particular concern is whether a time series is stationary, meaning that the analysis variable has a constant mean and variance and that its autocorrelation depends only on the number of time periods between data points Time series can be made stationary by transformations such as the difference or ratio of values in consecutive periods and there are several statistical tests that can be used to test for stationary time series Many important macroeconomic variables exhibit seasonality, trend effects, and cyclical behavior, which complicate their use as explanatory variables in regression analysis 25

26 Challenges encountered: Time series issues In spite of the wide array of statistical tests, determining whether a given time series is stationary can be quite challenging Unfortunately, the tests often give conflicting conclusions about stationarity The standard statistical tests have different forms depending upon whether the time series has a mean of zero, a non zero mean, or trend effects Test results are often unhelpful in determining which form of the test is most appropriate for a particular time series Tests for stationary time series cannot distinguish truly non stationary data from data that has structural breaks in the mean or variance over time Hidden Markov Models are a class of stationary time series models in which the analysis variable can be characterized as exhibiting a small number of distinct regimes Within a regime, the variable is stationary, but across regimes, mean, variance, and other characteristics can vary The analysis variable switches randomly from one regime to another 26

27 Market Excess Return Challenges encountered: Time series issues Example: The market excess return can be characterized by a two state Hidden Markov Model variables in regression analysis 15% 10% 5% 0% 5% 10% 15% 20% 25% 30% 35% 2000Q1 2001Q1 2002Q1 2003Q1 "Normal" Markets "Stressed" Markets 2004Q1 2005Q1 2006Q1 2007Q1 2008Q1 2009Q1 2010Q1 2011Q1 2012Q1 In Normal Markets, excess returns Are positive on average Exhibit modest volatility In Stressed Markets, excess returns Are slightly negative on average Exhibit much greater volatility Evaluating stationarity of a macroeconomic variable can be easier with the Hidden Markov Model concept than with standard time series tests 27

28 Challenges encountered: Lack of suitable predictive model with direct macroeconomic effects What can be done if for a segment of operational loss data, no specification of a stress testing model empirical model is found? Some institutions have used two stage empirical models to indirectly impart macroeconomic stress to loss categories where a direct approach failed to satisfy bank model selection criteria The two stage approach leverages the predictive power macroeconomic factors in one loss category (category D for direct impact) along with a modest correlation of outcomes across categories to indirectly pass macroeconomic stress to the other loss category (category I for indirect impact) The predicted values for loss category D are used an explanatory variables in the model for loss category I 28

29 Challenges encountered: Lack of suitable predictive model with direct macroeconomic effects What can be done if for a segment of operational loss data, no specification of a stress testing model empirical model is found? Assumptions for estimation of two stage frequency model Directly modeled category has non zero impact from explanatory variables Z t 0, Indirectly modeled category has zero impact from explanatory variables Z t 0, Indirectly modeled category has non zero impact using from explanatory variables The models can be simultaneously estimated using the Generalized Method of Moments or can be sequentially estimated using Maximum Likelihood Estimation 29

30 Next Topic 9:00 10:30 Introduction to stress testing operational risk Presenters: Alexander Cavallo, Risk Specialist, Operational Risk and Risk Analytics, NORTHERN TRUST Mark Weber, Manager, Model Risk Management, NORTHERN TRUST Karl R. Chernak, Director, Operational Risk, RBS CITIZENS FINANCIAL GROUP 11:00 12:30 Empirical models for stress testing operational risk Lead Presenter: Alexander Cavallo, Risk Specialist, Operational Risk and Risk Analytics, NORTHERN TRUST 1:30 3:00 Model Validation Issues for Operational Risk Projection Models in CCAR/CapPR Lead Presenter: Mark Weber, Manager, Model Risk Management, NORTHERN TRUST 3:30 4:30 Deriving Economic Value from Stress Testing Lead Presenter: Karl R. Chernak, Director, Operational Risk, RBS CITIZENS FINANCIAL GROUP 4:30 5:00 Closing Session: Panel Discussion/Q&A Presenters: Alexander Cavallo, Risk Specialist, Operational Risk and Risk Analytics, NORTHERN TRUST Mark Weber, Manager, Model Risk Management, NORTHERN TRUST Karl R. Chernak, Director, Operational Risk, RBS CITIZENS FINANCIAL GROUP 30