1. Challenges for early warning models in the policy process. 2. The evaluation approach: What is a good early warning model?

Transcription

1

2 Outline 1. Challenges for early warning models in the policy process 2. The evaluation approach: What is a good early warning model? 3. Types of early warning models: some examples 4. Caveats on thresholds to activate instruments 5. Conclusions 2

3 Challenges for Early Warning Models in the Policy Process 3

4 Challenges to overcome for EWMs Eichengreen (2002): Economic forecasting is like weather forecasting except that our knowledge of the underlying science is less complete, and with respect to early warning models even more scepticism: Structural relationships interact in nonlinear and state-contingent way Complex systems often have multiple equilibria sensitive to small perturbations Circularity, as forecast might affect outcome (Goodhart s Law: Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. ( ~ Lukas critique) => The role of fundamentals might be difficult to detect 4

5 Further challenges to overcome Out-of-sample failures due to in- sample over-fitting and variable selection bias Too few crises events / degrees of freedom After 1997 Asian crisis wave of optimism regarding new EWMs (data, methodologies, policy makers attention) very much like today So what is different today to increase our level of confidence in this type of models? 5

6 Reasons for optimism Out of sample methods for model selection and performance (e.g. cross-validation in random forests or recursive validation in discrete choice models, Lasso) (Pseudo) real time concepts Predict vulnerable states, not crisis date (e.g. crisis within next 3 years ) Focused EWMs for specific policy instruments (e.g. CCB analysis) Data availability; soon e.g. Credit registers, Corep/Finrep, Securities holdings statistics, Liquidity indicators (LCR, NSFR) Evaluation methods more refined: AUROC, Loss function, Usefulness measures [forget noise to signal ratio] Some key studies establishing that not every time is different. Reinhart/Rogoff (2009) and Schularick/Taylor (2009, 2014): It s credit and it s housing. See also: 6 ESRB OP No. 5.

7 The Evaluation Approach: What is a good early warning model? 7

8

9

10 Types of Early Warning Models: Some examples 10

11 Types of EWMs Three (operating) approaches to extract signals from indicators to predict crises: Signalling approach: Signal is extracted directly from the indicator when breaching a threshold. Both uni-variate and multi-variate approaches possible. Discrete-choice approach: A probit or logit mapping function transforms the variables into a continuous indicator variable. If the predicted probability exceeds a specified threshold a signal is issued. Decision trees / Random Forest: A partitioning algorithm that recursively identifies the indicators and the (optimal) thresholds that best split the sample into tranquil and pre-crises periods / then do some bragging. Work in progress: artificial neural networks, support vector machines 11

12

13 Discrete choice example: The bank early warning model Note: The aggregation is done by weighing the bank-specific distress probabilities by the respective bank shares in total euro area bank assets. The decomposition of individual distress probabilities into the different factors is done by using the (relative) distress probabilities that would prevail if all other variable blocks were set to their mean values. Source: Lang, Peltonen, Sarlin (2015), A Framework for Early-Warning Modeling, ECB, mimeo. 13

14 Real-time prediction of the Bank Early Warning Model Probability above threshold in 2007Q2 and below in 2014Q2 Note: Based on a recursive out-of-sample exercise starting in 2006Q1. A bank is signalled to be vulnerable if the predicted probability exceeds the optimal signalling threshold based on a policymaker with preference parameter of µ=0.9. The aggregation is done by weighing the bank-specific distress probabilities by the respective bank shares in total euro area bank assets. Source: Lang, Peltonen, Sarlin (2015), A Framework for Early-Warning Modeling, ECB, mimeo. 14

15

16 Caveats on thresholds to activate instruments 16

17 Problems in using models for activating instruments Model derived thresholds depend on policy makers preferences Indicators/models derived from pooled analysis might not work for individual countries Even if they do, pooled thresholds might not be reasonable for individual countries No simple solution for country specific thresholds: A) apply pooled optimal percentile to country series B) group countries => FSC/MPAG and ESRB CCB groups working on this (and other methods) Guided discretion (and suite of models) approaches Enter presentation title by changing the footer. 17

18

19 Pooled thresholds with different country groups Indicator Euro area Advanced Economies "Property" Rank T AUC Rank T AUC Rank T AUC Bank credit/gdp gap Total credit to HH/GDP gap Basel gap Real total credit growth HH (3year) Euro area : EMU 18 Advanced economies according to the ESRB OP definition: AT, BE, DE, DK, ES, FI, FR, GR, GB, IE, IT, NL, PT, SE Property : Countries with significant historical property sector developments: BE, DK, ES, FR, GB, IE, NL, PT, SE Source: Welz (2014), ECB mimeo, July. 19

20 Conclusions 20

21 Conclusions There are simple Early Warning Models out there which would have predicted the financial crisis like many other past crises [caveat: also significant number of false alarms ( below but close to 30%))] Policy makers preferences can and should be explicitly considered in the evaluation of EWMs (loss function, partial AUROC). A suite of models approach seems necessary (like for monetary policy) The guided discretion principle to trigger policy instruments is recommendable. But importantly: EWMs have the potential to support overcoming the this time is different -syndrom by shifting the burden of proof to authorities with inaction bias (just in case the latter exist ) 21