Prediction errors in credit loss forecasting models based on macroeconomic data Eric McVittie Experian Decision Analytics Credit Scoring & Credit Control XIII August 2013 University of Edinburgh Business School 2013 Experian Ltd. All rights reserved. No part of this copyrighted work may be reproduced, modified, or distributed in any form or manner without the prior written permission of Experian.
Abstract Loss forecasting models including aggregated economic variables may generate substantial and persistent forecast biases when fitted on limited historical data. Recent evidence from the UK, for example, suggests a bias towards over-prediction of credit losses from models estimated on short historical data periods including the 2008-2009 recession, when such models are applied to more recent economic conditions. This paper considers possible explanations for this pattern, and the potential for alternative approaches that are less prone to forecast errors of this type. 2
2006Q1 2006Q3 2007Q1 2007Q3 2008Q1 2008Q3 2009Q1 2009Q3 2010Q1 2010Q3 2011Q1 2011Q3 Prediction errors in macroeconomic credit loss forecasting models Outline Evidence of recent deterioration in forecast accuracy for portfolio loss / pd models including macroeconomic variables: Market, bureau & client data Variety of modelling methodologies Here concentrate on: Market card delinquency data Unemployment measures Look at some candidate explanations. Illustrate some issues around use of time series methods and aggregated economic variables in loss models. Contents: An evaluation framework borrowed from Hendry (1995) Dynamic Econometrics, OUP Some evidence lots of pictures Many questions Very few firm answers Some tentative suggestions Actual Predicted 3
Model Building & Forecasting Data Generation Process (DGP) Behaviour Not known or directly observed Measurement Robust forecasts require models that replicate key features of the (unknown) DGP Information is inevitably lost in moving from the DGP via observed data to empirical models Models provide better representations of the DGP the less information they lose and the more they retain Model design criteria should select models which minimize information loss Observed Data Empirical Models Predictions / Forecasts 4
Model Building & Forecasting: Information Losses Economic activity Measurement System Observed Data 1 Parameters of Interest Forecast Assumptions Data Transformations & Aggregation Data Partition Marginalization Lag Truncation Functional Form Derived Empirical Models Forecasts 2 3 4 5 6 Information losses 5
Model Building & Forecasting: Design Criteria Domain Criteria / Objectives Alternatives Measurement Data Accuracy Volatile, imprecise or Inaccurate data Past / insample Present Homoskedastic, innovation errors Weakly exogenous conditional variables for parameters of interest Residual autocorrelation; heteroskedasticity Invalid conditioning Future / Out of sample Rival models Constant, invariant parameters of interest Theory consistent / structural relationships Encompasses rival models / Forecasts dominate rival models Parameter non-constancy; predictive failure Implausible coefficients Relative poor fit / significant additional variables / alternative or additional variables valuable in forecasting / relatively poor forecasts Design criteria require minimum information losses at all stages of reduction relative to potential rival models. Generally highly demanding even when data availability is good. 6
Forecast errors: Hypotheses (1) The models were never right Invalid / excessive marginalization Variable Selection: Wrong drivers Missing variables? Economic drivers Quality of stock, lender behaviour Interactions Functional misspecification (2) The models were right but the world has changed Model parameters not constant / structural Fundamental regime shift resulting in permanent changes to relationships between economic factors and credit losses Continuously evolving relationships between economic factors and credit losses, resulting in gradual deterioration of calibration of models estimated on historical data. Some variants of (2) are observationally equivalent to some variants of (1) e.g. apparent regime shifts (in terms of observed variables) may be due to changes in unobserved (missing) variables. 7
Thinking about economic risk: Variable Selection Search space is large even if restricted to macroeconomic (aggregate) variables: Many potentially-relevant variables Functional forms? Lag structures? What about disaggregated data (e.g. for geographical areas)? Finding good models hampered by poor data availability: Limited historical time series for consistent credit loss data Missing variables related to credit quality, lending practices, etc. Weak tests for valid reductions. Grid Search / Data Mining Statistical Data Reduction Methods Theory, priors & expert judgement Macroeconomic factors proxy more direct influences on borrowers proxy quality varies. High risk of building models based on spurious correlations. Aggregated time series approaches lead to large information losses especially given short historical time series and force additional losses due to excess marginalization and lag truncation Data limitations weaken tests to enforce design criteria even if attempts are made to apply them rigorously. 8
Case Study: UK Credit Card delinquency & aggregate unemployment Look at models of UK market delinquency rate measure (derived from BoE data) over period 1992-2011 Focus on unemployment-related variables : In naive models including only unemployment measures Conditioned on other candidate drivers of card delinquency How do models perform in most recent time periods? Stability of coefficient estimates over sub-periods of the full sample Advantages of modelling market data: Longer (apparently) consistent history Better matching of borrower population to macroeconomic aggregates? (although even here population for macro variables may not match borrower population) I can show you the results! Disadvantages: More missing variable issues not possible to condition on quality of stock or allow for changes in lender behaviour Rules out anything other than simple time series modelling approaches Analysis and fitted models are purely illustrative useful case study to consider some of the issues involved linking credit outcomes to economic variables as basis for loss forecasting. 9
Case Study: UK Credit Card delinquency & aggregate unemployment 3 2 1 0-1 -2 Normalized series - levels 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 Covariance Analysis Card Delinquency Rate ILO Unemployment Rate Covariance -0.375 Correlation -0.242 t-statistic -2.298 Probability 0.024 Claimant Rate Covariance -0.808 Correlation -0.400 t-statistic -4.020 Probability 0.000 Credit Card Delinquency Rate ILO Unemployment Rate Claimant Unemployment Rate 10
Moving Correlation Coefficient Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment Moving correlations between ILO rate & card delinquency rate 1 0.5 0-0.5-1 5 year 7 year 10 year 15 year Full Sample Card Delinquency Rate ILO Unemployment Rate Covariance -0.375 Correlation -0.242 t-statistic -2.298 Probability 0.024 Claimant Rate Covariance -0.808 Correlation -0.400 t-statistic -4.020 Probability 0.000-1.5 1997Q1 2000Q1 2003Q1 2006Q1 2009Q1 End of rolling sample period 11
Case Study: UK Credit Card delinquency & aggregate unemployment 4 3 2 1 0-1 -2-3 Normalized series delinquency rate levels, quarterly changes in unemployment rates (smoothed) 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 Covariance Analysis Card Delinquency Rate Quarterly Change: ILO Rate Covariance 0.136 Correlation 0.755 t-statistic 10.545 Probability 0.000 Quarterlly Change: Claimant Rate Covariance 0.145 Correlation 0.676 t-statistic 8.417 Probability 0.000 Credit Card Delinquency Rate Quarterly Change in ILO Rate (Smoothed) Quarterly Change in Claimant Rate (Smoothed) 12
Moving Correlation Coefficient Prediction errors in macroeconomic credit loss forecasting models Credit Card Delinquency & Unemployment Moving correlations between quarterly changes in ILO rate & card delinquency rate 1 0.8 0.6 0.4 0.2 5 year 7 year 10 year 15 year Full Sample Card Delinquency Rate Quarterly Change: ILO Rate Covariance 0.136 Correlation 0.755 t-statistic 10.545 Probability 0.000 Quarterlly Change: Claimant Rate Covariance 0.145 Correlation 0.676 t-statistic 8.417 Probability 0.000 0-0.2 1997Q1 2000Q1 2003Q1 2006Q1 2009Q1 End of rolling sample period 13
Extended model of credit card delinquency rate, includes: Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment Quarterly change in ILO rate Annual growth of household real disposable income Quarterly change in interest rates What can we learn from residuals? E.g. Tendency to underpredict delinquency rate in post-recession periods (after 1992 and 2009)? 1.2 0.8 0.4 0.0-0.4-0.8-1.2 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 Residual Actual Fitted Strong autocorrelation in model residuals could be addressed using alternative specification in full sample but may reflect more fundamental specification issues (wrong or missing variables). 6 5 4 3 2 1 14
Case Study: UK Credit Card delinquency & aggregate unemployment 4.5 4.0 3.5 Recursive regression coefficients (progressively extending estimation sample to include later dates) Constant 6 5 4 3 Change in ILO Rate 3.0 2 2.5 1 0 2.0 92 94 96 98 00 02 04 06 08 10-1 92 94 96 98 00 02 04 06 08 10 Recursive C(1) Estimates ± 2 S.E. Recursive C(2) Estimates ± 2 S.E..3.2 Income Growth 1.2 0.8 Interest rate.1 0.4.0 -.1 0.0 -.2 92 94 96 98 00 02 04 06 08 10-0.4 92 94 96 98 00 02 04 06 08 10 Recursive C(3) Estimates ± 2 S.E. Recursive C(4) Estimates ± 2 S.E. 15
Moving Regression Coefficient Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment Moving regression coefficients between changes in ILO rate & card delinquency rate 3 2.5 2 1.5 1 0.5 0 5 year 7 year 10 year Full Sample Coefficient 2.737 Std. Error 0.359 t-statistic 7.621 Prob. 0.000-0.5-1 1997Q1 2000Q1 2003Q1 2006Q1 2009Q1 End of rolling sample period 16
Case Study: UK Credit Card delinquency & aggregate unemployment Predictive accuracy for model estimated on sub-sample: 2003Q1 2009Q4 6 5 4 3 2 1 0-1 2004 2005 2006 2007 2008 2009 2010 2011 Actual Predicted Residual 17
Forecast Error for CCD Delinq Rate Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment Predictive accuracy for model estimated on alternative 10 year sub-samples 1 0.5 0-0.5-1 2009 2010 2011-1.5-2 all 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Model estimated on 10 year sample ending: 18
Some conclusions on aggregate analysis: Estimated relationships between credit outcomes and unemployment (and other factors) vary greatly depending on the estimation sample Even full-sample models tend to over-estimate losses in recent history (whether with or without other economic conditioning variables) Autocorrelation issues in full sample model could be resolved using more sophisticated estimation procedures in the full data sample, but may indicate more basic problems with model specification. Alternative estimation procedures are generally not viable in samples available in practice. Over-estimation problem worse for models fitted on more recent history Changes in borrower population and/or lender behaviour probably play some role Not possible to control for these in aggregated models estimated over longer history Similar results in other applications using macroeconomic data: Prediction errors in macroeconomic credit loss forecasting models Case Study: UK Credit Card delinquency & aggregate unemployment Loss models fitted on aggregated or account-level lender data PD models fitted on bureau data Argues against explanations based on model specification Problem is (partially) mitigated in models using disaggregated economic data e.g. unemployment by local area or by age 19
Conclusions Good forecasting models replicate key features of data generation process with minimal information losses Criteria for effective models are challenging even with abundant data Aggregate loss forecasting modelling approaches emphasizing time series movements in economic data suffer from: Excess / invalid aggregation of economic (and possibly credit) data Excess / invalid marginalization: Difficult to effectively control for changes in borrower characteristics, lender & borrower behaviour Limited degrees of freedom forces concentration on small set of economic metrics that dominate time series movements in credit losses Limited scope for testing against design criteria weak tests for innovation errors (including stationarity), exogeneity, parameter constancy, etc. Disaggregated approaches (for both credit and economic data) reduce information losses resulting in more stable, robust forecast models. 20
21