Back Testing ALM Models April 17, Back Testing ALM Models: Concepts, Practice, and Compliant Business Solutions

Transcription

1 Back Testing ALM Models: Concepts, Practice, and Compliant Business Solutions Presented by: William J. McGuire Chairman Emeritus McGuire Performance Solutions, Inc N. Scottsdale Rd, Ste 290 Scottsdale, AZ slide 1 Poll Question Please select your current level of experience with today s topic: 1

2 Back Testing ALM Models: Concepts, Practice, and Compliant Business Solutions Presented by: William J. McGuire Chairman Emeritus McGuire Performance Solutions, Inc N. Scottsdale Rd, Ste 290 Scottsdale, AZ slide 3 Goals of the Session Review fundamental concepts and methodologies relating to establishing NII IRR model and behavior input accuracy Examine multiple approaches to confirming the consistency of model outcomes and inputs with subsequent actual values Explore a way to obtain quantified dynamic guidance of changes in balance sheet behaviors when interest rates rise Provide a conduit for questions and comments slide 4 2

3 Disclaimers Information provided in this presentation is the professional opinion of MPS, based on personal and client experience Information provided in this presentation is not represented as official regulatory guidance from any agency slide 5 A Refocus on Balance Sheet Management Controlling risk, while creating adequate earnings and capital, now requires a refocus on balance sheet management IRR modeling solutions include traditional and new elements A verified and validated IRR model and strong model governance Compliance analysis and decisions based on model forecasts Alternate outcomes analyses to explore ranges of potential risk IRR models now expected to assess interest rate related risk accurately, requiring that the model and its behavior inputs are correct - not conservative or otherwise intentionally biased slide 6 3

4 Accurate IRR Modeling: New Mandates Previous conservative IRR models and their behavior inputs were easy to define, with inputs often qualitatively-based IRR models are now required to be accurate Independent third party verification of model technical structure and validation of forecasted outcomes is a clear mandate Model behavior input accuracy must also now be proven Behavior inputs must normally be reasonable relative to peers Comparisons of forecasted as modeled behavior inputs versus actual outcomes in subsequent periods must be very close slide 7 Benchmarking versus Back Testing Benchmarking establishes the general reasonableness of IRR model outcomes and underlying behavior inputs Compares IRR model outputs to simple comparatives and inputs to alternatives available from recognized behavior input sources Goal is to ensure that values fall within a logically expected band Back Testing establishes the specific accuracy of IRR model outcomes and behavior inputs Compares model inputs to recent actual institution experience Goal is to prove that values closely compare to actual outcomes slide 8 4

5 Benchmark Earnings at Risk Profiles Peer NII IRR profile insights are obtained from validations of well constructed models $ Common Current NII IRR Profiles Most institutions have direct NII IRR profiles (upward sloping) due to high levels of asset side liquidity holdings and large core deposit bases NII will often decline in extreme high rate scenarios due to options, which alter first year repricing magnitudes Base Case + Interest Rates Peer data is not always a strong benchmark as model quality varies slide 9 Benchmark Earnings at Risk Profiles Peer EVE/NEV IRR profile insights are taken from validations of well constructed models $ Common Current EVE/NEV IRR Profiles Most institutions have direct equity at risk profiles (upward sloping) due to high levels of shorter term assets and large core deposit bases EVE/NEV will very often decline in extreme rate high scenarios due to options, which alter long term cash flow positioning Base Case + Interest Rates Peer data is not always a strong benchmark as model quality varies slide 10 5

6 Benchmark Behavior Inputs: Peer Data Some industry data available, mainly mortgage prepayments Mortgage loan prepayment rates found on multiple web sites National Mortgage Market information from Securities Industry and Financial Markets Association (SIFMA) is an example Index and analog inputs use institution statistical forecast data to create low cost IRR model behavior input benchmarks Core Deposit Index and Loan Index describe selected categories at a national level of aggregation Analog Core Deposit or Loan Input Reports tune national data to match institution size, location, footprint type, and charter slide 11 Alternate IRR Rate Test Benchmarks Non-Linear Ramp Yield Curve Shape Change +100 bp +100 bp/+10 bp +400 bp/+40 bp UST 1M UST 3M UST 6M UST 1Y UST 2Y UST 3Y UST 5Y UST 7Y UST 10Y Fed Funds Prime Mort Mort ARM Auto SmartRamp relations are ramp or shock projections of financial sector history Short rates move by more than long rates for any given amount of Fed action to 3 month rate Yield curve shape change relations can surprise: Short rates move by more than long rates but relative magnitudes are very large if major flattening of the curve takes place slide 12 6

7 Back Testing Approaches Back tests assess the accuracy of prior NII model forecasts (and underlying behavior inputs) used in financial decisions Model risk occurs if forecasts are inaccurate => poor decisions Short term and (usually annual) formal back tests required Retrocast tests assess NII model forecast and behavior input accuracy as if future conditions had been known at each point along the prior forecast They are reforecasts using actual interest rates, pricing, etc. The innate accuracy of underlying projected values is tested slide 13 NII IRR Model Forecast Back Testing NII IRR model back testing compares a prior earnings at risk forecast to subsequent actual outcomes Normally only very short term (e.g. 3 month) comparisons of forecast actuals are assessed, to control background noise NII IRR model first month margin review is used to affirm the near term accuracy of model forecasts Compares first month model forecasts to actual yields and costs Variance limits are often established in Policy for NII summary level back tests and first month margin reviews slide 14 7

8 Example NII Forecast Back Test 09/30/12 12/31/12 Variance Variance Ref Actual Forecast Actual (dollars) (percent) Total Interest Income 71,394 70,936 71,950 - (1,014) -1.41% Total Interest Expense 12,619 12,025 11, % Net Interest Income 58,775 58,911 59,969 - (1,058) -1.76% Total Interest Income Total Interest Expense Net Interest Income 80,000 14,000 70,000 70,000 60,000 50,000 40,000 30,000 20,000 10,000 12,000 10,000 8,000 6,000 4,000 2,000 60,000 50,000 40,000 30,000 20,000 10, Ref Actual Forecast Actual Ref Actual Forecast Actual Ref Actual Forecast Actual Source: MPS Back Test Analysis Figures in Thousands 3-Jun-13 slide 15 NII Forecast Back Test: Drill Down 09/30/12 12/31/12 Variance Variance Ref Actual Forecast Actual (dollars) (percentage) Interest Income Overnight Funds (0) -0.01% Total Investments 22,714 22,713 22, % Total Loans 48,589 48,153 49,405 - (1,252) -2.53% Total Interest Income 71,394 70,936 71,950 - (1,014) -1.41% Interest Expense Deposits 4,402 3,925 3, % Borrowings 8,217 8,101 8,110 - (9) -0.11% Total Interest Expense 12,619 12,025 11, % Net Interest Income 58,775 58,911 59,969 - (1,058) -1.76% Source: MPS Back Test Analysis Figures Thousands 4-Jun-13 in Back test chart of accounts can be kept simple to focus analysis on key contributors to NII variances Drill down to the underlying yield and cost, and balance sheet variance sources Rate, Volume, Mix Analysis Yield and Cost Variance Data Balance Sheet Variance Data Interest Rate Variance Data slide 16 8

9 First Month Margin Review Category Margin Report Model Margin Variance 03/31/12 04/30/12 Federal Reserve Stock (0.10) Comments FHLB Stock Significant variance Agency Bonds (0.04) MBS Bonds (2.15) Significant variance Ag Real Estate Variable (0.06) Ag Real Estate Fixed (0.15) Total Ag Real Estate (0.07) Ready Reserve-Consumer Review suggested Consumer Installment (0.01) Consumer Other (0.15) Total Consumer Loans Earning Assets (0.21) Aggregated Category Statement Savings - Regular Savings - Gold Money Market (0.00) Total Savings (0.00) Christmas Club (1.55) Significant variance Brokered CD's (0.02) CDARS Review suggested Interest Bearing Liabilities (0.02) Net Interest Margin (0.19) Aggregated Category Model margin benchmarking is a simple first month forecast comparison The controlled environment tests pricing spreads and related inputs A type of back test, but here forward-looking slide 17 Notes: NII IRR Model Back Testing Model back testing can have value in selected situations New model installation or in-house developed model testing Reality check on the general accuracy of model NII forecasts Back testing is a blunt instrument for model risk assessment Outcomes are often sensitive to test design and time period Back tests really only confirm that the basic model math works Preferred approach is to assess summary and category level model forecasts across scenarios to confirm that outcomes accurately depict underlying contractual specifications slide 18 9

10 Behavior Inputs Back Testing The time period of the assessment is a crucial specification Early alert back tests provide quarterly reviews of variances Annual back testing is standard practice for formal reporting Retrocasts are produced annually or on an as needed basis Special model governance upgrades may be required Specify types/timing of behavior input accuracy tests in Policy Variance limits (dollar or percentage) can be established for early alert and formal back tests, but they are not common slide 19 Loan Prepayment Back Test Mechanics Back test requires construction of an actual outcomes history Obtain loan level balances and maturity dates (at a minimum) and stratify data into appropriate loan categories In each month, identify any loans that disappeared from the prior month s list prior to their maturity date Sum the prepaid balances to create dollar outcomes and plus divide by average balance to produce a prepayment rate Back tests of indeterminate maturity loan payoff behaviors are similar to those for core deposit decay/retention slide 20 10

11 Early Alert: Loan Prepayments 17.50% 17.00% Actual Prepayment Rate/Base Case Forecast Prepayment Rate 17.50% 17.00% Ongoing monitoring of behaviors is needed to maintain confidence 16.50% 16.00% 15.50% 16.50% 16.00% 15.50% Compare forecasted to actual monthly values and produce a quarterly review by category 15.00% 01/13 02/13 03/ % Monitor levels and trends and understand patterns Month-to-month or seasonal patterns are often main variance drivers Sudden large mismatch may signify a change in underlying conditions slide 21 1,000, , , ,000 New Auto - Dealer Back Test: Loan Prepayments Green bars are actual prepayments and red line is the IRR model input 200, /12 02/12 03/12 04/12 05/12 06/12 07/12 08/12 09/12 10/12 11/12 12/12 T o tal Me dian Variability is expected but consistent mismatches identify an input problem Average 0.00 New Auto - Dealer 0.00 New Auto - Dealer 0.00 New Auto - Dealer Actual Base Case Actual Base Case Actual Base Case slide 22 11

12 Core Deposit Decay Back Test Concepts Compares a core deposit retention forecast at a prior point in time against subsequent actual retention to back test decay Prior Forecasted Retention and Implied Decay Close matches prove decay input accuracy Time Subsequent Actual Retention and Implied Decay Starting Balance Retention Defined by Decay Input Actual Retention Time slide 23 Core Deposit Decay Back Test Mechanics Back test requires construction of an actual retention history Obtain account level balances and rates paid (at a minimum) and stratify data into appropriate deposit categories In each month, sum the balances of accounts that remain open from the fixed pool of accounts defined in the initial month Divide each month s aggregated balance by the balance from the initial month to create a retention ratio (and implied decay) Retention ratio is a common denominator for comparing back tests Back tests of core deposit repricing and supply behaviors can also be produced and these often have value slide 24 12

13 Early Alert: Core Deposit Retention/Decay Total Core Deposits: Forecasts versus Actual Retention Ratios Constant monitoring of retention is needed to maintain confidence in EVE/NEV IRR analyses Jun-07 Jul-07 Aug-07 Sep-07 Actual Retention MPS: Base Constrained Actual and Forecasted Retention Ratio Variances MPS Base Case Forecast-Constraine Jul-07Act - Fcst Aug-07Act - Fcst Sep-07Act - Fcst IB CHECKING (0.0130) G G G SAVINGS (0.0174) G (0.0116) G (0.0783) Y Caution LOW TIER MMDA G G G HIGH TIERMMDA G G G OVERALL (0.0125) G (0.0077) G (0.0190) G Compare forecasted to actual values monthly and produce a quarterly review of variances Seasonal patterns can be variance drivers if not quantified in inputs slide 25 Back Test: Core Deposit Retention/Decay Forecast Date First Forecast Month Changes in the retention ratio over time illustrate the magnitude of decay Yellow bars are actual retention ratio outcomes and red line is as forecasted Variability is expected but consistent mismatch indicates an input problem slide 26 13

14 Core Deposit Back Testing: Advanced Exhibit BT-S Back Test of MPS Deposit Retention Forecasts: All Categories Sample Bank B ($450 Million Assets) 10/31/11-10/31/12 Ba se Case Forecasts versus Actual Retention Ratios Oct-11 Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12 May-12 Jun-12 Jul-12 Aug-12 Sep-12 Oct-12 Actual Retention MPS: Base Case FDICIA 305 Inputs DN Root 100 bp Mean Forecast Square versus Error Actual Statistics: Retention All Scenarios Ratios Oct-11 Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12 May-12 Jun-12 Jul-12 Aug-12 Sep-12 Oct Actual Best Forecast: RetentionMPS Base MPS: Case Base Case OCC: Optimistic FDICIA OCC: 305 Inputs Pessimistic Source: MPS Analyses of Sample Bank B Core Deposit Data The upper display illustrates a standard retention back test Root Mean Square Error data in lower display quantifies the relative size of the variances In each month, square the forecast actuals variance. Sum all squared variances and take the square root RMSE values are directly comparable. Adding alternate forecast values is optional but it can make a point slide 27 Back Tests: CD Early Withdrawals 400, Month CD Early Withdrawals , , , /04 01/05 04/05 07/05 10/05 01/06 04/06 07/06 10/06 01/07 04/07 07/07 10/07 01/08 04/08 07/08 10/08 01/09 04/ Early Withdrawals Spread: New Volume Rate - Coupon Rate Meaningful back tests of recent CD early withdrawals are not feasible See: Managing Interest Rate Risk: The IRR Implications of CD Early Withdrawals and Core Deposit Surge Balances - FMS White Paper 2013 slide 28 14

15 Retrocast Test Concepts and Mechanics Retrocast testing requires construction of alternate forecasts based on an equations based behavior input source Beginning in month 1 of the retrocast period, enter current period values for all independent variables (e.g. interest rates, rates paid or new volume rates, etc.) Recalculate the input forecast, now based on current period data Compare retrocast input values to actuals to assess accuracy Retrocast tests eliminate some of the background noise found in back tests, and as such provide a very strong innate test of absolute behavior input forecast accuracy slide 29 Retrocast Test: Loan Prepayments Forecast Date First Retrocast Month IRR Model Input Actual Prepayment Rate / Base Case Retrocast Prepayment Rate /12 04/12 05/12 06/12 07/12 08/12 09/12 10/12 11/12 12/12 01/13 02/13 03/ Retrocast forecasts incorporate exact refinance incentives, etc. by month and normally produce closer approximations to actuals slide 30 15

16 Retrocasts as Quantified Dynamic Guidance Retrocasts produce a continually evolving representation of expected input behaviors if history were repeating itself Consistency of retrocast values and actuals confirms that current conditions are generally following historic precedent A marked change in retrocast value actual variance or a steady trend of widening variances indicate a change in conditions The quantified dynamic guidance provided to managers indicates whether the future is or is not replicating history; reaction needed? Retrocast testing can rapidly identify the New Normal if such occurs when interest rates rise, plus provides other insights slide 31 Quantified Dynamic Guidance: Example 1 Actual Outcome Retrocast Value As interest rates rise, actual repricing lags original projection Retrocast supply forecasts are below original forecast But actual supply outcomes are above retrocast values Supply is shown to be rich to history slide 32 16

17 Quantified Dynamic Guidance: Example 2 Actual Outcome Retrocast Value As interest rates rise, actual repricing is pushed above the original projection Retrocast supply outcomes are below original forecast But actual supply outcomes are below retrocast values Time for Plan B? slide 33 Types of Quantified Dynamic Guidance Retrocast-based quantified dynamic guidance has its highest value where model risk is potentially highest Core deposit supply and repricing as interest expense drivers Core deposit retention/decay as EVE/NEV hedging determinants Quantified dynamic guidance can provide insights into loan prepayment and paydown behaviors as well, and CD early withdrawals if they become an IRR factor Understanding evolving balance sheet dynamics will be vital to performance and risk control when interest rates rise slide 34 17

18 Summary Guide to IRR Model Back Testing Multiple dimensions of NII IRR model outcomes and behavior inputs need have their accuracy formally conformed NII IRR model outcomes are validated by: Summary short term back tests of NII outcomes versus actuals Detailed monthly model margin comparisons versus actuals IRR model behavior inputs are validated by: Early alert quarterly back tests of forecast versus actuals Formal (at least) annual back tests of forecast versus actuals Periodic assessments of retrocast projections versus actuals slide 35 Contact Points and Key Services Jeff Wildenthaler President Katerina Steinberg Vice President, Sales Trina LaRocco Vice President, Sales McGuire Performance Solutions, Inc N. Scottsdale Rd., Suite 290 Scottsdale, AZ (tel) (fax) ALM model and specialized financial model verification and validation Advanced statistical analyses of core deposit supply, repricing, and runoff (decay) behaviors and loan prepayments/paydowns Liquidity/Contingency Funding Plan and ALLL function verification and validation ASC 820 (FAS 157) compliant CDI, CD, and loan valuations for fair value applications Goodwill and CDI impairment testing Statistically based loan probability of default analyses for fair value applications and ALLL model validation slide 36 18

19 Please take a moment to complete the brief evaluation. We appreciate your feedback. Upon completion of the evaluation, you will be provided with a link to the CPE form. 19