Session 80 PD, Model Validation Framework and Best Practices. Moderator: Joshua David Dobiac, JD, MS, CAIA

Transcription

1 Session 80 PD, Model Validation Framework and Best Practices Moderator: Joshua David Dobiac, JD, MS, CAIA Presenters: James Stuart McClure, FSA, MAAA Zohair A. Motiwalla, FSA, MAAA SOA Antitrust Disclaimer SOA Presentation Disclaimer

2 Scope of Model Validation Life and Annuity Symposium Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Life Insurance Insurance Company Company and Accordia and Accordia Life and Annuity Life and Company. Annuity Company. 1

3 What are the 6 parts of MV Plan? Output & Reporting Model Plan Mapping Assumptions Model Point Structure Product Features Application Code Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 2

4 What is Model Plan Mapping? Model Plan Mapping is the process of assigning policy records to plans coded in the model. EXAMPLE: In the model you coded up the largest Variable Annuity product with a GMWB rider with sales of over $1 billion. There are a bunch of state variations of the same product. One has different set of WB percentages. (sales of $100M) Another for some marketing requirements. (sales of $50k) In the model, two state variations are mapped to large product A slight difference, WB percent, small difference in results. The marketing requirements have no effect. Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 3

5 Questions & Evidence (Q&E) Model Plan Mapping Key Question 1. Are risk characteristics/product features preserved? Evidence/Documentation/Suggestions 1. Plan Maps organize in a spreadsheet 2. Suggest tracking size of mapped plans, may want to code separate when hit critical size. Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 4

6 What is Model Point Structure? Model Point Structure is the framework, including the methods and tools, used to translate policy data into model records. The popular term is compression. Which is a way of grouping polices with similar characteristics. EXAMPLES: For life insurance they may not code every single possible age. They may do quinquennial ages (every 5 th age). Use average fund balance to combine similar sized policies. Cluster modeling, used in ALFA to combine like polices, to reduce the number of points ran through model. Instead of running 300,000 seriatim polices, may be only 3,000 clusters. Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 5

7 Q&E for Model Point Structure Key Questions 1. How is each model point defined? 2. How are points grouped? Evidence 1. Tools used for record generation. 2. Rationale for structure, document. Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 6

8 Application Code Application Code is the set of operations that define the model calculations. But what is important to test with Application Code? If you are using a purchased projection system, you obviously need to be comfortable with how it performs calculations. The larger risk is the code that is modified or added to fit your needs. Backend spreadsheets should be included in the code review. Replicating the code is pretty key to confirming the code is working correctly. Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 7

9 Q&E Application Code Key Questions 1. Is model code black box, open code, made from scratch? 2. What simplifications/limitations are in the code? 3. Are there back end spreadsheets that are an extension of the model? Why? Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 8

10 Product Features Product Features are fixed characteristics of a product, including guarantees. Review? Compare product coding versus the policy contract. Other sources: the administration system, pricing documents, and marketing materials. Each product feature must be reviewed. Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 9

11 Q&E Product Features Questions 1. Do policy contracts exist for modeled plans? 2. What features are not modeled? Why? Evidence 1. Policy Contracts, Marketing Materials 2. Product documents used for coding administration systems, supplied by Pricing. Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 10

12 Assumptions Assumptions are model inputs estimated or projected based on experience, economic conditions or expert judgment. Review? Evaluated for reasonableness. Reconciled with assumption documentation to ensure correctly coded into the model. Changes to assumptions generally should include sensitivity testing to quantify potential impacts of experience changes. Testing should include stress testing. This type of testing validates model stability. The stresses are far outside what would be expected. Examples: dramatic rise/fall in interest rates, equity markets, and extreme mortality or lapse events. Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 11

13 Q&E Assumptions Questions 1. Are assumptions supported by experience studies? 2. Was Expert Judgment used? 3. Are same assumptions used in all models? Why not? (Pricing, Valuation, Hedging, other) Evidence 1. Assumption inputs & Experience Studies 2. Results from different models implementing same assumption. Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 12

14 Output and Reporting Output and Reporting includes the procedures used to translate, review, and prepare model output as useful business information. This includes using a database, excel or other software. In most cases this will concentrate on backend processes in Excel. Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 13

15 Q&E Output and Reporting Questions 1. Format of model output? Can it feed ledger directly, or need back end spreadsheet? 2. Templates used to organize results? 3. Who receives model deliverable? Evidence 1. SOX binder for Valuation models Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 14

16 What are the 6 parts of MV Plan? Output & Reporting Model Plan Mapping Assumptions Model Point Structure Product Features Application Code Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Insurance Company and Accordia Life and Annuity Company. 15

17 Ready to start a Model Validation? James.McClure@GAFG.com Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Global Atlantic Financial Group (Global Atlantic) is the marketing name for Global Atlantic Financial Group Limited and its subsidiaries, including Forethought Life Life Insurance Insurance Company Company and Accordia and Accordia Life and Annuity Life and Company. Annuity Company.

18 A Model Validation Framework and Best Practices Zohair Motiwalla, FSA, MAAA Principal and Consulting Actuary

19 Key Points U.S. industry is taking model validation and governance seriously Spurred by regulatory scrutiny, product complexity, and valuation changes Establishing a framework for validation and governance requires significant effort Most U.S. companies have now formalized this, but only recently The worth of ongoing good governance is capped at the initial level of model validation Governance and validation should focus on key product risks

20 Common Governance Related Challenges Drive to better understand financials Silo organizational structure Inconsistency in approaches Regulatory challenges Documentation challenges Assumption management Resource issues Model validation Key person risk Model change management

21 Major Findings from Model Validation Exercises Misuse of assumptions, data or results delivered between different functions Key risks ignored or insufficiently modeled Non-modeled business or features Model assumptions or approaches applied outside of actuarial guidelines or recommended practices Unexpectedly material simplifications Errors in back-end processes used in support of the primary projection model Legacy business using assumptions and/or methodology that is different to the rest of the business

22 Why is model validation important? Provides a robust framework to mitigate model risk Ensures that the results of models can be relied upon by its users Helps to mitigate model risk and promote confidence that the results are right Allows for a thorough understanding of strengths, gaps and limitations of existing processes and identifies areas of remediation Adds credibility in the eyes of senior management Meets emerging regulatory requirements Creates a culture of collaboration

23 Industry Benchmarking Based on recent survey results: A formally adopted model governance framework is relatively new (but generally is in place) 3 out of 4 companies have established a governance/risk committee spanning the organization Nearly all companies have attempted to automate their reporting process to varying degrees 2 out of 3 companies set up a formal test environment, change management, distribution process for performing production model updates, and 4 of 5 have designated a model stewardship role Half of companies have independently baselined High and Medium risk models Only a minority of companies have completed a Baseline model validation effort A comprehensive, end-to-end model validation undertaking More effective than an ad hoc review of narrow scope, which can be biased in terms of focus

24 What do we mean by Model Validation? Model Validation Sensitivity Testing Stress testing Ladder/waterfall approach for assumption changes Boundary conditions Controls Review of the model governance process and controls Identification of any manual processes

25 What do we mean by Model Validation? Model Validation Micro Validation Input review (unit testing) Application code review (unit testing) Reporting review Macro Validation Reasonability of Aggregate Results A priori impact, no harm testing Methodology Review Other types of Model Validation Parallel model rebuild Reconciliation to other platforms Assumption review DOCUMENTATION!!!

26 Model Life Cycle Model development Update to model documentation Model validation and creation of methodology/ assumption documentation Model change management

27 Types of Deliverables for the Initial Validation Overview of Models Including: Purpose and historical use Description of key risks, methodology and assumptions Data flow and processes Report: Justification for plan mapping and model compression Review of model inputs for product specifications and assumptions Application code review for intended purposes, including unit testing Appropriate use of reporting output Continued Monitoring: Model change documentation with impacts, peer review, and approvals

28 Post-Baselining Model Validation A formalized process for continuing validation controls after baselining Examples: Separate testing environment for model changes Roll-forward analysis via a waterfall or ladder of results Maintenance of documentation suited to each model aspect Archival of current assumptions Documentation of model change management with levels of governance: Development Technical Review Unit Testing, Aggregate Testing, No Harm Tests Peer Review Model owner and Management sign-off Process is much easier to build upon the baseline deliverables

29 Model Validation - Best Practices Start with models subject to highest risk first The Baseline report should stand on its own without the need to access the models (include work papers where needed) Ensure independence during the review process, but obtain feedback and buy-in on findings from model owners Establish a formalized process for continued governance going forward Identify and cover the gaps between: Initial generation of administration data, assumptions and inputs to the model Model outputs to final delivery of results to stakeholders (e.g. management, accounting) Recommendations for increasing automation and reducing reliance on manual processes Move onto to medium and low risk models Pass on lessons learned to other areas of the organization

30 Case Study Examples Based on actual model validation work for variable annuity business Same general principals apply to other lines of business

31 Test Plan Development Questions Purpose Testing / Process Evidence / Documentation What assumptions are being used? What support exists for each assumption, i.e. experience studies? Do the assumption inputs match the assumptions provided? Confirm the assumptions used by the business unit. Clearly identify support. Confirm that assumptions are correctly input into the model; using assumptions identified in step List all assumptions in the model and document current assumption used based on hierarchy of assumptions below. 1. For each assumption, gathering support from the business unit; including expert judgment. 1. Input review - Comparing assumption inputs to the source from step 1. Document method and any tools used to input assumptions 2. Relativity check - If applicable, using Greg's relativity check tool, we will confirm that inputs satisfy predetermined relativity constraints. 1. Assumption Grid showing all assumptions in the model and source of current assumption (varies by project or projection type, where necessary) 1. Updated Assumption Grid documenting the available support for each assumption. Include references to experience study from which assumption derived in assumption grid, if any. 1. Input checking documentation that includes: A: A list of all relavent values coded to modeled assumptions. B: An indication of which component of the model structure applies to the model assumption (e.g. MG- ALFA plan, Business Unit) C: A coding system for indicating the result of the check (Green - verified, Red - Mismatch, Orange - Not definative or no source) D: Name of checker E: Name of reviewer F: Followup plan for items in Red or Orange G: Check exceptions (spot or relativity checks) 2. Results of relativity checks, if applicable. Designed with sufficient detail to provide marching orders for validation staff

32 Color Coding Examples for Input Checking Color Code Schema Located input in spec, model input matches to spec Unable to definitively locate input in spec, but input appears correct Input not applicable to particular product Unable to definitively locate input in spec, but input appears wrong Located input in spec, model input does not match to spec

33 Product Input Examples Plan Bus GMDB Admin GMDB % Fund % Fund % Fund % Fund % Fund % Fund % Fund Product Description Code Unit Code Code Description Dur1 Dur2 Dur3 Dur4 Dur5 Dur6 Dur7 SL Enterprise 98-5% Growth DB ME98 MIC G FEMLG2 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% MIC G FEMLG1 Death Benefit Option A - 5% % 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% MIC G FUMLG5 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% MIC G FUMLG8 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% Documented as 7, 6, MIC 5, 4, 3, G FUMLG7 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% MIC G FSMLG1 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% 2, 1, 0 by the pricing MIC team. G FUMLG1 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% MIC G FSMLG3 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% MIC G FSMLG2 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00%

34 Scenario Reasonability Checks Models Worksheet Note MG-ALFA VarIndex Return Fund Map Var1Index Var2Index Var3Index Var4Index Var5Index Var6Index Var7Index Indices Nasdaq 100 S&P 500 EAFE Russ mo UST US Inv Grade US High Yld 25% LTCORP 60% LTCORP AAA Mapping AGGR US INTL SMALL MONEY AG43SS.Ain Per AG43 Assumptions Specified -13.5%/4%/5.5%-13.5%/4%/5.5%-13.5%/4%/5.5%-13.5%/4%/5.5% 0%/4.85% 0%/4.85% 0%/4.85% C3P2SS.Ain Per C3P2 Assumptions Specified -20%/3% -20%/3% -20%/3% -20%/3% 0%/4.85% 0%/4.85% 0%/4.85% DAC.Ain IFRS_DAC 400 RW scenarios 6.9% 6.9% 6.9% 6.9% 2.1% 4.7% 5.1% IFRSFV_1.Ain IFRS_FV 1000 RN scenarios 0.0% 0.0% 0.0% 0.0% 2.8% 2.6% 1.1% IFRSFV_2.Ain IFRS_GAAP 1000 RN scenarios 0.0% 0.0% 0.0% 0.0% 2.8% 2.6% 1.1% StatStoch.Ain Stoch 1000 AAA scenarios 8.1% 7.4% 7.7% 7.6% 2.3% 2.6% 2.7% Vacarvm_Tax.Ain Per AG43 Assumptions Specified -13.5%/4%/5.5%-13.5%/4%/5.5%-13.5%/4%/5.5%-13.5%/4%/5.5% 0%/4.85% 0%/4.85% 0%/4.85% GLB_Tax.Ain Per AG43 Assumptions Specified -13.5%/4%/5.5%-13.5%/4%/5.5%-13.5%/4%/5.5%-13.5%/4%/5.5% 0%/4.85% 0%/4.85% 0%/4.85%

35 Scenario Reasonability Checks (Risk Neutral) RN Scenario Martingale Tests Martingale Test Ratio S&P500 Nasdaq Bond Month

36 Scenario Selection for Testing The scenarios used are as follows: Scenario 1: An initial equity drop of 20% followed by an increase of 3% per year each year beginning in year 2 Scenario 2: Equity increases by 5% per year each year Scenario 3: An initial equity jump of 200%, followed by a 66% drop after 5 years, remaining flat thereafter Scenario 4: An initial equity drop of 13.5% followed by an increase of 5.5% per year each year beginning in year 2 Scenario 1 20% Drop and Recovery Clarity of test can be enhanced by removing decrements

37 Macro Checks GLWB Excess Cash Flows (Statutory stochastic model): Delayed impact of stress test.

38 Thank you