Session 40 PD, Model Risk and Governance. Moderator: Zohair A. Motiwalla, FSA, MAAA

Transcription

1 Session 40 PD, Model Risk and Governance Moderator: Zohair A. Motiwalla, FSA, MAAA Presenters: Arnold I. Behrmann, FSA, FIA James M. Hedreen, FSA, MAAA Zohair A. Motiwalla, FSA, MAAA SOA Antitrust Disclaimer SOA Presentation Disclaimer

2 2017 Life & Annuity Symposium Arnold Behrmann Model Governance in Practice May 8, 2017

3 SOCIETY OF ACTUARIES Antitrust Notice for Meetings Active participation in the Society of Actuaries is an important aspect of membership. However, any Society activity that arguably could be perceived as a restraint of trade exposes the SOA and its members to antitrust risk. Accordingly, meeting participants should refrain from any discussion which may provide the basis for an inference that they agreed to take any action relating to prices, services, production, allocation of markets or any other matter having a market effect. These discussions should be avoided both at official SOA meetings and informal gatherings and activities. In addition, meeting participants should be sensitive to other matters that may raise particular antitrust concern: membership restrictions, codes of ethics or other forms of self-regulation, product standardization or certification. The following are guidelines that should be followed at all SOA meetings, informal gatherings and activities: DON T discuss your own, your firm s, or others prices or fees for service, or anything that might affect prices or fees, such as costs, discounts, terms of sale, or profit margins. DON T stay at a meeting where any such price talk occurs. DON T make public announcements or statements about your own or your firm s prices or fees, or those of competitors, at any SOA meeting or activity. DON T talk about what other entities or their members or employees plan to do in particular geographic or product markets or with particular customers. DON T speak or act on behalf of the SOA or any of its committees unless specifically authorized to do so. DO alert SOA staff or legal counsel about any concerns regarding proposed statements to be made by the association on behalf of a committee or section. DO consult with your own legal counsel or the SOA before raising any matter or making any statement that you think may involve competitively sensitive information. DO be alert to improper activities, and don t participate if you think something is improper. If you have specific questions, seek guidance from your own legal counsel or from the SOA s Executive Director or legal counsel. 2

4 Presentation Disclaimer Presentations are intended for educational purposes only and do not replace independent professional judgment. Statements of fact and opinions expressed are those of the participants individually and, unless expressly stated to the contrary, are not the opinion or position of the Society of Actuaries, its cosponsors or its committees. The Society of Actuaries does not endorse or approve, and assumes no responsibility for, the content, accuracy or completeness of the information presented. Attendees should note that the sessions are audio-recorded and may be published in various media, including print, audio and video formats without further notice. 3

5 Model Governance in Practice 4

6 Overview Models Controls Data Reporting Assumptions 5

7 Models: Production Single Model for all purposes (EV, CFT, IFRS, Budget) Difficulties with a Production environment Solution: Use of automation: Starting Model - Production Define changes Assumptions Database version Scenarios In Force file Sensitivities Build and save new model Run the model 6

8 Models: Validation Cycle of model reviews in 3 year cycle Review of product parameters Back testing Given the right inputs, can model reproduce the past Actual compared to Model Static and Dynamic validation Debug output review for changes Sample cell testing Tests both input and process 7

9 Models: Oversight Model Steward Roles Documentation of Models and Changes Validation of changes Ensure Best practices followed Resource allocation Ensure Change control practice Chair User Group Model Oversight Group Set policy Monitor compliance with policy Ensure comprehensiveness (multiple users) 8

10 Models: Change management Maintain strict version control Ensure documentation is full and complete What has changed? Why was the change necessary? Impacts on all affected reporting bases Testing performed Sign-offs from all affected parties Electronic signatures ensure smooth and quick process Move to Production 9

11 Controls Access review read or modify access Inventory control policy count, Face Amount, Account Value Examine trend of EV by plan/ EV by model cell Actual vs. model issue ages within tight range EV Analysis of Change no surprises Scenario rates checking Learn from mistakes: Modify or add new controls 10

12 Data Single source for all data Validation effort only required once Complete and appropriate Common definitions Documentation Source Limitations 11

13 Reporting Standardized Output format Feeds into Reporting system Review of results anticipate questions from Corporate Reasonability checks on results Corporate oversight Comfort Explanations 12

14 Assumptions Process for development of assumptions Evaluation committee to oversee assumption setting and debate changes Formal approval process (with impacts) Central inventory of all assumptions Automated input to models 13

15 Other Aspects Strong partnership with external auditors Strong Systems team to enable robust reporting and automation effort New controls added due to lessons learned Models used for multiple purposes requires additional robustness Structure suited to PBR effort 14

16 MODEL RISK AND GOVERNANCE 2017 SOA Life & Annuity Symposium James Hedreen, CFA, FSA Chief Actuary, Farmers New World Life

17 Background on Farmers New World Life We are the company with the goofy ads Located here in the rainy Pacific Northwest Indirect subsidiary of Zurich Insurance Group Large European insurer with business units in many countries Have reported externally audited MCEV (market consistent embedded value) figures for many years As a result, robust model governance framework in place But..we seek to continuously improve, and we recognize we have room for improvement 2

18 Model Governance Challenges (Random thoughts from the chief actuary s perspective) One model, multiple purposes Backtesting Explanation of movements Controls Model validation nuances Assumption setting Working with external reviewers Governance in a low rate environment 3

19 One Model, Multiple Purposes Your team may want to have multiple copies of models and promise to keep them consistent This may appear to be more straightforward Also gives each practitioner flexibility Do not allow this. It does not work Need one model for each product / line of business, with multiple purposes For our company, multiple purposes means FAS 97 DAC calculations, budget and plan work, statutory cashflow testing, principles based reserves, market consistent embedded value, and pricing / product development Not feasible to keep multiple models consistent and up to date Instead we use one model with overlays (different assumptions, etc.) for various uses of the model Making this work was time consuming and challenging, but well worth the effort 4

20 Backtesting Backtesting means, can your model predict the past? If you start with a historical inforce file and roll it forward, how close do you get to actuals? As part of model governance, need to establish and enforce backtesting parameters and tolerances If your model cannot predict the past, it s not going to predict the future Backtesting can be overdone Necessary but not sufficient Can lead to complacency Won t catch issues at later policy durations if existing inforce has not yet reached those durations 5

21 Explanation of Movements Also known as analysis of change Part of model governance is being able to rigorously decompose movements from one reporting period to the next Expected return Experience variance Yield curve changes Etc. For large model with many model points, analysis of change is complex and challenging Tempting to come up with creative reasons for unexplained variance Leading theory until disproven should always be that an error is present 6

22 Controls At any large company, documentation of control signoff will by necessity be somewhat bureaucratic Changes to control design may also be cumbersome Nonetheless, controls should be useful Make sure you have the right controls that actually provide comfort Avoid check the box type thinking Controls should be constantly updated Errors can highlight shortcomings New technology Changes to product portfolio New risks 7

23 Model Validation Some Nuances Focus on the hard stuff sounds obvious, but tendency is to stress the more easy to check items Inhouse customization to handle unusual product features or innovative actuarial methodology needs to be constantly revalidated. Make sure the validator has excellent understanding of the methodology Why it is needed at all What it is intended to do May not be the case if the validator did not develop the methodology, especially if the developer is no longer with your company 8

24 Assumption Setting Bad assumptions = bad model = bad results Need rigorously controlled, well-documented assumption setting process Also consistent, thoughtful approach to measuring credibility, which will drive how much emerging experience will move your assumptions All assumptions reviewed annually But prioritize and focus on larger blocks Sometimes small changes can drive big changes in results Make sure you have sound basis for change Example: For the first time you now have duration 10 experience on one of your leading products, and lapse rates are much lower than assumed. Be careful before extrapolating that improvement into all successive durations Assumption changes should be directly tied to emerging experience Changes in methodology or approach should not drive material changes in assumptions (and cash flows and profits) Exception: correction of an error 9

25 Working With External Reviewers Ever increasing amount of external (outside of department) model review Examples: Internal audit, external audit, parent company, SOX (Sarbanes Oxley) team, risk, etc. Tempting to view as a painful exercise that must be endured Instead, get as much value as you can these external parties can be a great source of outside-the-box thinking Typically you will be asked a mix of seemingly stupid questions and very insightful questions Be sympathetic to the seemingly stupid questions they are not working with your products and models every day If a question is hard to answer, that probably means it is insightful and you should carefully consider and research Learn from what you never thought to ask 10

26 Governance in a Low Interest Rate Environment It s harder! High discount rates = the out years don t matter much Low rate environment means that cash flows many years in the future can have unexpectedly large effect on present value measures This creates additional model risk Note that later projection years are often dominated by policy years for which your company has no credible experience Judgment is key are margins reasonable? Is assumed policyholder behavior plausible? If your model shows ever increasing margins in later projection years be extremely skeptical 11

27 A Parting Thought 12

28 Model Risk and Governance Zohair Motiwalla, FSA, MAAA Principal and Consulting Actuary

29 Key Points U.S. industry is taking model governance and validation seriously Spurred by regulatory scrutiny, product complexity, and valuation changes Establishing a framework for Model governance requires significant effort Most U.S. companies have now formalized this, but only recently Model validation is an essential component of good governance The worth of ongoing good governance is capped at the initial level of underlying validation performed Model validation is more than just checking the model Use of inputs, assumptions, and back-end processing are all critical components Validation work should be focused on key product risks

30 Common Governance Related Challenges Drive to better understand financials Silo organizational structure Inconsistency in approaches Regulatory challenges Documentation challenges Assumption management Resource issues Model validation Key person risk Model change management

31 Major Findings from Model Governance Exercises Misuse of assumptions, data or results delivered between different areas of the company Key risks ignored or insufficiently modeled Non-modeled business or features Lack of economic rate variation Unexpectedly material simplifications Model assumptions or approaches applied outside of actuarial guidelines or recommended practices Legacy practices or calculations invalidated by new events Errors in back-end processes used in support of the primary projection model

32 Why is model governance and validation important? Provides a robust framework to mitigate model risk Ensures that the results of models can be relied upon by its users Helps to mitigate model risk and promote confidence that the results are right Allows for a thorough understanding of strengths, gaps and limitations of existing processes and identifies areas of remediation Adds credibility in the eyes of senior management Meets emerging regulatory requirements Creates a culture of collaboration

33 Industry Benchmarking Based on recent survey results: A formal adopted framework for model governance is relatively new, but now exists for most companies 3 out of 4 companies have established a governance/risk committee spanning the organization Nearly all companies have automated their reporting process to varying degrees 2 out of 3 companies set up a formal test environment, change management, distribution process for performing production model updates, and 4 of 5 have designated a model stewardship role Half of companies have independently baselined High and Medium risk models

34 Model Governance Life Cycle Supervisory authority and oversight Model life cycle Update to model documentation Model validation and creation of methodology/ assumption documentation Model change management

35 Observations Depending on the company, reviewing the back-end reporting process can be considerably more intensive than the model review Documentation can be very onerous but is necessary to show evidence Start with models subject to highest risk Baselining of all inputs, formulas, and reports is more effective than an ad hoc review on a narrower scope, which can be biased in terms of focus Establish a formalized process for continued governance going forward from the Baseline review Cover the gaps between: Initial generation of administration data & assumptions and inputs to the model Model outputs to final delivery of results to stakeholders (e.g. management, accounting, etc.)

36 What do we mean by Model Validation? Model Validation Sensitivity Testing Stress Testing Ladder/waterfall approach for assumption changes Boundary conditions Other types of Model Validation Parallel model rebuild Reconciliation to other platforms Assumption review

37 What do we mean by Model Validation? Model Validation Micro Validation Input Review Application Code Review Reporting Review Macro Validation Reasonability of Aggregate Results Methodology Review Controls Review of the Model Governance Process and Controls Identification of any Manual Processes

38 Types of Deliverables for Validation Effort Overview of Models Including: Historical use Description of key risks Description of methodology and assumptions Data flow and processes Report: Justification for plan mapping and model compression Review of model inputs for product specifications and assumptions Application code review for intended purposes Appropriate use of reporting output Test sample illustrations Continued Monitoring: Model change documentation with impacts, peer review, and approvals

39 Model Governance After Validation A formalized process for continuing validation controls after baselining Examples: Separate testing environment for model changes Roll-forward analysis via a waterfall or ladder of results Maintenance of documentation suited to each model aspect Archive of current assumptions Model change documentation with levels of peer review & acceptance: Development Technical Review Testing Documentation Peer Review Model owner/management sign-off Process is much easier to build upon the baseline deliverables

40 Best Practices Start with models subject to highest risk Baselining of all inputs, formulas, and reports is more effective than ad hoc review on a narrower scope, which can be biased in terms of focus The Baseline report should stand on its own without the need to access the models (include work papers where needed) Ensure independence during the review process, but obtain buy-in on findings from model owners Establish a formalized process for continued governance going forward Cover the gaps between: Initial generation of administration data & assumptions and inputs to the model Model outputs to final delivery of results to stakeholders (e.g. management, accounting, etc.)

41 Case Study Examples Based on model validation work for a variable annuity writer Same principals apply to other lines of business

42 Test Plan Development Questions Purpose Testing / Process Evidence / Documentation What assumptions are being used? What support exists for each assumption, i.e. experience studies? Do the assumption inputs match the assumptions provided? Confirm the assumptions used by the business unit. Clearly identify support. Confirm that assumptions are correctly input into the model; using assumptions identified in step List all assumptions in the model and document current assumption used based on hierarchy of assumptions below. 1. For each assumption, gathering support from the business unit; including expert judgment. 1. Input review - Comparing assumption inputs to the source from step 1. Document method and any tools used to input assumptions 2. Relativity check - If applicable, using Greg's relativity check tool, we will confirm that inputs satisfy predetermined relativity constraints. 1. Assumption Grid showing all assumptions in the model and source of current assumption (varies by project or projection type, where necessary) 1. Updated Assumption Grid documenting the available support for each assumption. Include references to experience study from which assumption derived in assumption grid, if any. 1. Input checking documentation that includes: A: A list of all relavent values coded to modeled assumptions. B: An indication of which component of the model structure applies to the model assumption (e.g. MG- ALFA plan, Business Unit) C: A coding system for indicating the result of the check (Green - verified, Red - Mismatch, Orange - Not definative or no source) D: Name of checker E: Name of reviewer F: Followup plan for items in Red or Orange G: Check exceptions (spot or relativity checks) 2. Results of relativity checks, if applicable. Designed with sufficient detail to provide marching orders for validation staff

43 Color Coding Examples for Input Checking Color Code Schema Located input in spec, model input matches to spec Unable to definitively locate input in spec, but input appears correct Input not applicable to particular product Unable to definitively locate input in spec, but input appears wrong Located input in spec, model input does not match to spec

44 Product Input Examples Plan Bus GMDB Admin GMDB % Fund % Fund % Fund % Fund % Fund % Fund % Fund Product Description Code Unit Code Code Description Dur1 Dur2 Dur3 Dur4 Dur5 Dur6 Dur7 SL Enterprise 98-5% Growth DB ME98 MIC G FEMLG2 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% MIC G FEMLG1 Death Benefit Option A - 5% % 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% MIC G FUMLG5 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% MIC G FUMLG8 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% Documented as 7, 6, MIC 5, 4, 3, G FUMLG7 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% MIC G FSMLG1 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% 2, 1, 0 by the pricing MIC team. G FUMLG1 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% MIC G FSMLG3 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00% MIC G FSMLG2 5% Growth 7.00% 7.00% 6.00% 6.00% 5.00% 4.00% 2.00%

45 Scenario Reasonability Checks Models Worksheet Note MG-ALFA VarIndex Return Fund Map Var1Index Var2Index Var3Index Var4Index Var5Index Var6Index Var7Index Indices Nasdaq 100 S&P 500 EAFE Russ mo UST US Inv Grade US High Yld 25% LTCORP 60% LTCORP AAA Mapping AGGR US INTL SMALL MONEY AG43SS.Ain Per AG43 Assumptions Specified -13.5%/4%/5.5%-13.5%/4%/5.5%-13.5%/4%/5.5%-13.5%/4%/5.5% 0%/4.85% 0%/4.85% 0%/4.85% C3P2SS.Ain Per C3P2 Assumptions Specified -20%/3% -20%/3% -20%/3% -20%/3% 0%/4.85% 0%/4.85% 0%/4.85% DAC.Ain IFRS_DAC 400 RW scenarios 6.9% 6.9% 6.9% 6.9% 2.1% 4.7% 5.1% IFRSFV_1.Ain IFRS_FV 1000 RN scenarios 0.0% 0.0% 0.0% 0.0% 2.8% 2.6% 1.1% IFRSFV_2.Ain IFRS_GAAP 1000 RN scenarios 0.0% 0.0% 0.0% 0.0% 2.8% 2.6% 1.1% StatStoch.Ain Stoch 1000 AAA scenarios 8.1% 7.4% 7.7% 7.6% 2.3% 2.6% 2.7% Vacarvm_Tax.Ain Per AG43 Assumptions Specified -13.5%/4%/5.5%-13.5%/4%/5.5%-13.5%/4%/5.5%-13.5%/4%/5.5% 0%/4.85% 0%/4.85% 0%/4.85% GLB_Tax.Ain Per AG43 Assumptions Specified -13.5%/4%/5.5%-13.5%/4%/5.5%-13.5%/4%/5.5%-13.5%/4%/5.5% 0%/4.85% 0%/4.85% 0%/4.85%

46 Scenario Reasonability Checks (Risk Neutral) RN Scenario Martingale Tests Martingale Test Ratio S&P500 Nasdaq Bond Month

47 Scenario Selection for Testing The scenarios used are as follows: Scenario 1: An initial equity drop of 20% followed by an increase of 3% per year each year beginning in year 2 Scenario 2: Equity increases by 5% per year each year Scenario 3: An initial equity jump of 200%, followed by a 66% drop after 5 years, remaining flat thereafter Scenario 4: An initial equity drop of 13.5% followed by an increase of 5.5% per year each year beginning in year 2 Scenario 1 20% Drop and Recovery Clarity of test can be enhanced by removing decrements

48 Macro Checks GMDB Excess Cash Flows (Statutory stochastic model): Delayed impact of stress test.

49 Sensitivity Testing of Results Sensitivity Testing of Results Present Values at 5%, Relative to Baseline Assumption Set Amounts in Thousands of USD Lapse Mortality Partial Withdrawals Statutory Reporting Item Down 10% Up 10% Down 10% Up 10% Down 10% Up 10% Total Income 213 (255) 67 (65) 3 (3) Net Surrender Benefits (2,012) 1,892 (633) 482 (28) 22 Annuitization Benefit 188 (132) 59 (34) 3 (2) Partial Withdrawals 679 (742) 214 (189) 10 (9) Retained Death Claims 657 (612) 207 (156) 9 (7) Increase in Reserves 21 (15) 7 (4) 0 (0) Net transfers to separate account 266 (260) 84 (66) 4 (3) Other Benefits and Expenses 439 (410) 138 (105) 6 (5) Total Benefit and Expenses 238 (279) 75 (88) 24 (28) Net Pre-tax Profits (25) 24 (8) 8 (2) 2 Symmetrical results are not necessarily correct (e.g., excess partial w/d penalties would not be symmetrical to profits)

50 Thank you