Model Risk Management. Henry Essert Gaurav Upadhya

Size: px

Start display at page:

Download "Model Risk Management. Henry Essert Gaurav Upadhya"

Victor Kelly
5 years ago
Views:

1 Model Risk Management Henry Essert Gaurav Upadhya

2 Model Risk Management Trends and Practices Henry Essert

3 History and Evolution Ad-Hoc Pre-2008 Reactive Proactive Now Productive TBD Modeling and testing of models have long been part of insurers operations. Peer reviews were common practice for some types of models. However, practices were widely dispersed within and amongst companies. Governance and independence was less developed. Beginnings of a more organized approach: US federal regulatory emphasis (SR 11-7); ORSA Guidance Manual European Solvency II and internal models. Insurers start to define models and develop inventory Insurance company senior management and boards have continued driving further developments. Model Risk Management Forum created in June 2015 Comprised of 14 leading North American insurers Actively developing improved practices November 2016 model risk scoring survey More emphasis on model stream and connectivity Viewing model risk management as a mechanism to ensure that models are performing at their best and generating measurable business value. Model Risk Management PwC 3

4 MRM Survey 2017 MRM programs by calendar year commenced 52 of 58 respondents that have, or plan to have, a MRM program in place 25% 23% 20% 15% 10% 5% 2% 8% 4% 12% 10% 6% 15% 15% 4% 2% 0% Model Risk Management PwC 4

5 What is driving the acceleration in MRM activities? 1. Insurers model validations are finding, and correcting, calculation errors 2. Process and control inadequacies and inefficiencies are being uncovered and resolved 3. Boards and senior management want better MRM: 79% of surveyed directors agree that more work needs to be done to validate their insurer s models and keep them informed on the validation process 86% of directors say more needs to be done to disclose models strengths and limitations 93% of CROs say more effort is needed to communicate how models are linked and used in important decisions 1. Regulators and rating agencies expect MRM programs to be in place: The Summary Report should demonstrate the insurer s process for model validation, including factors considered and model calibration. ORSA Guidance Manual, NAIC The "testing and validation" category addresses the quality of an insurer's approach to testing and validating all aspects of its ECM, including the insurer's methodologies, interpretation of data and ECM outputs into its ERM program A New Level Of Enterprise Risk Management Analysis: Methodology For Assessing Insurers' Economic Capital Models, S&P Global Ratings Model Risk Management PwC 5

6 MRM Survey 2017 Influencers in encouraging and maintaining your MRM program 33 of 58 respondents with a MRM program in place 70% 60% 50% 40% 30% 20% 10% 0% 39% 48% 61% 27% 9% 9% 3% 3% 3% 3% a. Board/Executive managementsupport b. Business unit/model ownersupport 21% c. Regulatory pronouncements 55% 55% 21% 30% d. Rating agency perspectives Very Unhelpful Somewhat Unhelpful No Impact Somewhat Helpful Very Helpful 12% Board/executive and business unit/model owner stand apart from regulator and rating agency. 12% no impact or unhelpful compared to 24% for regulators and 33% for rating agencies. Board/executive management support very helpful support. almost twice as high as for business unit/model owner responses. Model Risk Management PwC 6

7 Model Risk Management Basic Components Model Risk Management Function Model Risk Policy Model Inventory Risk Scoring Validation Charged with managing model risk Typically part of risk management function Interaction with other functions (actuarial, internal audit) Roles and responsibilities Model development and model change management Model validation Defining model Record of models and key model information Tracking validation status Typically defined via failure likelihood and loss severity Prioritize models based on risk Focus activities on higher risk models Periodic validation of model inputs, calculations, and outputs Key element for most insurers management of model risk Maximize efficiency as well as effectiveness Model Risk Management PwC 7

8 Data Set End-to-end validation 1 Input 2 Calculation 3 Output and Usage Historical information Regulatory models Model Calculations Trend analysis Benchmarking Future assumptions Internal models Model Theory Model Calculations User transparency 4 Upstream 4 and Downstream Model Risk Management PwC 8

9 MRM Survey 2017 Organizational design Consolidated All work is conducted by a consolidated MRM department, either centralized or dispersed (geographically or by BU), with essentially all validation being conducted by members of this department who have no substantive duties outside of MRM or other second line risk activities. Distributed Policy design, activity coordination and review/oversight is conducted by the MRM department but most validation work is conducted by utilizing staff outside of the department who have the required expertise and independence from the model being validated (and have other duties when not validating models). % Count 30% 10 37% 12 A blend of "consolidated" and "distributed" 21% 7 Other 12% 4 Total 100% 33 Model Risk Management PwC 9

10 Number of respondets MRM Survey 2017 Number of FTEs in MRM function 27 respondents, under all operational designs < Number of FTEs Model Risk Management PwC 10

11 No. of respondents MRM Survey 2017 Number of models per FTE 25 respondents, under all operational designs Models per FTE Model Risk Management PwC 11

12 Model Risk Management Potential areas for efficiencies Modular validation approach consider re-using validations of elements common to multiple models Better risk scoring some insurers are re-visiting scores and revising their view of model riskiness Better model documentation should be done by model owner as a matter of course, not a validation task/cost Use experienced model validators following defined procedures Model Risk Management PwC 12

13 Model Risk Management Strengthening Frameworks Gaurav Upadhya

14 Key Messages A. Actuaries and their teams are mandated to perform substantial work managing models & their risks as result of CSOP s B. Don t underestimate the risks in many existing models that may be complex, highly-customized, and poorly-documented Set up a Model Risk Management Framework to help! C. LICAT and soon IFRS 17 will bring additional model complexity, which will benefit from B already being in place, so all the more reason to get moving Model Risk Management Frameworks PricewaterhouseCoopers LLP 14

15 CSOP s (effective Jan 1, 2018) 1. When the work involves the use of a model, the actuary should: a. choose a model appropriate to the purpose and requirements of the work, and b. understand any limitations in the model which might make the results of the model inappropriate for the intended purpose or might produce a misleading result. (1) 2. Models will need to be risk rated 3. and mitigation activities will be required. (2) (1): Canadian Standards of Practice, January 2018, section 1535 (numbering added) (2): Use of Models Ed. Note, January 2017 (paraphrased) Model Risk Management Frameworks PricewaterhouseCoopers LLP 15

16 Where to begin? Start developing an MRM Framework! 1. Understand 2. Risk Rate 3. Mitigate Model Risk Management Frameworks PricewaterhouseCoopers LLP 16

17 Start developing an MRM Framework 1. Understand Understand your current models: a. Build a model inventory 2. Risk Rate 3. Mitigate b. Establish your appetite for model risk: How sure do you need to be in your models? (Achieve basic compliance, or be certain to the n th degree?) c. Assess if current models are appropriate for their intended purposes d. Delineate each model s limitations Model Risk Management Frameworks PricewaterhouseCoopers LLP 17

18 Start developing an MRM Framework 1. Understand 2. Risk Rate 3. Mitigate Risk rate models by assessing key attributes, such as: a. Complexity b. Level of decision making impact of the model c. Potential for human error (e.g. model s usability and level of manual manipulation) d. Degree of change management enforced e. Documentation quality f. Key person risk g. Model interdependencies Model Risk Management Frameworks PricewaterhouseCoopers LLP 18

19 Start developing an MRM Framework 1. Understand 2. Risk Rate 3. Mitigate Perform risk mitigation activities: a. Develop expertise in managing model risk b. Establish documentation and change management standards c. Perform governance activities such as compliance reviews Model Risk Management Frameworks PricewaterhouseCoopers LLP 19

20 Where can things go wrong in models? Following is an example of an AXIS valuation system, and some areas that contribute to risk Model Risk Management Frameworks PricewaterhouseCoopers LLP 20

21 Valuation Model Overview AXIS example Reporting Data Inforce Extracts Reinsurance Cession Data ETL (Extract, Transform, Load) EnterpriseLink DataLink AXIS Tables AXIS cells & objects DCAT Financial Statements Transaction Data Contracts Treaties Pricing Reports Economic Assumptions Experience Studies Product Features Assumptions AAR SOE LICAT Model Risk Management Frameworks PricewaterhouseCoopers LLP 21

22 Valuation Model Overview Areas for Caution 2 Data Inforce Extracts Reinsurance Cession Data! Model Risk Management Frameworks PricewaterhouseCoopers LLP Custom code ETL (Extract, Transform, Load) EnterpriseLink DataLink AXIS!! Transaction Data Contracts Treaties Pricing Reports Economic Assumptions Experience Studies Product Features Assumptions 1 Manual entry & hard-coding Assumptions change frequently (manual & custom code) Custom code Tables AXIS cells & objects 3 Custom code Custom objects! Documentation Custom tables 4 Reporting DCAT Financial Statements AA SOE LICAT 22

23 ! 1: Input Data Extract & Processing is Complex! Tips to reduce risk: 1. Use a detailed data dictionary which: a. Describes possible values and their interpretations b. Describes what each record represents 2. Document all data manipulations, and ensure kept up to date Model Risk Management Frameworks PricewaterhouseCoopers LLP 23

24 ! 2: Product Features & Assumptions Modelling # Bad Practice Good Practice Hardcode parameters within AXIS/Datalink No change management controls Store parameters in a database format outside of AXIS/Datalink Change management controls; periodic code reviews 3. Out of date, limited, or no documentation Up to date and multiple levels of documentation 4. Allow changes by minimally trained or less experienced staff Restrict changes to trained, experienced staff Model Risk Management Frameworks PricewaterhouseCoopers LLP 24

25 ! 3: AXIS Custom Coding Tips to reduce risk: 1. Implement a robust checking mechanism for tables and other objects, their definitions, and shapes 2. Set and enforce strong coding standards and practices 3. Maintain a test suite to support change management 4. Use code commenting & have documentation standards Model Risk Management Frameworks PricewaterhouseCoopers LLP 25

26 ! 4: Don t Over-Estimate Existing Documentation! Tips to reduce risk: 1. Follow good model design practices combined with concise and robust documentation 2. Use commenting functionality within AXIS 3. Employ layers of documentation in your MRM Framework such as model specifications, data transformation, model limitations, etc. 4. Implement a formal documentation standard with periodic compliance verification as part of overall MRM Framework Model Risk Management Frameworks PricewaterhouseCoopers LLP 26

27 In Summary A. CSOP s requires actuaries to choose appropriate models, understand any model limitations, risk rate the models, and perform mitigation activities. B. Establishing an MRM Framework will help achieve this. C. LICAT and soon IFRS 17 will bring additional complexity and scrutiny on actuarial models and their inherent risk (with more reliance on model outputs for financial statements) making the establishment of an MRM framework all the more critical. Model Risk Management Frameworks PricewaterhouseCoopers LLP 27

28 Thank you! This content is for general information purposes only, and should not be used as a substitute for consultation with professional advisers PricewaterhouseCoopers LLP, an Ontario limited liability partnership. All rights reserved. PwC refers to the Canadian firm, and may sometimes refer to the PwC network. Each member firm is a separate legal entity. Please see for further details.

Sections of the ORSA Report

Sections of the ORSA Report Lessons Learned From Orsa Reviews Impact on Risk Focused Examination NAIC Insurance Summit INS Companies Joe Fritsch, Director INS Companies Don Carbone, Exam Manager INS Companies Sections of the ORSA