Model change Guidance notes & 2016 submission requirements February 2016
Contents Introduction Page Background 3 Purpose 3 2016 Submission requirements Purpose of submission 4 Major model changes 4 Quarterly model change submissions 5 Lloyd s Guidance Change definitions 7 Mapping to Lloyd s change types 8 Combination of minor changes 8 Concurrent changes 8 Use of a changed model 9 Appendix 1: Template for approval of a major model change Appendix 2: Guidance for quarterly model change submissions Appendix 3: Mapping of model change definitions 1
2
INTRODUCTION Background Lloyd s undertook a dry run process for the approval of major model changes during 2015. Workshops were held with the market during December 2015 to obtain feedback on the dry run process and provide input to the process for 2016. Subsequent workshops were held in January 2016 to set out, at a high level, the approach to model change submissions in 2016. Purpose This document provides additional guidance to support managing agents in their model change submissions from now onwards. The guidance focuses on the 2016 submission requirements, as well as providing further guidance on model change definitions, the mapping of managing agent definitions to Lloyd s definitions, treatment of a combination of minor changes, how to deal with concurrent changes and the use of a changed model. It sets out the specific requirements for the quarterly model change report submissions and the submission of major model changes. This document provides practical guidance and does not seek to repeat existing Solvency II requirements on model change. Managing agents should continue to refer to the Solvency II requirements and the model change section of Lloyd s minimum standards. 3
2016 Submission Requirements Purpose of submission Model change reporting requirements are now of heightened importance as we operate in a live Solvency II environment. The overall objectives of the Lloyd s process are as follows: Meet regulatory requirements Provide an efficient mechanism for Lloyd s to stay up-to-date with syndicate models To act as a key input to the Lloyd s Internal Model (LIM) and potential LIM changes To facilitate the capital setting process The process for 2016 has been revised in order to address challenges identified and the feedback provided through the dry run. The key changes for 2016 centre on: A risk based approach to review and approval of major model changes A specific review timetable being agreed and communicated to managing agents for each major change Early engagement by Lloyd s to understand the changes planned through the year Risk Based Approach Each major change will be assessed and approved using a risk based approach. Factors which will be considered in this assessment include: Type of change e.g. quantitative, qualitative, methodology, parameters etc Materiality of the change to the LIM Impact on syndicate 1yr and Ultimate Solvency Capital Requirement (SCR) Impact on overall model compliance i.e. how material is the change to the model as a whole Review timetable Upon receipt of a major model change request, together with all supporting documentation, Lloyd s will determine a review timetable for approval of the major change. This timetable will be agreed by the Standards Assurance Group (SAG) and communicated to agents within 2 weeks of a complete model change submission being received. Early engagement During Q1 2016, Lloyd s will engage with all managing agents to consider the driver of any intended changes. The more Lloyd's can understand in advance, the smoother and more efficient the process should be through the remainder of the year. Major model changes Major model changes can be submitted to Lloyd s for approval at any time during the year. Lloyd s will only begin to process a major change once a complete application has been made. As such, in 2016, major model change requests need to be submitted with: A complete application form Board minutes evidencing approval or Senior Nominated Person (SNP) sign off that appropriate approval has been undertaken Supporting validation evidence 4
Application Form Lloyd s has amended the application form for a major model change application and has removed the pre-approval terminology from the application. The updated form is included in Appendix 1, and is also available to download from lloyds.com. Board Approval In line with Lloyd s Governance Minimum Standard 3.7 Managing agents shall ensure that as a minimum the following areas are approved at the full managing agency board: - Major internal model changes; Board approved minutes supporting the review and approval of the model change should be submitted with the application. To make the approval process as efficient as possible, if the only element delaying the application to Lloyd s is the finalisation of the Board minutes, it would be acceptable for a managing agent to make an initial application with a confirmation from the SNP that the appropriate Board approval has been followed, together with a draft copy of the minutes. If this approach is taken, this will initiate the Lloyd s process and approved Board minutes should be submitted to Lloyd s as soon as they are available. It is acceptable for the detailed review of a major model change to be delegated to a committee of the Board but the Board must discuss and challenge the change. An information paper from a delegated committee is not sufficient. Validation Major changes should be submitted with evidence of validation activity in line with a managing agent s validation policy. Lloyd s expectation is that certain changes would be subject to independent validation ahead of submission for approval. Lloyd s will issue further guidance, later in 2016, giving further clarity in relation to the validation activity which should be undertaken for differing change types. In the meantime all validation should be undertaken in accordance with an agent s model change policy. Submission to Lloyd s Managing agents should submit their major model change application with supporting documentation to risk.assurance@lloyds.com. Please note that incomplete applications will be returned to the managing agent and not submitted for initial SAG review. Managing agents should maintain the ability to reverse a major change should it not be approved by Lloyd s. Quarterly Model Change submissions Quarterly model change report submissions are due on the following dates in 2016: 3 March 7 July 13 September 30 December 5
Submission Report The format of the report has changed significantly from previous versions, largely to support ease of use and Lloyd s internal reporting requirements. However, the base data required for each model change is largely the same. Additional categorisation has been included to provide a much clearer link between syndicate model changes with the potential impact of these changes on the LIM. An additional tab has been included in the template with a small number of fields for agents to complete with information on model change policies and other standing information. For illustrative purposes, the template for this year s model change submissions is shown in Appendix 2 and includes specific guidance for its completion. The model change submission dates have purposely been aligned with those of the SCR submissions as we expect agents to be able to reconcile the changes in the SCR to the quantitative impact of the changes captured in the model change report. Where this does not reconcile exactly, agents should be able to explain the reasons for the difference. This could be achieved by highlighting in the SCR analysis of change those movements which have been reported as model changes. For quantitative changes, agents should report the impact on both the 1 year and to ultimate SCR. Lloyd s will provide templates to all agents for each syndicate under management. The templates will be prepopulated with the last agreed SCR and any changes that were reported to Lloyd s in Q4 2015. Submission to Lloyd s Managing agents should use the same report for all 4 quarterly submissions to Lloyd s, i.e. the July and September quarterly reports should build on the first report submitted in March. On completion of the capital setting process in Q4 2016, a new template will be issued to each agent. Where agents manage more than one syndicate, they should ensure that separate model change reports are provided for each syndicate. Submissions should be made on a nil return basis if no model changes have occurred during the reporting period. Managing agents should submit their quarterly model change reports to risk.assurance@lloyds.com. 6
Lloyd s guidance Change definitions Lloyd s definitions, which were included in our published guidance in 2014, covered the following change types: data updates, risk profile changes, parameterisation changes, changes to model methodology and model design, and changes in governance/controls. With the exception of data changes, these definitions are repeated below: Change type Definition Underlying risk profile significant changes to the nature, scale and complexity of the risk profile of the syndicate includes material changes in the business model, business strategy, products and lines of business, emerging risks and any other relevant changes to the risk profile significance should be justified on a quantitative and qualitative basis (i.e. using indicators specified in the policy) Model parameterisation represents a change in the managing agent s perception of a risk includes changes to the assumption and the parameter or the methodology and process to derive such assumption and parameter significance should be justified on a quantitative and qualitative basis Model methodology changes to the methods used to calculate the probability distribution forecast including external models and data Lloyd s interpretation is that this change reflects changes of the data sources and does not include data updates significance should be justified on a quantitative and qualitative basis Model design a change in the underlying structure or design of the model including use of IT systems and platforms, e.g. improvement of RI efficiency, moving to a newer version of the platform, increase/ decrease the number of simulations, switching from bootstrapping to claim simulation to measure reserving risk, moving from scenario testing to stochastic method for a certain risk module, change from RMS to AIR significance should be justified on a quantitative or qualitative basis. Governance/ controls a change to the control framework in which the model is operated e.g. switch from internal to external independent review, or to governance framework within which model output is used e.g. establishing a new committee the significance of these changes will always be determined qualitatively Inclusion of data changes in scope of the model change policy has to date been left to managing agent discretion with data changes reported to Lloyd s whether in or out of scope. Lloyd s is now comfortable that pure data changes e.g. foreign exchange rate changes, increase in line size, input of actual rather than estimate TPs, can be excluded from policies. Where data changes are out of scope of model change policies, they will no longer need to be reported to Lloyd s. 7
Mapping to Lloyd s change types It is not mandatory for managing agents to adopt the definitions set out above in their own model change policies. Lloyd s has previously required managing agents to map the Lloyd s change types to these definitions and this requirement is unchanged. Lloyd s have included a more granular list of change types within the quarterly model change submission template to aid analysis in support of the identification of potential changes to the LIM. A mapping of this more granular list to the main categories detailed above has been included in Appendix 3. There is no requirement for managing agents to map to this more granular list within their model change policies, the mapping undertaken previously remains sufficient. Major model change applications will also need to be categorised using this more granular list within the major model change application form. Combination of minor changes The way in which minor changes should be combined to ensure that significant changes are not overlooked has been an area of ongoing discussion with agents. Lloyd s suggested approach is to aggregate all minor changes on an absolute basis in terms of their impact on the SCR, as well as considering how qualitative changes may combine. The quantitative threshold for the aggregate changes to trigger a major change should be the same as the overall major change trigger. We recognise that the absolute basis approach may not be practical for all agents and therefore it is not mandatory. Whatever approach agents adopt, they should ensure that it is clearly explained in the model change policy together with a rationale as to why they consider it meets Solvency II requirements. Managing agents should ensure that they consider how qualitative as well as quantitative changes may combine to constitute a major change. Once a major change occurs, consideration of combination of changes re-starts from that point onwards. This is the case whether it is a major change driven from a combination of minor changes or a major change in its own right. When a major change application is made to Lloyd s the review will also include all the minor changes undertaken up to that point. For example, whether an agent is aggregating all minor changes on an absolute basis or an actual basis, the aggregation will start from zero at the point any major change occurs. When a syndicate makes an LCR submission to Lloyd s, this will be reviewed together with all changes up to the point of that submission. In March of each year, when Lloyd s restates syndicate SCRs, without an accompanying LCR submission from the managing agent, changes which have occurred up to that point will remain under review, and separate confirmation of approval will be provided subsequent to the restatement of the SCR. Concurrent changes It is possible that managing agents will be undertaking a number of major changes concurrently, or that a major change to the model will be undertaken together with a number of minor changes. In such circumstances, it is possible that a managing agent would trigger the major change threshold multiple times with one set of changes. In such circumstances, it is acceptable for one major change application to be made to Lloyd s, as long as sufficient detail and evidence on the various component changes are detailed within the form and supporting documentation. For quarterly reporting purposes, the concurrent changes should be reported separately. 8
Use of a changed model In the period between a major change application being made to Lloyd s and Lloyd s concluding its review and assessment, agents will need to have the ability to reverse the change if deemed appropriate by Lloyd s. In this interim period, managing agents should use the most current version of the model for decision making and make clear when making these decisions the impact on the results both pre and post the change. On this basis, there should be no issues in relation to ongoing compliance with the Use Test requirements and ensure that managing agents have a clear view of model output. 9
APPENDIX 1: TEMPLATE FOR APPROVAL OF A MAJOR MODEL CHANGE Agent Syndicate DATE MODEL CHANGE POLICY VERSION Change Type: Lloyd s detailed Change TYPE AGENT Change TYPE Qualitative/Quantitative/BOTH CHANGE CLASSIFICATION (MAJOR/MINOR/BOTH) Ultimate SCR (Against last approved SCR) 1 YEAR SCR (Against last approved SCR) PERCENTAGE IMPACT (%) FINANCIAL IMPACT( ) PERCENTAGE IMPACT (%) FINANCIAL IMPACT( ) IMPACT OF THE CHANGE: DRIVER FOR CHANGE (e.g. ADDRESSING MODEL LIMITATION/BEST PRACTICE) DESCRIPTION OF CHANGE AND IMPLICATION OF CHANGE ON DESIGN AND OPERATION OF THE MODEL
BRIEF DESCRIPTION OF INDEPENDENT VALIDATION UNDERTAKEN ON THE MODEL CHANGE (supported by evidence of validation) Confirm evidence of validation is attached AGENT APPROVAL OF CHANGE: BOARD APPROVAL DATE Confirm BOARD minutes are attached ANY FURTHER Information: Name of contact at agent Title of contact at agent
Appendix 2: Guidance for QUARTERLY Model Change submissions A screenshot of the template to be used for quarterly model change submissions is included below: Governance, Risk Management & Use Model change submission Agent: Syndicate: 1000 Date of return: For more information on how to complete each column please refer to the comments included in the headings of the table (this can also be found in the separate notes tab) Select date of row to Enter New Amend Model Delete Amend or Delete Ultimate SCR 1 Year SCR Record Change Record Select column G of Row New SCR Amend Delete Date Agreed Value m Date Agreed Value m you wish to Amend Last Agreed SCR 27/11/2015 100 27/11/2015 100 Date Syndicate number to which the model change applies Lloyd's change type Agent Change type Qualitative/ Quantitative Description and rationale for change Implication of change on design and operation of the internal model Classification (major/minor) Ultimate SCR impact 1 year SCR impact Committee approval Is combination of Does the entry minor changes fall within scope Model Change deemed to be a of the model Policy Version major change? change policy? (Yes/No) (Yes/No) Any further information Cumm % Cumm % Cumm m Cumm m Cumm % Cumm % Cumm m Cumm m Absolute (DD/MM/YY) Free Form at Drop down Free Form at Drop down Free Form at Free Form at Drop down % m Aggregate Absolute Aggregate Absolute % m Aggregate Absolute Aggregate Free Form at Drop down Drop down Free Form at Free Form at basis basis basis basis basis basis basis basis 01/01/2016 1234 Example EXAMPLE Quantitative EXAMPLE EXAMPLE Minor 0.00% 0.00 0.00% 0.00% 0.00 0.00 0.00% 0.00 0.00% 0.00% 0.00 0.00 EXAMPLE EXAMPLE EXAMPLE EXAMPLE EXAMPLE 01/01/2016 1234 Example EXAMPLE Quantitative EXAMPLE EXAMPLE Minor 0.00% 0.00 0.00% 0.00% 0.00 0.00 0.00% 0.00 0.00% 0.00% 0.00 0.00 EXAMPLE EXAMPLE EXAMPLE EXAMPLE EXAMPLE This template is issued separately in Excel format and additional guidance is included within the spreadsheet. The following table provides further guidance on the content to be included in each of the columns in the template above: Input Field Input Type Description Syndicate number to which the model change applies Automatically populated Displays the syndicate number to which the model change applies. A separate report should be produced for each managed syndicate. Date Input required in DD MM YYYY format The date the model change was considered by the relevant governance committee. Changes should be recorded in chronological order. Where changes were considered post implementation, e.g. by the Board, this can be captured under 'Any further information'. Lloyd's change type Drop down selection Change types as set out in Lloyd s guidance as shown below e.g. Design / Methodology, Risk Profile, Governance etc. Where an agent considers that their own model change type maps to more than one of the Lloyd s categories, they should select the Lloyd s type that they consider most appropriate and include a comment on any other Lloyd s change type considered relevant under Any further information. Agent Change type Free format field The change type as defined in the agent s model change policy, which will not necessarily be the same as Lloyd s guideline definitions. Qualitative/ Quantitative Drop down selection Whether the change is considered to be qualitative or quantitative. If the impact of the change can be quantified, then it should be classified as a quantitative change. Where relevant, comments on any qualitative aspects can be included under Description and rationale for change or Any further information.
Input Field Input Type Description Description and rationale for change Free format field Further detail on the change undertaken including a description of the change, the rationale for the change and any other relevant details, e.g. for qualitative changes the criteria used for classification of the change. Where small changes are batched and processed as one change e.g. by time, segment or other criteria this should also be detailed here. Implication of change on design and operation of the internal model Free format field The impact of the change on the design and operation of the internal model e.g. more stable output, change in run time. This field can also be used to capture further detail on the impact of qualitative changes to the model. Model Change Policy Version Free format field Details of the version of the policy in force at the time the change was considered through internal governance processes. Classification Drop down selection Whether the change is major or minor. Committee approval Free format field The managing agent governance committee which has approved the change being considered e.g. Internal Model Governance Committee, Risk Committee, Executive Committee, Board etc. Include detail here if the change was not approved when considered by the relevant committee. Note: All major model changes or combination of minor model changes to a major model change must have Board approval and full supporting evidence must be provided to Lloyd's. SCR impact m Numerical field For quantitative changes, the financial impact on the ultimate SCR and the one year regulatory SCR in millions. Enter zero for qualitative changes. Percentage impact on SCR will automatically be generated. Is combination of minor changes is deemed to be a major change? Drop down selection Indicator of whether the minor change being considered combines with prior minor changes to trigger a major change based on the approach to combination of changes currently adopted by the managing agent. Managing agents should select yes for the minor change that triggers the major change classification. For example if there are 4 minor changes that taken together are considered a major change, then yes should be selected for the last of the minor changes. Note: If Major Change is entered, this field will automatically default to "Yes" as a major model change must be combined with all minor changes before it that have not been approved by Lloyd's i.e. will be a combination of major and minor changes. The date of approval should reflect the date that the combined major change was considered by the board. Agents should include any further clarification or information they consider necessary under Any further information, including references to any supporting documentation that is available. Changes should be recorded in chronological order. Once a major change is triggered, any prior minor changes do not need to be taken into account when considering further combinations of changes. A major model change template (and any supporting documentation) should be submitted to Lloyd's in support of the combination of minor changes.
Input Field Input Type Description Does the entry fall within scope of the model change policy? Drop down selection This is a prompt box only. Lloyd's do not require entries out of scope to be reported within the model change template. Any further information Free format field Free form box to capture any additional information which a managing agent would like to submit to clarify the entries in the report or to support the change undertaken e.g. model version/reference number. Submit N/A Verifies the data input and adds row to model change report. The following table provides further detail on the Lloyd s change type definitions: Change Type Description Design/methodology - Dependencies Any change to the dependency structure used in the internal model, for example: Introduction of new dependency between classes of business and/or risk types or removal of existing dependency (whether driver or copula) Change in copula used e.g. from gumbel to student Design/methodology - Insurance - Underwriting (noncatastrophe) Any change to the methodology used in modelling non catastrophe insurance losses, for example: Change in modelling total losses to a frequency severity approach Change in the loss distribution used Change to methodology of generating 12 month losses compared to ultimate losses Changes in modelling of existing outwards reinsurance program - introduction of new program would be captured by changes in risk profile Design/methodology - Insurance - Underwriting (catastrophe) Any change to the methodology used in modelling catastrophe losses, for example: Change to vendor model used e.g. change in RMS model version or change from AIR to RMS Introduction of new perils Design/methodology - Insurance Reserves including Risk Margin Design/methodology - Operational Risk Design/methodology - Market Risk Any change to the modelling of technical provisions, for example: Change to distribution used Change to simulating incurred losses instead of paid losses Change to calculation of risk margin Any change in the modelling of operational risk, for example: Change to distribution used Introduction of new scenarios (if this falls within the definition of model change) Any change in the modelling of market risk, for example: Change to ESG provider Introduction of new asset classes or currencies
Change Type Description Design/methodology - Credit Risk Any change to the modelling of credit risk, for example Change to distribution used for generating defaults or loss given defaults Allowing for impact of security held against credit risk e.g. LOC s, funds with-held Inclusion of new source of credit risk Design/methodology - Other Any design/methodology change falling outside of the above - this must be explained under 'Description and rationale for change' Parameterisation & parameterisation methodology Change to the parameterisation or the methodology used to parameterise the internal model. For example Use of new data sources to parameterise assumptions Changing adjustments to historic data within the parameterisation process Changes to parameters as a result of data updates. Changing the degrees of freedom in a copula Risk Profile Changes to the risk profile used in the internal model as a result of business developments, for example: Introduction of a new reinsurance program New classes of business being written or a significant change to existing classes Change in asset allocation Governance IT changes including change in modelling platform Other Data changes Including changes to Model Change Policy, Validation Policy, Model Scope & Model Use For example change from ReMetrica to Igloo OR vice versa Anything outside of the above Where in scope, includes model data that are generated as a result of existing processes such as business planning process, underwriting process, reserving process etc.
APPENDIX 3: Mapping of Model Change Definitions Change type DETAILED CHANGE TYPE Data updates (excluded from model change reporting where out of scope of the agent s model change policy) Data Changes Changes as a result of movements in exchange rates Underlying risk profile Risk Profile Model parameterisation Parameterisation and Parameterisation Methodology Model methodology Dependencies Insurance Underwriting (Non-Catastrophe) Insurance Underwriting (Catastrophe) Insurance Reserves including Risk Margin Operational Risk Market Risk Credit Risk Other Model design Dependencies Insurance Underwriting (Non-Catastrophe) Insurance Underwriting (Catastrophe) Insurance Reserves including Risk Margin Operational Risk Market Risk Credit Risk IT changes including change in modelling platform Other Governance/controls Governance including Change Policy, Validation, Scope and Model Use Other Other