2018 Predictive Analytics Symposium Session 10: Cracking the Black Box with Awareness & Validation

Transcription

1 2018 Predictive Analytics Symposium Session 10: Cracking the Black Box with Awareness & Validation SOA Antitrust Compliance Guidelines SOA Presentation Disclaimer

2 Cracking the Black Box with Awareness & Validation Presenters: Kyle Nobbe, FSA, MAAA Vikram Kamath

3 SOA Antitrust Compliance Guidelines Active participation in the Society of Actuaries is an important aspect of membership. While the positive contributions of professional societies and associations are well-recognized and encouraged, association activities are vulnerable to close antitrust scrutiny. By their very nature, associations bring together industry competitors and other market participants. The United States antitrust laws aim to protect consumers by preserving the free economy and prohibiting anti-competitive business practices; they promote competition. There are both state and federal antitrust laws, although state antitrust laws closely follow federal law. The Sherman Act, is the primary U.S. antitrust law pertaining to association activities. The Sherman Act prohibits every contract, combination or conspiracy that places an unreasonable restraint on trade. There are, however, some activities that are illegal under all circumstances, such as price fixing, market allocation and collusive bidding. There is no safe harbor under the antitrust law for professional association activities. Therefore, association meeting participants should refrain from discussing any activity that could potentially be construed as having an anti-competitive effect. Discussions relating to product or service pricing, market allocations, membership restrictions, product standardization or other conditions on trade could arguably be perceived as a restraint on trade and may expose the SOA and its members to antitrust enforcement procedures. While participating in all SOA in person meetings, webinars, teleconferences or side discussions, you should avoid discussing competitively sensitive information with competitors and follow these guidelines: -Do not discuss prices for services or products or anything else that might affect prices -Do not discuss what you or other entities plan to do in a particular geographic or product markets or with particular customers. -Do not speak on behalf of the SOA or any of its committees unless specifically authorized to do so. -Do leave a meeting where any anticompetitive pricing or market allocation discussion occurs. -Do alert SOA staff and/or legal counsel to any concerning discussions -Do consult with legal counsel before raising any matter or making a statement that may involve competitively sensitive information. Adherence to these guidelines involves not only avoidance of antitrust violations, but avoidance of behavior which might be so construed. These guidelines only provide an overview of prohibited activities. SOA legal counsel reviews meeting agenda and materials as deemed appropriate and any discussion that departs from the formal agenda should be scrutinized carefully. Antitrust compliance is everyone s responsibility; however, please seek legal counsel if you have any questions or concerns.

4 Presentation Disclaimer Presentations are intended for educational purposes only and do not replace independent professional judgment. Statements of fact and opinions expressed are those of the participants individually and, unless expressly stated to the contrary, are not the opinion or position of the Society of Actuaries, its cosponsors or its committees. The Society of Actuaries does not endorse or approve, and assumes no responsibility for, the content, accuracy or completeness of the information presented. Attendees should note that the sessions are audio-recorded and may be published in various media, including print, audio and video formats without further notice.

5 Presenters Vikram Kamath Kyle Nobbe, FSA, MAAA Director of Analytics LIMRA Associate Actuary, Global Research and Data Analytics RGA Reinsurance Company

6 Understanding the SEMMA Process Individual Life Buyer/Non-Buyer Model

7 LIMRA Center of Excellence for Analytics 6

8 The SEMMA Process Sample Explore Modify Model Assess 7

9 8 Sample Acquire and clean data Take a sample (if needed) Split into training, testing, and validation 5000 person representative sample of US population 100+ questions: Financial situation Recent life insurance purchases Demographics Behavior

10 Explore 9 Find patterns and relationships in the data Select important variables Which variables influence buying behavior most? Built a gradient boosting machine to find the most influential variables Total Assets Total Debts Household Income Age Financial Advisor Children Employment Marital Status Recent Life Events Education Level Decision Making Power

11 Modify Create new variables Modify existing variables Bring in external data Created variables for presence of children and number of recent life events Did log transforms on highly skewed variables Tested variable interactions Linked external data (credit scores, Claritas Psycle scores) 10

12 Model Build the model that best fits the data Built logistic regression model from the most influential variables Model segments customers on a 1-10 scale: 1 = least likely 10 = most likely Focus on segments 8-10 to write more business Sample Average: 18% 34% 26% 53% 11

13 Assess 12 Evaluate model performance Draw conclusions and make recommendations Validated model using 10-fold cross-validation Set probability score cutoff at 0.25; Above = Buy, Below = No Buy Overall model accuracy: 76.8% Classification Predicted Actual Non-Buyer Buyer Non-Buyer 67.0% 8.4% Buyer 14.7% 9.8% ROC Curve Explainer: AUC of 0.5 means model is no better than chance 0.7 and above is a solid model Too close to 1 is an overfit model AUC = 0.762

14 Likely Buyer Profile The most likely buyers tend to have these qualities: Young to Middle Aged Upper Middle Class High Income/Assets but also High Debt Employed Full Time Have a Financial Advisor Own their Home Married with Children College Degree+ Had Significant Life Event(s) in the Last Two Years 13

15 Your Trusted Source for Industry Knowledge

16 Introduction to Validation 15

17 Introduction to Validation Basic Stats Review Root Mean Square Error P Value RRRRRRRR = nn=1 NN PPPPPPPPPPPPPPPPPPPP nn AAAAAAAAAAAAAA 2 NN Calculated probability of finding the observed results of target variable within CI Beware of p-hacking! Very common. Can be used for all types of models Sensitive to outliers and does not penalize overly complex models Degrees of Freedom Data points that are free to vary when estimating statistical parameters Reducing dimensions reduces df reduce overfitting 16

18 Introduction to Validation Data Splitting Standard Split Cross Validation Bootstrapping 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Holdout Test Trai n Modeling Data Example of 10 fold CV % 10 % 10 % 10 % 10 % 10 % 10 % 10 % 10 % % % 10 % 10 % 10 % 10 % 10 % 10 % 10 % 10 % % Test Data 30 % 70 % 70 % 30 % 70 % 30 % Test Data 30 % 70 % Train Data Train Data 17

19 Introduction to Validation Residuals A residual is the difference between the prediction and observation Residual plots are a tool for data scientists to find errors in their models Credit goes to Statswing for graphic 18

20 Introduction to Validation Information Loss AIC & BIC are most common Purpose is to optimize bias and variance BIC more heavily penalizes complex models compared to AIC (almost always) BIC is more likely to pick too simplistic of a model 19

21 Introduction to Validation Confusion Matrix and ROC Curve Terminology Sensitivity = True Positive Rate Specificity = True Negative Rate Fall out = False Positive Rate ROC Curve = A pretty graph (Receiver Operating Characteristic) AUC = Area Under Curve TRUE Positive Rate 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% ROC Curve and AUC AUC = 0.7 0% 0% 20% 40% 60% 80% 100% FALSE Positive Rate 20

22 Introduction to Validation Impurity Measures Most common impurity measures for building classification trees are Gini and entropy either Gini or entropy should be used when growing the tree The Elements of Statistical Learning studies have shown that the choice of impurity measure has little effect on model performance of a decision tree Introduction to Data Mining 21

23 Case Study Validating a Mortality Risk Model 22

24 Case Study Validating a Mortality Risk Model Introduction Create mortality risk score using socioeconomic data attributes. Research project to improve existing mortality rating techniques Model to validate is a GLM built on experience study data 23

25 Case Study Validating a Mortality Risk Model Case Study Validation Process 1. Lift Curves 2. Actual vs Predictions 3. Visualize Regression 4. Understand Documentation 24

26 Case Study Validating a Mortality Risk Model Case Study Validation Process Lift Curves 1. Lift Curves Relative A/E s by Score and Collar 2. Actual vs Predictions 3. Visualize Regression 4. Understand Documentation A/E (Traditional Expecteds) 140% 120% 100% 80% 60% 40% 20% 0% Deaths ,000 10,000 8,000 6,000 4,000 2,000 0 Number of Deaths Blue Collar Mixed Collar White Collar Collar and Mortality Score 25

27 Case Study Validating a Mortality Risk Model Case Study Validation Process Actual vs Prediction 1. Lift Curves Actual over Predictions by Weekly Income and Collar 2. Actual vs Predictions 3. Visualize Regression 4. Understand Documentation A/P (Raw Prediction) 120% 110% 100% 90% 80% 70% 60% 50% Blue Collar Mixed Collar White Collar < ,000 18,000 16,000 14,000 12,000 10,000 8,000 6,000 4,000 2,000 0 Number of Deaths Weekly Income 26

28 Case Study Validating a Mortality Risk Model Case Study Validation Process Visualize Regression 1. Lift Curves 2. Actual vs Predictions Visualize Regression by Collar and Weekly Income (holding all other covariates constant) Blue Collar White Collar Mixed Collar 3. Visualize Regression 4. Understand Documentation Weekly Income 27

29 Case Study Validating a Mortality Risk Model Case Study Validation Process Visualize Regression 1. Lift Curves 2. Actual vs Predictions NEW Visualize Regression by Collar and Weekly Income (holding all other covariates constant) Blue Collar White Collar Mixed Collar 3. Visualize Regression 4. Understand Documentation Weekly Income 28

30 Case Study Validating a Mortality Risk Model Case Study Validation Process Visualize Regression 1. Lift Curves NEW Visualize Regression by Collar and Weekly Income with Outliers (holding all other covariates constant) 2. Actual vs Predictions 3. Visualize Regression 4. Understand Documentation Blue Collar White Collar Mixed Collar Weekly Income 29

31 Case Study Validating a Mortality Risk Model Case Study Validation Process Understand Documentation 1. Lift Curves 2. Actual vs Predictions 3. Visualize Regression 4. Understand Documentation 45% 40% Percent Increase in AIC 35% 30% 25% 20% 15% 10% 5% 0% Percent Increase in AIC when Removing Covariate from GLM Age Gender Income Collar Education Collar:Income Covariate 30

32 Case Study Validating a Mortality Risk Model Case Study Validation Process Understand Documentation 1. Lift Curves 2. Actual vs Predictions 3. Visualize Regression 4. Understand Documentation Percent Increase in AIC when Removing Covariate from GLM Other Interactions Tested Df AIC P Value Final Model Gender:Collar Gender:Income Collar:Income^ Collar:Age Age:Income Gender:Age Gender:log(Age) Income Age

33 Case Study Validating a Mortality Risk Model Case Study Validation Process Understand Documentation 1. Lift Curves 2. Actual vs Predictions 3. Visualize Regression 4. Understand Documentation Thorough documentation of data consistent with ASOP 23; plus, handling of missing values, transformations, etc. List of other models tested with corresponding results and reasoning for model selection 32

34 Conclusion 33

35 34

36

37 Presenters Vikram Kamath Kyle Nobbe, FSA, MAAA Director of Analytics LIMRA Associate Actuary, Global Research and Data Analytics RGA Reinsurance Company