Statistical problems, statistical solutions. Glen Barnett

Transcription

1 Statistical problems, statistical solutions Glen Barnett

2 Summary of some issues I see a number of issues that I believe directly impact the ability of new actuaries to do their job well, to learn new ideas and to adapt to changing circumstances. - avoidance of serious critical examination the paradigms on which actuarial predictions and predictive modelling are based (when criticisms expressed outside the paradigm, frequently not understood. Within the paradigm can t describe the problem) - An unwillingness to subject specific models and predictions for given data to serious model criticism (subject to tests that cannot fail when tested at all)

3 Summary of some issues - Elevation of actuarial judgement to almost mythic status, without critical examination of its actual performance -Insularity: a lot of the actuarial literature moves forward (or not) largely independently of essentially exactly the same ideas and applications in other areas. [When ideas are taken up - often misapplied.] - Ignorance of its own history

4 The importance of criticism Criticism is critical to scientific endeavour, indeed to any form of not fooling ourselves. Criticism of models and assumptions is a central part of obtaining useful inference (including predictions). This is often entirely absent in actuarial work (at least that which I come into contact with), and it has serious, sometimes dramatic consequences.

5 The practice of building a model is not really taught to actuaries indeed, what is really meant by a model (in the statistical sense) is frequently misunderstood. Model building not a simple skill that is readily learned: it takes a great deal of practice on a variety of examples, must be accompanied by model assessment and criticism, (keeping in mind the purpose of the model) it needs to be practiced on real data.

6 Model building is an iterative process (Box) Important stage of model building: model criticism models carefully examined for misspecification When forecasting in time, particular kinds of misspecification are more critical than others

7 Different purposes for models A model can be used to - learn about a process find important features of a process; discover unanticipated effects (exploratory model building) - A model can be used to describe/understand a process a simplification that draws out essential features, separate important drivers from lesser effects - concise description of the historical

8 - as an inferential tool (for some models, some parameters or functions of them are of direct interest) - as a forecasting tool Different purposes lead to different requirements. If you re forecasting, the issues are entirely different from when you re trying to understand the process

9 Different uses require different strategies and often result in different models. - e.g.: - A known future legislation change will not impact a description of the past, but should impact a prediction. - Overparameterization can impact forecasting time series in a much more serious way than in say a static regression model (akin to forecasting outside the range of the data in a regression model). Variance quickly comes to dominate bias.

10 Actuarial Education an environment - education is not just notes, assignments, exercises or tutorials - the entire environment Let s consider a few aspects of the environment from say the last ten years or so.

11 Actuarial Educational environment e.g. - CT notes, and other standard documents, - professional standards, - prudential standards, - published research papers.

12 CT notes e.g. CT6 many errors, omissions and oddness e.g. chain ladder applied to data with inflation forecasts at some average of past rates (false) average incurred cost per reported claim presented as different to ratio methods. Actually just another one (with an unusual choice of estimator for ratio). -No attempt to place any kind of modelling in context emphasis on traditional methods for sake of tradition (no substantive reasons given for any of it)

13 CT notes e.g. CT3: lots on balanced ANOVA, partitioning variances & such - How many actuaries design experiments? Lots on hypothesis testing - rarely the best way of dealing with the needs of the actuary (much that is done as hypothesis testing is better done in other ways - often misused! e.g. see CT4 notes)

14 CT notes e.g. CT3: hardly anything on prediction intervals (mentioned in simple linear regression for a single prediction - but explored no further). statistical modelling? model criticism? - About 15 lines of text and one single plot of residuals vs fit for made up data not used in any of the numerical examples! What message do students take from this? Unimportant.

15 CT notes e.g. CT4: graduation -overemphasis on hypothesis testing - misguided, most of the advice is decades out of date. Actual exercise is one of model building, and while a few of the measures of model adequacy proposed are useful, formal hypothesis testing as suggested is counterproductive.

16 CT notes - any recent literature on fitting and testing models, statistical tools absent (looks like a rehash of notes from years ago common in CT notes) - such as fitting of many kinds of smooth-curve Poisson and Binomial models that are either directly equivalent to standard actuarial "methods" techniques, or close relatives, available in packages, including widely used free ones, is ignored e.g. R (free) or Splus (commercial), both widely used, can cover almost all this stuff easily. - Some parts of the notes are decades out of date (or misapplied. Or both) (But in other sections, relevant literature does come through)

17 Prudential standards e.g. GPS210 (2002) + GN210.1 central estimate plus one half of the coefficient of variation adding dollars and unit-free quantity GN is predicated on the assumption that mean plus half the s.d. only exceeds the mean when the distribution is heavily skew. (False) 75 th percentile repeatedly referred to as a risk margin or a risk estimate. Actually neither on its own.

18 Professional standards e.g. - Some old versions confused mean, median and mode PS300 (2002) - Tells us we can determine characteristics of what is explicitly a probability distribution (its mean, 75 th percentile, coefficient of variation, and so on), apparently without using statistical ideas at all. How this could mean anything is not clarified.

19 Professional standards - Importance of changing inflation in future explicitly detailed, but analyzing its impact on standard approaches in the past data is effectively unmentioned. - Focus is on allowance for effects, selection, assumptions and judgement Language about estimation, measurement, checking of models, critical assessment of past assumptions largely absent corresponding lack of consideration of these and related issues in actuarial work. - divorces the consideration of the future from its past.

20 Published papers Many highly lauded papers are filled with basic, even egregious errors. In a student, they would require serious remedial action. e.g. Patel and Raws: The basic idea of the paper is to apply a variety of judgementally-selected and -applied standard methods and then to form a central estimate, fit a distribution to their predicted outstandings in order to apply ML to find the mean. (This gets the process of statistics entirely backwards.)

21 Discussion focus on identifying the distribution from say 6-12 answers from variety of standard methods. (Even if meaningful, impossible with tiny samples.) The approach was claimed to give a range of uncertainty on the mean. (It doesn t) Methods to do what the paper was apparently attempting to achieve already exist in statistics, and had done for a fair while.

22 Discussion focus on identifying the distribution from say 6-12 answers from variety of standard methods. (Even if meaningful, impossible with tiny samples.) The approach was claimed to give a range of uncertainty on the mean. (It doesn t) Methods to do what the paper was apparently attempting to achieve already exist in statistics, and had done for a fair while. It won an award!

23 uncriticized models have consequences e.g. Lusk (1983) J. Forecasting This model criticism step is often not given sufficient attention, resulting in failure to isolate an appropriate model leading to predictable consequences

24 Actuarial students spend time learning topics of little value to them - spend a good amount of time learning balanced ANOVA, showing orthogonality, computing expectations of sums of squares, deriving distributions. - all very interesting if you are in the business of designing experiments - What do we find actuaries doing?

25 Fitting similar models to time series! - Loss development triangle is essentially a time series. Data arrives a diagonal at a time each year (/qtr/month). But one very popular model is a two-way crossclassification model essentially a main-effects two-way ANOVA (but with missing values). (chain ladder reproducing models like Mack s approach or the quasi-poisson GLM with row and column effects are of this type)

26 Ṙegularity in DY and AY direction ignored Time direction critical but ignored DY and AY treated as interchangeable This is astoundingly wrong Hugely overparameterized BUT poor fit Poor predictive behaviour has been repeatedly demonstrated

27 Experiment: Watson and Johnson-Laird (1972) Every card has a letter on one side & a number on the other. You see: A D 4 7 Suggested rule: "If a card has a vowel on one side, it has an even number on the other" Which cards do you turn over to test this rule?

28 Instinct is to check confirming instances Many people want to turn over the 4. - correct approach is to search for disconfirmation - In the previous experiment, turning over the 4 can only confirm the rule (if you get a vowel), but cannot disconfirm (non-vowel on the other side is irrelevant), so it s not testing it

29 Many actuaries - being good at mathematics can (eventually) figure out this problem But when it comes to testing their models, either turn over no cards, or only turn over the 4

30 Many papers don t turn over any cards - far too many papers to list fit models to data without examining the model at all. - No residual plots - No attempt to embed in a large model to look for additional effects - No assessment of the ability to predict the last year Comparing answers rather than whether the model bears any relationship to data

31 Many papers don t turn over any cards Some arbitrary examples (the first few I found in one of my subdirectories of papers) - England and Verrall (1999) Analytic and Bootstrap Estimates of Prediction error (Fits T&A data) - Pinheiro et al (2003) Bootstrap Methodology in Claims Reserving ( ) -Lowe (1994) Practical Guide to Measuring Reserve Variability Of the >600 pages of call papers presented at the 2008 CLRS, very few presented any model criticism.

32 Turning over 4 s: Wtd Std Res vs Dev. Yr Will look at a graph such as this, and declare victory Wtd Std Res vs Cal. Yr missing this! (Where s the future?)

33 For example: - Renshaw and Verrall (1998) A stochastic model underlying the chain ladder technique Plots residuals vs DY & AY CANNOT show lack of fit! No surprise - they find none!

34 Wtd Std Res vs Cal. Yr Common problems with very popular triangles: e.g. Taylor & Ashe data Used in dozens of papers <= Mack model ScDevRes vs CalYr <= quasi-poisson GLM Worse, Taylor &Ashe mentioned this issue!

35 Predicted and Actual (CY:90) vs DY. 100,000 Predicted 90,000 Observed 80,000 70,000 60,000 50,000 40,000-30,000 20,000 10, Leave out most recent year and predict it $ scale 100,000 Predicted and Actual (CY:90) vs DY. Predicted Observed 10,000 1,000 log scale underpredicting the past!

36 One step ahead predictive distributions Bootstrap Pred. Dist & Obs vs DY Bootstrap Pred. Dist & Obs vs DY % 75% 50% 25% 10% Obs 90% 75% 50% 25% 10% Obs DY DY Bootstrap allows you to check whether your predictive distribution is reasonably calibrated on recent past

37 Some common diagnostics are little help - - Small r 2 may indicate a good model, - even the best model Some residual plots will tell you little about predictive model adequacy depends on the model (against variables against which you are fully paramaterized; vs fitted when predictions from model are unlike fit)

38 Small r 2 but good model: perfect model, yet r 2 = here, higher r 2 worse predictive performance

39 toy example but actually relevant: 3,000 2,800 2,600 2,400 2,200 2,000 1,800 1,600 1,400 Incr.(1) vs Cum.(0) increm. vs cum - If chain ladder works what should it look like? -a better model has r 2 =0 for this pair of years

40 StdRes vs Increm Fit - quasi-poisson: resid vs fit misses predictive behaviour Need prediction errors Mack: Scaled Prediction Errors vs Predicted Wtd Std Res vs Fitted ,000 10,000 15,000 20,000 25,000

41 History is bunk GLMs, Bootstrap, computation of full predictive distributions are often called new. How long ago were they used in reserving? GLM? Bootstrap? Full predictive distributions?

42 Selection of ratios: Ratios imply a line through the origin If NOT a straight line through O => different ratios for every cell. (p=n) Cannot evaluate this - cannot compute standard errors; it is not a model in the statistical sense.

43 Is assumption E(p x) = a + (b-1) x tenable? Note: If corr(x, p) = 0, then corr((b-1)x, p) = 0 If x, p uncorrelated, no ratio has predictive power Ratio selection by actuarial judgement can t overcome zero correlation. p Corr. often close to 0 -Sometimes not. Does this imply ratios are a good model? x - ranges?

44 Bateup and Reed - used by students and practitioners (there are many issues with this document) e.g. discussion of the bootstrap: Says can select ratios and get std errors, etc. [Published papers seem to suggest the same thing (+ predictive distributions, etc)]

45 Bateup and Reed This is a misunderstanding of the bootstrap When the resampling stage occurs, for it to be a bootstrap, the process of fitting the original model has to be replicated - Cannot (independently) re select a set of ratios for each bootstrap resample. Consequently greatly underestimates uncertainty => underreserving/undercapitalized