Confusion in scorecard construction - the wrong scores for the right reasons

Transcription

1 Confusion in scorecard construction - the wrong scores for the right reasons David J. Hand Imperial College, London and Winton Capital Management September 2012 Confusion in scorecard construction - Hand 1

2 An example of financial model risk in another area Writing about the year an event such as the crash would not be anticipated to occur even if the stock market were to remain open for twenty billion years.. Sebastian Mallaby, in More Money than God Confusion in scorecard construction - Hand 2

3 Writing about the year the figures implied it would take a so-called ten sigma event... for the firm to lose all its capital in one year.. Roger Lowenstein, in When Genius Failed Confusion in scorecard construction - Hand 3

4 Writing about the year things were happening then that were only supposed to happen about once in every 100,000 years Bill Bonner, in Money Week Confusion in scorecard construction - Hand 4

5 Writing about the year What we witnessed yesterday was a series of movements of utterly unprecedented proportions, with currency price changes that are at the 6th and 7th and 8th standard deviations from the norm... David Gartman in The Gartman Letter Confusion in scorecard construction - Hand 5

6 But all of this makes sense to a statistician. If your model is wrong, your probability estimates are wrong And almost (but not quite) by definition: risk relates to the tails of distribution That is, to rare events Confusion in scorecard construction - Hand 6

7 e.g. Normal vs Cauchy, relative size of tail probabilities 24 Standard normal distribution P x Cauchy distribution P x A factor of Standard normal distribution P x Cauchy distribution P x A factor of Confusion in scorecard construction - Hand 7

8 Why credit scoring? - to decide who should receive a financial product - to protect applicants from overstretch - to decide interest rates - to detect fraud - to guide customer management - to make profit for the company - to increase shareholder value -... Confusion in scorecard construction - Hand 8

9 Data for credit scoring - application form data - past behaviour with loans - other sources Characterised by - potentially many predictors - mix of categorical and continuous Often (but not always) large data sets Typically (but not always) a small bad class Confusion in scorecard construction - Hand 9

10 Models for credit scoring Simplest scenario: predict who will default just two classes, labelled good and bad Two stages 1) compute credit score 2) choose decision threshold Most common form: Logistic regression tree e.g. 10 segments, each with 20 predictors, transformed and combined in various ways Confusion in scorecard construction - Hand 10

11 Modern credit scoring is data-driven (empirical): based on observed relationships in the data a statistical revolution Contrast with mechanistic: based on theory relating characteristics to default probability The fact that credit scoring is data-driven has sometimes worried the regulators - qualitative, mechanistic, modellers base models on narrative e.g. hypothesised causation - quantitative, data-driven, modellers base on data e.g. association Confusion in scorecard construction - Hand 11

12 Why data driven? Because our aim is to build the best models we can - use any characteristics which improve prediction - transform/combine in arbitrarily complex ways e.g. Linear discriminant analysis, quadratic discriminant analysis, naive Bayes, regularised discriminant analysis, logistic regression, SIMCA, DASCO, logistic regression, perceptrons, neural networks, support vector machines, tree classifiers, random forests, nearest neighbour, Parzen kernel methods,... Confusion in scorecard construction - Hand 12

13 Why the logistic regression tree? Tree because it is (equivalent to) including interactions Logistic because it gives a simple model of default probability expressed in terms of a weighted sum of predictors Confusion in scorecard construction - Hand 13

14 And because Confusion in scorecard construction - Hand 14

15 1) Logistic works - and often better than alternatives Prop linear = (Default - LDA) / (Default - Best) Confusion in scorecard construction - Hand 15

16 2) Easy to explain reasoning behind linear model For example:... a low score on characteristic XXX (e.g. prior credit record) is associated with a poor score... e.g. US Equal Credit Opportunity Act: applicant has the right to request the reason(s) for denial within sixty days of receipt of the creditor s notification, along with the name, address, and telephone number of the person who can provide the specific reason(s) for the adverse action Confusion in scorecard construction - Hand 16

17 Legal constraints on credit scoring What characteristics may be used? Anti-discrimination laws restrict choice Confusion in scorecard construction - Hand 17

18 The spirit of the law To prevent unequal treatment on grounds unrelated to the aim of estimating default probability Confusion in scorecard construction - Hand 18

19 The letter of the law (precise wording depends on legislature) Direct discrimination arises if one person is / has been / or would be treated less favourably than another in a comparable situation on grounds of racial or ethnic, sex, religion or belief, disability, age or sexual orientation Indirect discrimination arises when an apparently neutral provision, criterion or practice would put persons having a particular racial or ethnic origin, religion or belief, sex, disability, age or sexual orientation at a particular disadvantage compared with other people Except if this provision, criterion or practice is objectively justified by a legitimate aim and the means of achieving that aim are appropriate and necessary Confusion in scorecard construction - Hand 19

20 The consequences of the law It is illegal to treat certain groups differently, even if those groups are known to have different degrees of risk - women are less likely to default than men - older are less likely to default than younger But the law (appears to) mean(s) retail credit scorecards must not include sex as a predictor This disadvantages females - who have to pay larger interest rates - and have loan applications rejected more often than when decisions are based on their actual probability of defaulting Confusion in scorecard construction - Hand 20

21 Creditors want to use the most accurate models - so, if sex is prohibited, instead use a legal proxy variable - until the law prohibits that, too But recall: indirect discrimination prohibits situations where an apparently neutral provision, criterion or practice would put persons having a particular racial or ethnic origin, religion or belief, sex, disability, age or sexual orientation at a particular disadvantage compared with other people Confusion in scorecard construction - Hand 21

22 The crunch: Eliminating proxy variables also eliminates any legitimate predictive power they have e.g. any predictive power uncorrelated with sex The best risk estimates are not being used What to do? Confusion in scorecard construction - Hand 22

23 Solution A: the current solution Simply prohibit sex and proxies But this means that people are not assigned to risk category using best estimates of default probability Overestimates default probability for females Underestimates default probability for males Confusion in scorecard construction - Hand 23

24 Does not the failure to include sex put women at a particular disadvantage compared with men? - it unfairly penalises them by charging them at a rate which is greater than that at which they ought to be charged given their propensity to default - why should someone who is female (not a choice they made) be penalised in this way? - does the implementation of the law contravene the law? Meets the letter of law But contravenes the spirit Confusion in scorecard construction - Hand 24

25 Solution B: Treat men and women equally by allocating the same proportion of men and women to the good class by building separate models for men and women and choosing the default probability thresholds so that the same proportions are classified as good in each case But since the default probability distributions of men and women will be different, this means choosing different probability thresholds for the two sexes (lend to men if P(def)<0.8, to women if P(def)<0.7) Contravenes the law? (Equiv. incl. sex in interactions) Confusion in scorecard construction - Hand 25

26 Solution C: Treat men and women equally by allocating the men and women to the same class when they have the same probability of being good by building separate models for men and women and choose the same default probability thresholds But this means different proportions of men and women will be accepted (More men have P(def)>p than women) Contravenes the law? (Equiv. incl. sex in interactions) Confusion in scorecard construction - Hand 26

27 Solution D: Build the best predictive model we can, but one which does not let sex contribute to the classification by eliminating it statistically Linear model: build the predictive model using all the variables we can get hold of, including sex and any proxies for sex we like. But then make the prediction without including sex Nonlinear model: use residuals Contravenes the law? (Uses sex in building model) Confusion in scorecard construction - Hand 27

28 Recall: Indirect discrimination arises when an apparently neutral provision, criterion or practice would put persons having a particular racial or ethnic origin, religion or belief, sex, disability, age or sexual orientation at a particular disadvantage compared with other people By failing to take account of the known fact that someone is female, they are charged a higher rate than a male with the same (best estimate) probability of defaulting Confusion in scorecard construction - Hand 28

29 As if all that wasn t enough Last year the European Court of Justice ruled that it was unlawful sex discrimination for insurers to distinguish between men and women when deciding their premiums despite the fact that (for example) women are safer drivers than men... etc Confusion in scorecard construction - Hand 29

30 Context 1) People should be treated as individuals rather than stereotyped But probability estimates must be based on aggregation 2) Modern society is uneasy about letting some characteristics be used, even if they are predictive 3) It is intrinsic to civilised society that some risks subsidise others e.g. no fault risks - inherited diseases and the NHS Confusion in scorecard construction - Hand 30

31 The law of unintended consequences Suppose we equalise driving insurance cost at a weighted mean of current male and female values - then more higher risks will be encouraged onto the roads - increasing the risk to all of us On the other hand previously they may have driven without insurance At least these higher risks are now buying insurance And this might compensate for the lower risks who, now having to pay more, decide to drive without insurance Confusion in scorecard construction - Hand 31

32 Conclusions: - opening example of how bad model can lead to very misleading conclusions - nature of credit scoring models and data - legal anti-discrimination issues - the law seems to be confused - implementation contradicting spirit - insurance, context, and the law of unintended consequences Confusion in scorecard construction - Hand 32

33 Some references: Andreeva G (2011) The latest equality and anti-discrimination legal developments: implications for credit scoring. Edinburgh University Business School Chan and Seow (2012?) Legally scored. To appear in the Journal of Financial Regulation and Compliance. Dietrich J and Johannsson H (2005) Searching for age and gender discrimination in mortgage lending. OCC Economics Working Paper, Office of the Comptroller of Currency, US Treasury. Equality and Human Rights Commission Financial Services Inquiry: Follow-up report. Hand DJ (2012) Credit scoring, insurance, and discrimination. To appear in Proceedings of the Conference on Statistics, Science and Public Policy XVI: Risks, Rights and Regulations, April 17-20, Confusion in scorecard construction - Hand 33

34 thank you! Confusion in scorecard construction - Hand 34