Multi-criteria Analysis for Impact Assessment

Transcription

1 Multi-criteria Analysis for Impact Assessment The Maximum Likelihood Approach Michaela Saisana European Commission Joint Research Centre Econometrics and Applied Statistics Unit 1

2 Outline Applications of MCA and CBA Cost Benefit Analysis (+ limitations) Roots of MCA in Social Choice Theory 5 methods (Relative majority, Condorcet, Borda, Successive eliminations, Median ranking) Limitations of the Weighted Sum (most common approach) Weights as importance coefficients (BA and AHP) MCA: Maximum likelihood approach (steps, suitability) Sensitivity Analysis of MCA result Conclusion 2

3 Some Decision or Evaluation Problems Locating a new plant Human resources management Evaluating projects Selecting an investment strategy Electricity production planning Regional planning Evaluation of urban waste management systems Environmental applications Health Risk Prediction Systemic Risk Assessment ( JRC collaboration with the European Systemic Risk Board European Central Bank) 3

4 4

5 Basic steps of cost-benefit analysis (CBA) 1. Determine if CBA is worth doing 2. Identify objectives and policy alternatives 3. Determine stakeholders 4. Identify costs and benefits of each alternative 5. Sort into measurable and non-measurable costs and benefits 6. Estimate costs and benefits that can be measured in monetary terms 7. Conduct sensitivity analysis 8. Compare costs-benefits across alternatives 9. Adjust for non-measurable costs and benefits(?) 10. Make a decision 5

6 Cost-benefit guidelines UK Department of the Treasury, Appraisal and Evaluation in Central Government (The Green Book), London:2002, NZ Treasury guidelines Australian Government, Office of Best Practice Regulation, (see especially Handbook of Cost- Benefit Analysis, and Best Practice Regulation Handbook) Queensland Government, Department of Infrastructure and Planning, Cost Benefit Analysis, Government of Western Australia, Department of Treasury and Finance, 2005, Project Evaluation Guidelines, 6

7 Limitations of CBA Results often highly sensitive to specific assumptions, such as discount rate Difficult to balance non-quantifiable costs/benefits against quantifiable ones Anthropocentric in its underlying social vision How much is life, education (literacy), welfare, health, ecological sustainability, employment (business confidence) worthy? 7

8 Multi Criteria Analysis (MCA) - Definition Multi Criteria Analysis is a decision-making tool, developed for complex multi-criteria problems that include quantitative and/or qualitative aspects of the problem in the decision making process. (Center for International Forestry Research, CIFOR, 1999) 8

9 MCA - Steps 1. Establish the decision context 2. Identify the performance criteria and the options 3. Describe/rate the performance of each option against the criteria 4. Assign weights across criteria 5. Combine the information to obtain a ranking of the options 6. Examine the results and review 7. Conduct sensitivity analysis 8. Final decisions 9

10 MCA - Performance matrix Criteria should not be dependant on each other and not redundant (to avoid double counting) Criterion 1 (/20) Criterion 2 (rating) Criterion 3 (qual.) Criterion 4 (Y/N) Action G Yes Action B Yes Action VG No Action VB No Action G Yes 10

11 MCA - Performance matrix Who decides the ratings? MCA very flexible wrt who gets a say in either the criteria or rating the options: Democratic decision-making - all members of the decision-making body, or each organizational branch/unit, independently allowed to rate options Panel of experts asked to make judgments; can use different panel to judge different criteria Consensus model - decision-making body thrash it out Stakeholder inclusion Different groups can rate options on different criteria 11

12 MCA - Result The outcome of MCA can be used to: Identify a single, most-preferred option Rank options Short-list a limited number of options for subsequent detailed appraisal through other methods such as CBA Distinguish acceptable from unacceptable options Combine different options based on relative strengths 12

13 Social Choice Theory Problem: A group of voters have to select a candidate among a group of candidates (election) Each voter has a personal ranking of the candidates according to his/her preferences Which candidate must be elected? Best interest of society Analogy with multi-criteria analysis: Candidates actions Voters criteria What is the «best» voting procedure? 13

14 Social Choice Theory Social choice theory methods would be ideally suited for assessing multiple options through multiple criteria and were already available between the end of the XIII and the XV century, 14

15 Ramon Llull (ca ca. 1315) proposed first what would then become known as the method of Condorcet. 2. Nicolas de Condorcet, ( ) His Sketch for a Historical Picture of the Progress of the Human Spirit (1795) can be considered as an ideological foundation for evidence based policy (modernity at its best!). 3. Nicholas of Kues ( ), also referred to as Nicolaus Cusanus and Nicholas of Cusa developed what would later be known as the method of Borda. 4. Jean-Charles, chevalier de Borda ( ) developed the Borda count. 15

16 Five methods (among many others) 1. Relative majority 2. Condorcet 3. Borda 4. Successive eliminations 5. Median ranking 16

17 Method 1 : Relative majority 3 candidates: Adam, Brian, Carlos 30 voters: voters voters voters A B C B C B C A A A 11 B 10 C 9 Adam is elected 17

18 Method 1 : Relative majority 3 candidates: Adam, Brian, Carlos 30 voters: voters voters voters A B C B C B C A A Problem: B and C preferred to A by a majority of voters! A 11 B 10 C 9 Adam is elected 18

19 Method 2 : Condorcet 3 candidates: Adam, Brian, Carlos 30 voters: voters voters voters A B C B C B C A A B preferred to A B preferred to C C preferred to A Brian is elected 19 votes 21 votes 19 votes 19

20 Method 2 : Condorcet 3 candidates: Adam, Brian, Carlos 9 voters: voters voters voters A B C Problem: Nobody is elected! (cycle) A preferred to B B preferred to C 6 votes 7 votes B C A C A B C preferred to A 5 votes 20

21 Method 3 : Borda 3 candidates: Adam, Brian, Carlos 11 x x 1 31 x x 1 81 voters: voters voters voters voters voter voter A C C B A B C A B A B C B B A C C A Points Scores A 101 B 33 C 109 Carlos is elected! 39 x x 1 21

22 Method 3 : Borda 4 candidates: Adam, Brian, Carlos, David 7 voters: voters voters voters C B A B A D A D C D C B Points Adam is elected Scores A 13 B 12 C 11 D 6 Ranking A B C D 22

23 Method 3 : Borda 4 candidates: Adam, Brian, Carlos, David 7 voters: Problem: Fully Dependant on irrelevant alternatives (easy to manipulate) voters voters voters C B A Points 2 Scores A 6 Ranking C B A C 1 B 7 B A C B 0 C 8 A Carlos is elected 23

24 Method 4 : Successive eliminations 3 candidates: Adam, Brian, Carlos 11 voters: voters voters voters A C C C A B B B A Ranking A C B A tour-wise procedure, whereby the worst candidate (most voted in the last position) is eliminated progressively until one is left. 24

25 Method 5 : Median ranking 3 candidates: Adam, Brian, Carlos 11 voters: Ranking of candidates for each voter Median rank for each candidate across voters voters voters voters A C C C A B B B A A: B: C: Ranking A C B 25

26 5 candidates: Adam, Brian, Carlos, David, Edison? 25 voters: voters voters voters voters voters A B E D C C D C E E D C D B D B E B C B E A A A A Relative majority Adam elected Condorcet: Carlos elected Borda: David elected Successive eliminations: Edison elected Median ranking: Carlos elected 26

27 27

28 Kenneth Arrow (Nobel prize in economy, 1972) Impossibility theorem (1952): With at least 2 voters and 3 candidates, it is impossible to build a voting procedure that simultaneously satisfies the 5 following properties: Non-dictatorship Universality Independence with respect to third parties Monotonicity Non-imposition 28

29 Most common approach: Weighted sum weights I1 (50%) I2 (50%) a b c V(a) = V(b) = V(c) = 50 Problems: 1) Fully compensatory (elimination of conflicts) 29

30 Most common approach: Weighted sum weights I1 (50%) I2 (50%) a b c V(a) = V(b) = 55, V(c) = 50 Problems: 2) Does not encourage improvement in the weak dimensions 30

31 Most common approach: Weighted sum Y = 0.5 X X 2 R 1 2 = 0.08, R 2 2 = 0.83, corr(x 1, X 2 ) = 0.151, V(x 1 ) = 116, V(x 2 ) = 614, V(y) = 162 Problems: 3) Weights are used as if they were importance coefficients while they are trade off coefficients 31

32 Most common approach: Weighted sum Weighted sum approach only possible under special circumstances (eg standardized variables, uniform covariance matrix ) Hence we need to move away from weighted sums Effective weights are compared with nominal weights to ensure coherence between the two. [Paolo Paruolo, Michaela Saisana, Andrea Saltelli, 2013, Ratings and rankings: Voodoo or Science?, J. R. Statist. Soc. A, 176 (3), ] 32

33 MCA: Maximum likelihood Approach Features: no impact of outliers; no need for data normalisation; no need for uniform covariance matrix; no need to attach monetary value and use of both continuous and categorical variables; no use of any linear or multiplicative formula; use of the weights attached to the indicators as importance coefficients; a compromise between conflicting opinions; reasonably resistant to manipulation; produces a ranking that is statistically optimal (anonymous, neutral, Pareto optimal, satisfies reinforcement and local independence of irrelevant alternatives) [Kemeny (1959), Young and Levenglick (1978)] Led to: Condorcet-Kemeny-Young-Levenglick (C-K-Y-L) ranking procedure 33

34 MCA - Performance matrix Criteria should not be dependant on each other and not redundant (to avoid double counting) Criterion 1 (20%) Criterion 2 (30%) Criterion 3 (20%) Criterion 4 (30%) Action G Yes Action B Yes Action VG No Action VB No Action G Yes Where do weights come from? ( next couple of slides) 34

35 Consumption Access Stability Nutrition Quality Quality Availability Access Status Access Quality Toilet Facilities Waste Management Practices Quality Facilities Energy Quality Availability Access Tenure Quality Inputs Skills Services Assets Exposure Coping ability Recovery ability Food Education Healthcare Michaela Saisana Weights based on Budget Allocation (42 experts) In 4 dimensions of poverty, the average expert weight is similar to equal weighting Tiredness in filling in the questionnaire on weights?? Education Farm Assets Exposure & Resilience to Shocks Gender Equality 0 35

36 Weights based on Analytic Hierarchy Process USING PAIRWISE COMPARISONS, THE RELATIVE IMPORTANCE OF ONE CRITERION OVER ANOTHER CAN BE EXPRESSED 1 EQUAL 3 MODERATE 5 STRONG 7 VERY STRONG 9 EXTREME Questionnaire Which Indicator Do You Feel Is More Important? To What Degree? Patents vs. x Royalties x Patents vs. Internet x x x Patents vs. Technology exports x x Patents vs. Telephones x x Patents vs. Electricity x Patents vs. Schooling x years x Patents vs. x University Students x Royalties vs. Internet x x Royalties vs. Technology x exports x Royalties vs. Telephones x x x Royalties vs. Electricity x Royalties vs. Schooling x years x Royalties vs. x University Students x Internet vs. x Technology exports x Internet vs. Telephones x x x Internet vs. Electricity x Internet vs. Schooling years x x Internet vs. x University Students x Technology vs. Telephones x exports x x Technology exports vs. Electricity x Technology exports vs. Schooling x years x Technology exports vs. x University Students x Telephones vs. Electricity x x Telephones vs. Schooling x years x Telephones vs. x University Students x Electricity vs. x Schooling years x Electricity vs. x University Students x Schooling years vs. University Students x x Patents Royalties Internet Tech.Exports Telephones Electricity Schooling University St. Patents 1 1/ /6 1/8 Royalties / /3 1/4 Internet 1/5 1/3 1 1/ /7 1/6 Tech.Exports 1/ /4 1/5 Telephones 1/3 1/5 1/2 1/ /9 1/9 Electricity 1/9 1/9 1/2 1/9 1/7 1 1/9 1/9 Schooling University St /2 1 solve for the Eigenvector Weights Patents Royalties Internet hosts Tech exports Telephones Electricity Schooling University st Inconsistency 17.4 % 36

37 Weights based on Analytic Hierarchy Process USING PAIRWISE COMPARISONS, THE RELATIVE IMPORTANCE OF ONE CRITERION OVER ANOTHER CAN BE EXPRESSED 1 EQUAL 3 MODERATE 5 STRONG 7 VERY STRONG 9 EXTREME Patents Royalties Internet Tech.Exports Telephones Electricity Schooling University St. Patents 1 1/ /6 1/8 Royalties / /3 1/4 Internet 1/5 1/3 1 1/ /7 1/6 Tech.Exports 1/ /4 1/5 Telephones 1/3 1/5 1/2 1/ /9 1/9 Electricity 1/9 1/9 1/2 1/9 1/7 1 1/9 1/9 Schooling University St /2 1 P=5I R=3I We expect: P > R Expert said: R > P (R=3P) Inconsistency 37

38 MCA: Maximum likelihood Approach Step 1 - Input matrix to the multicriteria analysis Example: Three options need to be ranked according to five criteria. The importance of the criteria is reflected in the respective weights. Performance matrix Criterion 1 Criterion 2 Criterion 3 Criterion 4 Criterion 5 Weights 10% 20% 10% 30% 30% Option A Option B Option C

39 MCA: Maximum likelihood Approach Step 2 Options are compared pairwise For each comparison, e.g. option A versus option B, all the weights corresponding to the criteria that favour A versus B are added up (abbreviated as AB). In this case AB gets the weight of Criterion 2 only (=0.2). The comparison BA gets the sum of the weights of the remaining criteria: 1, 3, 4, 5 (=0.8). For n options, there are n (n-1) comparisons to be made. All the values from the pairwise comparisons are entered in a so called outranking matrix. Outranking matrix Option A Option B Option C Option A Option B Option C

40 MCA: Maximum likelihood Approach Step 3 Calculate support for all permutations and select the maximum All 3! (=6) permutations of the options are considered and the support score for each ranking is calculated. ABC has a support of 1.6 (= ), which is the sum of elements above the diagonal in the outranking matrix. Support scores for all six rankings: ABC= 1.6 ACB=1.4 BAC=2.2 BCA=1.6 CAB=0.8 CBA=1.4 The ranking selected is the one with the maximum likelihood score: BAC Outranking matrix Option A Option B Option C Option A Option B Option C

41 MCA: Maximum likelihood Approach Important to assess sensitivity of results to the weights How coupled stairs are shaken in most of available literature How to shake coupled stairs 41

42 Frequency matrix Sensitivity of the final ranking to the assumptions (e.g. weights) 42

43 MCA: Maximum Likelihood Approach The main limitation of this method is the difficulty in computing the ranking when the number of options grows (e.g. 100). For 10 options 10 = 3,628,800 permutations still trivial for today s PCs To solve this NP-hard problem when the number of options is very large there are plenty of numerical algorithms (JRC works on them!) 43

44 Concluding: How to use MCA in your work 1. Decide on the criteria that you want to use in your evaluation; 2. Identify appropriate indicators for each of the criteria (more than one indicator for each criteria is OK); 3. Score the alternatives on each criterion based on their performance on that criterion; 4. Determine the weights of all the criteria (use for instance AHP); 5. Calculate the overall ranking of the alternatives using Maximum Likelihood; 6. Examine the results: try to explain why some options turn out to be the better than others; 7. Do a Sensitivity Analysis: what happens if you assign other weights to the criteria? Does it affect the overall results? 8. Make a final decision 44

45 The more important issue is whether the (maximum likelihood) method is intuitively easy to grasp, and whether it improves on methods currently in use. On both these counts I think that the answer is affirmative, and I predict that the time will come when it is considered a standard tool for political and group decision making. [Peyton Young, 2002, Optimal Voting Rules, Journal of Economic Perspectives 9:51-64] Peyton Young Professor Emeritus, Research Professor in Economics, Johns Hopkins University 45

46 More reading at 46