Measuring Impact. Impact Evaluation Methods for Policymakers. Sebastian Martinez. The World Bank

Transcription

1 Impact Evaluation Measuring Impact Impact Evaluation Methods for Policymakers Sebastian Martinez The World Bank Note: slides by Sebastian Martinez. The content of this presentation reflects the views of the author, and not necessarily those of the World Bank. December 2007.

2 Measuring Impact 1) Causal Inference Counterfactuals Counterfeit Counterfactuals: Before and After (pre-post) Enrolled-not enrolled (apples and oranges) 2) IE Methods Toolbox: Randomized Controls Randomized Promotion (IV) Discontinuity Design (RDD) Difference in Difference (Diff-in-diff) Matching (P-score matching) 2

3 Impact Evaluation Logical Framework Theory Measuring Impact Identification Strategy Data Operational Plan Resources 3

4 Measuring Impact 1)Causal Inference Counterfactuals Counterfeit Counterfactuals: Before and After (pre-post) Enrolled-not enrolled (apples and oranges) 2) IE Methods Toolbox: Randomized Controls Randomized Promotion (IV) Discontinuity Design (RDD) Difference in Difference (Diff-in-diff) Matching (P-score matching) 4

5 Our Objective: Estimate the CAUSAL effect (impact) of intervention P (program or treatment) on outcome Y (indicator, measure of success) Example: what is the effect of a cash transfer program (P) on household consumption (Y)? 5

6 Causal Inference What is the effect of P on Y? Answer: α= (Y P=1)-(Y P=0) Can we all go home? 6

7 Problem of MISSING DATA α= (Y P=1)-(Y P=0) For a program beneficiary: we observe (Y P=1): Consumption level (Y) with a cash transfer program (P) but we do not observe (Y P=0): Consumption level (Y) without a cash transfer program (P) 7

8 Solution Estimate what would have happened to Y in the absence of P We call this the COUNTERFACTUAL Hint: The key to a good impact evaluation is a valid counterfactual! 8

9 Estimating Impact of P on Y α= (Y P=1)-(Y P=0) OBSERVE (Y P=1) Intention to Treat (ITT) - Those offered treatment Treatment on the Treated (TOT) Those receiving treatment ESTIMATE counterfactual for (Y P=0) Use comparison or control group IMPACT = outcome with treatment - counterfactual 9

10 The perfect Clone Beneficiary Control 6 Candies 4 Candies Impact = 6-4 = 2 Candies 10

11 In reality, use statistics Beneficiary Control Average Y = 6 Candies Average Y = 4 Candies Impact = 6-4 = 2 Candies 11

12 Getting Good Counterfactuals Understand the DATA GENERATION process Behavioral process by which program participation (treatment) is determined How are benefits assigned? What are the eligibility rules? The treated observation and the counterfactual: have identical characteristics, except for benefiting from the intervention Hint: With a good counterfactual, the only reason for different outcomes between treatments and controls is the intervention (P) 12

13 Case Study What is the effect of a cash transfer program (P) on household consumption (Y)? PROGRESA/OPORTUNIDADES Program National anti-poverty program in Mexico Started million beneficiaries by 2004 Eligibility based on poverty index Cash transfers conditional on school and health care attendance Rigorous impact evaluation with rich data 506 communities, 24K households Baseline 1997, follow-up 2008 Many outcomes of interest. Here we consider: Standard of living: consumption per capita 13

14 Case Study Eligibility and Enrollment Ineligibles (Non-Poor) Not Enrolled Eligibles (Poor) Enrolled 14

15 Measuring Impact 1) Causal Inference Counterfactuals Counterfeit Counterfactuals: Before and After (pre-post) Enrolled-not enrolled (apples and oranges) 2) IE Methods Toolbox: Randomized Controls Randomized Promotion (IV) Discontinuity Design (RDD) Difference in Difference (Diff-in-diff) Matching (P-score matching) 15

16 Counterfeit Counterfactuals Two common counterfactuals to be avoided!! Before and After (pre-post) Data on the same individuals before and after an intervention Enrolled-not enrolled (apples and oranges) Data on a group of individuals that enrolled in a program, and another group that did not We don t know why Both counterfactuals may lead to biased results 16

17 Counterfeit Counterfactual #1 Y Before and After A A-C = 2 IMPACT? C (counterfactual) A-B = 4 B T=0 Baseline T=1 Endline Time 17

18 Case 1: Before and After What is the effect of a cash transfer program (P) on household consumption (Y)? 2 Points in Time Measure beneficiaries : Consumption at T=0 Consumption at T=1 Estimate of counterfactual (Y i,t P=0) = (Y i,t-1 P=0) Impact = A-B = 35 Y 268 A 233 B α =35 T=0 T=1 Time 18

19 Case 1: Before and After Case 1 - Before and After Control - Before Treatment - After t-stat Mean Case 1 - Before and After Linear Regression Multivariate Linear Regression Estimated Impact on CPC 35.27** 34.28** (2.16) (2.11) ** Significant at 1% level 19

20 Case 1: Before and After What s the Problem? 2 Points in Time Only measure beneficiaries: Consumption at T=0 Consumption at T=1 Estimate of counterfactual (Y i,t P=0) = (Y i,t-1 P=0) Impact = A-B = 35 Does not control for time varying factors Boom: Impact = A-C A-B = overestimate Recession: Impact = A-D A-B = underestimate Y 268 A Impact α =35 C? Impact 233 B D? T=0 T=1 Time (1997) (1998) 20

21 Measuring Impact 1) Causal Inference Counterfactuals Counterfeit Counterfactuals: Before and After (pre-post) Enrolled-not enrolled (apples and oranges) 2) IE Methods Toolbox: Randomized Controls Randomized Promotion (IV) Discontinuity Design (RDD) Difference in Difference (Diff-in-diff) Matching (P-score matching) 21

22 Counterfeit Counterfactual #2 Enrolled-not enrolled Post-treatment data on 2 groups Enrolled: treatment group Not-enrolled: control group (counterfactual) Those ineligible to participate Those that choose NOT to participate Selection Bias Reason for not enrolling may be correlated with outcome (Y) Control for observables But not unobservables!! Estimated impact is confounded with other things 22

23 Case 2: Enrolled- not enrolled Measure outcomes in post-treatment (1998) Ineligibles (Non-Poor) Not Enrolled Eligibles (Poor) Y = 290 Enrolled Y = 268 In what ways might enrolled/not enrolled be different, other than program? 23

24 Case 2: Enrolled- not enrolled Case 2 - Enrolled/Not Enrolled Not Enrolled Enrolled t-stat Mean CPC Case 2 - Enrolled/Not Enrolled Linear Regression Multivariate Linear Regression Estimated Impact on CPC -22.7** (3.78) (4.05) ** Significant at 1% level 24

25 Case Study What is going on?? Linear Regression Case 1 - Before and After Multivariate Linear Regression Case 2 - Enrolled/Not Enrolled Multivariate Linear Regression Linear Regression Estimated Impact on CPC 35.27** 34.28** -22.7** (2.16) (2.11) (3.78) (4.05) ** Significant at 1% level Which of these do we believe? Problem with Before-After: Can not control for other time-varying factors Problem with Enrolled-Not Enrolled: Do no know if other factors, beyond the intervention, are affecting the outcome 25

26 Measuring Impact 1) Causal Inference Counterfactuals Counterfeit Counterfactuals: Before and After (pre-post) Enrolled-not enrolled (apples and oranges) 2)IE Methods Toolbox: Randomized Controls Randomized Promotion (IV) Discontinuity Design (RDD) Difference in Difference (Diff-in-diff) Matching (P-score matching) 26

27 Choosing your methods.. To identify an IE method for your program, consider: Prospective/retrospective Eligibility rules Roll-out plan (pipeline) Is universe of eligibles larger than available resources at a given point in time? Budget and capacity constraints? Excess demand for program? Eligibility criteria? Geographic targeting? Etc. Hint: Choose the most robust strategy that fits the operational context 27

28 Choosing your methods Identify the best possible design given the operational context Best design = fewest risks for contamination Have we controlled for everything? Internal validity Is the result valid for everyone? External validity Local versus global treatment effect 28

29 Measuring Impact 1) Causal Inference Counterfactuals Counterfeit Counterfactuals: Before and After (pre-post) Enrolled-not enrolled (apples and oranges) 2)IE Methods Toolbox: Randomized Controls Randomized Promotion (IV) Discontinuity Design (RDD) Difference in Difference (Diff-in-diff) Matching (P-score matching) 29

30 Randomized Controls When universe of eligibles > # benefits: Randomize! Lottery for who is offered benefits Fair, transparent and ethical way to assign benefits to equally deserving populations Oversubscription: Give each eligible unit the same chance of receiving treatment Compare those offered treatment with those not offered treatment (controls) Randomized phase in: Give each eligible unit the same chance of receiving treatment first, second, third. Compare those offered treatment first, with those offered treatment later (controls) 30

31 Randomization 1. Universe 2. Random Sample of Eligibles External Validity 3. Randomize Treatment Internal Validity Ineligible = Not Enrolled = Eligible = Enrolled = 31

32 Unit of Randomization Choose according to type of program: Individual/Household School/Health Clinic/catchment area Block/Village/Community Ward/District/Region Keep in mind: Need sufficiently large number of units to detect minimum desired impact. Spillovers/contamination Operational and survey costs Hint: As a rule of thumb, choose to randomize at the minimum viable unit of implementation. 32

33 Case 3: Randomization Oportunidades Evaluation Sample Unit of randomization: community Random phase in: 320 treatment communities (14,446 households) First transfers distributed April control communities (9,630 households) First transfers November

34 Case 3: Baseline Balance Variables Consumption per capita Treatment (4,670) RANDOMIZATION Control (2727) t-stats Head's age Head's education Spouse's age Spouse's education Speaks an indigenous language Head is female Household at baseline Bathroom at baseline Total hectareas of land Min. Distance locurban

35 Case 3: Randomization Mean CPC Baseline Mean CPC Followup Case 3 - Randomization Control Treatment t-stat Case 3 - Randomization Linear Regression Multivariate Linear Regression Estimated Impact on CPC 29.25** 29.79** (3.03) (3.00) ** Significant at 1% level 35

36 Case Study Case 1 - Before and After Multivariate Linear Regression Case 2 - Enrolled/Not Enrolled Multivariate Linear Regression Case 3 - Randomization Multivariate Linear Regression Estimated Impact on CPC 34.28** ** (2.11) (4.05) (3.00) ** Significant at 1% level 36

37 Measuring Impact 1) Causal Inference Counterfactuals Counterfeit Counterfactuals: Before and After (pre-post) Enrolled-not enrolled (apples and oranges) 2) IE Methods Toolbox: Randomized Controls Randomized Promotion (IV) Discontinuity Design (RDD) Difference in Difference (Diff-in-diff) Matching (P-score matching) 37

38 Randomized Promotion (IV) Common scenarios: National Program with universal eligibility Voluntary inscription in program Can we compare enrolled to not enrolled? Selection Bias! 38

39 Randomized Promotion (IV) Possible solution: provide additional promotion, encouragement or incentives to a random subsample: Information Encouragement (small gift or prize) Transport Other help/incentives Necessary conditions: 1. Promoted and non-promoted groups are comparable: Promotion not correlated with population characteristics Guaranteed by randomization 2. Promoted group has higher enrollment in the program 3. Promotion does not affect outcomes directly 39

40 Randomized Promotion Universal Eligibility Randomize Promotion Enrollment No Promotion Promotion Eligible = Enroll = Never Always 40

41 Randomized Promotion Promoted NOT Promoted IMPACT Enrolled = 80% Y = 100 Enrolled = 30% Y = 80 Enrolled= 0.5 Y=20 Never Enroll Impact = 40 Enroll if Encouraged Always Enroll 41

42 Examples Maternal Child Health Insurance in Argentina Intensive information campaigns Employment Program in Argentina Transport voucher Community Based School Management in Nepal Assistance from NGO Health Risk Funds in India Assistance from Community Resource Teams 42

43 Randomized Promotion Pilot test promotion strategy vigorously! Produces additional information of interest: How to increase enrollment Don t have to exclude anyone, but.. Strategy depends on success and validity of promotion Produces a local average treatment effect Randomized Promotion is an Instrumental Variable (IV) A variable correlated with treatment but nothing else (i.e. random promotion) More details in the appendix 43

44 Case 4: IV Randomized Treatment (Promoted) Enrolled = 92% Y = 268 Randomized Control Enrolled = 0% Y = 239 IMPACT Enrolled= 0.92 Y=29 TOT Impact = 31 Never Enroll Enroll if Encouraged 44

45 Case 4: IV - TOT Estimate TOT effect of Oportunidades on consumption Run 2SLS regression Linear Regression Case 4 - IV Multivariate Linear Regression Estimated Impact on CPC 29.88** 30.44** (3.09) (3.07) ** Significant at 1% level 45

46 Measuring Impact 1) Causal Inference Counterfactuals Counterfeit Counterfactuals: Before and After (pre-post) Enrolled-not enrolled (apples and oranges) 2) IE Methods Toolbox: Randomized Controls Randomized Promotion (IV) Discontinuity Design (RDD) Difference in Difference (Diff-in-diff) Matching (P-score matching) 46

47 Discontinuities in Eligibility Social programs many times target programs according to an eligibility index: Anti-poverty programs: targeted to households below a given poverty index Pension programs: targeted to population above a certain age Scholarships: targeted to students with high scores on standardized test Hint: For a discontinuity design, you need: -Continuous eligibility index -Clearly defined eligibility cut-off 47

48 Example: Eligibility index (score) from 1 to 100 Based on pre-intervention characteristics Score <=50 are eligible Score >50 are not eligible Offer treatment to eligibles 48

49 Regression Discontinuity Design - Baseline Outcome Score 49

50 Regression Discontinuity Design - Baseline Outcome Eligible Not Eligible Score 50

51 Regression Discontinuity Design - Post Intervention Outcome Score 51

52 Regression Discontinuity Design - Post Intervention Outcome IMPACT Score 52

53 Case 5: Discontinuity Design Oportunidades assigned benefits based on a poverty index Where Treatment = 1 if score <=750 Treatment = 0 if score >750 53

54 Case 5: Discontinuity Design Baseline No treatment Fitted values puntaje estimado en focalizacion yi = β0 + β1 Treatmenti + δ( score) + εi 54 2

55 Case 5: Discontinuity Design Treatment Period Fitted values puntaje estimado en focalizacion Estimated Impact on CPC ** Significant at 1% level Case 5 - Regression Discontinuity Multivariate Linear Regression 30.58** (5.93) 55

56 Potential Disadvantages of RD Local average treatment effects not always generalizable Power: effect is estimated at the discontinuity, so we generally have fewer observations than in a randomized experiment with the same sample size Specification can be sensitive to functional form: make sure the relationship between the assignment variable and the outcome variable is correctly modeled, including: Nonlinear Relationships Interactions 56

57 Advantages of RD for Evaluation RD yields an unbiased estimate of treatment effect at the discontinuity Can many times take advantage of a known rule for assigning the benefit that are common in the designs of social policy No need to exclude a group of eligible households/individuals from treatment 57

58 Measuring Impact 1) Causal Inference Counterfactuals Counterfeit Counterfactuals: Before and After (pre-post) Enrolled-not enrolled (apples and oranges) 2) IE Methods Toolbox: Randomized Controls Randomized Promotion (IV) Discontinuity Design (RDD) Difference in Difference (Diff-in-diff) Matching (P-score matching) 58

59 Diff-in-Diff Compare change in outcomes between treatments and non-treatment Impact is the difference in the change in outcomes Impact = (Y t1 -Y t0 ) - (Y c1 -Y c0 ) 59

60 Outcome Treatment Group A C B D Average Treatme nt Effect Control Group Treatment Time 60

61 Outcome Treatment Group Average Treatment Effect Estimated Average Treatmen t Effect Control Group Treatment Time 61

62 Diff in diff Fundamental assumption that trends (slopes) are the same in treatments and controls Need a minimum of three points in time to verify this and estimate treatment (two pre-intervention) 62

63 Case 6: Diff-in-Diff Case 6 - Diff in Diff Not Enrolled Enrolled t-stat Mean CPC Case 6 - Diff in Diff Linear Regression Multivariate Linear Regression Estimated Impact on CPC 27.66** 25.53** (2.68) (2.77) ** Significant at 1% level 63

64 Case Study Case 1 - Before and After Case 2 - Enrolled/Not Enrolled Case 3 - Randomization Case 4 - IV (TOT) Case 5 - Regression Discontinuity Case 6 - Diff in Diff Multivariate Linear Regression Multivariate Linear Regression Multivariate Linear Regression 2SLS Multivariate Linear Regression Multivariate Linear Regression Estimated Impact on CPC 34.28** ** 30.44** 30.58** 25.53** (2.11) (4.05) (3.00) (3.07) (5.93) (2.77) ** Significant at 1% level 64

65 Measuring Impact 1) Causal Inference Counterfactuals Counterfeit Counterfactuals: Before and After (pre-post) Enrolled-not enrolled (apples and oranges) 2) IE Methods Toolbox: Randomized Controls Randomized Promotion (IV) Discontinuity Design (RDD) Difference in Difference (Diff-in-diff) Matching (P-score matching) 65

66 Matching Pick up the ideal comparison that matches the treatment group from a larger survey. The matches are selected on the basis of similarities in observed characteristics This assumes no selection bias based on unobservable characteristics. Source: Martin Ravallion 66

67 Propensity-Score Matching (PSM) Controls: non- participants with same characteristics as participants In practice, it is very hard. The entire vector of X observed characteristics could be huge. Rosenbaum and Rubin: match on the basis of the propensity score= P(Xi) = Pr (Di=1 X) Instead of aiming to ensure that the matched control for each participant has exactly the same value of X, same result can be achieved by matching on the probability of participation. This assumes that participation is independent of outcomes given X. 67

68 Steps in Score Matching 1. Representative & highly comparables survey of non-participants and participants. 2. Pool the two samples and estimated a logit (or probit) model of program participation. 3. Restrict samples to assure common support (important source of bias in observational studies) 4. For each participant find a sample of nonparticipants that have similar propensity scores 5. Compare the outcome indicators. The difference is the estimate of the gain due to the program for that observation. 6. Calculate the mean of these individual gains to obtain the average overall gain. 68

69 Density Density of scores for participants Region of common support 0 1 Propensity score 69

70 PSM vs an experiment Pure experiment does not require the untestable assumption of independence conditional on observables PSM requires large samples and good data 70

71 Lessons on Matching Methods Typically used when neither randomization, RD or other quasi-experimental options are not possible (i.e. no baseline) Be cautious of ex-post matching Matching on endogenous variables Matching helps control for OBSERVABLE heterogeneity Matching at baseline can be very useful: Estimation: combine with other techniques (i.e. diff in diff) Know the assignment rule (match on this rule) Sampling: selecting non-randomized evaluation samples Need good quality data Common support can be a problem 71

72 Case 7: P-Score Matching Case 7 - PROPENSITY SCORE: Pr(treatment=1) Variable Coef. Std. Err. Age Head Educ Head Age Spouse Educ Spouse Ethnicity Female Head _cons P-score Quintiles Quintile 1 Quintile 2 Quintile 3 Quintile 4 Quintile 5 Xi T C t-score T C t-score T C t-score T C t-score T C t-score Age Head Educ Head Age Spouse Educ Spouse Ethnicity Female Head

73 Case 7: P-Score Matching Case 7 - Matching Linear Regression Multivariate Linear Regression Estimated Impact on CPC (3.59) (3.65) ** Significant at 1% level, + Significant at 10% level 73

74 Case Study: Results Summary Case 1 - Before and After Multivariate Linear Regression Case 2 - Enrolled/Not Enrolled Multivariate Linear Regression Case 3 - Randomization Multivariate Linear Regression Case 4 - IV (TOT) 2SLS Case 5 - Regression Discontinuity Multivariate Linear Regression Case 6 - Diff in Diff Multivariate Linear Regression Case 7 - Matching Multivariate Linear Regression Estimated Impact on CPC 34.28** ** 30.44** 30.58** 25.53** (2.11) (4.05) (3.00) (3.07) (5.93) (2.77) (3.65) ** Significant at 1% level 74

75 Methods Summary Randomization Randomized Promotion IV Discontinuity Design Diff-in-Diff Matching Internal Validity External Validity Risks 75

76 Measuring Impact 1) Causal Inference Counterfactuals Counterfeit Counterfactuals: Before and After (pre-post) Enrolled-not enrolled (apples and oranges) 2) IE Methods Toolbox: Randomized Controls Randomized Promotion (IV) Discontinuity Design (RDD) Difference in Difference (Diff-in-diff) Matching (P-score matching) Combinations of the above 76

77 Remember.. Objective of impact evaluation is to estimate the CAUSAL effect of a program on outcomes of interest In designing the program we must understand the data generation process behavioral process that generates the data how benefits are assigned Fit the best evaluation design to the operational context 77

78 Appendix 1: Two Stage Least Squares (2SLS) Model with endogenous Treatment (T): y = + T + x+ α β β ε 1 2 Stage 1: Regress endogenous variable on the IV (Z) and other exogenous regressors T = δ + δ x+ θ Z + τ Calculate predicted value for each observation: T hat 78

79 Appendix 1 Two stage Least Squares (2SLS) Stage 2: Regress outcome y on predicted variable (and other exogenous variables) Need to correct Standard Errors (they are based on T hat rather than T) In practice just use STATA - ivreg Intuition: T has been cleaned of its correlation with ε. ^ y = α + β ( T) + β x+ ε