Measuring Impact. Paul Gertler Chief Economist Human Development Network The World Bank. The Farm, South Africa June 2006

Transcription

1 Measuring Impact Paul Gertler Chief Economist Human Development Network The World Bank The Farm, South Africa June 2006

2 Motivation Traditional M&E: Is the program being implemented as designed? Could the operations be more efficient? Are the benefits getting to those intended? Monitoring trends Are indicators moving in the right direction? NO inherent Causality Impact Evaluation: What was the effect of the program on outcomes? Because of the program, are people better off? What would happen if we changed the program? Causality

3 Motivation We are interested in measuring the impact of a program on: the welfare of beneficiaries Perhaps other things as well (nonbeneficiaries, the environment, economy, etc) For example: What is the effect of micro-lending on income? What is the effect of scholarships on school attendance and performance (test scores)? What is the effect of health insurance on child health?

4 Motivation Objective in evaluation is to estimate the CAUSAL effect of intervention X on outcome Y What is the effect of a cash transfer on household consumption? For causal inference we must understand the data generation process For impact evaluation, this means understanding the behavioral process that generates the data how benefits are assigned

5 Causation versus Correlation Recall: correlation is NOT causation Necessary but not sufficient condition Correlation: X and Y are related Change in X is related to a change in Y And. A change in Y is related to a change in X Causation if we change X how much does Y change A change in X is related to a change in Y Not necessarily the other way around

6 Causation versus Correlation Porter s criteria for causation: Independent variable precedes the dependent variable. Independent variable is related to the dependent variable. There are no third variables that could explain why the independent variable is related to the dependent variable External validity Generalizability: causal inference to generalize outside the sample population or setting

7 Motivation The word cause is not in the vocabulary of standard probability theory. Probability theory: two events are mutually correlated, or dependent if we find one, we can expect to encounter the other. Example age and income For impact evaluation, we supplement the language of probability with a vocabulary for causality.

8 Statistical Analysis & Impact Evaluation Statistical analysis: Typically involves inferring the causal relationship between X and Y from observational data Many challenges & complex statistics Impact Evaluation: Retrospectively: same challenges as statistical analysis Prospectively: we generate the data ourselves through the program s design evaluation design makes things much easier!

9 How to assess impact e.g. How much does an education program improve test scores (learning)? What is the beneficiary s test score with the program compared to without the program? Formally, program impact is: α = (Y P=1) - (Y P=0) Compare same individual with and without programs same point in time

10 Solving the evaluation problem Problem: we never observe the same individual with and without program at same point in time Need to estimate what would have happened to the beneficiary if he or she had not received benefits Counterfactual: what would have happened without the program Difference between treated observation and counterfactual is the estimated impact

11 Finding a good counterfactual The treated observation and the counterfactual: have identical factors/characteristics, except for benefiting from the intervention No other explanations for differences in outcomes between the treated observation and counterfactual The only reason for the difference in outcomes is due to the intervention

12 Measuring Impact Impact Evaluation design options: Randomized Experiments Quasi-experiments Regression Discontinuity Difference in difference panel data Other (using Instrumental Variables, matching, etc) In all cases, these will involve knowing the rule for assigning treatment

13 Choosing your design For impact evaluation, we will identify the best possible design given the operational context Best possible design is the one that has the fewest risks for contamination Omitted Variables (biased estimates) Selection (results not generalizable)

14 Case Study Effect of cash transfers on consumption Estimate impact of cash transfer on consumption per capita Make sure: Cash transfer comes before change in consumption Cash transfer is correlated with consumption Cash transfer is the only thing changing consumption Example based on Oportunidades

15 Oportunidades National anti-poverty program in Mexico (1997) Cash transfers and in-kind benefits conditional on school attendance and health care visits. Transfer given preferably to mother of beneficiary children. Large program with large transfers: 5 million beneficiary households in 2004 Large transfers, capped at: $95 USD for HH with children through junior high $159 USD for HH with children in high school

16 Oportunidades Evaluation Phasing in of intervention 50,000 eligible rural communities Random sample of of 506 eligible communities in 7 states - evaluation sample Random assignment of benefits by community: 320 treatment communities (14,446 households) First transfers distributed April control communities (9,630 households) First transfers November 1999

17 Oportunidades Example

18 Counterfeit Counterfactual Number 1 Before and after: Assume we have data on Treatment households before the cash transfer Treatment households after the cash transfer Estimate impact of cash transfer on household consumption: Compare consumption per capita before the intervention to consumption per capita after the intervention Difference in consumption per capita between the two periods is treatment

19 Case 1: Before and After Compare Y before and after intervention α i = (CPC it T=1) - (CPC i,t-1 T=0) CPC Estimate of counterfactual Before After (CPC i,t T=0) = (CPC i,t-1 T=0) A Impact = A-B B t-1 t Time

20 Case 1: Before and After Case 1 - Before and After Control - Before Treatment - After t-stat Mean Case 1 - Before and After Linear Regression Multivariate Linear Regression Estimated Impact on CPC 35.27** 34.28** (2.16) (2.11) ** Significant at 1% level

21 Case 1: Before and After Compare Y before and after intervention α i = (CPC it T=1) - (CPC i,t-1 T=0) CPC Estimate of counterfactual Before After (CPC i,t T=0) = (CPC i,t-1 T=0) A Impact = A-B D? Does not control for time varying factors Recession: Impact = A-C B C? Boom: Impact = A-D t-1 t Time

22 Counterfeit Counterfactual Number 2 Enrolled/Not Enrolled Voluntary Inscription to the program Assume we have a cross-section of postintervention data on: Households that did not enroll Households that enrolled Estimate impact of cash transfer on household consumption: Compare consumption per capita of those who did not enroll to consumption per capita of those who enrolled Difference in consumption per capita between the two groups is treatment

23 Case 2: Enrolled/Not Enrolled Case 2 - Enrolled/Not Enrolled Not Enrolled Enrolled t-stat Mean CPC Case 2 - Enrolled/Not Enrolled Linear Regression Multivariate Linear Regression Estimated Impact on CPC -22.7** (3.78) (4.05) ** Significant at 1% level

24 Those who did not enroll. Impact estimate: α i = (Y it P=1) - (Y j,t P=0), Counterfactual: (Y j,t P=0) (Y i,t P=0) Examples: Those who choose not to enroll in program Those who were not offered the program Conditional Cash Transfer Job Training program Cannot control for all reasons why some choose to sign up & other didn t Reasons could be correlated with outcomes We can control for observables.. But are still left with the unobservables

25 Impact Evaluation Example: Two counterfeit counterfactuals What is going on?? Linear Regression Case 1 - Before and After Multivariate Linear Regression Case 2 - Enrolled/Not Enrolled Multivariate Linear Regression Linear Regression Estimated Impact on CPC 35.27** 34.28** -22.7** (2.16) (2.11) (3.78) (4.05) ** Significant at 1% level Which of these do we believe? Problem with Before-After: Can not control for other time-varying factors Problem with Enrolled-Not Enrolled: Do no know why the treated are treated and the others not

26 Possible Solutions We need to understand the data generation process How beneficiaries are selected and how benefits are assigned Guarantee comparability of treatment and control groups, so ONLY difference is the intervention

27 Measuring Impact Experimental design/randomization Quasi-experiments Regression Discontinuity Double differences (diff in diff) Other options

28 Choosing the methodology.. Choose the most robust strategy that fits the operational context Use program budget and capacity constraints to choose a design, i.e. pipeline: Universe of eligible individuals typically larger than available resources at a single point in time Fairest and most transparent way to assign benefit may be to give all an equal chance of participating randomization

29 Randomization The gold standard in impact evaluation Give each eligible unit the same chance of receiving treatment Lottery for who receives benefit Lottery for who receives benefit first

30 Population Randomization Sample Treatment Group Randomization Control Group

31 External & Internal Validity external validity first-stage sampling ensure sample results represent population at a defined level of sampling error. internal validity second-stage random assignment ensure observed (estimated) difference in the dependent variable is due to the treatment rather than some other confounding factors

32 Case 3: Randomization OPORTUNIDADES Randomized treatment/controls Community level randomization 320 treatment communities 186 control communities Pre-intervention characteristics well balanced

33 Baseline characteristics Variables Treatment (4,670) RANDOMIZATION Control (2727) t-stats Consumption per capita Head's age Head's education Spouse's age Spouse's education Speaks an indigenous language Head is female Household at baseline Bathroom at baseline Total hectareas of land Min. Distance locurban

34 Case 3: Randomization Mean CPC Baseline Mean CPC Followup Case 3 - Randomization Control Treatment t-stat Case 3 - Randomization Linear Regression Multivariate Linear Regression Estimated Impact on CPC 29.25** 29.79** (3.03) (3.00) ** Significant at 1% level

35 Impact Evaluation Example: No Design v.s. Randomization Case 1 - Before and After Multivariate Linear Regression Case 2 - Enrolled/Not Enrolled Multivariate Linear Regression Case 3 - Randomization Multivariate Linear Regression Estimated Impact on CPC 34.28** ** (2.11) (4.05) (3.00) ** Significant at 1% level

36 Does reducing class size improve elementary school education? Project STAR (Student-Teacher Achievement Ratio): 4-year experiment designed to evaluate the effect on learning of small class sizes. Focus of the experiment: 3 different class arrangements for kindergarten through third grade. Treatment levels: 1. Regular class size: students and a single teacher. 2. Small class: students and a single teacher. 3. Teacher s aide: regular-sized class plus a teacher s aide.

37 Does reducing class size improve elementary school education? Each school had at least one class of each type. Students entering kindergarten in a participating school were randomly assigned to one of these three groups. Teachers were also assigned randomly.

38 Y RegAide + u i = β0 + β1smallclassi + β2 Project STAR: Differences Estimates of Effect on Standardized Test Scores of Class Size Treatment Group i i Grade Regressor K Small Class 13.90*** (2.45) 29.78*** (2.83) 19.39*** (2.71) 15.59*** (2.40) Regular Size with aide 0.31 (2.27) 11.96*** (2.65) 3.48 (2.54) (2.27) Intercept *** (1.63) 1,039.39*** (1.78) 1,157.81*** (1.82) 1,228.51*** (1.68) Number of Observations 5,786 6,379 6,049 5,967

39 Does reducing class size improve elementary school education? The estimates presented suggest that: 1. Reducing class size has an effect on test performance, 2. But adding an aide to a regular sized class has a much smaller effect, possible zero. However, the estimates presented ignore both attrition and non-compliance. These two nuisances were high, and hence, the results might be biased.

40 Example Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment Angrist et al. (2002) AER

41 Motivation This paper presents evidence on the impact of one of the largest school voucher programs to date: Programa de Apliacion de Cobertura de la Educacion Secundaria (PACES). Treatment: 125,000 pupils with vouchers covering somewhat more than half the cost of private secondary school. Vouchers were renewable as long as students maintained satisfactory academic performance.

42 Design The authors interviewed 1,600 PACES applicants in 1998, stratifying to obtain approximately equal numbers of winners and losers. For practical reasons, interwieving was limited to the 1995 and 1997 applicant cohorts from Bogota and the 1993 applicant cohort from Jamundi, a suburb of Cali. Telephones were used for the majority of interviews. Approximately 60% response rate. Response is independent of treatment assignment.

43 Dependent variable Personal Characteristics and Voucher Status Bogotá 1995 Bogotá 1997 Jamundi 1993 Loser means Won voucher Loser means Won voucher Loser means Won voucher Age at time of survey 15.0 (1.4) (0.078) 13.2 (1.4) (0.171) 17.2 (1.4) (0.217) Male (0.029) (0.061) (0.077) Mother s highest grade completed 5.9 (2.7) (0.166) 5.9 (2.7) (0.371) 4.4 (2.7) 1.46 (0.494) Father s highest grade completed 5.9 (2.9) (0.199) 5.5 (2.5) (0.388) 5.2 (2.9) (0.640) Mother s age 40.7 (7.3) (0.426) 38.7 (6.6) (0.808) 43.6 (8.8) (1.42) Father s wage 44.4 (8.1) (0.533) 41.9 (7.3) (0.973) 45.5 (9.1) 1.92 (1.61) Father s wage (>2 min wage) (0.021) (0.043) (0.056) N 583 1, Notes: The table reports voucher losers means and the estimated effect of wining a voucher. Numbers in parentheses are standard deviations in columns of means and standard errors in columns of estimated voucher effects.

44 Dependent variable Educational Outcomes and Voucher Status (I) Loser means (1) No controls (2) Bogotá 1995 Basic controls (3) Basic +19 barrio controls (4) Combined sample Basic controls (5) Basic +19 barrio controls (6) Using any scholarship in survey year (0.232) 0.509*** (0.023) 0.504*** (0.023) 0.505*** (0.023) 0.526*** (0.019) 0.521*** (0.019) Ever used a scholarship (0.430) 0.672*** (0.021) 0.663*** (0.022) 0.662*** (0.022) 0.636*** (0.019) 0.635*** (0.019) Started 6th grade in private (0.328) 0.063*** (0.017) 0.057*** (0.017) 0.058*** (0.017) 0.066*** (0.016) 0.067*** (0.016) Started 7th grade in private (0.470) 0.174*** (0.025) 0.168*** (0.025) 0.171*** (0.024) 0.170*** (0.021) 0.173*** (0.021) Currently in private school (0.499) 0.160*** (0.028) 0.153*** (0.027) 0.156*** (0.027) 0.152*** (0.023) 0.154*** (0.023) Highest grade completed 7.5 (0.960) 0.164*** (0.053) 0.130*** (0.051) 0.120*** (0.051) 0.085** (0.041) 0.078** (0.041) Currently in school (0.375) (0.022) (0.020) (0.020) (0.016) (0.016) Sample size 562 1,147 1,577 Notes: The table reports voucher losers means and the estimated effect of wining a voucher. Numbers in parentheses are standard deviations in columns of means and standard errors in columns of estimated voucher effects. *** significant at 1% ** significant at 5% * significant at 10%

45 Dependent variable Educational Outcomes and Voucher Status (II) Loser means (1) No controls (2) Bogotá 1995 Basic controls (3) Basic +19 barrio controls (4) Combined sample Basic controls (5) Basic +19 barrio controls (6) Finished 6th grade (0.232) 0.026** (0.012) 0.023* (0.012) 0.021* (0.011) (0.011) (0.010) Finished 7th grade (excludes Bogotá 97) (0.360) 0.040** (0.020) (0.019) (0.019) (0.018) (0.018) Finished 8th grade (excludes Bogotá 97) (0.483) 0.112*** (0.027) 0.100*** (0.027) 0.094*** (0.027) 0.077*** (0.024) 0.074*** (0.024) Repetitions of 6th grade (0.454) *** (0.024) ** (0.024) ** (0.024) *** (0.019) *** (0.019) Ever repeated after lottery (0.417) *** (0.023) ** (0.023) ** (0.023) *** (0.019) *** (0.019) Total repetitions since lottery (0.508) *** (0.028) ** (0.027) ** (0.027) *** (0.022) *** (0.022) Years in school since lottery 3.7 (0.951) (0.052) (0.050) (0.050) (0.044) (0.043) Sample size 562 1,147 1,577 Notes: The table reports voucher losers means and the estimated effect of wining a voucher. N umbers in parentheses are standard deviations in columns of means and standard errors in columns of estimated voucher effects. *** significant at 1% ** significant at 5% * significant at 10%

46 Variable OLS results (1) Test Results OLS results with covariates (2) RE (3) RE with covariates (4) Sample size (5) Total Points 0.217* (0.116) Math scores (0.120) Reading scores 0.204* (0.115) Writing scores (0.116) 0.205* (0.108) (0.114) 0.203* (0.114) (0.105) Pooled test scores 0.170* (0.095) 0.148* (0.088) 846 Math and 0.192* reading scores (0.101) Robust standard errors are reported in parentheses. *** significant at 1% ** significant at 5% * significant at 10% 0.162* (0.096) 568

47 Measuring Impact Experimental design/randomization Quasi-experiments Regression Discontinuity Double differences (diff in diff) Other options

48 Case 4: Regression Discontinuity Assignment to treatment A score is assigned to person based on a continuous quantifiable and observable index Eligibility is above (below) a known cutoff (discontinuity) of the index Observations are ordered along index Estimated effect is comparing observations just above cutoff to those just below the cutoff Estimated impact around the cutoff may not generalize to entire population

49 Indexes are common in targeting of social programs Anti-poverty programs targeted to households below a given poverty index Pension programs targeted to population above a certain age Scholarships targeted to students with high scores on standardized test CDD Programs awarded to NGOs that achieve highest scores

50 Example: effect of cash transfer on consumption Target transfer to poorest households Construct poverty index from 1 to 100 with pre-intervention characteristics Households with a score <=50 are poor Households with a score >50 are non-poor Cash transfer to poor households Measure outcomes (i.e. consumption) before and after transfer

51 Regression Discontinuity Design - Baseline Outcome Score

52 Regression Discontinuity Design - Baseline Outcome Poor Non-Poor Score

53 Regression Discontinuity Design - Post Intervention Outcome Score

54 Regression Discontinuity Design - Post Intervention Outcome Treatment Effect Score

55 Case 4: Regression Discontinuity Oportunidades assigned benefits based on a poverty index Where Treatment = 1 if score <=750 Treatment = 0 if score >750

56 Case 4: Regression Discontinuity Baseline No treatment Fitted values puntaje estimado en focalizacion yi = β0 + β1 Treatmenti + δ( score) + εi 2

57 Case 4: Regression Discontinuity Treatment Period Fitted values puntaje estimado en focalizacion Estimated Impact on CPC ** Significant at 1% level Case 4 - Regression Discontinuity Multivariate Linear Regression 30.58** (5.93)

58 Potential Disadvantages of RD Local average treatment effects not always generalizable Power: effect is estimated at the discontinuity, so we generally have fewer observations than in a randomized experiment with the same sample size Specification can be sensitive to functional form: make sure the relationship between the assignment variable and the outcome variable is correctly modeled, including: Nonlinear Relationships Interactions

59 Advantages of RD for Evaluation RD yields an unbiased estimate of treatment effect at the discontinuity Can many times take advantage of a known rule for assigning the benefit that are common in the designs of social policy No need to exclude a group of eligible households/individuals from treatment

60 Measuring Impact Experimental design/randomization Quasi-experiments Regression Discontinuity Double differences (Diff in diff) Other options

61 Case 5: Diff in diff Compare change in outcomes between treatments and non-treatment Impact is the difference in the change in outcomes Impact = (Y t1 -Y t0 ) - (Y c1 -Y c0 )

62 Outcome Treatment Group Average Treatment Effect Control Group Treatment Time

63 Outcome Average Treatment Effect Treatment Group Estimated Average Treatmen t Effect Control Group Treatment Time

64 Diff in diff Fundamental assumption that trends (slopes) are the same in treatments and controls Need a minimum of three points in time to verify this and estimate treatment (two pre-intervention)

65 Case 5: Diff in Diff Case 5 - Diff in Diff Not Enrolled Enrolled t-stat Mean ΔCPC Case 5 - Diff in Diff Linear Regression Multivariate Linear Regression Estimated Impact on CPC 27.66** 25.53** (2.68) (2.77) ** Significant at 1% level

66 Impact Evaluation Example Summary of Results Case 1 - Before and After Multivariate Linear Regression Case 2 - Enrolled/Not Enrolled Multivariate Linear Regression Case 3 - Randomization Multivariate Linear Regression Case 4 - Regression Discontinuity Multivariate Linear Regression Case 5 - Diff in Diff Multivariate Linear Regression Estimated Impact on CPC 34.28** ** 30.58** 25.53** (2.11) (4.05) (3.00) (5.93) (2.77) ** Significant at 1% level

67 Measuring Impact Experimental design/randomization Quasi-experiments Regression Discontinuity Double differences (Diff in diff) Other options Instrumental Variables Matching

68 Other options for Impact Evaluation There are a few others out there Common scenario: Voluntary inscription in program Can t control who enrolls and who does not Possible solution: random promotion or incentives into the program Information Money Other help/incentives

69 Random Promotion Those who get promotion are more likely to enroll But who got promotion was determined randomly, so not correlated with other observables/non-observables Compare average outcomes of two groups: promoted/not promoted Effect of offering the program (ITT) Effect of the intervention (TOT) TOT = effect of offering program/proportion of those who took up

70 Example Community Based School Management Chaudhury, Gertler, Vermeersch (work in progress) Estimate effect of decentralization of school management on learning outcomes Grant for funding of community based management Community management of hiring, budgeting, oversight 1500 schools in the evaluation Each community chooses whether to participate in program Community submits proposal for program participation

71 Evaluation Design Community based school management Provision of technical assistance and training by NGOs for submission of grant application Random selection of communities with NGO support Random promotion is an Instrumental Variable

72 Technique called Instrumental Variables Some fancy statistics: Find a variable Z which satisfies two conditions: 1. Correlated with T: corr (Z, T) 0 2. Uncorrelated with ε: corr (Z, ε) = 0 Z is the random promotion in our example

73 Indirect least squares Case 1 Promotion No- Promotion Change Takeup (T) Test Score (S) ΔS 20 Impact P ΔS = Δ = = = 40 ΔT ΔT 0.5 ΔP

74 Indirect least squares Case 2 Promotion No- Promotion Change Takeup (T) Test Score (S) ΔS 10 Impact P ΔS = Δ = = = 20 ΔT ΔT 0.5 ΔP

75 Two Stage Least Squares (2SLS) Model with endogenous Treatment (T): y = + T + x+ α β β ε 1 2 Stage 1: Regress endogenous variable on the IV (Z) and other exogenous regressors T = δ + δ x+ θ Z + τ Calculate predicted value for each observation: T hat

76 Two stage Least Squares (2SLS) Stage 2: Regress outcome y on predicted variable (and other exogenous variables) Need to correct Standard Errors (they are based on T hat rather than T) In practice just use STATA - ivreg Intuition: T has been cleaned of its correlation with ε. ^ y = α + β ( T) + β x+ ε 1 2

77 Instrumental Variables A variable correlated with treatment but nothing else (i.e. random promotion) Again, we really just need to understand how the data are generated Don t have to exclude anyone

78 Case 6: IV Estimate TOT effect of Oportunidades on consumption Run 2SLS regression Linear Regression Case 6 - IV Multivariate Linear Regression Estimated Impact on CPC (3.09) (3.07) ** Significant at 1% level

79 Measuring Impact Experimental design/randomization Quasi-experiments Regression Discontinuity Double differences (Diff in diff) Other options Instrumental Variables Matching

80 Matching Pick up the ideal comparison that matches the treatment group from a larger survey. The matches are selected on the basis of similarities in observed characteristics This assumes no selection bias based on unobservable characteristics. Source: Martin Ravallion

81 Propensity-Score Matching (PSM) Controls: non- participants with same characteristics as participants In practice, it is very hard. The entire vector of X observed characteristics could be huge. Rosenbaum and Rubin: match on the basis of the propensity score= P(Xi) = Pr (Di=1 X) Instead of aiming to ensure that the matched control for each participant has exactly the same value of X, same result can be achieved by matching on the probability of participation. This assumes that participation is independent of outcomes given X.

82 Steps in Score Matching 1. Representative & highly comparables survey of non-participants and participants. 2. Pool the two samples and estimated a logit (or probit) model of program participation. 3. Restrict samples to assure common support (important source of bias in observational studies) 4. For each participant find a sample of nonparticipants that have similar propensity scores 5. Compare the outcome indicators. The difference is the estimate of the gain due to the program for that observation. 6. Calculate the mean of these individual gains to obtain the average overall gain.

83 Density Density of scores for participants Region of common support 0 1 Propensity score

84 PSM vs an experiment Pure experiment does not require the untestable assumption of independence conditional on observables PSM requires large samples and good data

85 PSM vs other QE methods In compsarisons with results of a randomized experiment on a US training program, PSM gave a good estimation Better that non-experimental regressionbased methods studied by Lalonde for the same program Robustness has been questioned.

86 Lessons on Matching Methods Typically used when neither randomization, RD or other quasi-experimental options are not possible (i.e. no baseline) Be cautious of ex-post matching Matching on endogenous variables Matching helps control for OBSERVABLE heterogeneity Matching at baseline can be very useful: Estimation: combine with other techniques (i.e. diff in diff) Know the assignment rule (match on this rule) Sampling: selecting non-randomized evaluation samples Need good quality data Common support can be a problem

87 Case 7: Matching Case 7 - PROPENSITY SCORE: Pr(treatment=1) Variable Coef. Std. Err. Age Head Educ Head Age Spouse Educ Spouse Ethnicity Female Head _cons P-score Quintiles Quintile 1 Quintile 2 Quintile 3 Quintile 4 Quintile 5 Xi T C t-score T C t-score T C t-score T C t-score T C t-score Age Head Educ Head Age Spouse Educ Spouse Ethnicity Female Head

88 Case 7: Matching Case 7 - Matching Linear Regression Multivariate Linear Regression Estimated Impact on CPC (3.59) (3.65) ** Significant at 1% level, + Significant at 10% level

89 Impact Evaluation Example Summary of Results Case 1 - Before and After Multivariate Linear Regression Case 2 - Enrolled/Not Enrolled Multivariate Linear Regression Case 3 - Randomization Multivariate Linear Regression Case 4 - Regression Discontinuity Multivariate Linear Regression Case 5 - Diff in Diff Multivariate Linear Regression Case 6 - IV (TOT) 2SLS Case 7 - Matching Multivariate Linear Regression Estimated Impact on CPC 34.28** ** 30.58** 25.53** 30.44** (2.11) (4.05) (3.00) (5.93) (2.77) (3.07) (3.65) ** Significant at 1% level

90 Measuring Impact Experimental design/randomization Quasi-experiments Regression Discontinuity Double differences (Diff in diff) Other options Instrumental Variables Matching Combinations of the above

91 Remember.. Objective of impact evaluation is to estimate the CAUSAL effect of a program on outcomes of interest In designing the program we must understand the data generation process behavioral process that generates the data how benefits are assigned Fit the best evaluation design to the operational context

92 Remember.. Objective of impact evaluation is to estimate the CAUSAL effect of a program on outcomes of interest In designing the program we must understand the data generation process behavioral process that generates the data how benefits are assigned Fit the best evaluation design to the operational context