DIME WORKSHOP OCTOBER 13-17, 2014 LISBON, PORTUGAL

Transcription

1 DIME WORKSHOP OCTOBER 13-17, 2014 LISBON, PORTUGAL

2 Impact Evaluation Workshop Experimental Methods Daniel Stein Economist, DIME June 2014 Kigali, Rwanda

3 What we will learn... Impact Evaluation looks to asses the causal impact of a project To assess causality of a project, we must gather data from a control group Randomly selecting people into a treatment and control group is the gold standard for causal inference Opportunities for randomization abound, even in large infrastructure projects!

4 Randomization? That s Not For Me Randomization can t offer all questions, but it can be a useful tool in many types of projects Maybe you can t randomize the placement of water connections, but you might be able randomize: Access price Maintenance Contracts Monitoring Etc Randomization allows clear answers to YOUR questions!

5 Overview of the Presentation Control groups and Causality Selection Bias and Randomization Opportunities for Randomization Sampling

6 Room for improvement: before-after comparison Historically, many projects measure impact by looking at project indicators before and after the project implementation This is not good enough! Many things change over time naturally Impact evaluation seeks to improve on this strategy

7 Using monitoring For impact Before Forest Cover (km^2) After Treatment Group Is this the impact of the program? 7

8 up3/results_and_discussion

9 Before-After Comparison In most cases, comparing populations before and after a project is not a good measure of impact! Lots of things can affect indicators over time that have nothing to do with the project General economic growth/recession Weather World prices of commodities Need a measurement of the counterfactual: what would have happened in the absence of the project, with everything else the same

10 Impact Evaluation and Causality Impact evaluation seeks to understand the causal effect of a program Separate the impact of the program from other factors Need to find out what would have happened without the program, or with an alternative strategy 10

11 The Value of a Control Group Control Group Treatment Group (+) Impact of other (external) factors (+) Impact of the program Before After 11

12 Control Group Quality Control Group has to be good! Projects started at specific times and places for particular reasons What is a good control group? By design treatment and comparison have the same characteristics (observed and unobserved), on average Only difference is treatment Control group represents what would have happened to the treatment population if the project has not occurred 12

13 Selection Bias Can we just compare people who received the project to anyone who didn t receive the project? Danger of Selection Bias What was the reason that some people received it and others didn t? Selection bias a major issue for impact evaluation Projects started at specific times and places for particular reasons Participants may select into programs (eligibility criteria) First farmers to adopt a new technology are likely to be very different from the average farmer, looking at their yields will give you a misleading impression of the benefits of a new technology

14 Danger of Selection Bias 1) Village Offered Payment Less Deforestation for Environmental Services OR 2) Forest conservation in the area is politically important Village Offered Payment for Environmental Services Reduced deforestation from other sources

15 How to create Control Group? Need to find a group of non-treated people/(trees?) who can proxy for people who received treatment This is hard: there is normally some reason why some people received treatment and others not, meaning any differences might not be due to the project Unless

16 Randomized Experimental Design Randomization is the best way to create a good control group Randomly assign potential beneficiaries to be in the treatment or comparison group By design treatment and comparison have the same characteristics (observed and unobserved), on average Only difference is treatment With large sample, all characteristics average out 16

17 Steps to Randomizaion 1. Choose sample for impact evaluation These are people who are eligible for project and can be in treatment or control Selection of sample affects external validity only 2. Randomize into treatment and control group This step affects internal validity, allowing you to assess the impact of your project

18 Start with sample of all possible program beneficiaries

19 Choose who will be part of impact evaluation IE Sample: Could be part of treatment of control group Out of IE sample: Ineligible for program, or must be treated

20 Randomize IE sample into treatment and control IE Sample: Randomly selected treatment AND control group Out of IE sample: Ineligible for program, or must be treated

21 Actual Randomization Example: Coin Flip Village A B Treatment Group Control Group

22 Why randomize? To a greater and greater extent, randomization is becoming a necessity to definitively convince audiences of program impact While randomization is not always possible, randomized evaluations are viewed as a different level of quality the non-random ones

23 Can we Randomize? Randomization does not mean denying people the benefits of the project Usually there are constraints within project implementation that allow randomization

24 Opportunities for Randomization Budget constraints prevent full coverage Random assignment (lottery) is fair and transparent Limited implementation capacity Randomized phase-in gives all the same chance to go first No evidence on which alternative is best Random variation in treatment with equal ex ante chance of success Take up of existing program is not complete Encouragement design: Randomly provide information or incentive for some to sign up 24

25 Example: Connection to the Electrical Grid The government of Umbastan wants to undertake a project electrifying rural villages that are close to a major transmission line It has identified 100 villages where the project is feasible and the community would likely benefit What types of randomized designs might be possible?

26 Example: Connection to the Electrical Grid Possible Constraint: The government only has money to fund 50 villages Opportunity for Randomized Assignment: 50 villages to receive project could be randomly selected from 100 eligible This is a fair way to select beneficiaries Other 50 serve as control group

27 Example: Connection to the Electrical Grid Possible Constraint: The government can fund 100 villages eventually, but only has the manpower run the power lines the first year in 50 villages Opportunity for Randomized Phase-In: 50 villages to receive project in the first year This is a fair way to select who gets project in first year Other 50 serve as control group for first year Drawback is that it would be difficult to measure long term effects

28 Example: Connection to the Electrical Grid Possible Constraint: Worry that even if a village is connected, community members won t connect electricity to their house and may not benefit Opportunity for Encouragement Design: Within villages that receive a access to the projects, villagers are randomly offered credit for household connection This show the effect of the credit Also allows us to identify the effect of electrification in general

29 Example: Connection to the Electrical Grid Possible Constraint: There are worries that electricity use will cause damaging debts in the village Opportunity for randomized Variation in Treatment: Households that electrify could randomly receive text messages about their electricity use and costs. Helps figure out how to optimize project design

30 Is Randomization Ethical? No project affects everyone all at oncesomeone is generally left out. Randomization can make the targeting more fair. Structured targeting helps avoid nepotism A good IE can produce better results for everyone in the long run!

31 Thank you