Daniel Stein (DIME) Using Randomized Evaluations to Improve Policy Development Impact Evaluation Initiative innovations & solutions in infrastructure, agriculture & environment naivasha, april 23-27, 2011 in collaboration with Africa region, SD network, GAFSP and AGRA 1
What we will learn... Impact Evaluation looks to asses the causal impact of a project To assess causality of a project, we must gather data from a control group Randomly selecting people into a treatment and control group is the gold standard for causal inference Opportunities for randomization abound, even in large infrastructure projects!
Randomization? That s Not For Me There are opportunities for randomization in almost every project Maybe you can t randomize the placement of roads, but you might be able randomize: Access price Maintenance Contracts Monitoring Etc Randomization allows clear answers to YOUR questions!
Overview of the Presentation Control groups and Causality Selection Bias and Randomization Opportunities for Randomization Unit of Randomization Encouragement Design
Using monitoring For impact 14 12 10 8 6 4 2 0 Before Treatment Group After Treatment Group Is this the impact of the program? 5
Impact Evaluation and Causality Impact evaluation seeks to understand the causal effect of a program separate the impact of the program from other factors Need to find out what would have happened without the program, or with an alternative strategy 6
What is Impact Evaluation? Counterfactual analysis Compare same individual with & without subsidy, information etc. at the same point in time to measure the difference This is impossible! The solution: Use a control group Need to identify people that represent what the treatment group would have been like if there was no project
The Value of a Control Group 14 12 Control Group Treatment Group (+) Impact of the program 10 8 (+) Impact of other (external) factors 6 4 2 0 Before After 8
Control Group Quality But Control Group has to be good! Projects started at specific times and places for particular reasons What is a good control group? By design treatment and comparison have the same characteristics (observed and unobserved), on average Only difference is treatment Control group represents what would have happened to the treatment population if the project has not occurred 9
Selection Bias Can we just compare people who received the project to anyone who didn t receive the project? Danger of Selection Bias What was the reason that some people received it and others didn t? Selection bias a major issue for impact evaluation Projects started at specific times and places for particular reasons Participants may select into programs (eligibility criteria) First farmers to adopt a new technology are likely to be very different from the average farmer, looking at their yields will give you a misleading impression of the benefits of a new technology
Danger of Selection Bias 1) Village Electrification Higher Income OR 2) Village is Politically Influential Home Electrification Higher Income from Other Sources
How to create Control Group? Need to find a group of non-treated people who can proxy for people who received treatment This is hard: there is normally some reason why some people received treatment and others not, meaning any differences might not be due to the project Unless
Randomized Experimental Design Randomly assign potential beneficiaries to be in the treatment or comparison group By design treatment and comparison have the same characteristics (observed and unobserved), on average Only difference is treatment With large sample, all characteristics average out Unbiased impact estimates 13
Can we Randomize? Randomization does not mean denying people the benefits of the project Usually there are constraints within project implementation that allow randomization
Opportunities for Randomization Budget constraints prevent full coverage Random assignment (lottery) is fair and transparent Limited implementation capacity Randomized phase-in gives all the same chance to go first No evidence on which alternative is best Random variation in treatment with equal ex ante chance of success Take up of existing program is not complete Encouragement design: Provide information or incentive for some to sign up 15
Example: Irrigation Canal Project The government of Umbastan wants to undertake a project creating irrigation canals to farming communities It has identified 100 villages where the project is feasible and the community would likely benefit What types of randomized designs might be possible?
Example: Irrigation Canal Project Possible Constraint: The government only has money to fund 50 villages Opportunity for Randomized Assignment: 50 villages to receive project could be randomly selected from 100 eligible This is a fair way to select beneficiaries Other 50 serve as control group
Example: Irrigation Canal Project Possible Constraint: The government can fund 100 villages eventually, but only has time to build the canals the first year in 50 villages Opportunity for Randomized Phase-In: 50 villages to receive project in the first year This is a fair way to select who gets project in first year Other 50 serve as control group for first year Drawback is that it would be difficult to measure long term effects
Example: Irrigation Canal Project Possible Constraint: There are worries that there will not be equitable distribution of water within a village Opportunity for randomized Variation in Treatment: 50 villages could receive water meters, and 50 could be organized into water user groups If the best system is unknown ex-ante, randomization can provide evidence for which is best
Example: Irrigation Canal Project Possible Constraint: Worried people will not connect their fields to main canal Opportunity for Encouragement Design: Within villages that receive a canal, farmers are randomly given a course on how to use and benefit from irrigation This show the effect of the training session Also allows us to identify the effect of the irrigation project
Lottery among the qualified Must get the program whatever Randomize who gets the program Not suitable for the program
Unit of Randomization For statistical power, randomizing at the individual level is best Randomizing at higher level sometimes necessary: Political constraints on differential treatment within community Practical constraints confusing for one person to implement different versions Spillover effects may require higher level randomization Randomizing at group level requires many groups because of within community correlation 22
Group or individual randomization? Sample size and unit of randomization Individual randomization Group randomization N=16 N=4
A good choice when there is incomplete take-up of the product of service Those who get/receive promotion or marketing are more likely to enroll But who got promotion or marketing was determined randomly, so not correlated with other observables or non-observables Compare average outcomes of two groups: promoted/not promoted Effect of offering the encouragement (Intent-To-Treat) Effect of the intervention on the complier population (Local Average Treatment Effect) LATE= effect of offering program (ITT)/proportion of those who took it up
Assigned to treatment Assigned to control Difference Impact: Average treatment effect on the treated Non-treated Treated Proportion treated 100% 0% 100% Impact of assignment Mean outcome 103 80 23 Intent-to-treat estimate 100% 23/100%=23 Average treatment on the treated
Encouraged Not encouraged Difference Impact: Average treatment effect on compliers Non-treated (did not take up program) Treated (did take up program) Proportion treated 70% 30% 40% Impact of encouragement Outcome 100 92 8 Intent-to-treat estimate 100% 8/40%=20 Average treatment on the compliers
Thank you