Evaluation Design: Assignment of Treatment

Transcription

1 Evaluation Design: Assignment of Treatment Megha Pradhan Policy and Training Manager, J-PAL South Asia Kathmandu, Nepal 29 March 2017

2 What can be randomized? Access : We can choose which people will be offered the program Timing of access: We can choose when to provide access to the program Encouragement: We can decide which people will be provided encouragement to participate in the program

3 Types of Randomization Designs Simple lottery Phase in Encouragement

4 Types of Randomization Designs Simple lottery Phase in Encouragement

5 Simple Lottery Randomly sample from area of interest Randomly assign to treatment and control Randomly sample from both treatment and control

6 Example: Slum relocation program in Gujarat Tested the impact of relocating slum residents to suburban housing on general well being, tenure security and children s educational attainment. Eligible women who applied (497 women) Treatment (110 women) Control (387 women) Barnhardt, Sharon, Erica Field, and Rohini Pande. Moving to Opportunity or Isolation? Network Effects of a Randomized Housing Lottery in Urban India. NBER Working Paper No , July 2015.

7 Recap: Simple Lottery Design Most useful when Advantages Disadvantages Basic Lottery Program oversubscribed Easy to understand Easy to implement Can be implemented in public Control group may not cooperate Differential attrition

8 Types of Randomization Designs Simple lottery Phase in Encouragement

9 Phase-in Takes advantage of program expansion (example: the NGO cannot implement in all villages in the first year) Everyone gets program eventually Natural approach when expanding program faces resource constraints

10 Phase in: Laptop Programme

11 Phase 0: No one treated yet All control

12 Phase 1: 1/4 th treated 3/4 ths control

13 Phase 2: 2/4 ths treated 2/4 ths control

14 Phase 3: 3/4 ths treated 1/4 th control

15 Phase 4: All treated (experiment over)

16 Example: Primary School Deworming in Kenya School based mass deworming programme tested the impact of providing deworming drugs and worm prevention education on education outcomes Rationale for Phase in: Logistical and financial constraints 75 Schools in Southern Busia District Year 1 Group 1 (25 schools) Group 2 (25 schools) Group 3 (25 schools) Year 2 Group 1 Group 2 Group 3 Year 3 Group 1 Group 2 Group 3 Miguel, Edward, and Michael Kremer "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities." Econometrica 72(1):

17 Recap: Phase-In Design Most useful when Advantages Disadvantages Phase-In Expanding over time Everyone must receive treatment eventually Easy to understand Constraint is easy to explain Control group complies because they expect to benefit later Anticipation of treatment may impact short-run behavior Difficult to measure long-term impact

18 Types of Randomization Designs Simple lottery Phase in Encouragement

19 Randomized Encouragement Sometimes it s not possible to randomize program access (vaccines, savings program, entitlement programs etc.) But many programs have less than 100% take-up undersubscribed Randomize encouragement to receive treatment Treatment differential comes from higher program takeup among the encouraged

20 What is encouragement? Something that makes some individuals more likely to use program than others Not in itself a treatment E.g., SMS messages, information campaigns, visit from agent, etc

21 Encouragement design Encourage Do not encourage Participated Did not participate compare encouraged to not encouraged These must be correlated do not compare participants to nonparticipants

22 Example: Household water connections in Tangier, Morocco Does an increase in access to water, with no change in water quality, change household s value of a private water connection and household s welfare? Amendis Loan Program: Both T and C were eligible for interest free loans from Amendis to buy a home water connection, paid back over 3, 5, or 7 years. Randomized Encouragement Design Treatment Group: Door to door awareness campaign to encourage households to buy a connection Pre approval for the loan from the authorities Assistance in preparing paperwork Branch officer visited households to collect down-payment Comparison Groups: No application assistance given Devoto, Florencia, Esther Duflo, Pascaline Dupas, William Parienté, and Vincent Pons "Happiness on Tap: Piped Water Adoption in Urban Morocco." American Economic Journal: Economic Policy 4(4):

23 Recap: Encouragement Design Design Most useful when Advantages Disadvantages Encouragement Program has to be open to all comers When takeup is low, but can be easily improved with an incentive Can randomize at individual level even when the program is not administered at that level Measures impact of those who respond to the incentive Need large enough inducement to improve take-up Encouragement itself may have direct effect

24 Real World Challenges Challenge Implication Solution Service provider can t distinguish between T & C Control group finds out, benefits Resources to treat all Program is an entitlement Sample size is small Crossovers Change Unit of Randomization Spillovers Crossovers Attrition No control group Can t force/deny program Insufficient power Change Unit of Randomization Phase in Encouragement Design Change unit of randomization Stratification J-PAL HOW TO RANDOMIZE 25

25 Real World Challenges Challenge Implication Solution Service provider can t distinguish between T & C Control group finds out, benefits Resources to treat all Program is an entitlement Sample size is small Crossovers Change Unit of Randomization Spillovers Crossovers Attrition No control group Can t force/deny program Insufficient power Change Unit of Randomization Phase in Encouragement Design Change unit of randomization Stratification J-PAL HOW TO RANDOMIZE 26

26 When to stratify 1. When we want to achieve balance 2. When we need balance for political/logistical feasibility 3. When we want to increase statistical power 4. When we want to analyze the impact by subgroup

27 How to stratify Dividing the sample into different subgroups Do random assignment for each subgroup Assign to treatment and control

28 Stratification by Gender

29 Stratification by Gender: Randomize Women separately

30 Stratification by Gender: Randomize Women separately

31 Randomize the men separately J-PAL HOW TO RANDOMIZE 32

32 Stratification by Gender J-PAL HOW TO RANDOMIZE 33

33 Which stratification variables should we use? Stratify on variables that could have important impact on outcome variable Stratify on subgroups that you are particularly interested in (where may think impact of program may be different) Can get complex to stratify on too many variables

34 Thank you!