MEASURING IMPACT Impact Evaluation Methods for Policy Makers

Size: px

Start display at page:

Download "MEASURING IMPACT Impact Evaluation Methods for Policy Makers"

Milton Walker
6 years ago
Views:

MEASURING IMPACT Impact Evaluatin Methds fr Plicy Makers This material cnstitutes supprting material fr the "Impact Evaluatin in Practice" bk.

1 MEASURING IMPACT Impact Evaluatin Methds fr Plicy Makers This material cnstitutes supprting material fr the "Impact Evaluatin in Practice" bk. This additinal material is made freely but please acknwledge its use as fllws: Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluatin in Practice: Ancillary Material, The Wrld Bank, Washingtn DC ( The cntent f this presentatin reflects the views f the authrs and nt necessarily thse f the Wrld Bank.

2 Randmized Assignment Randmized Offering/Prmtin Discntinuity Design Difference-in-Differences Diff-in-Diff Matching P-Scre matching IE Methds Tlbx

3 Chsing yur IE methd(s) Key infrmatin yu will need fr identifying the right methd fr yur prgram: Prspective/Retrspective Evaluatin? Eligibility rules and criteria? Rll-ut plan (pipeline)? Is the number f eligible units larger than available resurces at a given pint in time? Pverty targeting? Gegraphic targeting? Budget and capacity cnstraints? Excess demand fr prgram? Etc.

4 Chsing yur IE methd(s) Chse the best pssible design given the peratinal cntext: Best Design Have we cntrlled fr everything? Is the result valid fr everyne? Best cmparisn grup yu can find + least peratinal risk Internal validity Gd cmparisn grup External validity Lcal versus glbal treatment effect Evaluatin results apply t ppulatin we re interested in

5 Randmized Assignment Randmized Offering/Prmtin Discntinuity Design Difference-in-Differences Diff-in-Diff Matching P-Scre matching IE Methds Tlbx

6 Randmized Treatments & Cmparisn Eligibles > Number f Benefits Randmize! Lttery fr wh is ffered benefits Fair, transparent and ethical way t assign benefits t equally deserving ppulatins. Oversubscriptin Give each eligible unit the same chance f receiving treatment Cmpare thse ffered treatment with thse nt ffered treatment (cmparisns). Randmized Phase In Give each eligible unit the same chance f receiving treatment first, secnd, third Cmpare thse ffered treatment first, with thse ffered later (cmparisns).

7 Randmized treatments and cmparisns 3. Randmize treatment 1. Ppulatin 2. Evaluatin sample X Cmparisn Treatment = Ineligible = Eligible External Validity Internal Validity

8 Unit f Randmizatin Chse accrding t type f prgram Individual/Husehld Schl/Health Clinic/catchment area Blck/Village/Cmmunity Ward/District/Regin Keep in mind Need sufficiently large number f units t detect minimum desired impact: Pwer. Spillvers/cntaminatin Operatinal and survey csts

9 Case 3: Randmized Assignment Prgresa CCT prgram Unit f randmizatin: Cmmunity 506 cmmunities in the evaluatin sample Randmized phase-in 320 treatment cmmunities (14446 husehlds): First transfers in April cmparisn cmmunities (9630 husehlds): First transfers Nvember 1999

10 Case 3: Randmized Assignment 320 Treatment Cmmunities T=0 T=1 Time 186 Cmparisn Cmmunities Cmparisn Perid

11 Case 3: Randmized Assignment Hw d we knw we have gd clnes? In the absence f Prgresa, treatment and cmparisns shuld be identical Let s cmpare their characteristics at baseline (T=0)

12 Case 3: Balance at Baseline Case 3: Randmized Assignment Treatment Cmparisn T-stat Cnsumptin ($ mnthly per capita) Head s age (years) Spuse s age (years) Head s educatin (years) ** Spuse s educatin (years) Nte: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

13 Case 3: Balance at Baseline Case 3: Randmized Assignment Treatment Cmparisn T-stat Head is female= Indigenus= Number f husehld members Bathrm= Hectares f Land Distance t Hspital (km) Nte: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

14 Case 3: Randmized Assignment Treatment Grup (Randmized t treatment) Cunterfactual (Randmized t Cmparisn) Impact (Y P=1) - (Y P=0) Baseline (T=0) Cnsumptin (Y) Fllw-up (T=1) Cnsumptin (Y) ** Estimated Impact n Cnsumptin (Y) Linear Regressin 29.25** Multivariate Linear Regressin 29.75** Nte: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

15 Prgresa Plicy Recmmendatin? Impact f Prgresa n Cnsumptin (Y) Case 1: Befre & After Case 2: Enrlled & Nt Enrlled Multivariate Linear Regressin 34.28** Linear Regressin -22** Multivariate Linear Regressin Case 3: Randmized Assignment Multivariate Linear Regressin 29.75** Nte: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

16 Keep in Mind! Randmized Assignment In Randmized Assignment, large enugh samples, prduces 2 statistically equivalent grups. We have identified the perfect clne. Feasible fr prspective evaluatins with versubscriptin/excess demand. Mst pilts and new prgrams fall int this categry. Randmized beneficiary Randmized cmparisn

17 Randmized Assignment Randmized Offering/Prmtin Discntinuity Design Difference-in-Differences Diff-in-Diff Matching P-Scre matching IE Methds Tlbx

18 What if we can t chse? It s nt always pssible t chse a cntrl grup. What abut: Natinal prgrams where everyne is eligible? Prgrams where participatin is vluntary? Prgrams where yu can t exclude anyne? Can we cmpare Enrlled & Nt Enrlled? Selectin Bias!

19 Randmly ffering r prmting prgram If yu can exclude sme units, but can t frce anyne: Offer the prgram t a randm sub-sample Many will accept Sme will nt accept Randmized ffering If yu can t exclude anyne, and can t frce anyne: Making the prgram available t everyne But prvide additinal prmtin, encuragement r incentives t a randm sub-sample: Additinal Infrmatin. Encuragement. Incentives (small gift r prize). Transprt (bus fare). Randmized prmtin

20 Randmly ffering r prmting prgram Necessary cnditins: 1. Offered/prmted and nt-ffered/ nt-prmted grups are cmparable: Whether r nt yu ffer r prmte is nt crrelated with ppulatin characteristics Guaranteed by randmizatin. 2. Offered/prmted grup has higher enrllment in the prgram. 3. Offering/prmtin f prgram des nt affect utcmes directly.

21 Randmly ffering r prmting prgram 3 grups f units/individuals Never Enrll WITH ffering/ prmtin X WITHOUT ffering/ prmtin X Only Enrll if ffered/ prmted X Always Enrll

22 Randmly ffering r prmting prgram Eligible units Randmize prmtin/ ffering the prgram Enrllment N Offering/ N Prmtin X X Offering/ Prmtin 0 Never Only if ffered/ prmted Always

Grup %Enrlled=30% Average Y fr entire grup=80 Impact Enrlled=50%

23 Randmly ffering r prmting prgram Never Enrll Offered /Prmted Grup %Enrlled=80% Average Y fr entire grup=100 Nt Offered/ Nt Prmted Grup %Enrlled=30% Average Y fr entire grup=80 Impact Enrlled=50% Y=20 Impact= 20/50%=40 - Only Enrll if Offered/ Prmted Always Enrll -

24 Examples: Randmized Prmtin Maternal Child Health Insurance in Argentina Intensive infrmatin campaigns Cmmunity Based Schl Management in Nepal NGO helps with enrllment paperwrk

25 Cmmunity Based Schl Management in Nepal Cntext: A centralized schl system 2003: Decisin t allw lcal administratin f schls The prgram: Cmmunities express interest t participate. Receive mnetary incentive ($1500) What is the impact f lcal schl administratin n: Schl enrllment, teachers absenteeism, learning quality, financial management Randmized prmtin: NGO helps cmmunities with enrllment paperwrk. 40 cmmunities with randmized prmtin (15 participate) 40 cmmunities withut randmized prmtin (5 participate)

26 Maternal Child Health Insurance in Argentina Cntext: 2001 financial crisis Health insurance cverage diminishes Pay fr Perfrmance (P4P) prgram: Change in payment system fr prviders. 40% payment upn meeting quality standards What is the impact f the new prvider payment system n health f pregnant wmen and children? Randmized prmtin: Universal prgram thrughut the cuntry. Randmized intensive infrmatin campaigns t infrm wmen f the new payment system and increase the use f health services.

27 Case 4: Randmized Offering/ Prmtin Randmized Offering/Prmtin is an Instrumental Variable (IV) A variable crrelated with treatment but nthing else (i.e. randmized prmtin) Use 2-stage least squares (see annex) Using this methd, we estimate the effect f treatment n the treated It s a lcal treatment effect (valid nly fr ) In randmized ffering: treated=thse ffered the treatment wh enrlled In randmized prmtin: treated=thse t whm the prgram was ffered and wh enrlled

28 Case 4: Prgresa Randmized Offering Offered grup %Enrlled=92% Average Y fr entire grup = 268 Nt ffered grup %Enrlled=0% Average Y fr entire grup = 239 Impact Enrlled=0.92 Y=29 Impact= 29/0.92 =31 Never Enrll - Enrll if Offered Always Enrll - - -

29 Case 4: Randmized Offering Estimated Impact n Cnsumptin (Y) Instrumental Variables Regressin 29.8** Instrumental Variables with Cntrls 30.4** Nte: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

30 Keep in Mind! Randmized Offering/Prmtin Randmized Prmtin needs t be an effective prmtin strategy (Pilt test in advance!) Prmtin strategy will help understand hw t increase enrllment in additin t impact f the prgram. Dn t exclude anyne but Strategy depends n success and validity f ffering/prmtin. Strategy estimates a lcal average treatment effect. Impact estimate valid nly fr the triangle hat type f beneficiaries.

31 Appendix 1 Tw Stage Least Squares (2SLS) Mdel with endgenus Treatment (T): y T x 1 2 Stage 1: Regress endgenus variable n the IV (Z) and ther exgenus regressrs: T x Z Calculate predicted value fr each bservatin: T hat

32 Appendix 1 Tw Stage Least Squares (2SLS) Stage 2: Regress utcme y n predicted variable (and ther exgenus variables): y ( T) x ^ 1 2 Need t crrect Standard Errrs (they are based n T hat rather than T) In practice just use STATA ivreg. Intuitin: T has been cleaned f its crrelatin with ε.

MEASURING IMPACT. Impact Evaluation Methods for Policy Makers - I. Bénédicte de la Brière The World Bank

MEASURING IMPACT. Impact Evaluation Methods for Policy Makers - I. Bénédicte de la Brière The World Bank MEASURING IMPACT Impact Evaluatin Methds fr Plicy Makers - I Bénédicte de la Brière The Wrld Bank These slides cnstitute supprting material t the Impact Evaluatin in Practice Handbk : Gertler, P. J.; Martinez,