MEASURING IMPACT. Impact Evaluation Methods for Policy Makers - I. Bénédicte de la Brière The World Bank

Size: px

Start display at page:

Download "MEASURING IMPACT. Impact Evaluation Methods for Policy Makers - I. Bénédicte de la Brière The World Bank"

Hugh Chapman
5 years ago
Views:

1 MEASURING IMPACT Impact Evaluatin Methds fr Plicy Makers - I Bénédicte de la Brière The Wrld Bank These slides cnstitute supprting material t the Impact Evaluatin in Practice Handbk : Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluatin in Practice: Ancillary Material, The Wrld Bank, Washingtn DC ( The cntent f this presentatin reflects the views f the authrs and nt necessarily thse f the Wrld Bank.

2 Reference Spanish & French Versins als available, Prtuguese sn.

3 Impact Evaluatin Lgical Framewrk Measuring Impact Hw the prgram wrks in thery Methd t identify impact Data Operatinal Plan Resurces

4 Cunterfactuals Incrrect Cunterfactuals Causal Befre & After (Pre & Pst) Enrlled & Nt Enrlled (Apples & Oranges) Inference

5 Randmized Assignment Randmized Prmtin Discntinuity Design Difference-in-Differences Diff-in-Diff Matching P-Scre matching IE Methds Tlbx

6 Cunterfactuals Incrrect Cunterfactuals Causal Befre & After (Pre & Pst) Enrlled & Nt Enrlled (Apples & Oranges) Inference

7 Our Objective Estimate the causal effect (impact) f interventin (P) n utcme (Y). (P) = Prgram r Treatment (Y) = Indicatr, Measure f Success Example: What is the effect f a Cash Transfer Prgram (P) n Husehld Cnsumptin (Y)?

8 Causal Inference What is the impact f (P) n (Y)? α= (Y P=1)-(Y P=0) Are we dne?

9 Prblem f Missing Data α= (Y P=1)-(Y P=0) Fr a prgram beneficiary: we bserve (Y P=1): Husehld Cnsumptin (Y) with a cash transfer prgram (P=1) but we d nt bserve (Y P=0): Husehld Cnsumptin (Y) withut a cash transfer prgram (P=0)

10 Slutin Estimate what wuld have happened t Y in the absence f P. We call this the Cunterfactual.

11 Estimating impact f P n Y α= (Y P=1)-(Y P=0) OBSERVE (Y P=1) Outcme with treatment ESTIMATE (Y P=0) The Cunterfactual IMPACT = Outcme with treatment - cunterfactual Intentin t Treat (ITT) Thse ffered treatment Treatment n the Treated (TOT) Thse receiving treatment Use cmparisn r cntrl grup

12 Example: What is the Impact f giving Sanjay additinal pcket mney (P) n Sanjay s cnsumptin f sweets (Y)?

13 The Perfect Clne Sanjay Sanjay s Clne 6 sweets 4 sweets IMPACT=6-4=2 Sweets

14 In reality, use statistics Treatment Cmparisn Average Y=6 sweets Average Y=4 sweets IMPACT=6-4=2 Sweets

15 Finding gd cmparisn grups We want t find clnes fr the Sanjays in ur prgrams. The treatment and cmparisn grups shuld have identical characteristics except fr benefiting frm the interventin. In practice, use prgram eligibility & assignment rules t cnstruct valid estimates f the cunterfactuals

16 Case Study: Prgresa (1) Natinal anti-pverty prgram in Mexic Cash Transfers cnditinal n schl and health care attendance in rural areas. Operatinal Rules: Targeting: Eligibility based n a pverty index Timing: Started 1997 Phased Rll-ut, 5 millin beneficiaries by 2004, nw Oprtunidades 6.5 millin beneficiaries including in urban areas

17 Case Study: Prgresa (2) Rigrus impact evaluatin with rich data 506 cmmunities, 24,000 husehlds (eligible and ineligible) Baseline 1997, fllw-up 1998, 2000 with randm assignment and up until nw with ther strategies Many utcmes f interest Here: Cnsumptin per capita What is the effect f Prgresa (P) n Cnsumptin Per Capita (Y)? If impact is a increase f $20 r mre, then scale up natinally

18 Eligibility and Enrllment Ineligibles (Nn-Pr) Eligibles (Pr) Nt Enrlled Enrlled

19 Cunterfactuals Incrrect Cunterfactuals Causal Befre & After (Pre & Pst) Enrlled & Nt Enrlled (Apples & Oranges) Inference

20 Incrrect Cunterfactual #1 Befre & After Y A A-C = 2 IMPACT? C (cunterfactual) A-B = 4 B T=0 Baseline T=1 Endline Time

21 Case 1: Befre & After What is the effect f Prgresa (P) n cnsumptin (Y)? Y (1) Observe nly beneficiaries (P=1) 268 A (2) Tw bservatins in time: Cnsumptin at T=0 and cnsumptin at T= B α = $35 T=1997 T=1998 Time IMPACT=A-B= $35

22 Case 1: Befre & After Cnsumptin (Y) Outcme with Treatment (After) Cunterfactual (Befre) Impact (Y P=1) - (Y P=0) 35.3*** Estimated Impact n Cnsumptin (Y) Linear Regressin 35.27** Multivariate Linear Regressin 34.28** Nte: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

23 Case 1: What s the prblem? Ecnmic Bm: Real Impact=A-C A-B is an verestimate Ecnmic Recessin: Real Impact=A-D A-B is an underestimate Y A C? B D? Impact? α = $35 Impact? T=0 T=1 Time

24 Cunterfactuals Incrrect Cunterfactuals Causal Befre & After (Pre & Pst) Enrlled & Nt Enrlled (Apples & Oranges) Inference

25 Incrrect Cunterfactual #2 Enrlled & Nt Enrlled If we have pst-treatment data n Enrlled: treatment grup Nt-enrlled: cmparisn grup (cunterfactual) Thse ineligible t participate. Thse that chse NOT t participate. Selectin Bias Reasn fr nt enrlling may be crrelated with utcme (Y) Cntrl fr bservables. But nt un-bservables! Estimated impact is cnfunded with ther things.

26 Case 2: Enrlled & Nt Enrlled Measure utcmes in pst-treatment (T=1) Ineligibles (Nn-Pr) Eligibles (Pr) Nt Enrlled Y=290 Enrlled Y=268 In what ways might E&NE be different, ther than their enrllment in the prgram?

27 Case 2: Enrlled & Nt Enrlled Cnsumptin (Y) Outcme with Treatment (Enrlled) 268 Cunterfactual (Nt Enrlled) 290 Impact (Y P=1) - (Y P=0) -22** Estimated Impact n Cnsumptin (Y) Linear Regressin -22** Multivariate Linear Regressin Nte: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

28 Prgresa Plicy Recmmendatin? Will yu recmmend scaling up Prgresa? B&A: Are there ther time-varying factrs that als influence cnsumptin? E&BNE: Case 1: Befre & After Case 2: Enrlled & Nt Enrlled Impact n Cnsumptin (Y) Linear Regressin 35.27** Multivariate Linear Regressin 34.28** Linear Regressin -22** Multivariate Linear Regressin Are reasns fr enrlling crrelated with cnsumptin? Selectin Bias. Nte: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

29 Keep in Mind! B&A Cmpare: Same individuals Befre and After they receive P. Prblem: Other things may have happened ver time. E&NE Cmpare: Grup f individuals Enrlled in a prgram with grup that chses nt t enrll. Prblem: Selectin Bias. We dn t knw why they are nt enrlled. Bth cunterfactuals may lead t biased estimates f the cunterfactual and the impact.

30 Randmized Assignment Randmized Prmtin Discntinuity Design Difference-in-Differences Diff-in-Diff Matching P-Scre matching IE Methds Tlbx

31 Chsing yur IE methd(s) Key infrmatin yu will need fr chsing the right methd fr yur prgram: Prspective/Retrspective Evaluatin? Targeting rules and criteria? Timing: Rll-ut plan (pipeline)? Pverty targeting? Gegraphic targeting? Mney: Is the number f eligible units larger than available resurces at a given pint in time? Budget and capacity cnstraints? Excess demand fr prgram? Etc.

32 Chsing yur IE methd(s) Chse the best pssible design given the peratinal cntext: Best Design Have we cntrlled fr everything? Is the result valid fr everyne? Best cmparisn grup yu can find + least peratinal risk Internal validity Gd cmparisn grup External validity Lcal versus glbal treatment effect Evaluatin results apply t the ppulatin yu re interested in

33 Randmized Assignment Randmized Prmtin Discntinuity Design Difference-in-Differences Diff-in-Diff Matching P-Scre matching IE Methds Tlbx

34 Randmized Treatments & Cmparisn Eligibles > Number f Benefits Randmize! Lttery fr wh is ffered benefits Fair, transparent and ethical way t assign benefits t equally deserving ppulatins. Oversubscriptin Give each eligible unit the same chance f receiving treatment Cmpare thse ffered treatment with thse nt ffered treatment (cmparisns). Randmized Phase-In Give each eligible unit the same chance f receiving treatment first, secnd, third Cmpare thse ffered treatment first, with thse ffered later (cmparisns).

35 Randmized sampling and randm assignment assignment assignment

36 Randmized treatments and cmparisns 1. Ppulatin 2. Evaluatin sample 3. Randmize treatment X Cmparisn Treatment = Ineligible = Eligible External Validity Internal Validity

37 Unit f Randmizatin Chse accrding t type f prgram Individual/Husehld Schl/Health Clinic/Catchment area Blck/Village/Cmmunity Ward/District/Regin Keep in mind Need sufficiently large number f units t detect minimum desired impact: Pwer. Spillvers/cntaminatin Operatinal and survey csts

38 Case 3: Randmized Assignment Prgresa CCT prgram Unit f randmizatin: Cmmunity 506 cmmunities in the evaluatin sample Randmized phase-in 320 treatment cmmunities (14,446 husehlds): First transfers in April cmparisn cmmunities (9,630 husehlds): First transfers Nvember 1999

39 Case 3: Randmized Assignment 320 Treatment Cmmunities T=0 T=1 Time 186 Cmparisn Cmmunities Cmparisn Perid

40 Case 3: Randmized Assignment Hw d we knw we have gd clnes? In the absence f Prgresa, treatment and cmparisns shuld be identical Let s cmpare their characteristics at baseline (T=0)

41 Case 3: Balance at Baseline Case 3: Randmized Assignment Treatment Cmparisn T-stat Cnsumptin ($ mnthly per capita) Head s age (years) Spuse s age (years) Head s educatin (years) ** Spuse s educatin (years) Nte: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

42 Case 3: Balance at Baseline Case 3: Randmized Assignment Treatment Cmparisn T-stat Head is female= Indigenus= Number f husehld members Bathrm= Hectares f Land Distance t Hspital (km) Nte: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

43 Case 3: Randmized Assignment Treatment Grup (Randmized t treatment) Cunterfactual (Randmized t Cmparisn) Impact (Y P=1) - (Y P=0) Baseline (T=0) Cnsumptin (Y) Fllw-up (T=1) Cnsumptin (Y) ** Estimated Impact n Cnsumptin (Y) Linear Regressin 29.25** Multivariate Linear Regressin 29.75** Nte: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

44 Prgresa Plicy Recmmendatin? Impact f Prgresa n Cnsumptin (Y) Case 1: Befre & After Case 2: Enrlled & Nt Enrlled Multivariate Linear Regressin 34.28** Linear Regressin -22** Multivariate Linear Regressin Case 3: Randmized Assignment Multivariate Linear Regressin 29.75** Nte: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

45 Keep in Mind! Randmized Assignment In Randmized Assignment, large enugh samples, prduce 2 statistically equivalent grups. We have identified the perfect clne. Feasible fr prspective evaluatins with versubscriptin/excess demand. Mst pilts and new prgrams fall int this categry. Randmized beneficiary Randmized cmparisn

46 Randmized assignment with different benefit levels Traditinal impact evaluatin questin: What is the impact f a prgram n an utcme? Other plicy questin f interest: What is the ptimal level fr prgram benefits? What is the impact f a higher-intensity treatment cmpared t a lwer-intensity treatment? Randmized assignment with 2 levels f benefits: Cmparisn Lw Benefit High Benefit X

47 Randmized assignment with different benefit levels 1. Eligible Ppulatin 2. Evaluatin sample 3. Randmize treatment (2 benefit levels) X = Ineligible = Eligible

48 Randmized assignment with multiple interventins Other key plicy questin fr a prgram with varius benefits: What is the impact f an interventin cmpared t anther? Are there cmplementarities between varius interventins? Randmized assignment with 2 benefit packages: Interventin 1 Treatment Cmparisn Interventin 2 Treatment Cmparisn Grup A Grup B Grup C Grup D X

49 Randmized assignment with multiple interventins 3. Randmize interventin 1 1. Eligible Ppulatin 2. Evaluatin sample 4. Randmize interventin 2 = Ineligible X = Eligible

50 Cunterfactuals Incrrect Cunterfactuals Causal Befre & After (Pre & Pst) Enrlled & Nt Enrlled (Apples & Oranges) Inference

51 Randmized Assignment Randmized Prmtin Discntinuity Design Difference-in-Differences Diff-in-Diff Matching P-Scre matching IE Methds Tlbx

52 THANK YOU

MEASURING IMPACT Impact Evaluation Methods for Policy Makers

MEASURING IMPACT Impact Evaluatin Methds fr Plicy Makers This material cnstitutes supprting material fr the "Impact Evaluatin in Practice" bk. This additinal material is made freely but please acknwledge