The PROGRESA/Oportunidades program of Mexico and its Impact Evaluation (II) Emmanuel Skoufias The World Bank PRMPR May 2007
PROGRESA/OPORTUNIDADES: Evaluation Design ζ ζ ζ ζ ζ EXPERIMENTAL DESIGN: Program randomized at the locality level (Pipeline experimental design) IFPRI not present at time of selection of T and C localities Report examined differences between T and C for more than 650 variables at the locality level (comparison of locality means) and at the household level (comparison of household means) Sample of 506 localities 186 control (no program) 320 treatment (receive program) 24, 077 Households (hh) ψ78% beneficiaries ψdifferences between eligible hh and actual beneficiaries receiving benefits ψdensification (initially 52% of hh classified as eligible)
Table: A Decomposition of the Sample of All Households in Treatment and Control Villages Localities: 320 Households:14,856 Localities: 186 Households: 9,221 Household Eligibility Status Discriminant Score ( puntaje ) TREATMENT LOCALITY where PROGRESA is in operation (T=1) CONTROL LOCALITY where PROGRESA operations are delayed (T=0) Eligible for PROGRESA benefits (B=1) Low Below Threshold A B=1, T=1 B B=1, T=0 Non-Eligible for PROGRESA benefits (B=0) Above Threshold High C B=0, T=1 D B=0, T=0
E (Y) Treatment 2DIF impact estimate Control Before After
Using regressions to get 2DIF estimates: Limit sample to eligible households in treatment and control and run regression: Y ( i, t) α + β T( i) + β R2+ β ( T( i) * R2) + θ X + η ( i, v, t) = T R TR j Y(i,t) denotes the value of the outcome a indicator in household (or individual) i in period t, alpha, beta and theta are fixed parameters to be estimated, T(i) is an binary variable taking the value of 1 if the household belongs in a treatment community and 0 otherwise (i.e., for control communities), R2 is a binary variable equal to 1 for the second round of the panel (or the round after the initiation of the program) and equal to 0 for the first round (the round before the initiation of the program), X is a vector of household (and possibly village) characteristics; last term is an error term summarizing the influence random disturbances. j j
( ) [ ] + + + + = = = j j j TR R T X R T Y E θ β β β α X 1, 2 1, ( ) [ ] + + = = = j j j T X R T Y E θ β α X 0, 2 1, ( ) [ ] + + = = = j j j R X R T Y E θ β α X 1, 2 0, ( ) [ ] + = = = j j j X R T Y E θ α X, 0 2, 0
[ ] = β T + β TR CSDIF = E( Y T = 1, R2 = 1, X ) E( Y T = 0, R2 = 1, X) [ ] = β R + βtr BADIF = E( Y T = 1, R2 = 1, X ) E( Y T = 1, R2 = 0, X) 2 DIF = β TR = [ E ( Y T = 1, R 2 = 1, X ) E ( Y T = 1, R 2 = 0, X )] [ E ( Y T = 0, R 2 = 1, X ) E ( Y T = 0, R 2 = 0, X )]
Evaluation Tools ζ ζ ζ Formal surveys (Semi)-structured observations and interviews Focus groups with stakeholders (beneficiaries, local leaders, local PROGRESA officials, doctors, nurses, school teachers, promotoras)
PROGRESA Evaluation Surveys/Data ζ BEFORE initiation of program: ζ AFTER initiation of program Oct/Nov 97: Household census to select beneficiaries March 98: consumption, school attendance, health Nov 98 June 99 Nov/Dec 99 Included survey of beneficiary households regarding operations
PROGRESA Evaluation Surveys ζadditional Info Sources ψ School & clinic survey ψ School and clinic administrative data ψ Nutrition survey conducted independently by Min. of Health and INSP ψ Student achievement test scores by Min of Education ψ Record of payments distributed to beneficiary households
Topics of PROGRESA s Evaluation ζ Targeting accuracy and impact on poverty ζ School enrollment, attendance, child labor, achievement scores ζ Health and utilization of health facilities ζ Child Nutrition ζ Household Consumption & Nutrition
Topics of PROGRESA s Evaluation cont d ζ Operation of the program and perceptions of stakeholders ζ Cost-Analysis and Cost Effectiveness ζ Status of women, community relations ζ Adult labor supply, leisure ζ Impact on short-run poverty ζ intrahousehold transfers
Evaluation Results-Targeting ζ Geographic targeting of the program in rural areas is good ζ Method of selecting poor households within localities is generally accurate (undercoverage of 7% ) ζ PROGRESA s targeting decreases the poverty gap P(1) by 30% and the severity of poverty P(2) by 45%
NOTE: Program Operation & Impact ζ Linkage between program operation & implementation & estimated program impact ψ Delays in Benefit Distribution ψ Mistakes in lists of Beneficiaries ψintention to Treat Effect vs Treatment Effect on those who actually received treatment
Figure D.1 Average Cash Transfers from PROGRESA Mar-00 Feb-00 Jan-00 Dec-99 Nov-99 Oct-99 Sep-99 Aug-99 Jul-99 Jun-99 May-99 Apr-99 Mar-99 Feb-99 Jan-99 Dec-98 Nov-98 Oct-98 Sep-98 Aug-98 Jul-98 Jun-98 May-98 0 100 200 300 400 500 600 October 1998 Pesos
Table D.1 PROGRESA Transfers to Beneficiary Households from November 1998 to October 1999 Household size Total value of consumption (Food) [Nonfood] All poor households 5.81 1190 (947) [242] Households with preschoolers Households with school aged children Households with heads aged 60 or older Beneficiary Households Average monthly transfers received Average monthly alimento transfer Average monthly beca transfer Average monthly school utilities transfer Poor Households Residing in Control Localities Household Total size expenditures (Food) [Nonfood] 197 99 91 8 5.47 1039 (806) [233] Transfers as a percentage of nonbeneficiaries expenditures 19.54% 6.58 1289 202 101 93 8 6.41 1092 18.7% 6.59 1311 239 101 128 11 6.40 1155 20.9% 4.35 936 138 93 41 3 4.23 880 16.5% Source: Calculations based on transfer data provided by PROGRESA averaged across the 12 months period between November 1998 and October 1999 (deflated to November 1998 prices). Consumption and family size averaged across the 3 rounds of the ENCEL surveys in November 1998, June 1999, and November 1999.
Education ζ Are more children attending school because of PROGRESA? ζ How much can schooling be expected to increase? ζ Are there more cost effective ways of bringing children to school? ζ Does PROGRESA have more impact in certain grades? ζ Any effects on drop-out rates, grade progression, repetition, reentry?
All Boys 12-17 Years Old Percent Attending School Last Week 0.7 0.6 0.5 0.4 0.3 0.2 Nov-97 Nov-98 Jun-99 Nov-99 Survey Round Treatment Control
All Girls 12-17 Years Old Percent Attending School Last Week 0.7 0.6 0.5 0.4 0.3 0.2 Nov-97 Nov-98 Jun-99 Nov-99 Survey Round Treatment Control
All Boys 12-17 Years Old Percent working Last Week 0.40 0.35 0.30 0.25 0.20 Nov-97 Nov-98 Jun-99 Nov-99 Survey Round Treatment Control
All Girls 12-17 Years Old Percent Working Last Week 0.20 0.15 0.10 0.05 Nov-97 Nov-98 Jun-99 Nov-99 Survey Round Treatment Control
Evaluation Results: Education ζ PROGRESA has a positive effect the school attendance of both boys and girls in primary and secondary school ψboys in secondary: increase by 8 % ψ Girls in secondary: increase by 14% ζ Negative impact on children s labor market participation (especially boys) ζ No observed increase in the attendance rate (frequency) of children in school. ζ PROGRESA increases overall educational attainment by 10% (and 8% higher earnings)
Evaluation Results: Education ζ Program effective in keeping children in school especially during the critical transition from primary to secondary ζ Less effective in bringing back to and keeping in school children who were out. ζ Earlier entry ages, less grade repetition, better grade progression ζ PROGRESA more cost-effective than increasing access to junior secondary education
Health ζ Does it increase visits to public health clinics? ζ Does PROGRESA have an effect on child health? ζ On the health of adults?
Prob. Ill in Last Four Weeks 0.45 0.40 0.35 0.30 0.25 0.20 0.15 Incidence of Illness for 0-2 Year Olds Baseline 7 Months Post Baseline 14 Months Post Baseline 20 Months Post Baseline PROGRESA Controls
Incidence of Illness for 3-5 Year Olds Prob. Ill in Last Four Weeks 0.30 0.25 0.20 0.15 0.10 0.05 Baseline 7 Months Post Baseline 14 Months Post Baseline 20 Months Post Baseline PROGRESA Controls
Evaluation results: Health ζ Significant increase in visit rates in PROGRESA communities ψincreasing in nutrition monitoring visits, immunization rates and prenatal care in 1 st trimester (8% increase) ζ No substitution between private and public facilities
Evaluation results: Health ζ 12% lower incidence of illness in children between ages 0-5. ζ Significantly positive effects on adult health
Nutrition ζ Does PROGRESA impact of child growth? ζ Household consumption and food diet?
Evaluation results: Nutrition ζ Significant effect at increasing child growth (1cm higher growth) and reducing the probability of stunting among children 12-36 mo. ζ Household total consumption increases ζ PROGRESA households eat better (higher expenditures on fruits, vegetables, meats & animal products)
Impact of PROGRESA on Poverty (1): ζ Results so far: program has no adverse effects on labor income ζ Effects on total hh income and thus poverty depends on the direct and indirect costs associated with participation in PROGRESA. ζ Participation in PROGRESA ψ (a) income losses form children s work ψ (b) give up benefits from other programs (DIF, Ninos de Solidaridad, Abasto Social de Leche) in additional to the elimination of the Tortilla subsidy
Impact of PROGRESA on Poverty (2): ζ Figures 1 & 2: the effects of PROGRESA on hh income and poverty may not be adequately summarized by the size and incidence of the cash transfers ζ Econometric analysis based on individual and hh income in each round from a variety of sources: labor income, income from selfemployment, other income (pensions, rent, and community profits) and government transfers (Ninos de Solidaridad, ININ, PROBECAT, PET, PROCAMPO) +PROGRESA Cash Transfers (from program admin records)
where Impact on Poverty ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) t i R i T R R i T R R i T R i T t i P TR R TR R TR R T, 4 * 4 3 * 3 2 * 2,, 4 4 3 3 2 2 0 η β β β β β β β β α + + + + + + + + = ( ) ( ) ( ) t i Poor z t i y z t i P, *,,, α α =
Table 4: The impact of PROGRESA on poverty using two different poverty Poverty line: Canasta basica in Nov 97 Poverty line: Median of Nov 98 Consumption p.c. (a) Headcount Ratio P(0) (a) Headcount Ratio P(0) Var Coeff. st.error t-val. p-val. Impact* Var Coeff. st.error t-val. p-val. Impact* T 0.0017 0.012 0.13 0.893 T 0.0228 0.019 1.23 0.219 R2 0.0183 0.007 2.49 0.013 R2 0.0379 0.012 3.28 0.001 R2xT 0.0007 0.009 0.08 0.938 0.09 R2xT -0.0271 0.014-1.89 0.059-4.88 R3 0.0429 0.009 4.91 0 R3 0.1120 0.011 10.46 0 R3xT -0.0060 0.011-0.57 0.568-0.73 R3xT -0.0545 0.014-4.03 0-9.83 R4 0.0392 0.006 6.29 0 R4 0.0533 0.012 4.39 0 R4xT -0.0207 0.009-2.26 0.024-2.52 R4xT -0.1004 0.016-6.3 0-18.11 _cons 0.8199 0.010 78.2 0 _cons 0.5316 0.015 36.44 0
Table 4: The impact of PROGRESA on poverty using two different poverty Poverty line: Canasta basica in Nov 97 Poverty line: Median of Nov 98 Consumption p.c. (b) Poverty Gap P(1) (b) Poverty Gap P(1) Var. Coeff. st.error t-val. p-val. Impact* Var. Coeff. st.error t-val. p-val. Impact* T 0.0215 0.014 1.53 0.126 T 0.0333 0.014 2.32 0.021 R2 0.0399 0.008 4.7 0 R2 0.0425 0.010 4.36 0 R2xT -0.0284 0.011-2.61 0.009-5.70 R2xT -0.0457 0.013-3.6 0-15.92 R3 0.0992 0.008 11.97 0 R3 0.1248 0.009 14.19 0 R3xT -0.0445 0.011-4.2 0-8.95 R3xT -0.0717 0.012-6.08 0-24.99 R4 0.0375 0.008 4.59 0 R4 0.0306 0.009 3.33 0.001 R4xT -0.0794 0.012-6.74 0-15.94 R4xT -0.1074 0.014-7.71 0-37.40 _cons 0.4763 0.011 45 0 _cons 0.2538 0.010 26.6 0
Table 4: The impact of PROGRESA on poverty using two different poverty Poverty line: Canasta basica in Nov 97 Poverty line: Median of Nov 98 Consumption p.c. (c) Severity of Poverty P(2) (c) Severity of Poverty P(2) Var. Coeff. st.error t-val. p-val. Impact* Var. Coeff. st.error t-val. p-val. Impact* T 0.0287 0.014 2.1 0.037 T 0.0346 0.013 2.65 0.008 R2 0.0412 0.009 4.58 0 R2 0.0382 0.009 4.18 0 R2xT -0.0393 0.012-3.37 0.001-10.96 R2xT -0.0488 0.012-4 0-24.21 R3 0.1151 0.008 13.93 0 R3 0.1257 0.008 15.03 0 R3xT -0.0616 0.011-5.61 0-17.15 R3xT -0.0774 0.011-6.73 0-38.42 R4 0.0325 0.008 3.83 0 R4 0.0223 0.008 2.7 0.007 R4xT -0.0938 0.013-7.36 0-26.13 R4xT -0.0955 0.014-7.03 0-47.42 _cons 0.3301 0.010 34.68 0 _cons 0.1669 0.008 21.11 0
Table 4 results: (1) ζ PROGRESA had a significant impact in reducing poverty between November 1997 and November 1999. ψ E.g. using the 50th percentile of the value of consumption per capita as a poverty line, suggests that the headcount poverty rate declined by around 4.88% between November 1997 and November 1998 and by 18.11% in the November 1999 in treatment areas (using as base the 55.44% headcount poverty rate in treatment localities in November 1997). ψ Over the same period, and using as base the corresponding value of the poverty gap and squared poverty gap indices in treatment areas in November 1997, the poverty gap measure declined by 37.40%, and the severity of poverty measure (squared poverty gap) declined by 47.42%. ψ The higher impacts of the program in reducing poverty over time are consistent with the findings of Gertler et al. (2006), who demonstrated that rural households increased their investments in microenterprises and agricultural activities which, improved the ability of households to generate income.
Table 4 results: (2) ζ Estimates are remarkably in line with the estimates obtained using ex-ante simulations. ψe.g. simulations based on the predicted consumption of each household in the evaluation sample in November 1997 and adding the maximum amount of PROGRESA cash transfers an eligible household could receive assuming full compliance with the program s requirements (see Skoufias et al., 2001).
Table 4 results: (3) ζ The poverty reduction effects are stronger for the poverty gap and severity of poverty measures, which put greater weight on the poorest of the poor, and our evidence suggests that these estimated poverty effects are robust to the choice of different poverty lines. ζ Figure 3 and Appendix A
Cost Analysis ζ Are the administrative costs of PROGRESA high? ζ What are the private costs associated with participation in the program? ζ What might be the indirect effects of the program on the national economy? (e.g. financing of the program)?
Evaluation Results: Cost Analysis ζ For every 100 pesos allocated to the program, 8.2 pesos are administration/program costs. ψ Very low compared to LICONSA (40 pesos per 100 pesos) and TORTIVALES (14 pesos per 100) ζ Targeting and conditioning of the program makeup 56% of program costs (4.6 pesos out of 8.2 pesos) ζ Private costs (3.8 pesos out of 8.2 pesos)
Evaluation Results: Cost Analysis ψeliminating distortionary food subsidies and using funds to finance a program like PROGRESA leads to substantial welfare gains.
The Contributions of Program Evaluation-1 ψ Program continued and improved ξ Fox administration (begun in 2001) kept and expanded the program ξ Early operations reports in PROGRESA identified implementation issues to be analyzed further (food supplements, intra-household conflict, targeting views) ξ Decision to maintain household targeting in PROGRESA expansion, but to add self-selection to administrative selection in urban areas
The Contributions of Program Evaluation-2 ψprogram Design improved: Program expanded to urban areas ξ Benefits extended to Preparatoria Secondary level ξ Jovenes con Oportunidades- aims to create income generating opportunities for poor households through preferential access to microcredit, housing improvements, adult education and access to social/health insurance.
Critical Issues to be Resolved on CCT ζ Do CCT programs break the intergenerational transmission of poverty? Need long time-series ζ What is the minimal CCT that may be paid? ψoportunidades: size of transfer based on opportunity cost of children (child wage/value of children s contribution to family )
Critical Issues to be Resolved: ζ Impact on Children s Achievement & Learning? ψdo CCT increase achievement or induce teachers to lower grade-passing standards? ζ Teacher & health worker incentives ζ Quality of Services
Critical Issues to be Resolved: ζ Do CCT generate Program/Welfare Dependency? ψso far no negative incentive effects on adult work ψtransfers & Income generation ζ Exit Rule? ψ Lack built-in flexibility to expand coverage to households falling below poverty during crisis
Final Issue ζ Long-Run Sustainability of Program Budget & Political Economy of Program Support
Thank you
TABLE 3a Estimates of Program Impact By Round (BOYS 12-16 yrs old) Experimental Estimates RDD Impact Estimates using different kernel functions 2DIF CSDIF CSDIF-50 Uniform Biweight Epanechnik Triangular Quartic Guassian SCHOOL (1) (2) (3) (4) (5) (6) (7) (8) (9) Round 1 n.a 0.013-0.001-0.053-0.016-0.031-0.018-0.016-0.050 st. error 0.018 0.028 0.027 0.031 0.029 0.031 0.031 0.021 Round 3 0.050 0.064 0.071 0.020 0.008 0.010 0.008 0.008 0.005 st. error 0.017 0.019 0.028 0.028 0.034 0.031 0.033 0.034 0.022 Round 5 0.048 0.061 0.099 0.052 0.072 0.066 0.069 0.072 0.057 st. error 0.020 0.019 0.030 0.028 0.032 0.030 0.032 0.032 0.021 Nobs 16331 4279 R-Squared 0.21 0.25 WORK Round 1 n.a. 0.018 0.007 0.012-0.016-0.004-0.013-0.016 0.025 st. error 0.019 0.029 0.027 0.032 0.029 0.031 0.032 0.021 Round 3-0.037-0.018-0.007 0.007-0.004 0.002 0.001-0.004 0.005 st. error 0.023 0.017 0.029 0.024 0.028 0.026 0.028 0.028 0.019 Round 5-0.046-0.028-0.037-0.031-0.029-0.030-0.029-0.029-0.028 st. error 0.025 0.017 0.025 0.024 0.028 0.026 0.027 0.028 0.019 Nobs 16331 4279 R-Squared 0.16 0.19 NOTES: Estimates in bold have t-values >=2 Treatment Group for Experimental & RDD Estimates: Beneficiary Households in Treatment Villages (Group A) Comparison Group for Experimental Estimates: Eligible Households in Control Villages (Group B) Comparison Group for RDD Estimates: NonEligible Households in Treatment Villages (Group C)
TABLE 3b Estimates of Program Impact By Round (GIRLS 12-16 yrs old) Experimental Estimates RDD Impact Estimates using different kernel functions 2DIF CSDIF CSDIF-50 Uniform Biweight Epanechnik. Triangular Quartic Guassian SCHOOL (1) (2) (3) (4) (5) (6) (7) (8) (9) Round 1 n.a. -0.001 0.000-0.027-0.025-0.026-0.025-0.025-0.035 st. error 0.020 0.030 0.029 0.036 0.033 0.034 0.036 0.023 Round 3 0.086 0.085 0.082 0.038 0.039 0.041 0.039 0.039 0.054 st. error 0.017 0.020 0.029 0.030 0.036 0.033 0.034 0.036 0.024 Round 5 0.099 0.098 0.099 0.078 0.114 0.097 0.107 0.114 0.084 st. error 0.020 0.019 0.028 0.031 0.036 0.033 0.035 0.036 0.025 Nobs 15046 3865 R-Squared 0.22 0.23 WORK Round 1 n.a. 0.034 0.000 0.033 0.026 0.027 0.027 0.026 0.030 st. error 0.017 0.024 0.019 0.022 0.020 0.021 0.022 0.015 Round 3-0.034 0.000 0.001 0.005 0.001 0.003 0.002 0.001-0.008 st. error 0.017 0.009 0.016 0.015 0.018 0.016 0.017 0.018 0.012 Round 5-0.042-0.008-0.025-0.019-0.034-0.029-0.033-0.034-0.025 st. error 0.019 0.009 0.018 0.015 0.018 0.017 0.018 0.018 0.013 Nobs 15046 3865 R-Squared 0.05 0.07 NOTES: Estimates in bold have t-values >=2 Treatment Group for Experimental & RDD Estimates: Beneficiary Households in Treatment Villages (Group A) Comparison Group for Experimental Estimates: Eligible Households in Control Villages (Group B) Comparison Group for RDD Estimates: NonEligible Households in Treatment Villages (Group C)
Concluding Remarks ζ Our analysis reveals that in the PROGRESA sample, the RDD performs very well (i.e. yields program impacts close to the ideal experimental impact estimates). ζ Critical to be aware of some of the limitations of the RDD approach: ψestimates treatment effects at the point of discontinuity (eligibility threshold). Impact on this group of households may be of less interest than impact of the program on the poorer households
Concluding Remarks ζ The integrity/quality of the control/comparison group is of vital importance. ψ Spillover effects do not necessarily lead to a violation of the RDD approach. As long as the local continuity assumption continues to hold even though there are spillover effects the presence of spillover effects would only affect the interpretation of the RD effect: It is the effect of being eligible for program participation in treatment villages net of spillover effects. ζ Social programs at the national scale may be very difficult to evaluate ex-post because of the difficulty in finding an adequate comparison group