Assessing the impact of 4Ps on school participation of Filipino children using Propensity Score Matching (PSM) Celia M. Reyes and Christian D. Mina Making Impact Evaluation Matter: Better Evidence for Effective Policies and Programs Asian Development Bank, Manila 5 September 2014
Average daily wage rate of wage/salary workers by educational attainment, 2011 Php 1,200 1,137 1,000 800 600 598 400 200 141 169 186 202 246 335 0 Source of basic data: July 2011 LFS
Data 2011 Annual Poverty Indicators Survey (APIS) and July 2011 Labor Force Survey (LFS) - for the estimation of probability of participating in the 4Ps and for matching 2003 and 2006 Family Income and Expenditure Surveys (FIES), merged with January 2004 and January 2007 LFS, respectively - for the estimation of probability of being selected in the 4Ps
Data Data set Advantage(s) Disadvantages 2003 FIES/LFS 2006 FIES/LFS used by DSWD in the estimation of the PMT model most appropriate to provide the pre-program characteristics can only estimate the probability of being selected in the program predicted poor 4Ps the patterns found in 2003 might no longer hold in 2008 can only estimate the probability of being selected in the program predicted poor 4Ps 2011 APIS/FS can estimate directly the probability of program participation can easily satisfy the balancing tests characteristics of a large proportion 4Ps families that have been in the program in 2008 or 2009 might have been changed significantly
Estimated Logit Models Variable PS Model - 2003 FIES Parameter PS Model - 2006 FIES Dependent variable: Poverty status Independent variables: Household head profile Sex 0.0739 (0.0539) 0.0113 (0.0243) Age -0.0142 (0.0082)* -0.0109 (0.0043)** Square of age 0.0002 (0.0001)* 0.0001 (0.0000)** Household composition Natural logarithm of family size -2.1956 (0.0551)*** -2.0653 (0.0267)*** Proportion of members aged 6-14 -0.0024 (0.0015)* -0.0003 (0.0008) Proportion of members aged 15-64 0.0031 (0.0017)* 0.0087 (0.0010)*** Proportion of members aged 65 & above -0.0031 (0.0020) 0.0056 (0.0011)*** Education of members Proportion of members who are currently attending school 0.0000 (0.0000)*** -0.0012 (0.0003)*** Proportion of members who are elementary undergraduate 0.0068 (0.0011)*** 0.0032 (0.0006)*** Proportion of members who are elementary graduate 0.0132 (0.0012)*** 0.0087 (0.0007)*** Proportion of members who are high school undergraduate 0.0174 (0.0013)*** 0.0105 (0.0007)*** Proportion of members who are elementary undergraduate 0.0244 (0.0015)*** 0.0182 (0.0008)*** Proportion of members who are elementary undergraduate 0.0354 (0.0020)*** 0.0325 (0.0010)*** Proportion of members who are elementary undergraduate 0.0500 (0.0037)*** 0.0456 (0.0016)*** Proportion of members who are elementary undergraduate 0.0393 (0.0231)* 0.1492 (0.0245)*** Employment of members Proportion of employed members 0.0103 (0.0019)*** 0.0040 (0.0010)*** Proportion of laborers & unskilled workers -0.0133 (0.0013)*** -0.0152 (0.0006)*** Proportion of members who are agricultural workers -0.0100 (0.0014)*** -0.0115 (0.0009)***
Estimated Logit Models Variable Parameter PS Model - 2003 FIES PS Model - 2006 FIES Proportion of members who are wage/salary workers 0.0173 (0.0015)*** 0.0209 (0.0007)*** Proportion of members who are employers 0.0085 (0.0030)*** 0.0170 (0.0016)*** Proportion of members who are self-employed -0.0008 (0.0018) 0.0043 (0.0010)*** Proportion of members who are permanent workers 0.0049 (0.0010)*** 0.0079 (0.0005)*** Proportion of members who are overseas contract workers 0.0385 (0.0094)*** 0.0732 (0.0032)*** Housing characteristics and tenure Single house -0.2091 (0.0923)** -0.2784 (0.0408)*** House constructed by strong/predominantly strong materials 0.0793 (0.1147) 0.2100 (0.0486)*** House/lot owned/rented with owner's consent -0.1838 (0.0814)** 0.1320 (0.0376)*** Access to basic amenities Safe water -0.0652 (0.0382)* -0.0511 (0.0184)*** Sanitary toilet facility 0.2139 (0.0392)*** 0.3230 (0.0202)*** Electricity 0.5100 (0.0436)*** 0.2250 (0.0226)*** Ownership of asset Television set 0.5886 (0.0460)*** 0.3836 (0.0213)*** VTR/CD/DVD player 0.6426 (0.0571)*** 0.4579 (0.0195)*** Refrigerator 0.8120 (0.0629)*** 0.7373 (0.0226)*** Washing machine 0.6906 (0.0785)*** 0.7619 (0.0272)*** Airconditioner 0.6228 (0.1821)*** 0.7597 (0.0758)*** Car/motor vehicle 0.7226 (0.1030)*** 0.8075 (0.0306)*** Telephone/cellular phone 1.0485 (0.0762)*** 0.8108 (0.0180)*** Computer 1.2785 (0.4142)*** 1.1468 (0.1118)*** Microwave oven 1.5315 (0.4765)*** 1.7374 (0.1421)***
Estimated Logit Models Variable Parameter PS Model - 2003 FIES PS Model - 2006 FIES Location Region (base category: Region XIII (NCR)) Region I (Ilocos Region) -1.1086 (0.1137)*** -0.8763 (0.0429)*** Region II (Cagayan Valley) -0.1757 (0.1241) -0.0577 (0.0482) Region III (Central Luzon) -0.5050 (0.1121)*** -0.3714 (0.0410)*** Region IV-A (CALABARZON) -0.8022 (0.1078)*** -0.6330 (0.0394)*** Region IV-B (MIMAROPA) -0.9830 (0.1167)*** -1.0279 (0.0469)*** Region V (Bicol Region) -1.1520 (0.1123)*** -0.7215 (0.0435)*** Region VI (Western Visayas) -0.9677 (0.1108)*** -0.5198 (0.0420)*** Region VII (Central Visayas) -0.1837 (0.1124) -0.2809 (0.0417)*** Region VIII (Eastern Visayas) -0.4348 (0.1162)*** -0.7757 (0.0457)*** Region IX (Zamboanga Peninsula) -1.2336 (0.1204)*** -0.4428 (0.0479)*** Region X (Northern Mindanao) -1.4535 (0.1164)*** -0.9591 (0.0480)*** Region XI (Davao Region) -0.6904 (0.1171)*** -0.5801 (0.0454)*** Region XII (SOCCKSARGEN) -0.6776 (0.1170)*** -0.5302 (0.0480)*** Region XV (ARMM) -0.3902 (0.1193)*** -0.5430 (0.0471)*** Region XIV (CAR) -0.4760 (0.1281)*** -0.5968 (0.0509)*** Region XVI (Caraga) -1.7697 (0.1180)*** -1.4525 (0.0462)*** Urban/rural -0.0155 (0.0400) 0.1372 (0.0177)*** Constant 2.4762 (0.2546)*** 1.3423 (0.1217)*** *** significant at 1%; ** significant at 5%; * significant at 10% Source of basic data: Merged files of 2003 FIES & January 2004 LFS; Merged files of 2006 FIES & January 2007 LFS, NSO
Propensity score (PS) models PS Model 2003 FIES (Y = poverty status) PS Model 2006 FIES (Y = poverty status) PS Model 2011 APIS (Y = 4Ps) Matching methods Nearest neighbor (one-to-one), without replacement Nearest neighbor (one-to-one), with replacement Nearest neighbor (one-to-two), with replacement Radius (caliper = 0.01) Kernel, normal, bandwidth = 0.01 Kernel, epanechnikov Local linear regression, normal, bandwidth = 0.01 Local linear regression, epanechnikov
Diagnostic checking Common support - ensured through imposition of common support through the Stata syntax psmatch2, common 0.2.4.6.8 Propensity Score Untreated Treated
Diagnostic checking Balancing property - checked through the following criteria: 1. % reduction in bias (after matching) in: o o propensity scores: 2003 FIES: 99.4-100% 2006 FIES: 99.5-100% 2011 APIS: 86-100% all covariates; 2003 FIES: 68-77% 2006 FIES: 68-75% 2011 APIS: 91-96%
Diagnostic checking Balancing property - checked through the following criteria: 2. Two-sample t-tests of means in the covariates (between treatment and comparison) > 50% are of all the covariates are not significant 3. % reduction in Pseudo R-square after matching reduced by more than 50%
Diagnostic checking Conditional independence - checked through sensitivity tests of hidden bias through Mantel-Haenszel bounds
Example: PS Model - 2011 APIS; Age 6; 1-to-1 matching without replacement Mantel-Haenszel (1959) bounds for variable prop_mem6_attsch Gamma Q_mh+ Q_mh- p_mh+ p_mh- ------------------------------------------------- 1 2.18725 2.18725.014362.014362 1.05 1.92101 2.45619.027365.007021 1.1 1.66706 2.71273.047751.003337 1.15 1.42501 2.95884.077077.001544 1.2 1.19373 3.19545.116292.000698 1.25.972256 3.42336.165462.000309 1.3.759742 3.64329.223704.000135 1.35.555448 3.85586.289294.000058 1.4.358718 4.06163.359903.000024 1.45.168974 4.2611.432909.00001 1.5 -.014301 4.4547.505705 4.2e-06 Gamma : odds of differential assignment due to unobserved factors Q_mh+ : Mantel-Haenszel statistic (assumption: overestimation of treatment effect) Q_mh- : Mantel-Haenszel statistic (assumption: underestimation of treatment effect) p_mh+ : significance level (assumption: overestimation of treatment effect) p_mh- : significance level (assumption: underestimation of treatment effect)
Mean comparison tests By matching method - Estimated treatment effects are not significantly different across matching method for all age groups By PS model - Estimated treatment effects significantly vary across PS models
Best matching estimators, by age group and data set used Age Group 2003 FIES 2006 FIES 2011 APIS 6-14 Nearest neighbor (1), wr Nearest neighbor (1), wr Kernel, normal, bw=0.01 6 LLR, normal, bw=0.01 LLR, normal, bw=0.01 Radius (cal=0.01) 7 Nearest neighbor (2), wr LLR, normal, bw=0.01 Radius (cal=0.01) 8 Kernel, epanechnikov LLR, normal, bw=0.01 Kernel, epanechnikov 9 Radius (cal=0.01) Kernel, epanechnikov Nearest neighbor (1), wor 10 LLR, epanechnikov Radius (cal=0.01) Radius (cal=0.01) 11 Nearest neighbor (1), wor Radius (cal=0.01) Radius (cal=0.01) 12 Nearest neighbor (2), wr Kernel, normal, bw=0.01 Kernel, normal, bw=0.01 13 Kernel, normal, bw=0.01 Radius (cal=0.01) Radius (cal=0.01) 14 Nearest neighbor (1), wor Nearest neighbor (1), wr Radius (cal=0.01) 15-18 Nearest neighbor (2), wr Radius (cal=0.01) Kernel, normal, bw=0.01 15 Radius (cal=0.01) Nearest neighbor (1), wor Kernel, normal, bw=0.01 16 Nearest neighbor (1), wor Nearest neighbor (1), wor Radius (cal=0.01) 17 Nearest neighbor (1), wor Nearest neighbor (2), wr LLR, normal, bw=0.01 18 Kernel, normal, bw=0.01 Kernel, epanechnikov Radius (cal=0.01)
Average treatment effect on the treated (ATT) estimates, by age group and by PS model ATT estimate 10 8 PS Model- 2003 FIES PS Model- 2006 FIES PS Model- 2011 APIS 6 4 2 0-2 -4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Age
Average treatment effect on the treated (ATT) estimates~, by age group and by PS model Age group PS Model- 2003 FIES PS Model- 2006 FIES PS Model- 2011 APIS 6-14 4.3 * 4.6 * 3.0 * 6 7.8 * 7.4 * 5.3 * 7 3.3 * 3.1 * 1.5 ^ 8 2.3 * 1.9 * 1.1 ^ 9 2.4 * 2.8 * 2.3 * 10 3.3 * 2.7 * 1.4 * 11 2.5 * 2.4 * 1.9 * 12 3.5 * 3.4 * 2.0 * 13 4.5 * 3.6 * 3.1 * 14 7.8 * 7.5 * 7.6 * 15-18 6.6 * 5.5 * 2.1 15 3.3 3.8 2.8 16 1.6 1.0 3.5 17-3.0 2.6-2.3 18 3.5 4.4 4.4 ~ Results of the best matching estimators; * significant at 5%; ^ significant at 10%
School participation rate of children aged 6-18, 2007 and 2011 % 100 97.3 92.0 90 94.3 84.8 80 98.4 98.3 98.2 97.8 96.9 96.3 97.3 96.9 93.9 96.3 94.4 90.6 90.4 86.8 82.2 85.2 2007 2011 70 60 50 40 70.1 67.0 57.3 54.4 47.0 44.5 6 7 8 9 10 11 12 13 14 15 16 17 18 Age Sources of basic data: 2007 & 2011 APIS
Proportion of children in 4Ps families who are attending school, by age group % 100 90 80 70 60 50 40 30 20 10 0 92.6 98.0 98.4 98.9 98.8 98.3 96.4 93.6 89.7 77.5 60.0 43.6 33.8 6 7 8 9 10 11 12 13 14 15 16 17 18 Age Source of basic data: APIS 2011, NSO
Reasons for not attending school among children (aged 6-18) in 4Ps families Lack of personal interest 41.4 34.5 High cost of education 24.8 30.9 Employment/looking for work 15.7 15.4 Illness/Disability 3.9 4.5 Marriage 2.9 4.5 Housekeeping 2.7 3.0 Cannot cope with school work 2.2 1.3 School are very far 2.0 1.4 Others 1.9 1.9 No regular transportation 0.8 0.4 Problem with school record 0.5 0.5 Problem with birth certificate 0.4 0.7 Too young to go to school 0.3 0.2 No school within the barangay 0.2 0.2 Finished schooling 0.2 0.5 Source of basic data: APIS 2011, NSO 0 10 20 30 40 50 4Ps Non-4Ps
Proportion of children in matched 4Ps families who are attending school and/or working, by single year of age, 2011 % 92.1 96.9 96.5 95.7 100 94.4 92.6 88.7 90 84.8 80 75.2 70 63.1 60 48.4 46.10 50 38.29 40 26.15 30 24.3 14.88 33.6 20 0.00 0.00 0.00 0.00 0.10 0.54 0.91 2.52 5.39 10 0 6 7 8 9 10 11 12 13 14 15 16 17 18 Age out-of-school, working studying, not working Source of basic data: Matched files of 2011 APIS and July 2011 LFS
Recommendations (in 2012) Extend the period of assistance to existing children-beneficiaries to go beyond 5 years Cover families with children aged 15-18
Policy influence In the President s State of the Nation Address in July 2013, the President announced that the 4Ps will be extended to high school to cover children up to 18 years old or up to when they finish high school whichever comes first On June 2014, the DSWD started with the Modified Conditional Cash Transfer (MCCT)- Expanded Age Coverage (EAC) of the 4Ps The MCCT-EAC covers beneficiaries which are not covered by the regular CCT. As of 31 July 2014, there are 888,350 households with children 15-18 and 924,066 children aged 15-18 included in the program.
www.pids.gov.ph