Survey Design Third Party Monitoring and Evaluation (M&E) of UNICEF s Unconditional Cash Transfer Program Jose Ramon Toots G. Albert, Ph.D. Immediate Past President, PH Statistical Association, Inc. And Senior Research Fellow, Philippine Institute for Development Studies Email: jrgalbert@gmail.com Presentation of Third Party Monitoring Results
2 Agenda Project Purpose Survey Objectives Sample Methodology Sample Selection Household Weights Sampling Operations
3 1. Project Purpose To conduct an independent third party monitoring and evaluation of UNICEF s Unconditional Cash Transfer Program in Tacloban and Leyte In Dec 2013, UNICEF & ACF* entered into agreement to provide assistance to 10,000 household-beneficiaries in Tacloban and five municipalities of Leyte Vulnerable sectors : pregnant and lactating women (PLWs), children suffering from acute malnutrition (MAM, SAM), persons with disabilities (PWDs), persons with chronic illnesses; elderly persons, single female headed households, and child-headed households * Action Contre La Faim (ACF) also conducted monitoring activities
4 1. Project Purpose The PSAI evaluation activities followed a mixed methods study design to examine (a) design of UCT program, including targeting; (b) implementation of the program; (c) impacts of the program on households; and (d) impact on markets, traders and other economic actors.
5 1. Project Purpose From among the list of 10,000 household beneficiaries, a probability sample was drawn: 500 sample households Monitoring UCT, processes, usage, impact Using the CBMS questionnaire (that collects demographic information, education and health of members, income, employment, physical security, assets, food consumption, nutritional status of children) with added questions on the impact of the UCT on the household beneficiaries Note: households were interviewed three times thus forming panel data ; survey supplemented by FGDs
1.1. Survey Objectives 6 To describe the UCT household beneficiaries, in terms of socio-economic background; To collect data from household beneficiaries for monitoring and assessing the implementation of UCT, incl. targeting system employed; To obtain information for monitoring and assessing the impacts of the cash transfer on the household s nutrition and other dimensions of well-being, including education, access to water and sanitation
7 2. Sampling Methodology The target population for the evaluation is the entire list of UCT household-beneficiaries, but rather than conduct a full enumeration of these beneficiaries, a stratified random sample of around 500 households is designed to be tracked for interviews. Use of sample surveys provides a cost effective mechanism of conducting survey objectives
8 2.1. Why a Sample Survey? Reasons for Sampling 1. Reduced Cost (relative to complete enumeration) 2. Greater Speed or Timeliness 3. Greater Efficiency and Accuracy 4. Greater Scope (more items with fewer observations) 5. Convenience 6. Necessity (e.g. life-testing equipment) 7. Ethics (e.g. blood sampling among infants)
2.2. Sample Selection 9 Since beneficiaries came from a mixture of vulnerable households, the sampling selection involved stratifying the targeted household population into 11 strata, with these strata accounting for the UNICEF selection criteria of groups of vulnerable households. In some cases, the vulnerable groups were merged with other groups owing to their rareness. We suggested to UNICEF that we try to work out a random controlled trial (RCT) for the entire study but UNICEF had concerns about its implementation
2.2. Sample Selection 10 A stratified sampling design is used when within strata, we find more homogeneity, and across strata, there is heterogeneity. Oversampling is typically done in one stratum which may have a lot more variability than other strata. Since there is no prior information about the extent of variability within strata, a random sample of households is chosen per stratum accounting for the size of the strata.
2.2. Sample Selection 11 Based on design, 502 sample households were are targeted for interview. The percentage distribution of sample households roughly mirrors the UNICEF listing (see next slide). Subsequent information revealed 130 double counts in the list of 10,000 UCT beneficiaries. An inspection of the 502 sample households suggests that 3 out of 502 are part of the 130 double-counted households.
Distribution of Beneficiaries (in Study Population and Sample) by Municipality 12 HOUSEHOLD POPULATION SAMPLE HOUSEHOLDS Municipality Frequency Percent Frequency Percent Burauen 2027 20.25 101 20.7 Dagami 1504 15.03 86 16.94 Julita 518 5.18 19 3.7 La Paz 1260 12.59 61 11.91 Pastrana 648 6.47 35 6.91 Tacloban City 4052 40.48 200 39.84 Total Households 10009 100.00 502 100.00
2.3. Household Weights 13 From the design, household weights are estimated: represent number of households that each sample household represents among the 9870 UCT beneficiary-households. weights are inversely proportion to selection probabilities. preliminary design weights are to be further adjusted to incorporate non-response adjustments in the final household weights.
14 2.3. Household Weights A mere re-weighting of the remaining 499 households, accounting for the removal of the three households from the originally selected 502 households, yields the final list of sample households, and provides a mechanism for the (preliminary) estimation of household weights. Subsequent information provided to the PSAI suggested a final list of 10,009 UCT household beneficiaries.
15 3. Sampling Operations Conducted training of personnel April 28-30, 2014 in Tacloban City Trainors: CBMS Philippines Team and PSAI Project Team Field Team: PSAI Members in Tacloban City Actual sampling operations for round 1 conducted from May 9 until July 15
16 3.1. Sampling Operations for Round 1 Municipality Interviewed Nonresponse Total Targeted Burauen 99 2 101 Dagami 85 0 85 Julita 17 2 19 La Paz 58 1 59 Pastrana 35 0 35 Tacloban City 196 4 200 Total 490 499 Response rate = 98% ; 1 HHs in Burauen actually in Macarthur; 1 supposed in La Paz actually in Burauen.
3.1. Sampling Operations for Round 1 17 As a general rule, a response rate of 30 per cent or greater for a postal/externally sent questionnaire is generally regarded as reasonable. A goal of 50 per cent or more responses should be attempted in any questionnaire that involved face-to-face interviews. (In rich countries, this is usually not attained). For the project, the target response rate (of 50% of more) was achieved.
18 3.2. Sampling Operations for Round 2 Municipality Interviewed Nonresponse Total Targeted Burauen 98 3 101 Dagami 85 0 85 Julita 17 2 19 La Paz 56 3 59 Pastrana 34 1 35 Tacloban City 196 4 200 Total 486 499 Attrition rate = Less than 1 % (0.82%)
19 3.3. Sampling Operations for Round 3 Municipality Interviewed Nonresponse Total Targeted Burauen 98 3 101 Dagami 85 0 85 Julita 17 2 19 La Paz 55 4 59 Pastrana 34 1 35 Tacloban City 195 5 200 Total 484 499 Attrition rate = 1.22% (from round 1)
20 3.4. Focus Group Discussions Results from the rounds of sample surveys were supplemented with information from other data sources, including focus group discussions (FGDs). Three rounds of FGDs were conducted in each of the project sites to collect qualitative information. Different beneficiary households were invited to each FGD.
21 END of Presentation Maging miyembro po sana tayo ng Philippine Statistical Association Inc. (PSAI)! (Join us @ PSAI ) http://www.philstat.org.ph/ psai.sec@gmail.com