FINAL EVALUATION REPORT

Ministry of Community Development and Social Services (MCDSS) German Technical Cooperation (GTZ) Social Safety Net Project FINAL EVALUATION REPORT KALOMO SOCIAL CASH TRANSFER SCHEME Lusaka, September 2007

EVALUATION TEAM The monitoring and evaluation of the Social Cash Transfer Scheme (SCTS) in Kalomo district was conducted by a consultant team. Mr. W. Colenbrander, M&E consultant, designed and managed the external monitoring and evaluation system. Ms. E. Schüring, technical advisor to the Social Safety Net Project with the assistance of Mr. E. Boonstoppel, M&E consultant, and Mr. S. Michelo, M&E specialist in the Ministry of Community Development and Social Services, conducted the quantitative analysis and wrote the evaluation report. Mr. W. Colenbrander was assisted by his research assistant Mr. Nicholas Chooma and by enumerators Mr. N. Mbandama and Ms. K. Mwiimba, who were trained by Mr. Zimba and Mr. Tembo, both from Central Statistical Office (CSO) in Livingstone. The later also participated as an enumerator. With regards to database design and data entry, the evaluation survey team was assisted by Mr. L. Banda of Trainezee Computer Consultancy Limited under the overall responsibility of Mrs. G. Kamfwa. ACKNOWLEDGEMENT People at community level often get interviewed without receiving any acknowledgement for their time and readiness to respond to all the questions asked. So firstly, we would like to express our gratitude to the respondents. Furthermore, we hope that this report contributes to further improving the service delivery by the public service and non-governmental sector to the target group that is at the centre of this evaluation, i.e. the most vulnerable and incapacitated households in society. This report is a product born out of the input from a wide array of persons, organizations and institutions. Although it is impossible to list them individually, we would still like to extend our appreciation for their tireless efforts. We are particularly grateful for the support received from the Ministry of Community Development and Social Services (MCDSS) at all levels. Special thanks also go to Mr. P.-E. Couralet, Mr. B. Schubert, Mr. M. Rompel for their valuable comments and input in the evaluation.

TABLE OF CONTENTS LIST OF ACRONYMS... 1 LIST OF FIGURES... 2 LIST OF TABLES... 3 EXECUTIVE SUMMARY... 4 1. INTRODUCTION... 7 2. METHODOLOGY... 9 2.1 Data Collection... 9 2.2 Quantitative Analysis... 10 2.3 Qualitative Analysis... 11 2.4 Constraints... 12 3. TARGETING MECHANISM... 15 3.1 Targeting Method... 15 3.2 Household Characteristics... 17 3.3 Effectiveness of Targeting... 22 3.4 Conclusion... 26 4. DELIVERY MECHANISM... 28 4.1 Delivery Mechanism... 28 4.2 Extraction of Funds... 28 4.3 Accessibility of Transfers... 29 4.4 Client Satisfaction... 30 4.5 Conclusion... 31 5. MANAGEMENT OF THE SCHEME... 32 5.1 Cost-effectiveness of the scheme... 32 5.2 Quality of management... 33 5.3 Conclusion... 35 6. IMPACT ON EDUCATION... 37 6.1 School Enrolment... 37 6.2 Levels and Reasons for Absenteeism... 38 6.3 Conclusion... 39 7 IMPACT ON NUTRITION & HEALTH... 41 7.1 Nutrition... 41 7.2 Health... 43 7.3 Conclusion... 44 8 LIVELIHOODS... 46 8.1 Cash Income... 46 8.2 Debt & assets... 47 8.3 Consumption... 49 8.4 Investment Pattern... 51 8.5 Misuse... 53 8.6 Conclusion... 54 9 IMPACT ON THE COMMUNITY... 55 9.1 Positive impact... 55 9.2 Negative impact... 56 9.3 Conclusion... 57

10 SOCIAL POSITION AND SELF-ESTEEM... 58 10.1 Social Position within the Community... 58 10.2 Self esteem and future perspective... 60 10.3 Conclusion... 61 11. CONCLUSION... 63 12. LITERATURE... 65 Annex 1 - Research questions, indicators, targets and data sources... 66 Annex 2 - Sample by Agricultural Block and Geographic Location... 71 Annex 3 Baseline survey and Evaluation survey Questionnaire... 72 Annex 4 FGDs and Topics... 74 Annex 5 Consumption of Food per Household during a Week... 75 Annex 6 Sources of cash income... 76 Annex 7 Percentage of households per consumption item... 77

LIST OF ACRONYMS ACC CSO CWAC DR DSW DSWO DSW Office DWAC FGD GTZ LCMS MCDSS OVC PRSP PSWO PWAS SCTS SSNP ZMK Area Coordinating Committee Central Statistics Office Community Welfare Assistance Committee Dependency Ratio Department of Social Welfare District Social Welfare Officer District Social Welfare Office District Welfare Assistance Committee Focus Group Discussions German Technical Cooperation Living Conditions Monitoring Survey Ministry of Community Development and Social Services Orphans and Vulnerable Children Poverty Reduction Strategy Paper Provincial Social Welfare Office Public Welfare Assistance Scheme Social Cash Transfer Scheme Social Safety Net Project Zambian Kwacha 1

LIST OF FIGURES Figure 1: Kalomo District, Base Map 9 Figure 2: Targeting Flow Chart 16 Figure 3: Age Pyramids for Beneficiary Household Members and All Zambia 18 Figure 4: Marital Status of Beneficiary Household Heads 19 Figure 5: Prevalence of Disability of Beneficiary Household Members by Age 20 Figure 6: Composition of Beneficiary Households 21 Figure 7: Reasons for Difficult Access stated by Beneficiary Households 30 Figure 8: Average Number of Meals per Agricultural Block 41 Figure 9: Levels of Satiation of Beneficiary Households 42 Figure 10: Percentage of Beneficiary Households consuming Vitamins and Proteins every day and Meat and Fish at least once per week 43 Figure 11: Distribution of Beneficiary Households by Sources of Cash 47 Figure 12: Average Amount of Debt by Location 47 Figure 13: Distribution of Beneficiary Households by Number of Assets owned 48 Figure 14: Distribution of Beneficiary Households by Number of Times Households sold Assets over 3 Months 49 Figure 15: Average Consumption per Week by Location in ZMK 49 Figure 16: Percentage of Overall Consumption on Different Consumption Categories 50 Figure 17: Percentage of Households making Investments by Location 51 Figure 18: Average Investment per Quarter by Location 51 Figure 19: Percentage of Households with Return on Investment 52 Figure 20: Reasons stated by Beneficiary Households for no Return on Investment 53 Figure 21: Distribution of Beneficiary Households Begging from the Community 55 Figure 22: Frequency of Begging by Beneficiary Households comparing Baseline to Evaluation 56 Figure 23: Perception of Beneficiary Households on whether they are accepted by the Community 58 Figure 24: Tendency of Acceptance by the Community over Time 59 Figure 25: Participation of Beneficiary Households in Community Activities 59 Figure 26: Perception of Beneficiary Households on how are they are viewed by the community 60 Figure 27: Percentage of Beneficiary Household being hopeful about the Future 60 Figure 28: Future Plans of Beneficiary Households 61 2

LIST OF TABLES Table 1: Timing of Baseline and Evaluation Survey 9 Table 2: Number of Focus Group Discussions per Group and Number of Participants 11 Table 3: Number of People Interviewed per Key Informant Group 12 Table 4: Population Distribution by Age and Ratio of Female 17 Table 5: Marital Status of SCTS Beneficiary Household Members by Age and Sex 18 Table 6: Age of Heads of Beneficiary Households by Age and Sex Ratio 19 Table 7: Prevalence of Orphanhood by Type and Age Group 20 Table 8: Percentage Distribution of Disabilities of Beneficiary Household Members 20 Table 9: Distribution of Beneficiary Households by Dependency Ratio 22 Table 10: Distribution of Beneficiary Households by Disqualifiers of the Scheme 23 Table 11: Perception of Exclusion Error according to Status after Retargeting in SCTS 25 Table 12: Satisfaction of Beneficiary Households with CWAC Performance 25 Table 13: Satisfaction about CWAC Performance disaggregated by Retargeting Status 26 Table 14: Receipt of Payments by Beneficiary Households 29 Table 15: Assessment by Beneficiary Households of whether Access to Transfers was difficult disaggregated by Bank and Pay Point 29 Table 16: Satisfaction by Beneficiary Households with Performance of Pay Points and Bank 30 Table 17 Development of Costs for Transfers and Administration 32 Table 18: School Enrolment for Primary & Secondary Education (%) by Age Groups for Beneficiary Households and by Location and Poverty Status for National Average 37 Table 19: Net & Gross Enrolment Rate for Evaluation 38 Table 20: Levels of Absenteeism by Sex and by Number of Days for the 3 months preceding the Survey 39 Table 21: Reason for Absenteeism 39 Table 22: Average Number of Meals per Day per Beneficiary Household 41 Table 23: Prevalence of Sickness of Beneficiary Households in the past month by Age and Gender 43 Table 24: Prevalence of Disabilities among Beneficiary Household who are in Baseline and Evaluation 44 Table 25: Distribution of Beneficiary Households by different Income Groups and by Location 46 Table 26: Percentage of Beneficiary Households Having Debt 47 Table 27: Distribution of Beneficiary Households by Ownership of Assets 48 Table 28: Percentage of Households per Consumption Category 50 Table 29: Average Investment per Item during the 3 months preceding the Survey 52 3

EXECUTIVE SUMMARY While conditional and unconditional cash transfer schemes have long been established and tested in middle income settings, they are rather recently implemented and remain contested in low income countries. The Kalomo social cash transfer scheme, which has been set up as a social protection mechanism in reaction to the HIV/AIDS pandemic in Zambia, is the first of its kind to verify whether cash transfer schemes are feasible and can have an impact in a low income country with weak administrative structures and constrained financial resources. OPERATIONAL EFFECTIVENESS Effectiveness of community based targeting A general analysis of the beneficiary household characteristics compared to the national or regional averages shows that the Age Pyramid, SCTS Beneficiary Household Members targeted group of beneficiary households 65+ 60-64 55-59 is proportionally more vulnerable, and match the main entry criteria for the 50-54 scheme which require households to 45-49 40-44 have no or limited self-help potential. 35-39 The beneficiary population has 30-34 25-29 proportionally more younger as well as 20-24 15-19 elderly persons, showing the underrepresentation of a productive 10-14 5-9 0-4 generation. 23.4% of all households are 12 10 8 6 4 one generation households - out of 2 0 2 4 6 8 10 12 Population (%) which the majority (17.5%) are elderly Men Women one person households - while 32.6% of Source: SCTS, Baseline Survey, 2004 all households are generation gap households, where the elderly take care of the younger children. More than half of all household heads are aged 65 and above, 74% of them are single and 62% are female. The level of orphanhood among the beneficiary children is with 68.4% considerably higher than the national average of 17%. Relatively more household members are disabled with 15.6% compared to 2.4% nationwide. While the characteristics of the targeted households reflect a high degree of vulnerability and the perception of beneficiary households and the community on the quality of targeting has generally been positive, the findings on the effectiveness of the targeting system are not conclusive yet. Based on the calculation of the dependency ratio according to objective criteria (as opposed to the judgement of the CWAC members), we have 29% of households who do not meet the criterion of having a dependency ratio above 300. Nevertheless, this inclusion error is mainly explained by the discretion of CWAC members to classify someone as fit or unfit and could also be due to the fact that the dependency ratio requirement of above 300 is not adequate to identify the most destitute and incapacitated households in Zambia. More research is currently underway to look at the effectiveness of community based targeting and contrast it with other targeting mechanisms. Effectiveness of the delivery mechanism Despite limited infrastructure and accessibility problems, the scheme has proven that cash can be delivered effectively in a low-income country like Zambia. Most beneficiary households received the social cash timely and in full through different pay points spread throughout the pilot region. None of the beneficiary households declared any of the money to be extracted by the bank, pay point managers, CWAC members or other stakeholders in the scheme. In addition, 69% of the beneficiary households mentioned that it was not difficult to collect their 4

money, which is generally a positive result given that most household heads are old, some of them are disabled and many suffer from sicknesses. Overall, 90% of the beneficiary households were happy or very happy about the method of payment. Quality of management The Social Cash Transfer Scheme proves to be cost-effective as the administrative costs represented on average 13 % of the total costs of the scheme excluding government salaries and 19% including government salaries. Ongoing technical assistance by GTZ has not been factored in. The cost-effectiveness is first of all a structural advantage of cash transfer programmes, which are less costly in delivery than in-kind transfers and can secondly be ascribed to the utilization of an already existing government structure for the implementation of the scheme. At the beginning of the scheme a lot of capacity building and backstopping was needed to improve the quality of management. After 1.5 years of implementation the general management has now reached satisfactory levels where the scheme is managed by the District Social Welfare Office without any direct technical support. IMPACT ANALYSIS Education Enrolment rates rose by 3% points to 79.2% at evaluation. This increase is statistically significant and could be an effect of the SCTS. Following the sampled youth of school-going age over time, 50% of all youth (7 17) who were not in school at baseline were enrolled at evaluation. Half of them started school while the other half returned to a higher grade in school, meaning that they had dropped out of school before the scheme started. The impact of the SCTS on absenteeism is not conclusive. While there was an overall increase in the number of days absent, absenteeism reduced for shorter periods and increased for longer periods. The longer periods of absenteeism might be a result of the drought which took place in 2004/05 and which increased the opportunity costs to schooling. Qualitative research on the contrary revealed a positive change in school attendance of beneficiary children after the inception of the scheme. Nutrition and health Household members living on one meal a day decreased from 19.3% to 13.3% at evaluation and even more importantly, beneficiary households felt more satiated after having eaten. The percentage of households indicating that 60% 50% 40% 30% 20% 10% 0% Satiation levels of beneficiary households 56.3% Baseline survey 41.9% Evaluation survey 34.8% 28.5% 23.3% 14.1% 1.1% 0.0% Still Hungry after Meal Just Enough Enough Don't Know 90 80 70 60 50 40 30 20 10 0 Prevalence of sickness in the past month by age groups Baseline Evaluation 0-5 6-18 19-64 65 and above they were still hungry after each meal decreased from 56.3% at baseline to 34.8% at evaluation. Due to the higher purchasing power households were also able to have a more varied diet: the number of households consuming vitamins (vegetables and fruits) and proteins (fish and meat) increased at evaluation. Most-likely due to improved nutrition, the incidence of illnesses reported during the 5

month preceding the surveys went down from 42.8% to 35%. In addition, a reduction of partial sightedness among beneficiary households from 7.2% to 3.3% could be observed, potentially due to the fact that beneficiary households could afford small eye surgeries. It appears unlikely that these positive improvements in nutrition and health would have taken place without the scheme. Livelihoods The scheme has most likely positively impacted upon the livelihoods of the beneficiary households. The number of beneficiary households relying on cash-income from external sources such as relatives and neighbours decreased at evaluation, pointing towards increased self-reliance of the beneficiary households rather than reliance on the community. Ownership of assets by households Type of The average debt went down from roughly 13,000 to Baseline Evaluation asset 8,000 ZMK, and asset ownership developed positively Cattle 5.5% 5.2% from 4.2 assets at baseline to 5.2 assets at evaluation. Donkeys 1.5% 0.4% Particularly, the increase in smaller livestock is Goats Pigs Chicken 8.5% 1.5% 42.4% 41.7% 1.5% 57.6% noteworthy: 7 times as many households owned goats and the ownership for chickens increased by 15% points. In addition, households who needed to sell assets in order Guinea fowls 1.9% 2.2% to buy food during the 3 months preceding the surveys Bicycle 3.7% 5.2% decreased by 4 percentage points. Ox-cart 3.0% 3.3% Radio 4.4% 8.1% House 74.9% 91.1% Furniture 16.6% 15.5% Pans/ pots 87.1% 93.4% Plough 14.4% 17.7% Hoe 86.0% 88.9% Axe 70.5% 77.5% Grinding Mill 0.0% 10.0% Others 0.0% 1.1% Wider community More households both consumed and invested more. The number of beneficiary households making investments quadrupled from roughly 14% to 50% and the average amount invested doubled. 71% of all households mentioned that they had invested part of the social cash and 52% of them even stated to have generated some extra income. Concerning misuse of money, FGDs and quantitative research show that it has been rare. When recorded, CWAC members reacted quickly to reverse this trend. The wider community has benefited from the SCTS. The scheme has presented a financial relief to the community with the prevalence as well as the frequency of begging reducing considerably. The percentage of households begging from community members decreased from 86.7% to 69.3% and 75% of households who were still begging at evaluation claimed to beg less. The trend of reduced begging is confirmed by the qualitative research. In addition, it is likely that the SCTS injected cash into the local economy and stimulated economic activities. The majority of the money used on consumption items at evaluation (81.3%) is spent locally. It is therefore likely that neighbours as well as local shopkeepers have also benefited from the additional inflow of cash. Jealousy even though expressed does not seem to have caused disturbances in communities and did not tilt the overall positive perception of the scheme by the community. Social status Self Esteem While the impact of the scheme with respect to the social position of beneficiary households is not conclusive, households have definitely gained more self-confidence over time: they think that they are considered less poor by the community at evaluation, look more positive into the future (households being hopeful increased from 37% at baseline to 49% at evaluation) and more households have plans for the future (increase from 50% at baseline to 73% at baseline), which is a crucial prerequisite for breaking through the vicious cycle of poverty. 6

1. INTRODUCTION While conditional and unconditional cash transfer schemes have long been established and tested in middle income settings, they are rather recently implemented and remain contested in low income countries. The Kalomo social cash transfer scheme, which has been set up as a social protection mechanism in reaction to the HIV/AIDS pandemic in Zambia, is the first of its kind to verify whether cash transfer schemes are feasible and can have an impact in a low income country with weak administrative structures and constrained financial resources. In order to generate the necessary information on the feasibility, costs and impact of the scheme, the MCDSS/GTZ Social Safety Net Project, which set up the social cash transfer scheme, commissioned a monitoring and evaluation consultancy. The scheme delivers a small amount of cash every month in form of grants to households in Kalomo District without any or limited self-help potential. Most of these households are affected by HIV/AIDS. All targeted households receive 30,000 ZMK 1 (US$ 7.5 2 ) on a monthly basis while targeted households with children get an additional child bonus of 10,000 ZMK (US$ 2.5). These critically poor and incapacitated households are identified through a community based targeting process carried out by grassroots structures of Public Welfare Assistance Scheme (PWAS). Through cash transfers the pilot project aims to establish a manageable and affordable system of basic social protection that reduces extreme poverty, enables people to care for their basic needs and empowers beneficiaries to take their own decisions without creating dependency or lack of initiative to improve their own livelihood. The pilot scheme pursues the following two main objectives 3 : Reduce extreme poverty, hunger and starvation in the 10% most destitute and incapacitated (non-viable) households in the pilot region (approximately 1,000 households at the first pilot stage) 4 ; The focus of the scheme is on generation gap households, which are headed by the elderly caring for orphans and vulnerable children (OVCs) because the breadwinners are chronically sick or have died due to HIV/AIDS or other reasons. Generate information on the feasibility, costs and benefits and all positive and negative impacts of a Social Cash Transfer Scheme as one component of a comprehensive Social Protection Strategy for Zambia. After a research, design and test phase 5 the scheme was launched on 4 May 2004 by the Honourable Marina Nsingo MP Minister of Community Development and Social Services. By January 2005 the scheme had been rolled out to the whole pilot area of Kalomo Central and Kanchele Agricultural Blocks comprising 143 villages, 5 township sections and 11,349 households (approximately 70,000 persons). The scheme currently involves 6 Area Coordinating Committees (ACCs), 39 Community Welfare Assistance Committees (CWACs), 27 Pay Points and the Kalomo Branch of Finance Bank. The intention is to cover the entire district by mid 2007. Since it is equally important to learn whether such a cash transfer scheme is first of all feasible in a low income country and secondly whether it produces the desired impact, the evaluation report is divided into two main parts: 1) Operational effectiveness, 2) Impact analysis. The first 1 The amount of social cash transfer was at the onset of the scheme the equivalent of a 50KG bag of maize. 2 Exchange rate: 4000 ZMK = 1 US$ 3 Details on the preceding research and on the rationale, design and organization of the scheme have been documented in a number of reports, which can be accessed through the SCTS website www.socialcashtransferszambia.org. 4 The 10% limit is based on the PWAS National Household Survey 2003, which concluded that 10.5% of the Zambian Households are destitute households. 5 March 2003 to April 2004 7

part sheds light on the quality of the community-based targeting mechanism (Chapter 3), the effectiveness of the delivery mechanism (Chapter 4) and the quality of management at the different levels of PWAS (Chapter 5). This part mainly analyzes whether the right beneficiary households were targeted, whether any households were wrongfully excluded, whether beneficiary households received their transfer on time, regularly and in full and whether the scheme has been cost-effective and professionally managed. The second part of the evaluation report focuses on the impact that the scheme has had on beneficiary households and the wider community. It looks at whether children of beneficiary households have a higher educational attainment (Chapter 6), whether the nutrition and health status of beneficiary households has improved over time (Chapter 7), whether negative cooping mechanisms could be avoided and in what way the cash transfer impacted consumption and investment patterns (Chapter 8). It also regards to what extent the scheme has had an impact on the wider community (Chapter 9), whether cash transfers have changed the status of beneficiary households in the community and whether the self-esteem of beneficiary households has been strengthened (Chapter 10). This report is the final evaluation report that has incorporated feedback, comments and suggestions from a number of stakeholders on the first version of the evaluation report. 8

2. METHODOLOGY 2.1 Data Collection The SCTS started in 2004 in two agricultural blocks in Kalomo District, Kanchele and Kalomo Central, and is still ongoing. The baseline survey and the first round of focus group discussions and interviews were conducted before the beneficiaries started receiving the social cash transfers, but after the beneficiaries were targeted. Due to the fact that not all of the beneficiary households were targeted at the same time, the baseline survey was split in two parts. The first half of the baseline survey was done in September 2004 in Kanchele Agricultural Block while the second half of the baseline survey was conducted in December 2004 in Kalomo Central Agricultural Block. One year after the baseline study, the data collection for the evaluation was conducted in Kanchele Agricultural Block in September 2005 and in Kalomo Central Agricultural Block in December 2005. As with the baseline study, the evaluation includes a quantitative survey 6 as well as focus group discussions and interviews with relevant stakeholders. Table 1 outlines the sample distribution and timing of the surveys. Table 1 Timing of Baseline and Evaluation Survey Surveys Block Sep 04 Dec 04 Sep 05 Dec 05 Kanchele Baseline survey 157 HHs Kalomo Baseline Central 146 HHs Source: Baseline and Evaluation Survey survey Evaluation survey 146 HHs Evaluation 128 HHs survey Figure 1 Kalomo District, Base Map Kalomo Central Agricultural Block Kanchele Central Agricultural Block Source: Zambia Ministry of Local Government, 2004 6 The surveys took on average 45 minutes and were carried out in the local language, Tonga. The FGDs were done in the local language or English depending on how the participants mastered English. 9

Two different agricultural blocks were chosen for the pilot SCTS to enable an analysis of the performance of the scheme in a rural and urban environment: Kanchele Agricultural Block is entirely rural, while Kalomo Central Agricultural Block is urban, peri-urban and rural. 7 Both agricultural blocks are indicated in figure 1. Since the majority of beneficiary households live in rural areas (~80%), the sample has a larger population of rural households: 89 households are considered urban or peri-urban, while 214 households are rural. See Annex 2 for a disaggregation of the sample population by geographical location compared to the population in each CWAC. 2.2 Quantitative Analysis The quantitative part of the evaluation is based on a randomized sample of 303 households without control group. Given that the SCTS reaches approximately 1000 households, a minimum sample size of 278 was determined to be representative. 8 Since the same households were followed for baseline and evaluation, a slightly larger sample size was chosen to mitigate attrition over time. 9 After one year the remaining sample size for evaluation amounted to 274 households. In order to compare the baseline data to national household data and more particularly to data from Southern Province, the existing national household surveys such as the Living Conditions Monitoring Survey (LCMS) 2002/2003 and 2004 were used for the analysis. The questionnaire was designed together with the Social Safety Net Project (GTZ), the PWAS Management Unit, a number of non-governmental stakeholders and a Consultant from the Central Statistical Office in Lusaka. The questions asked in the evaluation survey are identical to the questions in the baseline survey, although a number of questions were added to provide more detail on certain topics of particular interest such as retargeting, the returns on investments made, the impact of cash transfers on begging and other forms of community assistance, the performance of the scheme when compared to other programmes etc. See Annex 3 for the baseline and evaluation questionnaires. When conducting the baseline and evaluation survey, every household head 10 of the sample was interviewed. 11 The respondent answered both the questions at household and individual level for all household members. Questions at individual level relate mainly to background characteristics, education, health and nutrition. 2.2.1 Attrition Any evaluation that utilizes panel data experiences attrition of households (deaths, dissolution and relocation). Attrition of households between baseline and evaluation amounted to 11% of the total sample. In total 271 out of the 303 households from baseline were found at evaluation, meaning that 32 households left the sample. Of the households that left the sample from September 2004 till September 2005, 9 households were dissolved after the death of the household head, 5 households migrated to a different area, no household members were found for the interview in 9 households, 1 household could not be located and there was no information available for 8 households. Attrition is more prevalent in urban areas: 17% of urban households were no longer present at evaluation, while 8% of rural households. A general 7 Urban: <1km from centre Kalomo; peri-urban: 1 5 km from centre; rural: >5km from centre 8 According to Table for determining sample size from given populations developed by Krijcle, R.V and Morgan D.W (1970) for the USA National Education Association. 9 For sampling the list of all beneficiary households from each Community Welfare Assistance Committee (CWAC) was used. Every third name on the list was chosen. 10 The household head is the person all members of the household regard as the head and who normally makes dayto-day decisions concerning the running of the household (LCMS 2004). 11 In case of his/her absence another household member was asked to respond to the questions. The household head is the person all members of the household regard as the head and who normally makes day-to-day decisions concerning the running of the household (LCMS 2004). 10

analysis of households that left the sample in comparison with the total sample shows no significant differences. The analysis with the exception of the section on household characteristics is based on the 271 households. 2.2.2 Fluctuation Those households who stayed in the scheme are quite dynamic and over the course of a year, a number of fluctuations of household members can be noticed. Only 32% of all households remained either the same or grew naturally through the arrival of new-borns. 28% of all households had new household members who joined the household as well as household members who left. 40% of all households had either members leaving or members arriving during the year between baseline and evaluation. For the analysis, this means that even though the same number of households is compared, the number of individuals varies from baseline to evaluation. All together, 75% of all individuals in baseline remained in the sample at evaluation. Those household members who either left or joined the beneficiary households are mainly young people: roughly ¾ of fluctuating household members are below 20 years of age. Consequently the death of older household members cannot largely account for the fluctuations in household membership. Infant mortality or child migration within extended family networks however might be possible explanations why there is a high percentage of under 5s (about 30.3%) who left beneficiary households. More research would be necessary to draw any definite conclusions. Interesting to note is that in particular smaller size households (1 5 members) grew in size while the exact opposite development took place for larger households. This can be explained by the fact that smaller households have naturally a higher probability of growing with members finding partners or having children than bigger households who split up once they have reached a certain size. In addition, the cash transfer could have created an additional incentive for either households to take in more household members or for relatives to send their dependents to those households who have additional support. 2.3 Qualitative Analysis Next to the quantitative survey, Focus Group Discussions (FGDs) and interviews with various stakeholders were carried out in order to triangulate findings. In total 36 FGDs were held for the evaluation. The number of FGDs was evenly spread over the two agricultural blocks, although slightly more FGDs were conducted in Kanchele Agricultural Block (21 FGDs). FGDs were conducted with headmen and CWAC members who are directly involved in the targeting of the beneficiary households, but are not supposed to be beneficiaries themselves. Separate FGDs were organized with beneficiary households and non-beneficiary households in the community as well as with women and men from beneficiary households. The composition of the FGDs was at random and depended on who happened to be in the area at this particular moment. The following table gives an overview of the number of discussions per category and the number of participants. The FGDs for Headmen and CWAC members were predominantly composed of men. The gender composition of discussions with the community is exactly the opposite: about 70% of the participants were women. Table 2 Number of Focus Group Discussions per Group and Number of Participants Headmen CWAC Total members Non-Beneficiary community members Beneficiary HH members (female) Beneficiary HH members (male) Number of FGD 4 7 8 9 8 36 Total number of 22 40 163 87 36 348 participants Source: Report on Qualitative Research during Evaluation of SCTS 11

FGDs covered the following topics: Frequency and intensity of begging/ asking for favours in relation to SCTS Increase or decrease in community coherence (jealousy) in relation to SCTS Changes in livelihoods of the beneficiaries in relation to SCTS Relevance of SCTS in comparison with other projects that are implemented in the area Perceived use and misuse of the social cash that is transferred to the beneficiaries Perceptions on inclusion and exclusion errors with regards to the targeting of the beneficiaries For a more detailed overview of the topics addressed in the specific FGDs see Annex 4. Apart from the discussions on the predetermined topics, the FGDs provided the opportunity to the participants to raise issues that were not addressed in the quantitative survey. Regarding interviews with key informants and stakeholders, a little more than half of the interviews were conducted with people from rural areas (55%) and a few more interviews were conducted in Kalomo Central Agricultural Block (56%). Interviews were conducted with a range of people that were directly or indirectly connected to the SCTS. The interviews were held to acquire a broader opinion base and to get more detailed information on the management, targeting and transfer delivery of the scheme. The selection of people for interviews mainly depended on who was present in the area while conducting the quantitative research. In contrast to the FGDs, where most participants were women, 84% of the people interviewed were men. Almost 80% of the CWAC members interviewed were men 12 and most administrative stakeholders at district level were male. The following table shows in more detail who was interviewed. Table 3 Number of People Interviewed per Key Informant Group Key Informants Headman CWAC chairpersons ACC chairpersons CWAC members Constituency MP Councillors Pay point managers No of key informant interviews 6 4 3 2 1 2 3 5 Source: Report on Qualitative Research during Evaluation survey SCTS See again Annex 4 for the checklist with questions that were used for interviews with the various stakeholders of the Kalomo Pilot SCTS. With the exception of inclusion and exclusion errors, the same topics were covered in FGDs and interviews. others 13 2.4 Constraints Despite the choice of combining qualitative and quantitative research methods in order to produce a richer evaluation, there were a number of limitations experienced with the research setup. The greatest limitation is that there is no control group. At the design stage it was felt very strongly that for ethical reasons it would be inappropriate to have a control group. 14 Stakeholders felt that it would not be appropriate to monitor the suffering of households who are currently not benefiting from this intervention. However, without a control group no definite conclusions on the impact of the SCTS can be drawn since the influence of external factors, such as droughts, harvests, interventions of other organizations, macroeconomic shocks etc 12 The majority of CWAC members are male. 13 Chairperson RDC, NGO staff, senior Council staff (2) and senior government official. 14 When it comes to an extension of the scheme and more beneficiary households are gradually integrated into the scheme, it is recommended to use those beneficiary household who will be included at a later stage (after one year) as a control group. In this way the evaluation would gain in strength while not overlooking ethical concerns. 12

cannot be factored out. The lack of a control group is even more unfortunate given that Zambia experienced a drought in the year before the evaluation was conducted while rainfall was favourable in the year before the baseline study. Consequently, while this evaluation attempts to draw some conclusions on the impact of the cash transfer scheme, a stronger focus will be on the operational effectiveness, while the upscaling of the scheme will be used to draw more definite conclusions on the impact. The absence of a control group could be compensated if a national household survey was available for the year of the baseline as well as the evaluation study. Unfortunately, no household survey was conducted in 2005. Disaggregated data for Southern Province concerning education and health could also not be obtained from the respective ministries. With respect to the qualitative research it became clear that at times community members had preconceived responses in answering questions and providing information. In FGDs and interviews concerning the relevance of SCTS in relation to other interventions such as food aid for instance, respondents in rural areas mentioned that they preferred cash above food aid. However, a similar study conducted by CARE International, an organization providing food aid, drew the exact opposite conclusion. Although the latter study was conducted in an area where the SCTS was not implemented, this example stresses the importance for triangulation between quantitative and qualitative methods as well as with other research. 13

PART I OPERATIONAL EFFECTIVENESS 14

3. TARGETING MECHANISM The SCTS aims at assisting the 10% most vulnerable and incapacitated households in the community. These households are identified through a community based targeting system that involves the grassroots -, area - as well as the district level. In order to verify the assumption that 10% is the right cut-off point and in order to prove that the chosen targeting mechanism manages to keep leakages low, it is important to calculate the exclusion and inclusion error. Even though there is no fixed international standard for targeting (exclusion and inclusion) errors, the generally accepted error is a maximum of 20% (for each error), a target which has been adopted by the cash transfer scheme. This chapter introduces first of all the targeting method employed by the scheme. It then looks at characteristics of households targeted and compares them to the national average or to the average for Southern Province where possible. Based on this analysis, the calculation of the dependency ratio of beneficiary households and beneficiary households perception of the exclusion and inclusion errors, the targeting error is estimated and the effectiveness of the targeting mechanism assessed. 3.1 Targeting Method In determining the right targeting mechanism for Zambia, several factors played a role. First of all Zambia is a vast and sparsely populated country with certain areas that are difficult to access, making it almost impossible to register all households or obtain sufficient and verifiable information about them at affordable costs. Secondly, the majority of households (61%) live in rural areas in Zambia and poverty levels are noted to be very similar among them. Therefore a standardized questionnaire might not capture the small but nevertheless meaningful difference among those households. Thirdly, Zambia already has a Public Welfare Assistance Scheme (PWAS) in place that coordinates social welfare interventions down to the grassroots level. It operates on a voluntary basis, has experience with targeting and with harmonizing interventions. Considering those factors, a community based targeting system, operated through the PWAS seemed to be the most rational choice. It is certainly cost-effective due to the fact that the community committees work on a voluntary basis and are active in even remote and difficult to access areas. Community committees also know about the difference in poverty status among community members and they can easily verify whether information provided by applicants is correct. Working through PWAS guarantees that the cash transfer scheme is embedded in the landscape of existing social welfare interventions and that government takes ownership of a service that has to be provided by government. In addition, it allows communities themselves to participate actively in interventions that have a direct impact on them. The community based targeting system involves different stakeholders as is outlined in Figure 2. The Community Welfare Assistance Committee (CWAC) establishes a list with all potential destitute and incapacitated households who are supposed to fulfil all of the following eligibility criteria: 1) no self-help potential 15, 2) no valuable assets, 3) no steady source of substantial income. The identified households are interviewed individually by CWAC members and the information is recorded in an application form. The headman then checks each and every application form and verifies with his signature that all information provided is correct. The above mentioned eligibility criteria are necessary but not sufficient for inclusion into the scheme since the scheme has fixed a cap of 10%. 10% is not a deliberate cut-off point but originates from a survey carried out by PWAS and by a study commissioned by GTZ, concluding that about 10% of all households in Zambia are destitute and incapacitated. 16 Even 15 This translates into either no productive household member or a dependency ratio over 300. 16 The Incapacitated Poor in Zambia, 2004. Available from: http://www.socialcashtransferszambia.org/pageid_2466950.html 15

though the limit ignores regional differences, it allows for a greater simplicity in management and targeting. Consequently, after all the potentially eligible households have been selected, the CWAC meets again and decides which households among them fall within the 10% most destitute in the community. In order to differentiate between households the application forms include questions on external support, assets, livelihood strategies and on the problems the household faces. The ranking is then presented to the community who can propose changes and additions. After the community has reached consensus on the targeting, the application forms are checked by the Area Coordinating Committee (ACC) for completeness, consistency and correctness and are then forwarded to the District Social Welfare Office (DSW Office). The District Social Welfare Officer (DSWO) together with a representative from the District Welfare Assistance Committee (DWAC) and with a CWAC member who serves as a resource person review the application forms and either approve or reject them in case a household is neither incapacitated nor destitute (i.e. has self-help potential, valuable assets or a steady source of substantial income). Cases that are critical or deserve further discussion in a broader committee are referred to the DWAC meeting. Given regional differences in poverty, there will sometimes be more than 10% of households that could be included under the criteria and sometimes fewer. The local knowledge of the CWAC is therefore an important factor in helping to decide who should be included. Figure 2: Targeting Flow Chart Community information meeting CWAC members interview those HH and fill in application form CWAC meeting ranks HH based on application form CWAC lists all extremely needy & incapacitated HH Headman verifies that the information on application form is correct Community meeting discusses ranking DSWO together with DWAC member and assisted by respective CWACs approves or rejects DSWO informs Bank, Pay Points and CWACs on approval or disapproval Beneficiary HH access transfers at Pay Points DWAC decides over critical cases, forwarded by the DSWO CWAC informs applicants on approval and disapproval Since a community based targeting mechanism is prone to nepotism, it was decided to integrate checks and balances in the system that help to prevent leakages to the non-poor from the beginning and that allow for revisions later on. That is why the CWAC does not decide on its own who in the community can benefit. The decisions taken by the CWAC are not only checked by local leaders and the community but also by the ACCs, the DSWO and the DWAC. This minimizes the chances of including households that do not fit the criteria. If a household that does not qualify for one reason or another has been included in the scheme, there is still a chance to exclude the household at a later stage. Whoever has doubts about a targeting choice made by the CWAC can report to the DSWO who will then verify the information provided and 16

decide whether a household should still to be in the scheme or should be excluded. There is furthermore a chance to exclude households which do not fit the criteria or are too borderline (given the situation of other households) during the retargeting process, which takes place every other year. The retargeting serves to reassess the situation of beneficiary households and to eventually include new beneficiary households who have recently fallen into destitution and become incapacitated. The CWAC repeats the same targeting process as outlined in figure 2 but only includes new households in case existing households are able to graduate. Since the scheme does not want to create a disincentive for households to use the money wisely and improve their livelihoods, households whose situation has improved over the course of the two years, mainly due to the cash transfers, will be retargeted for a second time. Households whose structure has changed and who have new productive members are those who can be expected to graduate. The first retargeting in the pilot scheme took place after only one year and 23.3% of all beneficiary households in the sample were excluded, mainly based on the impression of CWACs that those households were slightly better off than other community members and based on the assumption that the annual retargeting could facilitate a rotation system. In reaction to this misconception, a decision was taken by the Department of Social Welfare in collaboration with the DSWO, the PSWO and with inputs from GTZ to slightly modify the procedures for the retargeting. A two year rhythm was established that mirrors better the potential of beneficiary households to graduate. In addition, the inclusion of new beneficiary households is only possible if any of the existing households have graduated from the scheme. 3.2 Household Characteristics Looking at household characteristics and comparing them to national averages gives us a first indication of whether the characteristics of beneficiary households match the household profile that the scheme intends to target. 17 The following section therefore sets the stage for the analysis of the effectiveness of targeting, looking at the characteristics of beneficiary household members, at the household structures as well as more precisely at the prevalence of orphanhood and disability among beneficiary households. 3.2.1 Population Distribution by Age and Ratio of Female Table 4: Population Distribution by Age and Ratio of Female Age Group % Total % Female of Total Baseline LCMS 2004 Baseline LCMS 2004 0 4 8.8 14 50.5 51.7 5 9 14.3 15 52.6 51.6 10 14 19.3 15 50.0 50.0 15 19 13.4 12 52.5 50.0 20 24 5.2 11 36.5 47.6 25 29 3.1 8 47.4 50.0 30 34 2.9 6 51.4 53.8 35 39 2.6 5 56.3 50.0 40 44 1.8 4 68.2 50.0 45 49 2.4 3 75.9 50.0 50 54 1.9 2 78.3 50.0 55 59 3.6 2 69.8 50.0 60 64 4.7 1 66.7 50.0 65 + 15.9 2 69.8 60.0 17 For the analysis of household characteristics, only the data from the 303 households in the baseline survey is used since the households in the evaluation survey are the same, with the exception of attrition. 17

Total 100 100 56.3 50.0 No of observations 1210 681 Source: SCTS Baseline Survey & LCMS 2004 As evidenced by the age pyramids in figure 3, the beneficiary population compared to national data has proportionally more younger as well as elderly persons, showing the underrepresentation of a productive generation and the predominance of more vulnerable groups in the respective area. Whereas the sample contains globally the same percentage of young people (0 19) than the LCMS 2004 (~ 56%), there are fewer children from 0 4 and more children from 10 14 among beneficiary households. This could be related to the fact that beneficiary households have fewer newborns because there are proportionally fewer members of reproductive age and that they have more orphans who join the household at an age of 10 and above. Looking at the productive age classes (20 65), we can see that the sample has a considerable lower percentage than the national average, which is almost 50% higher. The most striking difference is among the elderly (65 +), whose percentage in the sample is almost 8 times higher than the national average. Figure 3: Age Pyramids for SCTS Beneficiary Households Members and All Zambia Age Pyramid, SCTS Beneficiary Household Members Age Pyramid, All Zambia 65+ 60-64 55-59 50-54 45-49 40-44 35-39 30-34 25-29 20-24 15-19 10-14 5-9 0-4 65+ 60-64 55-59 50-54 45-49 40-44 35-39 30-34 25-29 20-24 15-19 10-14 5-9 0-4 12 10 8 6 4 2 0 2 4 6 8 10 12 Population (%) 12 10 8 6 4 2 0 2 4 6 8 10 12 Population (%) Source: SCTS, Baseline Survey, 2004 Men Women Source: SCTS Baseline Survey & LCMS 2004 Source: LCMS 2004 Men Women The female population in the sample is overall higher (56%) than at national level (50%). Disaggregating sex by age group we find that the proportion of women from 20 24 is lower among beneficiary households, showing that women in this group probably leave the destitute households to start a household on their own or join their husbands households. It is on the contrary significantly higher for age group 40 and above, alluding to the fact that women in destitute households are often coping on their own with partners either having died or left them. The percentage of women in the age group 65 and above in the sample converges to the national average due to the fact that women have on general a higher life expectancy. The information on the marital status of women confirms the finding that women in the higher age categories are often on their own: 64% of all women above 40 are widowed and 8.5% are divorced and interestingly even among the younger women from 40 54, only 41% are married. Table 5: Marital Status of SCTS Beneficiary Household Members by Age and Sex Marital status Age 12 19 (%) Age 20 39 (%) Age 40 54 (%) Age 55 + (%) Female Male Female Male Female Male Female Male Never married 96.4 99.0 34.9 73.7 5.9 16.7 1.8 2.3 Married 1.6 0.7 23.0 17.1 40.6 44.4 20.5 61.3 Separated 0.3 0.0 5.9 0.6 1.0 0.0 0.3 4.1 Divorced 0.3 0.3 9.9 3.4 13.9 19.4 7.1 9.3 Widowed 1.3 0.0 26.3 5.1 38.6 19.4 70.4 23.1 No of observations 304 306 152 175 101 36 395 173 Source: SCTS Baseline Survey 18