IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-43960) ON A CREDIT IN THE AMOUNT OF SDR 10.9 MILLION (US$17.0 MILLION EQUIVALENT) TO THE

Transcription

1 Public Disclosure Authorized Document of The World Bank Report No.: ICR Public Disclosure Authorized Public Disclosure Authorized IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-43960) ON A CREDIT IN THE AMOUNT OF SDR 10.9 MILLION (US$17.0 MILLION EQUIVALENT) TO THE PLURINATIONAL STATE OF BOLIVIA FOR THE INVESTING IN CHILDREN AND YOUTH PROJECT December 23, 2015 Public Disclosure Authorized Social Protection and Labor Global Practice Latin America and the Caribbean Region

2 CURRENCY EQUIVALENTS (Exchange Rate Effective October 26, 2015) Currency Unit = Bolivian (BOB) Bs 1.00 = US$0.145 US$1.00 = Bs 6.91 FISCAL YEAR January 1 - December 31 ABBREVIATIONS AND ACRONYMS AAA BJA CCT CONAPES DALY ESNUT ICR FEP FM GDP GoB IADB IERR KPI M&E MCP MIS MoE MoL MDP MPED PAD PDO PRF RPS-DIC RUB SIMPS TCU UDAPE UGESPRO Analytic and Advisory Activities Juana Azurduy Grant (Bono Juana Azurduy) Conditional Cash Transfer National Council for Economic and Social Policy (Consejo Nacional de Política Económica y Social) Disability-adjusted Life Year Health and Nutrition Evaluation Survey (Encuesta de Evaluación de Salud y Nutrición) Implementation Completion and Results Report First Employment Program Financial Management Gross Domestic Product Government of Bolivia Inter-American Development Bank Internal Economic Rate of Return Key Performance Indicator Monitoring and Evaluation Mothers and Children Program Management Information System Ministry of Education Ministry of Labor Ministry of Development Planning My First Job with Dignity (Mi Primer Empleo Digno) Project Appraisal Document Project Development Objective Project Result Framework Social Protection and Integrated Community Development Network (Protección Social y Desarrollo Integral Comunitario) Unified Register of Beneficiaries (Registro Unico de Beneficiarios) Social Program Monitoring System Technical Coordination Unit Economic and Social Policy Analysis Unit (Unidad de Análisis de Políticas Económicas) Project Management Unit (Unidad de Gestión de Proyectos)

3 WDR WHO ZMP World Development Report World Health Organization Zero Malnutrition Program Senior Global Practice Director: Arup Banerji Country Director: Alberto Rodriguez Practice Manager: Margaret Grosh Project Team Leader: Gastón Mariano Blanco ICR Primary Author: Maria Cecilia Zanetta

4 PLURINATIONAL STATE OF BOLIVIA Investing in Children and Youth Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design Key Factors Affecting Implementation and Outcomes Assessment of Outcomes Assessment of Risk to Development Outcome Assessment of Bank and Borrower Performance Lessons Learned Comments on Issues Raised by Borrower/Implementing Agencies/Partners Annex 1. Project Costs and Financing Annex 2. Outputs by Component Annex 3. Economic and Financial Analysis Annex 4. Bank Lending and Implementation Support/Supervision Processes Annex 5. Beneficiary Survey Results Annex 6. Stakeholder Workshop Report and Results Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Annex 9. List of Supporting Documents MAP... 50

5 A. Basic Information Country: Bolivia Project Name: BO: Investing in Children and Youth Project ID: P L/C/TF Number(s): IDA ICR Date: 12/23/2015 ICR Type: Core ICR Lending Instrument: Original Total Commitment: Revised Amount: Environmental Category: C Specific Investment Loan Borrower: Plurinational State of Bolivia SDR million Disbursed Amount: SDR 7.92 million SDR 8.40 million Implementing Agencies: Ministry of Health (MoH), Ministry of Labor (MoL), Economic and Social Policy Analysis Unit (Unidad de Análisis de Políticas Económicas [UDAPE]). Co-financiers and Other External Partners: Not applicable. B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 09/18/2007 Effectiveness: 06/30/ /13/2009 Appraisal: 01/14/2008 Restructuring(s): 1/31/ /23/ / Approval: 03/11/2008 Mid-term Review: 12/10/ /07/2012 Closing: 12/31/ /30/2015 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Borrower Performance: Moderately Unsatisfactory Moderate Moderately Satisfactory Moderately Unsatisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Moderately Government: Satisfactory Satisfactory Quality of Supervision: Moderately Implementing Moderately Satisfactory Agency/Agencies: Unsatisfactory Overall Bank Performance: Moderately Satisfactory Overall Borrower Performance: Moderately Unsatisfactory i

6 C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Performance (if any) Potential Problem Project Quality at Entry No at any time (Yes/No): (QEA): Problem Project at any Quality of Supervision Yes time (Yes/No): (QSA): DO rating before Moderately Closing/Inactive status: Unsatisfactory Rating None None D. Sector and Theme Codes Original Actual Sector Code (as percent of total Bank financing) Central government administration Other social services Theme Code (as percent of total Bank financing) Improving labor markets Nutrition and food security Social Safety Nets/Social Assistance & Social Care Services E. Bank Staff Positions At ICR At Approval Vice President: Jorge Familiar Pamela Cox Country Director: Alberto Rodriguez Carlos Felipe Jaramillo Practice Manager/ Manager: Margaret Ellen Grosh Helena Ribe Project Team Leader: Gastón Mariano Blanco Manuel Salazar ICR Team Leader: ICR Primary Author: Gastón Mariano Blanco Maria Cecilia Zanetta F. Results Framework Analysis Project Development Objectives The project development objectives (PDOs) were to: (a) reduce the prevalence of chronic malnutrition in children up to two years old living in the most vulnerable rural areas of the recipient s territory; (b) improve the management and operation of the recipient s skills development program for low-income youth to enhance its effectiveness in increasing the ability of low-income youth living in urban and peri-urban areas to find and maintain a formal job; and (c) improve the recipient s capacity to design and manage a coherent, measurable, and effective social protection system in the medium term, and develop instruments to improve inter- ii

7 institutional coordination and monitoring. 1 Revised Project Development Objectives (as approved by original approving authority) Not applicable. (a) PDO Indicators Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years PDO 1 - Social Protection Program for the Mother and Child Indicator 1 Percentage of two-year-old children participating in the program with a height-for-age below 2Z scores Value (Quantitative or 38 percent 22 percent 21.4 percent qualitative) Date achieved January 31, 2008 December 31, 2013 December 31, 2012 Comments (incl. percent achievement) Indicator 2 Value (Quantitative or qualitative) Achieved (103 percent level of achievement). While the target was exceeded, the decrease could not be statistically attributed to the Bono Juana Azurduy (BJA) Program (2012 BJA impact evaluation). Percentage of mothers with children younger than two years of age living in the targeted areas receiving transfers Date achieved January 31, 2008 Comments (incl. percent achievement) 0 percent 70 percent 50 percent December 31, 2013 December 31, 2014 Estimated at 30.3 percent for December 31, 2012 Not Achieved (61 percent level of achievement). The end target was reduced from 70 to 50 percent under the 2013 restructuring, as enrollment of eligible children was significantly lower than anticipated (that is, 45.8 percent on average between ). Transfers for payment of benefits were also lower than anticipated (on average only 66.2 percent received at least one transfer during the same period). 1 The PDO included in the ICR is slightly different from the text of the PDO included in the PAD, which states: The proposed ICY Project seeks to assist the Government in strengthening the effectiveness of its social protection system by supporting the design, financing and implementation of two flagship interventions of the Morales administration s social protection strategy, while enhancing its capacity to coordinate and monitor programs that are part of the social protection network. In this context, the development objectives of the Project are to: (i) contribute to a reduction in the prevalence of chronic malnutrition in children ages 0-2 years old living in the most vulnerable rural areas of the country; (ii) improve the management and operation of the skills development program to enhance its effectiveness in increasing low income youths ability to find and maintain a good quality job; and (iii) improve Government capacity to design and manage a coherent, measurable, and effective social protection system in the medium term, and develop instruments to improve inter-institutional coordination and monitoring. iii

8 Indicator 3 Proportion of children participating in the program with anemia Value (Quantitative or 84 percent 40 percent 74.1 percent qualitative) Date achieved January 31, 2008 December 31, 2013 December 31, 2012 Comments (incl. percent achievement) Indicator 4 Not achieved. Although the target was not achieved, the 2012 impact evaluation indicated that the BJA resulted in a 7.8 percent reduction in the probability of BJA children living in rural areas having anemia with respect to their non-bja counterparts. Percentage of beneficiary women who know their rights to access services and coresponsibilities to participate in the program (Percentage) Value (Quantitative or 0 percent 90 percent 91 percent qualitative) Date achieved January 31, 2008 December 31, 2013 December 31, 2012 Comments (incl. percent Achieved (101 percent level of achievement). achievement) PDO 2 - Skills Development Program for Unemployed Youth Indicator 5 Percentage of beneficiaries with an employment contract four months after the end of the internship Value (Quantitative or 0 percent 50 percent 72 percent qualitative) Date achieved January 31, 2008 December 31, 2013 December 31, 2014 Comments (incl. percent Achieved (144 percent level of achievement). achievement) Indicator 6 Proportion of beneficiaries enrolled during the expansion of the program that are poor. Value (Quantitative or qualitative) 0 percent 80 percent 100 percent 100 percent Date achieved January 31, 2008 December 31, December 31, June 30, 2015 Comments (incl. percent achievement) Achieved (100 percent level of achievement). Based on self-reported income below the poverty line in addition to either: (a) an electric bill showing a consumption of less than 70 KW per month or (b) proof of having attended a public school. PDO 3 - Institutional Strengthening of Bolivia s RPS-DIC Annual progress reports on Social Protection and Integrated Community Development Indicator 7 Network (Protección Social y Desarrollo Integral Comunitario [RPS-DIC]) Programs submitted to the National Council for Economic and Social Policy (Consejo Nacional de Política Económica y Social [CONAPES]) Value (Quantitative or qualitative) No Yes Yes iv

9 Date achieved January 31, 2008 Comments (incl. percent achievement) December 31, 2013 Achieved (100 percent level of achievement). June 30, 2015 (b) Intermediate Indicators Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Component 1: Social Protection Program for the Mother and Child Indicator 1 Number of municipalities that signed agreements to enter the program. Value (Quantitative or Qualitative) Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. Achieved (100 percent level of achievement). percent achievement) Indicator 2 Management Information System (MIS) completed and operational Value (Quantitative or Qualitative) No Yes Yes Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. percent achievement) Achieved (100 percent level of achievement). Indicator 3 Number of municipalities with adequate IT-related equipment and connected with the MIS (Number) Value (Quantitative or Qualitative) Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. percent achievement) Achieved (100 percent level of achievement). Indicator 4 Proportion of children below two years of age who participate in the growth monitoring sessions in the areas of intervention (Percentage) Value (Quantitative or Qualitative) n.a. 80 percent 50 percent 45 percent Date achieved January 31, 2008 December 31, 2013 December 31, 2014 December 31, 2014 Comments (incl. Largely achieved (90 percent level of achievement). The end target was reduced percent achievement) from 80 to 50 percent under the 2013 restructuring given low coverage among children below two years of age in the targeted areas of intervention. Indicator 5 Proportion of pregnant women living in the targeted areas with complete prenatal controls (Percentage) Value (Quantitative or Qualitative) 43 percent 70 percent 30 percent 35 percent Date achieved January 31, 2008 December 31, 2013 December 31, 2013 December 31, 2014 v

10 Achieved (level of achievement is undetermined since the baseline is higher than the target). The end target was reduced from 70 to 30 percent under the 2013 restructuring given BJA low enrollment rates. The baseline was based on national Comments (incl. data (Health and Nutrition National Survey 2003 ENDSA-), for which percent achievement) implementation proved to be significantly higher than in target areas (that is, the country s most vulnerable rural municipalities). The indicator refers to women who participated in the BJA in the targeted area and left the program in the previous 12 months with complete prenatal controls. Indicator 6 Proportion of women receiving postnatal care within seven days of delivery (Percentage) Value (Quantitative or Qualitative) 13 percent 30 percent 22 percent Date achieved January 31, 2008 December 31, 2013 December 31, 2014 Comments (incl. Not achieved. Indicator refers to women who participated in the BJA in the targeted percent achievement) area, delivered in the previous 12 months, and had postnatal care within seven days of delivery. Indicator 7 Proportion of women participating in the program with sufficient knowledge about childcare, food consumption, and home hygiene. (Percentage) Value (Quantitative or Qualitative) n.a. 80 percent 75.6 percent Date achieved January 31, 2008 December 31, 2013 December 31, 2012 Comments (incl. Largely achieved (95 percent level of achievement). As determined by the 2012 percent achievement) impact evaluation. Indicator 8 Proportion of bi-monthly reports on compliance with co-responsibilities systematized through the MIS (Percentage) Value (Quantitative or Qualitative) 0 80 percent 100 percent Date achieved January 31, 2008 December 31, 2013 December 31, 2014 Comments (incl. Exceeded (125 percent level of achievement). percent achievement) Indicator 9 Proportion of payment agencies with enough funds to pay beneficiaries on time (Percentage) Value (Quantitative or Qualitative) percent 100 percent Date achieved January 31, 2008 December 31, 2013 December 31, 2014 Achieved (100 percent level of achievement). While access to payment points still Comments (incl. poses a challenge for BJA participants living in rural areas, payment agencies are percent achievement) fully funded according to the terms of the contract that was subscribed. Indicator 10 Proportion of complaints resolved by the program (Percentage) Value (Quantitative or Qualitative) 0 70 percent n.a. Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Undetermined. The Operational Manual envisioned three types of mechanisms to Comments (incl. submit complaints (that is, dedicated phone line, , and complaint boxes in percent achievement) departmental units). The phone line and the were never implemented and there were poor records of written complaints. Indicator 11 Proportion of indigenous beneficiaries satisfied with the program (Percentage) vi

11 Value (Quantitative or Qualitative) 0 80 percent n.a. Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. percent achievement) Undetermined. This data was to be collected through beneficiary surveys. Unfortunately, although the project included resources to finance beneficiary surveys, these were never conducted. Indicator 12 Proportion of registered women participating in Information and Education Campaigns (IEC) activities (Percentage) Value (Quantitative or Qualitative) 0 80 percent n.a. Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Undetermined. Data on education activities with beneficiaries held in health Comments (incl. centers were poorly registered and no records were included in the BJA percent achievement) Management Information System. The lack of beneficiary surveys also precluded the possibility to track this indicator from a different source. Component 2: Skills Development Program for Unemployed Youth Indicator 13 Number of low-income youth enrolled in the program during expansion (Number) Value (Quantitative or Qualitative) 0 3,800 1,750 1,367 Date achieved January 31, 2008 December 31, 2013 December 31, 2013 June 30, 2015 Comments (incl. Not achieved (78 percent level of achievement). percent achievement) Indicator 14 National Unit for the First Employment Program (FEP) fully staffed and operational (as described in the Operational Manual) Value (Quantitative or Qualitative) 0 Yes No Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. Not Achieved. percent achievement) Indicator 15 Number of departmental offices for the FEP with adequate staffing, operating and connected to the MIS (as described in the Operational Manual) Value (Quantitative or Qualitative) Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. Not achieved (60 percent level of achievement). percent achievement) Indicator 16 Management Information System designed and operational Value (Quantitative or Qualitative) Final design stage Yes Yes Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. Achieved (100 percent level of achievement). percent achievement) Indicator 17 Targeting mechanism to select low-income youths designed and tested Value (Quantitative or Qualitative) n.a. Yes Yes Date achieved January 31, 2008 December 31, 2013 June 30, 2015 vii

12 Comments (incl. Achieved (100 percent level of achievement). percent achievement) Indicator 18 Process evaluation designed and implemented Value (Quantitative or Qualitative) n.a. Yes Yes Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. Achieved (100 percent level of achievement). percent achievement) Indicator 19 Proportion of youth enrolled in the program completing the in-class phase of the program (Percentage) Value (Quantitative or Qualitative) n.a. 80 percent 93 percent Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. Exceeded (116 percent level of achievement). percent achievement) Indicator 20 Proportion of youths completing the internship (as a share of those who completed the in-class phase) (Percentage) Value (Quantitative or Qualitative) n.a. 80 percent 86 percent Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. Exceeded (108 percent level of achievement). percent achievement) Indicator 21 Share of women among beneficiaries who complete the internship (Percentage) Value (Quantitative or Qualitative) n.a. 40 percent 59 percent Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. Exceeded (148 percent level of achievement). percent achievement) Indicator 22 System to certify providers tested and operational Value (Quantitative or Qualitative) n.a. Yes Yes Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. Achieved (100 percent level of achievement). percent achievement) Component 3: Institutional Strengthening of the RPS-DIC Indicator 23 Unified Register of Beneficiaries (RUB) and Monitoring System designed Value (Quantitative or Qualitative) n.a. Yes Yes Date achieved January 31, 2008 December 31, 2013 June 30, 2015 Comments (incl. Achieved (100 percent level of achievement). percent achievement) Indicator 24 New Operational Manual of the BJA in use based on results of impact and process evaluations Value (Quantitative or Qualitative) n.a. Yes Yes Date achieved January 31, 2008 December 31, 2013 June 30, 2015 viii

13 Comments (incl. Achieved. A formal adoption through an MoH Resolution is in progress and pending percent achievement) signature from the MoH minister. G. Ratings of Project Performance in ISRs No. Date ISR Archived DO IP Actual Disbursements (SDR, millions) 1 06/07/2008 Satisfactory Satisfactory /01/2008 Satisfactory Moderately Satisfactory /07/2009 Satisfactory Moderately Satisfactory /09/2009 Moderately Unsatisfactory Moderately Unsatisfactory /23/2010 Moderately Unsatisfactory Moderately Unsatisfactory /26/2011 Moderately Unsatisfactory Moderately Unsatisfactory /10/2011 Moderately Unsatisfactory Moderately Unsatisfactory /31/2012 Moderately Unsatisfactory Unsatisfactory /13/2012 Moderately Unsatisfactory Unsatisfactory /10/2013 Moderately Unsatisfactory Unsatisfactory /01/2014 Moderately Unsatisfactory Unsatisfactory /28/2014 Moderately Unsatisfactory Moderately Satisfactory /26/2015 Moderately Unsatisfactory Moderately Satisfactory 5.98 H. Restructuring Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO IP Amount Disbursed at Restructuring in SDR millions 01/31/2011 MU MU /23/2013 MU U 4.51 Reason for Restructuring & Key Changes Made A Level 2 Restructuring, which introduced the following changes: (a) changes in institutional arrangements for Component 3; (b) the responsibility for overall coordination of the project by the Ministry of Development Planning (MDP) was eliminated; and (c) Component 3 and the corresponding intermediate indicators were modified to reflect the revised institutional arrangements. A Level 2 Restructuring, which introduced the following changes: (a) extension of the closing date by 12 months from December 31, 2013 to December 31, 2014; (b) reallocation of ix

14 Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO IP Amount Disbursed at Restructuring in SDR millions 10/31/2014 MU MS 5.05 I. Disbursement Profile Reason for Restructuring & Key Changes Made unused funds under Component 1 (BJA Program) to Components 2 and 3; (c) cancellation of all remaining unused funds for finance transfers to BJA beneficiaries; and (d) changes in the Project Result Framework (PRF), largely adjusting targets downwards given weak implementation. A Level 2 Restructuring, which introduced the following changes: (a) extension of the closing date by six months from December 31, 2014 to June 30, 2015; (b) creation of new expenditure categories under Components 1 and 2; (c) reallocation of funds within the expenditure categories of Components 1 and 2; and (d) elimination of the legal requirement to complete a procurement audit for Component 3 given, among other factors, the satisfactory track record of the implementing agency in procurement. x

15

16 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal Bolivia has traditionally ranked among the poorest and most unequal countries in Latin America. Despite a decrease in the incidence of poverty since the late 1990s, roughly 60 percent of the population was still living in poverty at the time of project preparation in Income disparities were also among the largest not only in the region, but also in the world, with the richest 10 percent of households accounting for 47 percent of total income, while the poorest decile accounted for less than 1 percent. These disparities were even larger in rural areas. Children and youth, who constitute the majority of the Bolivian population, are particularly vulnerable to social risks. At appraisal, the poverty rate for this group was well above the national average, surpassing 80 percent among children below 5 years of age in rural areas. The lack of access to quality health and nutrition services increased vulnerabilities among poor, rural, and indigenous children. In particular, chronic child malnutrition constituted a major stumbling block in the country s efforts to reduce infant mortality. In 2003, chronic malnutrition affected one-third of Bolivian children below five years of age, of which 42 percent of children were living in rural areas and over 50 percent of children belonged to indigenous groups. 2 It was an important contributor to infant mortality, which reached 86 per 1,000 live births in rural areas and 94 among indigenous communities in Chronic malnutrition also led to irreparable neural and physical damage to the survivors, hampering their lifelong development and productivity. The effects of poor nutritional status during infancy among low-income children were compounded by low quality and insufficient basic and secondary education later in life, resulting in poor Bolivian youth facing considerable disadvantages when entering the labor market. On average, only one in four Bolivians enrolled in tertiary education, and only 3 percent were among the lowest quintile. It was estimated that, in the largest cities, four out of ten students completing secondary education entered the labor market each year, as they did not enroll in tertiary or other post-basic education. In turn, the large influx of Bolivian youth into the labor market further exacerbated the concentration of unemployment among poor urban youth that is typically observed in other countries as a result of age, insufficient skills, and lack of job experience. While overall urban unemployment was 8 percent in 2005, unemployment of youth aged years reached more than 18 percent, and almost twice that level among the poorest at 34 percent. This operation the Investing in Children and Youth Project was designed to assist the government of Bolivia (GoB) in strengthening the effectiveness of its social protection system by supporting the design, financing, and implementation of two flagship interventions focusing on the poorest and most vulnerable children and youth, the Social Protection Program for Mothers and Children (MCP) which was later renamed Juana Azurduy Grant (Bono Juana Azurduy [BJA]) and the First Employment Program (FEP) which was later renamed My First Job with Dignity (Mi Primer Empleo Digno [MPED]). 2 Health and Nutrition National Survey 2003 ENDSA. 1

17 1.2 Original Project Development Objectives (PDO) and Key Indicators The project development objectives (PDOs) were to: (a) reduce the prevalence of chronic malnutrition in children up to two years old living in the most vulnerable rural areas of the recipient s territory; (b) improve the management and operation of Bolivia s skills development program for low-income youth to enhance its effectiveness in increasing the ability of low-income youth living in urban and peri-urban areas to find and maintain a formal job; and (c) improve the GoB s capacity to design and manage a coherent, measurable, and effective social protection system in the medium term, and develop instruments to improve inter-institutional coordination and monitoring. 3 The project s key performance indicators (KPIs) were defined as follows: PDO 1 - Reduce prevalence of chronic malnutrition in children up to two years old: KPI 1 - Percentage of two-year-old children with a height-for-age below 2Z scores in the intervention areas; KPI 2 - Percentage of mothers with children younger than two years of age living in the targeted areas receiving transfers (as a share of the estimated total number of mothers with children younger than two years of age in the targeted municipalities); KPI 3 - Proportion of children participating in the program with anemia; KPI 4 - Percentage of beneficiary women who know their rights to access services and co-responsibilities to participate in the program. PDO 2 - Increase employability of low-income youth: KPI 5 - Percentage of beneficiaries with an employment contract four months after the end of the internship phase (as a share of all beneficiaries who completed the internship every year); KPI 6 - Proportion of beneficiaries enrolled during the expansion of the program that are poor. PDO 3 - Enhance capacity to design and manage social protection strategy: KPI 7 - Annual reports on Social Protection and Integrated Community Development Network (Protección Social y Desarrollo Integral Comunitario [RPS-DIC]) programs progress submitted to the National Council for Economic and Social Policy (Consejo Nacional de Política Económica y Social [CONAPES]), including analysis of indicators, programs harmonization, and implementation. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification The PDOs were not modified during the project s lifetime. The KPIs also remained the same, with the exception of the end target for KPI 2, which was reduced from 70 to 50 percent, and for KPI 6, which was increased from 80 to 100 percent, both under the 2013 restructuring. 1.4 Main Beneficiaries Under Component 1, it was estimated that about 45,000 of the poorest families living in the 52 most vulnerable municipalities in the country would benefit from the implementation of the MCP/BJA Program under the Bank s support. Under Component 2, it was estimated that approximately 4,000 low-income youth would benefit from training and internships under the 3 Financial Agreement, Schedule 1, page 5; Project Appraisal Document (PAD), section IV on page 8. 2

18 FEP/MPED Program. Under Component 3, the direct beneficiary was initially the unit within the Ministry of Development Planning (MDP) responsible for the design and management of the country s social protection strategy and later the Economic and Social Policy Analysis Unit (Unidad de Análisis de Politicas Económicas [UDAPE]). 1.5 Original Components Component 1: Implementing and scaling up the MCP to combat chronic malnutrition (SDR 8 million). This component provided support to the government in implementing and scaling up a program of cash transfers to families with pregnant women and/or mothers of children below two years of age. The objective of this intervention was to create an incentive for extremely poor families to invest in the health and nutrition of young children and to provide financial support to increase the food consumption of beneficiary households. The MCP was part of the Zero Malnutrition Program (ZMP) and was to be implemented in the 164 municipalities targeted by the ZMP using the vulnerability index, with the project supporting the implementation of the MCP in the 52 most vulnerable municipalities. Activities were grouped into three subcomponents, which focused on: (a) benefits for mothers and children to combat chronic malnutrition; (b) information, communication, and education strategy; and (c) management and monitoring of the MCP. Component 2: Improving and expanding a skills development program for lowincome unemployed youth living in urban and peri-urban areas (SDR 2.3 million). The primary objective of this component was to improve the effectiveness of the existing MPED Program as an intervention to increase the employability of low-income youth living in poor urban and peri- urban areas who had completed at least the second year of secondary education. Activities were grouped into four subcomponents, which focused on: (a) improving the effectiveness of the FEP; (b) strengthening the institutional capacity and implementation arrangements of the Ministry of Labor (MoL) to administer and monitor the FEP; (c) providing opportunities for skills training and a first labor market experience for low-income youth; and (d) supporting the MoL to formulate a midterm strategy for youth unemployment. Component 3: Institutional strengthening of the RPS-DIC (SDR 0.6 million). This component aimed to help the government meet the long-term objective of the RPS-DIC, which was to build a sustainable and integrated social protection system. Activities were grouped into three subcomponents, which focused on: (a) strengthening the Technical Coordination Unit (TCU) of the MDP and developing a Midterm Social Protection Strategy; (b) designing and testing a Registry of Beneficiaries and a monitoring and evaluation (M&E) system for the RPS-DIC; and (c) evaluation of selected programs of the social protection network, including the pilot of the FEP and the initial expansion of the MCP to the first 10 municipalities. 1.6 Revised Components Component 1: Implementing and scaling up the MCP to combat chronic malnutrition (SDR 4.18 million). By the end of 2010, the GoB requested the project s expansion to support an additional 55 vulnerable municipalities (for a total of 107 municipalities) to assure the proper use of the credit resources in view of lower-than-expected enrollment rates in the original number of municipalities, which was caused by lack of adequate information about the program, congestion 3

19 of the health care system leading to long waiting times, and the requirement to have a valid identification document as a BJA enrollment condition. The request to expand was granted by the Bank. The 2013 restructuring included a partial cancellation of resources and a reallocation of the remaining resources from Component 1 to Components 2 and 3. The project continued to support the original second and third components till closing. During the last semester of project implementation, PDO 1 continued to receive project support under Component 3 through evaluation and dissemination activities focusing on the BJA Program. Component 2: Improving and expanding a skills development program for lowincome unemployed youth living in urban and peri-urban areas (SDR 3.4 million). No significant changes were introduced to Component 2. Component 3: Institutional strengthening of the RPS-DIC (SDR 0.82 million). Under the 2011 restructuring, the subcomponents were modified to reflect new institutional arrangements (that is, Subcomponent 3.1); the elimination of the development of an M&E system (that is, Subcomponent 3.2); and revised evaluation activities (that is, Subcomponent 3.3). The GoB decided to eliminate the M&E system from the project, because it was included in an Inter- American Development Bank (IADB) project. The subcomponents were redefined as follows: (a) strengthening of the MDP s UDAPE; (b) designing and testing of a Registry of Beneficiaries; and (c) evaluation of the BJA and MPED programs. 1.7 Other Significant Changes The credit was approved on March 11, 2008, and declared effective on June 30, It was amended three times during the project s lifetime, as follows: A Level 2 Restructuring was approved on January 21, 2011, which (a) transferred the responsibility of Component 3 implementation from the MDP s TCU to the UDAPE, the agency responsible for the design and analysis of social and economic policies; (b) eliminated the responsibility of the MDP s TCU of overall coordination of the project and transferred the responsibility to each executing agency (that is, the Ministry of Health [MoH], MoL, and UDAPE) to operate directly as the Bank counterpart for all administrative and financial aspects of their respective components; and (c) modified the description and costs of Component 3, as well as the corresponding intermediate indicators to reflect the revised institutional arrangements and the revisions to the evaluation work program. A Level 2 Restructuring was approved on December 23, 2013, which (a) extended the closing date by 12 months from December 31, 2013 to December 31, 2014; (b) reallocated unused funds (SDR million) of Component 1 to Component 2 (SDR 1.1 million to additionally fund the MPED Program) and Component 3 (SDR million to expand dissemination and evaluation activities); (c) cancelled all the remaining unused funds under the expenditure category of Component 1 which financed transfers to beneficiaries (that is, SDR million); and (d) changed in-end 4

20 targets of KPIs 2 and 6, changed in-end targets of intermediate indicators 4, 5, and 13, and created intermediate indicator 24. A Level 2 Restructuring was approved on October 31, 2014, which (a) extended the closing date by six months from December 31, 2014 to June 30, 2015; (b) created a new expenditure category under Component 2 to finance the stipends of MPED beneficiaries through a new payment system; (c) created a new expenditure category under Component 1 to finance the contractual procurement audit; (d) reallocated SDR 0.7 million within the expenditure categories of Component 2 and SDR 0.03 million within the expenditure categories of Component 1; and (e) eliminated the legal requirement to complete a procurement audit for Component 3 given the reduced number of procurement processes, the existence of ex-ante revision mechanisms, and the UDAPE s satisfactory track record in procurement. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry Soundness of the background analysis. Project design benefitted from the Social Protection Analytic and Advisory Activities (AAA) Program that proceeded in parallel with project preparation. The AAA provided evidence-based analysis on poverty, exclusion, and vulnerability, including nutrition, as well as an overview of existing social protection policies and programs. Assessment of the project design. Project design exhibited both strengths and weaknesses, which can be summarized as follows: Strength: Grounded in rich international experience. The design of the two programs supported under the project was informed by an exhaustive survey of similar programs that was conducted under the AAA Program. Thus, the BJA and MDPE programs reflected the experience and lessons learned from others focusing on cash transfers and youth skills development programs implemented in Latin America and other regions, as well as local conditions and priorities. Strength: Importance given to evaluation activities. Evaluation activities focusing on the BJA Program (both process and impact evaluations) as well as a beneficiary survey were envisioned at appraisal (see section 2.3). Strength: Overall project coordination under the MDP. In addition to the three implementing agencies, the original institutional arrangements assigned the MDP s TCU the responsibility to articulate and coordinate the activities under the three components, including those implemented by the MoH and MoL. Weaknesses: Overly ambitious scope. The project supported the implementation of two completely different programs the BJA and the MPED programs that were complex by themselves and involved two line ministries with limited technical and institutional capacity. 5

21 Weaknesses: Lack of clarity in the Project Result Framework (PRF). A critical weakness was the definition of PDO 1 that is, reduce chronic malnutrition which was overly ambitious and whose achievement was subject to a myriad of factors outside the orbit of the project, such as the evolution of the economy (see section 2.3). Relevance at appraisal. The operation exhibited high relevance at entry, as the two programs supported under the operation were a priority for the government, and adequately responded to the country s main human development challenges. The BJA and the MPED programs were flagship programs of the Morales administration and core components of the RPS- DIC (Decreto Supremo No /2007) under the umbrella of the National Development Plan Bolivia Digna, which focused on poverty and inequality reduction through social protection interventions. In addition, the BJA was a core component of an ambitious multi-sector strategy to reduce chronic malnutrition (that is, the ZMP), another priority program of the government. The operation was also consistent with the three pillars of the Bank s Interim Strategy Note for Bolivia for FY07-08: (a) improving good governance; (b) fostering job creation; and (c) promoting social inclusion. Specifically, the operation was expected to directly improve social inclusion by implementing and increasing the effectiveness of GoB s priority social programs targeted to children and youth and to have secondary benefits for the other two pillars. Assessment of risks. The project was correctly identified as having Substantial risks at appraisal. The main risks were: (a) fiduciary challenges and risks regarding the implementation of conditional cash transfers (CCTs) under the BJA Program; (b) insufficient technical and institutional capacity of implementing agencies, particularly given the innovative nature of the interventions and the lack of previous experience with Bank projects in two of them; and (c) uncertainties at the local level in the context of the decentralization process. While the main risks were addressed by the project design, technical assistance, phased implementation, and evaluation activities, the residual risk was still Substantial. 2.2 Implementation Beyond the Control of the Government Positive macroeconomic environment. The project was implemented in a context of fast economic growth, with gross domestic product (GDP) growth averaging 5.2 percent between 2008 and 2014 (World Development Indicators 2015). In turn, favorable economic conditions were conducive to the availability of counterpart funds as well as internship and employment opportunities for MPED beneficiaries. Conversely, they may have diminished the attractiveness of the cash transfers under the BJA Program. Subject to the Control of the Government Elimination of overall coordination function on the part of the GoB, and compound implementation and supervision challenges. As mentioned earlier, under the 2011 restructuring, the TCU s role of overall project coordination was eliminated. At that time, the GoB decided those responsibilities should be executed exclusively through the executing agencies. These changes resulted in an increased burden on the part of the Bank to supervise the project with three different counterparts. 6

22 High turnover of authorities at the MoH and MoL. The turnover in ministerial and project authorities, as well as the changes in project staff that went hand in hand, were one of the main factors contributing to the project s disjointed implementation, as they reduced implementation pace and prevented a sustained follow-up of agreements on project execution. Several government-level decisions negatively affected the implementation of the BJA Program. First, the GoB made the decision to implement it at the national level even though it had been originally conceived as a pilot program in selected municipalities. The implementation of a pilot would have permitted the fine-tuning of administrative and Information Technology and Communication (ITC) systems as well as the program s overall design before scaling it up. Likewise, a phased approach would have also been more in line with the MoH s implementation capacity. Second, the CCT amount the pivotal incentive of the program has not been modified since the program launch in 2009, which reduced its capacity to compensate for the costs of accessing heath care controls, particularly transport and opportunity costs for mothers living in rural areas, and the cash transfer s overall effectiveness. Third, the BJA Program was implemented within the orbit of the MoH and was not properly articulated with the existent health care delivery network at the departmental and municipal levels. Subject to the Control of the Implementing Agency Implementation weaknesses of the BJA Program. There were significant challenges in the BJA implementation in enrollment, payment of benefits, and fiduciary management. First, enrollment rates were significantly lower than anticipated and, for pregnant women, declined during the operation s lifetime. Several factors contributed to low enrollment, including lack of adequate information about the program, congestion of the health care system leading to long wait times, the requirement to have a valid identification document as a BJA enrollment condition, 4 and, as discussed above, the lack of attractiveness of the amount of the cash transfer. These weaknesses are still to be effectively addressed by the MoH, as reflected by the BJA s continuously low enrollment rates (that is, 54 percent of eligible children and only 18 percent of eligible pregnant women at the end of 2014). Second, several factors contributed to low benefit payment rates (61 to 66 percent), including the one-year delay in establishing an independent payment agent 5 and the lack of easy access to the points of payments for beneficiaries living in rural areas. 6 Third, there were significant delays in compliance with BJA fiduciary responsibilities, including substantial delays in the preparation of accounting records and the lack of financial statements and audits. As a result, by the end of 2012, the Bank, in agreement with the GoB, suspended the financing of BJA transfers. Implementation weaknesses of the MPED Program. The MPED Program built upon the experience gained during a pilot phase that was successfully implemented in four cities between 4 In Bolivia, a large proportion of the population lacks a government ID (up to 30 percent as reported by the National Police), with this percentage most likely being significantly higher among the BJA target population (that is, indigenous population living in rural areas [Harbitz and Tamargo 2009]). 5 During preparation, the GoB had requested that the army serve as the payment agent, mirroring the practice adopted under Bolivia s other CCT program (that is, the BJA in the education sector). After a thorough assessment, the Bank team opted to require the establishment of an independent payment agent, as reflected in the Financial Agreement. 6 The service fee paid to payment agents under the contract with the GoB does not adequately incentivize payment in remote areas, unless there is substantial mass of unpaid benefits that justifies the transport costs of the payment agent. 7

23 2009 and Results from a process evaluation completed in June 2010 were used to redesign the program, including aligning the training curriculum with the accreditation requirements of the Ministry of Education (MoE). With project support, the expansion phase of the program was launched in six cities in November Changes in the MoL administration delayed the initiation of training courses till the first quarter of The final number of beneficiaries, however, was significantly lower than anticipated at appraisal (1,367 versus 3,800) as a result of slow implementation, recurrent changes in MoL authorities and staff, and protracted procurement approval processes. In addition, there were significant increases in the cost of training activities due largely to an increase in the number of training hours to comply with the MoE s accreditation requirements; adjustments in stipends to ensure they accurately reflected participants transportation, meals, and childcare costs; and annual adjustments in the stipends during the internship following changes in the minimum salary (see section 3.3). Favorable institutional changes to Component 3. In 2011, the UDAPE was assigned responsibility for the implementation of Component 3, which was a suitable decision given that the activities under this component were compatible with the unit s responsibility for the design and analysis of social and economic policies within the MDP. Moreover, the stronger emphasis on evaluation under the 2011 restructuring, which incorporated evaluations of the BJA and MPED programs, contributed to strengthening the UDAPE s evaluation capacity as well as the strengthening of the two programs. 7 Financial administration and procurement as implementation bottlenecks. In the case of the MoL, limited technical capacity and overly centralized approval processes resulted in long delays in procurement processes (up to 12 months in the case of one contract), which negatively affected implementation. In the case of the MoH, the Project Management Unit (Unidad de Gestión de Proyectos [UGESPRO]) lacked the necessary agility to ensure the timely allocation of resources for payments to beneficiaries, which caused significant delays in payment and most likely led to an adverse opinion of the BJA Program among the targeted population. 2.3 Monitoring and Evaluation (M&E) Design, Implementation, and Utilization M&E design. Although the PRF served to track progress toward program readiness and program implementation, helping to detect implementation bottlenecks, it exhibited several weaknesses that prevented fully capturing the operation s impact. Specifically: Weakness in PDO definition and operationalization. While PDO 1 reflected the objective stated in the decree that established the BJA Program (that is, reducing chronic malnutrition), it was overly ambitious and affected by factors unrelated to the operation. Moreover, the BJA was just one component of the ZMP, which included other complementary interventions (that is, a massive food supplement program, nutrition education campaigns, and expanded access to drinking water and sanitation) aimed at decreasing malnutrition. A more adequate focus would have been increasing access to health care among the target population and the corresponding health 7 Originally, the scope of the evaluation activities supported under the project was more limited, as it called for the impact evaluation of just the pilot phase of the MPED Program and the initial expansion of the BJA Program to the first 10 municipalities (PAD, page 14). The BJA impact evaluation was going to be financed under the Health Adaptable Program Loan III Project. 8

24 outcomes, such as weight at birth. In addition, PDO 2 lacked consistency and clarity. In the Financial Agreement as well as some parts of the PAD (page 8), the PDO definition focuses on increasing FEP effectiveness in enhancing the employability of the targeted population. It is not specified, however, what baseline should be used to measure the enhanced effectiveness (presumably the pilot program). In other parts of the PAD (that is, annex 3), the focus is specifically on increasing the employability of the target population. This definition is considered more adequate and, thus, is the one being adopted for this Implementation Completion and Results Report (ICR). Finally, the definition of PDO 3 was vague and not easily operationalized. Some weaknesses in the KPIs. There is inconsistency with the set of KPIs described in the text of the PAD (pages 8 10) and those in annex 3. Not only is the list in the text incomplete, there are also some indicators that are listed in the text that are not included in the PRF defined in annex 3 (in fact, only one of the six key indicators listed in the text were included in the PRF). The PRF as defined in annex 3 is the one adopted for this ICR. Some weaknesses in the intermediate indicators. While the PRF includes a large number of intermediate indicators, particularly for Component 1, that aim to capture readiness and implementation progress, they lacked a clear logic and failed to systematically capture the level of achievement at different stages in the implementation. The battery of indicators recommended as part of the process evaluation of the BJA Program was far superior, as they were telescopic indicators that focused progressively on the subsets of the beneficiaries successfully completing the different stages of the program that is, enrolled pregnant women as a proportion of target universe of pregnant women; enrolled pregnant women with complete prenatal check-ups as a proportion of enrolled pregnant women; pregnant women that received full payments for prenatal check-ups as a proportion of pregnant women with complete prenatal check-ups as well as the overall synthesis percentage of pregnant women that received full payments for prenatal check-ups as a proportion of the target universe of pregnant women. With regard to evaluation, two process evaluations were programmed under the project to identify and address potential implementation problems in the two programs supported under the project. A process evaluation for the BJA Program was planned once the first 10 municipalities had been incorporated and households and communities were receiving the benefit. A process evaluation of the MPED Program was to be conducted during the first year of implementation to evaluate the pilot and, if required, adjust the program s design. In addition, an independent impact evaluation was envisioned to allow for the rigorous measurement of BJA outcomes in conjunction with the Health Adaptable Program Loan III operation that was implemented in the same target municipalities as those supported under the project. Finally, beneficiary surveys were anticipated during the implementation of the BJA Program. M&E implementation. The Management Information Systems (MISs) developed in the MoH and MoL to support the implementation of the BJA and MPED programs served to track physical and financial progress and to generate regular reports on program implementation. They 9

25 were also utilized for the payment of beneficiaries. Although there was room for improvement, the two systems were fully operational and served their purpose. With regard to evaluation activities, although there were delays in the implementation of the BJA evaluation (it was only carried out in 2012, with initial results by the end of 2013), it was methodologically sound, exhaustive, and highly representative, focusing on both beneficiaries and health care providers in four departments representing 73 percent of the beneficiaries. 8 In addition, a tracking survey of MPED beneficiaries was carried out with the objective of measuring the impact of the program with regard to employment and remuneration. 9 M&E utilization. The results of the BJA evaluation and MPED survey continue to be utilized to support the implementation of these programs, which indicates their sustainability. In addition, the results from the two impact evaluations allowed evaluation progress toward attaining the development objectives. Results from the BJA impact evaluation were also used, in combination with the findings from the BJA process evaluation, to modify the BJA Operational Manual to improve overall program performance and execution. 2.4 Safeguard and Fiduciary Compliance Safeguards. OP 4.10 Indigenous Peoples was triggered for Component 1 of the project, and the whole project was rated Category C. The project did not require an Indigenous Peoples Plan for Component 1 as the overwhelming majority of beneficiary population in the 52 municipalities was indigenous and therefore, the project as a whole was considered an indigenous peoples project. Features and requirements of an Indigenous Peoples Plan were incorporated into the design and therefore, project preparation complied with the requirements of the Bank s policy on indigenous peoples (OP 4.10). A communication strategy was designed and implemented during the project to (a) improve program understanding among all stakeholders at national and subnational levels, particularly with regard to its relevance as part of GoB efforts to tackle chronic malnutrition in rural areas, improve transparency, and boost and maintain credibility of the proposed model of intervention; (b) complement cash benefits with a pedagogic message linked to program co-responsibilities, outcomes, social control, and accountability; and (c) inform beneficiary communities about the program s goals and beneficiary rights and responsibilities. Communications were conducted in two mayor indigenous languages (that is, Aymara and Quechua) in addition to Spanish. 8 The BJA impact evaluation utilized data collected at the national level under the Health and Nutrition Evaluation Survey 2012 (Encuesta de Evaluación de Salud y Nutrición 2012 [ESNUT]). This evaluation adopted a quasiexperimental design, with the sample size of the ESNUT being determined so as to ensure that the experimental (BJA participants) and control (target population that did not enroll in the BJA Program) were comparable. A total of 8,433 interviews were conducted that is, 2,456 were in urban areas and 5,977 in rural areas. A process evaluation of the youth employment pilot was conducted in June 2010, which served to adjust the design of the expansion phase that was implemented under this operation. 9 In the MPED tracking survey, beneficiaries were interviewed at three different points in time (that is, all participants at the moment of enrollment in 2012, a sample of 577 participants in December 2013, and a sample of 326 beneficiaries in December 2014). 10

26 Financial management (FM). The project had three implementing agencies: the MoL, the MoH, and the UDAPE, each with its own FM arrangements, Operational Manual, and staff. Except for the UDAPE, limited institutional capacity prevented these implementing entities from putting in place and maintaining adequate FM arrangements to properly support project implementation. Weak coordination between technical and administrative teams, inadequate accounting practices, and a weak internal control system overall limited their capacity to provide reliable financial information required to monitor and facilitate the smooth flow of funds, especially under the cash transfer component. A similar situation happened in certain periods under the MoL component (Component 2). The lack of timely and acceptable financial records caused considerable and recurrent delays in the submission of interim financial reports and audited financial statements, as required in the Financing Agreement. More importantly, the serious weaknesses in their internal control, as observed by the external auditors in the annual audit reports numerous times, led to delays in project implementation and significantly increased the incidents of ineligible expenditures, as was the case for Component 1. Overcoming said limitations required significant and continuous support from the Bank team, starting with providing guidance and including the development and agreement of alternative arrangements where warranted. The Bank continued to support the fiduciary unit of the MoH in charge of the BJA during 2012 and 2013, with the MoH s fiduciary functions being restored by the end of A multi-annual audit ( ) was presented to the Bank in April Procurement. All procurement processes were done under the provisions stipulated in the Financing Agreement; the Guidelines: Procurement of Goods, Works, and Non-consulting Services under IBRD Loans and IDA Credits and Grants by World Bank Borrowers, published by the Bank in May 2004 and revised in October 2006; the Guidelines: Selection and Employment of Consultants under IBRD Loans and IDA Credits and Grants by World Bank Borrowers, published by the Bank in May 2004 and revised in October 2006; and procedures set in Operations Manuals for each component. According to the results of the procurement post review missions and procurement audits for Components 1 and 2, no serious issues that would be considered fraud or corruption were found. In general, performance in procurement was considered Moderately Satisfactory because of delays in contracting processes caused in part by changes in staff. For Component 3, performance was in general Satisfactory, and a procurement audit was not needed because most processes were done under prior review. Legal covenants. The borrower complied with all legal covenants. 2.5 Post-completion Operation/Next Phase The operation can be expected to have a smooth transition after closing. The operation was fully implemented within the GoB s public sector structure, with administrative expenditures being financed by government funds. With regard to key program costs, since the midterm review, the GoB has been fully responsible for the financing of BJA transfers to program beneficiaries, including the 52 municipalities that were initially supported under the project. The MPED Program continues to be operational under the MoL with additional support under a follow-on Bank project the Improving Employability and Labor Income of Youth Project, P that was approved on May 15, 2014 in the amount of US$20 million. 11

27 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design, and Implementation Relevance: High (during both pre- and post-restructuring, since the PDOs and design remained unchanged) The relevance of the project s development objectives is deemed High, as the BJA and MPED programs constitute clear priorities of the current administration as part of the strategy to consolidate social protection interventions, tackle poverty, promote human capital formation, and ensure sustainability of results. Specifically, the BJA Program continues to be a core component of the ZMP, one of the main programs of the Morales administration. The objective of the ZMP Program is to enable children below five years of age, with emphasis on those below two years of age and pregnant and lactating women, to meet their special nutrition-related needs, contributing to the eradication of malnutrition in the most vulnerable areas of the country. Likewise, the MPED Program has been scaled up with support from the aforementioned IBRD operation. Finally, there are ongoing efforts to continue to enhance the government s capacity to design and manage the country s social protection network. PDOs 1 and 2 are also consistent with the Human Development and Access to Basic Services results area of the FY12 15 Country Partnership Strategy for the Plurinational State of Bolivia (Report BO), discussed by the Board on December 1, PDO 3 is consistent with the results area focusing on enhancing the efficacy of the public sector. In addition, the objective of improving health care among pregnant women and children in the early stages through CCTs like the BJA is relevant as reflected in findings from international impact evaluations of similar programs in Latin America and the Caribbean and other regions. Finally, the objective of improving employability of poor youth through training programs like the MPED is relevant for many countries in Latin America and the Caribbean as it shows recent evaluations in Argentina, Brazil, Colombia, Chile, Dominican Republic, Panama, and Peru. 10 The operation s design also continues to be relevant, as the BJA and MPED programs reflect best practices and lessons learned from similar programs in the region. Except for the size of the cash transfers, which may have been insufficient to effectively serve as incentives, and the location of the program in the MoH, which may have reduced the level of demand to influence health service provision, the design of the BJA incorporates key features of successful conditioned cash transfers, such as a clear exit policy, and parallel efforts to strengthen the supply of basic services. Although the BJA Program is universal, this operation targeted poor households by focusing on the country s most vulnerable municipalities. Likewise, the MPED is aligned with recent research and best practices for youth skills development, including the World Development Report (WDR) 2013, Jobs. 11 The WDR recognizes the need to acknowledge the informal sector as the normal in many developing countries, taking into account that the formal sector is still small in these countries and will not grow fast enough to absorb the current large cohorts of youth looking for work. The WDR urges governments to stimulate job creation and productivity gains in both formal and informal private enterprises, to create more and better opportunities for youth in the labor market. 10 Vezza, E Escaneo de Políticas y Meta-Análisis: Juventud y Políticas de Empleo en América Latina. La Plata, Argentina: CEDLAS. 11 World Bank World Development Report 2013: Jobs. Washington, DC: World Bank. 12

28 3.2 Achievement of Project Development Objectives Rating: Moderately Unsatisfactory (both pre-restructuring and post-restructuring periods with respect to the 2013 restructuring) The overall level of PDO achievement is rated Moderately Unsatisfactory for the pre- and post-restructuring periods and is determined by weighting the partial ratings in proportion to the share of disbursements. Despite a decrease in malnutrition rates among children aged 0 2 years in the participating municipalities, the causality between the project and these reductions could not be established (PDO 1). The project has achieved its objectives of enhancing the employability of youth participating in skills development programs (PDO 2) as shown by the results of an impact evaluation. Likewise, the project has achieved the objective of enhancing the government s capacity to design and manage an effective social protection network (PDO 3), as evidenced by substantial improvements in its ability to monitor and evaluate social programs. Coverage, however, was significantly less than anticipated in both the BJA and the MPED, particularly with respect to the original targets. The specific outputs and outcomes corresponding to each PDO can be summarized as follows. PDO 1: Reduce prevalence of chronic malnutrition in children up to two years old - Moderately Unsatisfactory. Achievement of PDO 1 is considered Moderately Unsatisfactory for periods before and after the 2013 restructuring, when the coverage targets of PDO 1 were adjusted downwards. The CCTs approach supported under the BJA Program had strong merits as shown by the significant impacts with regard to improvements in utilization of health care services among participating pregnant women and children aged two years or younger, as well as in health outcomes such as weight at birth and in nutritional practices such as consumption of micronutrients. Unfortunately, the corresponding PDO 1 did not fully capture these impacts due to its sole focus on malnutrition reductions rather than intermediate outcomes as the ones noted above. In addition, lower-than-expected enrollment, low payment rates of benefits among participants, and the erosion of the transfer s incentive in a favorable macroeconomic environment that offered potential recipients increasing economic opportunities, hindered attainment of PDO 1. Project outcomes. Results from the 2012 impact evaluation indicated that the BJA had statistically significant impacts with regard to increasing the demand of pre- and postnatal health care services among its beneficiaries. These are important intermediate outcomes that are in line with the overall development objective of decreasing malnutrition. Specifically, the BJA had a positive impact on the demand for prenatal and postnatal care of BJA pregnant women compared to their non-bja counterparts, as follows: (a) 6 and 10.3 percentage points increase in the probability of having at least one and four prenatal check-ups, respectively; (b) 11 percent increase in the probability of having a health check-up before week 20 of pregnancy and a decrease equivalent to 2.3 weeks of pregnancy for the first prenatal control; and (c) an increase in the number of institutional deliveries (that is, an increase of 4.9 percentage points in the joint probability of having an institutional delivery and have the first post-partum check-up in rural areas). 13

29 There was also an increase in the demand for child health services among BJA children compared to their non-bja counterparts, as follows: (a) up to 3.6 additional check-ups among children aged 0 23 months during the 6 to 24-month period); (b) 11.7, 11.3, and 12 percentage points increase in the probability of having the yellow fever vaccine, Measles, Mumps and Rubella (MMR) vaccine against measles, rubella, and mumps, and the complete basic immunization schedule, respectively); and (c) nutritional improvements (that is, 10.8 percentage points increase in the likelihood of utilizing nutritional supplements Chispitas ). The BJA also had a statistically significant impact on health outcomes, including: (a) reduce low birth weight (that is, 8.3 percentage points decrease in the probability of weight at birth being below 2,500 grams in urban areas) and (b) achieve a 7.8 percent reduction in the probability of children in rural areas participating in the program having anemia with respect to their counterparts. Project outputs. According to the BJA MIS, enrollment rates were lower than anticipated, averaging 25 percent for eligible pregnant women and 50 percent for eligible children aged 0 2 years between 2009 and There was also a gap between the enrollment and the receipt of cash transfers, with only 61.2 percent of enrolled pregnant women and 66.2 percent of enrolled children receiving at least one payment. As a result, it is estimated that, on average, only 30.3 percent of mothers with children aged 0 2 years living in the targeted municipalities received BJA transfers (43 and 61 percent level of achievement with respect to original and revised target, respectively). See annex 2 for a more detailed description of outputs. PDO 2: Increase employability of low-income youth - Moderately Satisfactory. Achievement of PDO 2 is considered Moderately Satisfactory throughout the operation s lifetime. With project support, the GoB has improved the design and scaled up the implementation of the MPED Program in two additional cities, aimed at enhancing the employability of low-income youth. The intervention has been shown to have an impact with regard to the probability of obtaining employment as well as the earnings of participants. Coverage, however, was significantly lower than expected due to implementation delays, particularly with respect to prerestructuring targets. Project outcomes. Results from the 2015 impact evaluation indicated that the MPED had significant impacts with regard to increasing the employability and labor quality among its beneficiaries. The final survey indicated that 72 percent of beneficiaries had an employment contract four months or more after the end of the internship, exceeding the target (144 percent level of achievement). The impact evaluation showed that 21.9 percentage points of the employment rate could be attributed to the MPED. Additionally, the rate of formal employment (that is, with social security contributions) increased from 5.2 to 18 percent among male and from 2.2 to 10 percent among female participants, with an attribution of 10 percentage points to the MPED. Finally, earnings also improved among youth who participated in the MPED Program, by 49 percent and 100 percent among male and female participants, respectively. This increase is the result of both a higher hourly wage and more hours of work. With regard to earnings, the attribution to the MPED is roughly 60 percent of the average increment. Project outputs. A total of 1,367 MPED beneficiaries completed the training courses and received a medium-level technical degree accredited by the MoE in the areas of apparel 14

30 construction, gastronomy, construction, metallurgy, and domiciliary gas service installations. See annex 2 for a more detailed description of outputs. PDO 3: Institutional Strengthening of the RPS-DIC - Satisfactory. Achievement of PDO 3 is considered Satisfactory throughout the operation s lifetime. With project support, the UDAPE has developed considerable institutional capacity to fulfill its responsibility in building a sustainable and integrated social protection system, including evaluation capacity and the development of modern management tools to monitor social programs as well as socioeconomic conditions of their beneficiaries. Project outcomes. Seven social programs with national coverage are now being regularly monitored under the Social Program Monitoring System (SIMPS), including the ZMP and the BJA. The MPED Program is in the process of being incorporated into the system. Annual progress reports on RPS-DIC Programs are submitted to CONAPES utilizing data from the SIMPS, and the Unified Register of Beneficiaries (Registro Unico de Beneficiarios [RUB]) is now operational. Project outputs. A SIMPS was designed and implemented. 12 The SIMPS supports the management of social programs by systematically monitoring progress in outputs and outcomes under individual programs Logical Framework. An RUB has been created to improve the targeting of social programs. With project support, the UDAPE also conducted evaluations of the BJA and MPED programs (see annex 2 for a more detailed description of outputs). 3.3 Efficiency Rating: Substantial Overall, the project s efficiency is considered Substantial. BJA. Although a statically significant relationship between the observed reductions in malnutrition and the project could not be established, the economic analysis conducted as part of the impact evaluation of the BJA Program indicated that it was a cost-effective intervention based on other impacts, including: (a) reductions in low weight at birth; (b) reductions in the incidence of anemia in rural areas; and (c) reductions in perinatal mortality associated with the BJA. It is estimated that the annual implementation of the BJA yields a reduction of 47,512 disabilityadjusted life years (DALYs). 13 Taking into consideration an annual implementation cost of US$35.4 million (including also the costs incurred by beneficiaries, such as transportation), the unit cost of each DALY avoided amounts to US$745, which falls into the category of highly cost efficient under the World Health Organization (WHO) guidelines. 14 The robust economic 12 The design of the SIMPS and RUB was financed with resources from a loan from the IADB the Multi-Phase Program to Support the Plan to Eradicate Extreme Poverty, Loan No Component 3 supported the operationalization of SIMPS as well as the transfer of the RUB to the MDP. 13 The DALY is a measure of overall disease burden, expressed as the number of years lost due to illness, disability, or early death. It was developed in the 1990s as a way of comparing the overall health and life expectancy of different countries. 14 According to guidelines suggested under the Choosing Interventions that are Cost-effective (CHOICE) Program of the WHO, the unit cost of a DALY can be compared to the country s GDP per capita to determine an intervention s economic efficiency. In the case of the unit cost being below the GDP per capita, the intervention is deemed highly cost-effective. In the case of Bolivia, the GDP per capita was US$2,480 in 2012, roughly three times higher than the unit costs of the DALYs avoided as a result of the implementation of the BJA Program. 15

31 efficiency of the BJA was confirmed by the economic analyses conducted as part of this ICR based on the project s outcomes at closing, which renders a US$716.1 unit cost of DALY avoided. The cost-benefit ratio of is also robust, indicating that for every Bolivian peso invested for the project, Bs 3.46 was achieved with regard to income earned by DALYs (see annex 3). The cost of Component 1 represented 49.8 percent of the overall project cost. MPED. A cost-benefit analysis of the MPED Program was conducted as part of the program evaluation to determine its cost-efficiency. Results indicate that the MPED intervention was economically efficient, with an internal economic rate of return (IERR) of 52 percent and a cost-benefit ratio of 0.24, indicating that per each Bolivian peso invested in each beneficiary, the program obtains a benefit equal to Bs 4.25 with regard to wage gains assuming a 6 percent discount rate. The cost of Component 2 represented 40.5 percent of the overall project cost. While the economic efficiency of Component 3 was not formally estimated as part of this ICR, it can be expected to be highly economically efficient given the expected gains to be derived from more effective monitoring and design of social programs. 3.4 Justification of Overall Outcome Rating Rating: Moderately Unsatisfactory To determine the overall rating, this ICR utilizes the methodology described under the joint guidelines for restructured operations in which objectives or associated outcome targets have been formally revised. 15 In the case of this operation, the outcome targets for PDO 1 were revised. Thus, the project outcome is assessed against both the original and revised outcome targets (that is, preand post-restructuring periods). The overall outcome rating is subsequently determined by weighting the pre- and post- restructuring partial ratings in proportion to the share of actual disbursements corresponding to each period. 16 The operation s outcome is deemed Moderately Unsatisfactory during the pre-restructuring period to reflect its moderately unsatisfactory efficacy regardless of its high relevance and substantial efficiency. The operation s outcome during the post-restructuring period is deemed moderately satisfactory, to reflect the moderately satisfactory efficacy, high relevance, and substantial efficiency. Overall, the project is considered to be Moderately Unsatisfactory to reflect the weighted average of a Moderately Unsatisfactory and Moderately Satisfactory achievement during the pre-restructuring and post-restructuring periods, based on disbursements, respectively (see table 1). 15 ICR Guidelines, OPCR, August 2006, last updated on July 22, The disbursement amounts reflect execution rather than actual disbursements. Specifically, the pre-restructuring period reflects a reimbursement to the GoB in the amount of SDR million made on June 30, 2015 corresponding to the financing of BJA transfers made during the pre-restructuring period. 16

32 Disbursements Table 1. Project Rating Pre-restructuring Period ( ) Post-restructuring Period ( ) SDR 6.18 million out of SDR 7.96 million (78% of actual disbursements) Relevance High High Efficacy (*) - PDO 1 - PDO 2 - PDO 3 Moderately Unsatisfactory - Moderately Unsatisfactory (73%) - Moderately Satisfactory (21%) - Satisfactory (6%) Efficiency Substantial Substantial 17 SDR 1.78 million out of SDR 7.96 million (22% of actual disbursements) Moderately Satisfactory - Moderately Unsatisfactory (49%) - Moderately Satisfactory (41%) - Satisfactory (10%) Partial Rating Moderately Unsatisfactory (3 out of 6) Moderately Satisfactory (4 out of 6) Weighted Partial Rating (**) 3 x 0.78 = x 0.22 = 0.89 Overall Project Rating (***) Moderately Unsatisfactory (3.22) (*) The percentages represent the amount of credit resources corresponding to each PDO during the period. (**) The weighted partial rating considers the rating for the pre- and post-restructuring periods weighted by the proportion of disbursements corresponding to each period. (***) The overall project rating considers the weighted partial rating and the disbursement ratio for both periods. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development Poverty impact. The operation is expected to have a strong poverty impact, particularly the BJA. While the two programs supported under the operation targeted low-income beneficiaries, they had different targeting mechanisms. In the case of the BJA, targeting was based on two criteria: (a) geographical location (that is, the 107 poorest municipalities) and (b) lack of health insurance coverage. It is estimated that this approach has resulted in roughly 10.7 percent of beneficiaries that are non-poor for the program as a whole (that is, including both poor and nonpoor municipalities), which is considered an acceptable trade-off in view of the costs associated with the implementation of more stringent eligibility verification. This percentage is expected to be significantly lower in the 52 municipalities that were supported under this operation (Apella and Blanco 2015). In the case of the MPED, targeting was based on the following criteria: (a) self-reported household income below the poverty line in addition to either (b) an electric bill showing a consumption of less than 70 KW per month or (c) proof of having attended a public school. While these targeting criteria are considered effective from a theoretical perspective, in the event of income underreporting, up to 85 percent of non-poor youth can be included in the program (Apella and Blanco 2015). Gender impact. The operation is expected to have a strong impact on women. The two programs supported under the operation have a strong focus on women s welfare. In the case of the BJA, women constitute the primary beneficiaries, either directly if pregnant, or indirectly if

33 mothers of children aged 0 2 years. While the MPED targets youth in general, regardless of gender, 59 percent of those who successfully completed the program were female compared to the 40 percent target. It is likely that pro-women design features such as including the cost of day care into mothers stipends helped make the program more attractive to female participants. In addition, the MPED seems to have had a greater impact on female participants with regard to income, contributing to closing the gender gap in income between male and female participants in their post-program employment. (b) Institutional Change/Strengthening The operation had a significant institutional impact, ensuring that the design of the two social programs supported under the operation reflected lessons learned from best practices in the region. It also supported the development and implementation of effective institutional mechanisms and support systems, including M&E systems, to enable the implementation of the BJA and MPED programs. This is particularly important as these programs are quite complex, involving activities at the department and municipal levels and, in the case of the BJA, with a large number of beneficiaries in the entire country with limited permanence in the program (pregnant women with a maximum permanence of 12 months and children below 2 years old with a maximum permanence of 24 months). As noted by the BJA process evaluation (2013), while the BJA systems and processes offer many opportunities for improvement, they are succeeding in supporting the program s implementation. The operation also resulted in significant capacity building within the UDAPE with regard to evaluation as well as monitoring of social programs. In the context of the newly developed SIMPS, the UDAPE has signed technical cooperation agreements with other government agencies, including the MoH, MoL, MoE, the Pension and Insurance Overseeing Authority, and the housing agency. A technical team has been established within the UDAPE that is responsible for monitoring social programs. In addition, the newly developed consolidated beneficiary registry under the responsibility of MPD, RUB, can be expected to improve the targeting of both existing and future social programs. (c) Other Unintended Outcomes and Impacts (positive or negative) By supporting a rich evaluation agenda, the operation contributed to strengthening evaluation capacity within the UDAPE and the evaluation culture in Bolivia. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Not applicable. 4. Assessment of Risk to Development Outcome Rating: Moderate The GoB s strong support for the BJA and MPED programs positively affected the likelihood that outcomes achieved under this operation will be maintained over time. The BJA Program continues to be a priority for Bolivian authorities in the context of the ZMP, which encompasses not just the BJA, but a broad array of actions from other sectors. The government remains keen to expand coverage of MPED skills development programs and to incorporate other 18

34 training modalities for poor youth. With Bank support, the follow-on operation will continue to support the implementation of skills development programs in 15 urban centers while expanding MPED training modalities to target poor youth entrepreneurs as well as remedial secondary education programs. In the case of the BJA, it is also important to note that in addition to the cash transfers, the program sought to enhance knowledge and change behaviors regarding infant health and nutrition among beneficiary households through individual counseling and collective workshops to beneficiaries, which is expected to help sustain decreases in malnutrition and anemia attained under the operation. However, other factors pose a threat to the sustainability of the outcomes under the two programs. Specifically, the limited implementation capacity of the two implementing agencies can erode the programs effectiveness in the medium and long term. This risk is ameliorated in the case of the MPED Program, since continuous technical support will be provided under the followon operation. In addition, the possibility of less favorable economic conditions in the future can adversely affect project outcomes, particularly with regard to the availability of internships to complement training programs as well as post-training employment opportunities. Finally, less favorable economic conditions may revert some of the strides made in decreasing malnutrition particularly since these gains were not statistically associated with the BJA Program and the cash transfers might become relatively more attractive in such a context, thus incentivizing program participation. The sustainability of the outcomes under Component 3 is considered highly likely, as the UDAPE and the MDP, that are responsible for the operation of the SIMPS and RUB, respectively, exhibit strong institutional and technical capacity. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory With this operation, the Bank brought significant value to Bolivia s social programs, as it supported areas in which the Bank had substantial expertise in program design, implementation, and analysis. The AAA Program that was conducted in parallel with the operation s design served to maximize the Bank s contribution in enhancing policy dialogue and the design of the BJA and MPED programs. As noted earlier, the design of both programs was robust and reflected international experience as well as local conditions and priorities. Even though the Bank preparation team incorporated mechanisms to address some of the risks identified at appraisal for example, having a government unit responsible for overall project coordination, definition of pilot phases of the BJA and MPED, and the establishment of an independent payment agent for the BJA the commitment of the government on those mechanisms was over estimated by the Bank at appraisal. These features were changed in the first two years of implementation (pilot of the BJA and overall coordination unit were dismissed). Additionally, the Result Framework presented some weakness in the presentation of the KPIs 19

35 along with the PAD and some intermediate indicators for Component 1 did not totally capture readiness and implementation progress of the BJA. (b) Quality of Supervision Rating: Moderately Satisfactory The performance of the task team during supervision is deemed Satisfactory. The operation was regularly supervised and documented with the support of sector specialists. The entire list of PDOs and intermediate indicators was tracked in each supervision mission, including a description of the trends and an estimate of the likelihood of reaching the target at the end of the project. Supervision efforts also included close monitoring of fiduciary compliance, which, together with the training in FM and procurement, helped overcome serious implementation bottlenecks. Supervision became particularly challenging after the elimination of the TCU as the Project Coordinating Unit; the Bank supervision team had to coordinate actions with the three implementing agencies. Limited implementation capacity of two of these agencies also required additional attention on the part of the Bank s team. Throughout the project s lifetime, the Bank provided relevant technical expertise in the three substantive areas supported under the operation. Moreover, the Bank supervision team engaged in research activity that helped verify the efficacy of targeting mechanisms and efficacy of the programs supported under the operation. Strong support from local Bank staff also contributed to the close supervision of the project. The midterm review was conducted on schedule and served to identify and address critical implementation weaknesses. By restructuring the operation in 2013, the Bank exhibited flexibility and continuous commitment toward providing support to the GoB in the implementation of two of its flagship social programs. While Bank management followed project implementation closely, as reflected by their detailed comments in the Implementation Status and Results Reports and support provided to the project throughout its implementation, there were some management decisions that hindered project implementation. Specifically, agreeing with the GoB s decision in 2011 to eliminate the overall coordination function of the TCU made supervision much more challenging and costly. Moreover, having had supervision from the TCU may have helped overcome some of the implementation challenges within the MoH and MoL, increasing the accountability of responsible parties in critical issues such as excessive delays in contract approval and submission of financial statements and financial audits. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory Overall Bank performance is deemed Moderately Satisfactory to reflect the moderately satisfactory performance at both preparation and supervision. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 20

36 The government s performance is deemed Moderately Satisfactory. The GoB demonstrated strong ownership and commitment for the implementation and sustainability of the two programs supported under this operation. This is not surprising given that both the BJA and MPED programs were entirely identified by Bolivian authorities. Specifically, the MDP in conjunction with the MoH and MoL led the conceptualization of the BJA and MPED programs with support from the Bank, which helped inform program design by sharing knowledge regarding international best practices and lessons learned throughout project preparation with the government counterparts. Government support and ownership have been reflected in the availability of counterpart funding to support implementation as well as the decision to fully finance the cash transfers under the BJA Program once IBRD financial support for Component 1 was interrupted and subsequently cancelled following the midterm review. Finally, government decision to launch the BJA nationwide created some operational constraints in program implementation. (b) Implementing Agency or Agencies Performance Rating: Moderately Unsatisfactory The overall performance of the implementing agencies is considered Moderately Unsatisfactory, although uneven across agencies. Specifically, the performance of the MoH is considered Unsatisfactory due to the poor implementation performance of Component 1 over the project s lifetime. The MoH failed to provide enough resources to implement the program nationwide and to create synergies between the health care services and the management of the BJA at the local level. Additionally, the inability to manage fiduciary aspects of the project was partially explained by a decision of the MoH to create a central fiduciary unit which during the implementation period had lack of qualified staff and lack of coordination with the BJA technical unit in charge of program operation. Finally, high staff rotation at the project and ministerial levels, especially in the first three years of project implementation, also hindered implementation. The performance of the MoL is also considered Moderately Unsatisfactory due to the implementation delays that postponed the initiation of the program from 2011 to 2012 (change in MoL administration at the beginning of 2011 postponed the signature of contracts with training institutions for 10 months) and required an 18-month extension of the original closing date, the failure to fully disburse the resources reallocated to Component 2 under the 2013 restructuring, and the inability to reach the expected number of beneficiaries under the employment program over the extended implementation period. In addition, there were significant delays in complying with some FM requirements, which affected project implementation, and there were long periods of vacancies in key staff positions in the National Project Coordination Unit. However, the great dedication of the personnel in the National Project Coordination Unit as well as the high-quality final evaluation it produced should be noted. The performance of the UDAPE is deemed Satisfactory, as it accomplished all activities under Component 3 as envisioned. In addition, the UDAPE has to be commended for the significant inter-institutional efforts required to carry out the impact evaluations of the BJA and MPED programs. The BJA evaluation, in particular, required the design of the survey instrument to be utilized for the ESNUT 2012, as well as coordination with multiple sources of financing. 21

37 (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Unsatisfactory Consistent with the guidelines of the Independent Evaluation Group, overall borrower performance is deemed Moderately Unsatisfactory to reflect the moderately satisfactory performance of the government and the overall moderately unsatisfactory performance of the implementing units. 6. Lessons Learned Important lessons of experience emerged from the implementation of this operation that can be of use in the design of social protection projects in Bolivia as well as Latin America and the Caribbean and other regions. They can be summarized as follows. (a) Lessons of General Application It is of utmost importance for the PDOs to reflect outcomes that are within the orbit of the project. In the case of the BJA Program, the objective of reducing malnutrition was too ambitious, as it is determined by a wide array of factors that go beyond the intervention supported under the project. Other outcomes, such as reduced incidence of mortality and low weight at birth, would have been more appropriate. Favorable economic conditions can impact project outcomes both positively and negatively. In the case of the BJA Program, it is likely that growing income-generating opportunities among the targeted population may have eroded the financial incentives of the CCTs under the program. Conversely, fast economic growth is likely to have fueled the availability of internship opportunities for MPED trainees as well as subsequent opportunities for employment. For incentives to work, they have ben gauged correctly. The fact that incentives matter has long been accepted among the development community. The implementation experience of the BJA and MPED programs also highlights that, for incentives to work, they have be gauged correctly. As previously noted, the CCTs did not adequately take into consideration the transport costs incurred by intended low-income recipients in rural areas, a population that was one of the program s prime targets. They were also not adjusted to reflect the increasing opportunities for employment that became available in an environment of fast economic growth and that resulted in transfers not being so attractive. Alternatively, the stipends provided to the MPED proved to be attractive, as shown by the program s low desertion rates. They adequately took into consideration participants transport and opportunity costs as well as, in the case of mothers, childcare costs. Stipends were also adjusted in line with increases in the minimum wage. Pilot programs provide opportunities for critical learning and minimizing costly mistakes. The implementation experience of the MPED Program illustrates that, although time consuming, pilot experiences can be crucial in improving program design and provide the opportunity for developing sound support systems for program implementation. Alternatively, the BJA Program was implemented nationwide, even though it was originally designed as a pilot experience focusing on high-priority target population. The challenges of massively implementing a program that was highly innovative in the context of Bolivia overwhelmed the MoH s limited 22

38 implementation capacity. Moreover, shortcomings in the program s design as well as operational difficulties that could have been identified in a pilot experience may have caused lack of confidence and credibility to the program, eroding its ability to effectively draw participation. There are often difficult trade-offs between government decisions and program readiness. The case of the BJA Program posed a difficult choice for the Bank. While from a technical standpoint the program called for a limited pilot experience as the first step toward program implementation, the President s support for the program and his decision to implement it nationwide created a great challenge for the Bank s and the MoH s technical teams that could not be easily resolved. Establishing a CCT program within the agency responsible for providing the services requires clear internal agreements. In the case of the BJA, the Program Coordination Unit was established in the MoH. Program co-responsibilities were exclusively linked to services provided by the MoH. International experiences showed that implementation of CCTs creates the responsibilities to improve the coverage and quality of service provision. In the case of the BJA, lack of specific agreements between the program and the responsible unit for health services reduced the final coverage in some distant areas and also forced the BJA to hire doctors and nurses to guarantee the proper outreach, registration, and attention of beneficiaries. Women appear to be more likely to benefit from the potential positive impact from skills building programs, targeted to poor unskilled youth, such as the MPED. This is particularly important since poor young women tend to bear a disproportionate share of unemployment. Under the MPED, the needs of female participants received particular attention, as reflected in stipends including the costs of day care. Including a rich evaluation agenda at the time of project conception can enhance effectiveness. This operation benefitted from a set of evaluation activities, including both impact and process evaluations, which permitted to better assess the project s efficacy as well as the identification of operational bottlenecks. It is important to ensure the integration of program-supported health care services into the country s health delivery network. In the case of the BJA Program, creating a parallel structure that is not integrated with the health delivery network at the department and municipal levels may hinder program effectiveness and result in a suboptimal allocation of human resources. Operational arrangements affect project efficiency. In the case of the MPED, paying stipends and salaries directly to beneficiaries served to reduce the cost of contracts with private training institutions, improve transparency and efficiency of the payment mechanism, and reduce the time of justification of expenditures. Government and Bank portfolio decisions could result in overly ambitious project scope. The preference of both the Bank and smaller borrowing countries to bundle several dissimilar and often overly complex interventions under a single, larger operation is likely to result in a systematic bias in the country portfolios, as the probability of an overall project satisfactory performance is greatly diminished. 23

39 (b) Project-specific Lessons Under the follow-on operation, it is imperative to decentralize the approval of procurement documents within the MoL to avoid serious bottlenecks in the implementation of the MPED Program. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies See annex 7. (b) Co-financiers Not applicable. (c) Other partners and stakeholders Not applicable. 24

40 Annex 1. Project Costs and Financing (a) Project Cost by Component (in SDR million equivalent) Components Appraisal Estimate (SDR, millions) Actual/Latest Estimate (SDR, millions)* Percentage of Appraisal Component Component Component Total Project Costs Note: * The difference reflects a cancellation of SDR 2.5 million. (b) Financing Source of Funds Type of Cofinancing Appraisal Estimate (SDR, millions) Actual/Latest Estimate (SDR, millions) Percentage of Appraisal Borrower Counterpart funds IDA Credit

41 Annex 2. Outputs by Component Component 1 - Implementing and scaling up the MCP to combat chronic malnutrition The Maternal and Child Health program BJA seeks to provide incentives to encourage the use of maternal and child health services according to the health care protocols for pregnant women and children aged below 2 years, thus contributing to improve the health among this population group. The BJA is a CCT Program under the policy of social protection and integral community development. It was established by Supreme Decree No. 066 dated April 3, 2009, with implementation starting that same month nationwide. The BJA provides cash transfers to pregnant women as well as mothers or caregivers of children aged below 2 years, conditioned upon the use of preventive health services during pregnancy, birth attended by skilled health personnel, and compliance with health checks of children. In this way, the BJA seeks to improve the early detection and treatment of health conditions and nutrition deficiencies in pregnant women and children aged below 2 years. The BJA is open to all pregnant women and children aged below 2 years who do not have health coverage. The supply of health services is guaranteed at no cost to the beneficiaries under the Integrated Health Services Law (Ley de Prestaciones de Servicios de Salud Integral) through the public health system. The program has universal coverage for all pregnant women and children under 2 years of age who lack health insurance, regardless of their socioeconomic status, area of residence, or other criteria. In 2012, only 17.5 percent of the Bolivian population was covered by social security and only 2.2 percent had private health insurance. In this context, the universe of eligible women and children to the program was estimated at about 80 percent of pregnant women and children between the ages of 0 and 2 years. The BJA Program awards a bonus of Bs 50 for each completed prenatal care up to a maximum of 4 controls and a bonus of Bs 120 for births attended by skilled health personnel and have a postnatal control. In the case of children, the BJA Program transfers Bs 125 for each bimonthly check-up. The maximum amount of benefits for pregnant women amounts to Bs (US$46) during the gestation period, while the maximum amount of benefits for children under 2 years of age amounts to Bs 1, (US$217) during a 24-month period. The total amount of transfers offered by the BJA Program is equivalent to 1.2 times the minimum monthly wage. This operation provided support for the implementation of the BJA Program in the 107 most vulnerable municipalities between 2009 and 2012 (see Table 2.1). Subsequently, BAJ implementation continued with financial support from the GoB. 26

42 Table 2.1. Number of Participating Municipalities by Departmental Unit Departmental Units No. of Municipalities Chuquisaca 24 La Paz 31 Cochabamba 9 Oruro 3 Potosí 17 Tarija 4 Santa Cruz 4 Beni 3 Pando 12 Total 107 Source: Final Report Bono Juana Azurduy (MoH, 2015). The main outcomes of this component can be summarized as follows: According to the BJA administrative records, 46,500 pregnant women and 83,609 children aged 0 2 years enrolled in the program in the 107 participating municipalities, from its inception to December Of the overall BJA investments, 75.6 percent was used for cash transfers, 17.9 percent for payments to community doctors, and 6.5 percent for administrative expenses. Around 91 percent of beneficiary women who know their rights to access services and co-responsibilities participated in the program (101 percent level of achievement). It is estimated that, on average, only 30.3 percent of mothers with children ages 0 2 years living in the targeted municipalities received BJA transfers during the period. It is estimated that, on average, only 21 percent of pregnant women living in the targeted municipalities received BJA transfers during the period. While there was widespread awareness about the BJA Program (87 percent of women in reproductive age reported knowing about the BJA in 2012), enrollment rates were less than originally anticipated. According to the BJA administrative data, enrollment rates among pregnant women have ranged between 24.5 percent and 39.8 percent since the BJA inception, peaking at the time of program inception and exhibited a sustained decrease thereafter. The average enrollment rate among pregnant women for the period was 34 percent (see Figure 2.1). 27

43 Figure 2.1. Enrollment Rate among Eligible Pregnant Women and Children ages 0 to 2 years Source: ESNUT According to the BJA administrative data, enrollment rates among children aged 0 2 years ranged between 37.4 percent and 45.8 percent since the BJA inception, with enrollment rates increasing during the first year of implementation and remained roughly flat thereafter. The average enrollment rate among children for the period was 49.5 percent (see Figure 2.1). Respondents identified lack of information, required documentation, and long waiting times as the main barriers to BJA participation. There was also a gap between the enrollment and the receipt of cash transfers, with only 61.2 percent of enrolled pregnant women and 66.2 percent of enrolled children receiving at least one payment. As reported by beneficiaries, cash transfers were primarily used for the purchase of food, clothing, diapers, and milk. Component 2 - Improving and expanding a skills development program for low-income unemployed youth living in urban and peri-urban areas The MPED is a Skills Development Program, comprising a vocational in-class training session coupled with internships in private and public sector firms, in such a way that poor youth gain practical experience and acquire life skills that help them succeed in the workplace. Eligible MPED beneficiaries are poor youth aged years, who have completed at least primary education. The main outcomes of this component can be summarized as follows: A total of 1,367 low-income youth enrolled in the MPED Program under this operation (36 and 78 percent level of achievement with respect to original and revised targets, respectively). Ninety-three percent of those enrolled in the program completed the in-class training phase, exceeding the target (116 percent level of achievement). 28

44 Of those who completed the in-class training phase, 86 percent completed the internship, exceeding the target (108 percent level of achievement). Women represented 59 percent of those who completed the internship, amply exceeding the target (148 percent level of achievement). The 1,095 beneficiaries who completed the entire program (that is, equivalent to 80 percent of enrollment) received a medium-level technical degree accredited by the MoE. Out of the 46 training packages that were contacted, 44 were successfully completed, focusing on apparel construction, gastronomy, construction, metallurgy, and domiciliary service installations (plumbing, electric, and so on). Component 3 - Institutional strengthening of the RPS-DIC This component was implemented by the UDAPE. The main outcomes of this component can be summarized as follows: A Health and Nutrition Evaluation Survey (ESNUT) was conducted in 2012 to collect information to be used in the impact evaluation of the BJA and the mid-term evaluation of the ZMP. A representative sample of 8,433 households was interviewed nationally. The impact evaluation of the BJA Program was conducted in 2013 using data from ESNUT Results were published in Beneficiary surveys were conducted at the time of enrollment and subsequently in December 2013 and December The data collected was used to conduct an impact evaluation of the MPED Program (2015). An SIMPS was designed and implemented. 17 The SIMPS supports the management of social programs by systematically monitoring progress in outputs and outcomes under individual programs Logical Framework. An RUB has been created to improve the targeting of social programs. Annual progress reports on the RPS-DIC Programs are being regularly submitted to CONAPES using data from the SIMPS. Seven social programs with national coverage are now being regularly monitored under the SIMPS, including the ZMP and the BJA. The MPED program is in the process of being incorporated into SIMPS. The RUB is now operational under the orbit of the MDP. 17 The design of the SIMPS and RUB was financed with resources from a loan from the Inter-American Development Bank the Multi-Phase Program to Support the Plan to Eradicate Extreme Poverty, Loan No Component 3 supported the operationalization of SIMPS as well as the transfer of the RUB to the MDP. 29

45 Annex 3. Economic and Financial Analysis 18 This annex presents a review of the results of economic and financial analysis conducted for the proposed Bolivia Investment in Children and Youth Project (P101084). As part of the National Development Plan, the Plurinational State of Bolivia launched two programs in 2009, a CCT Program the BJA and a youth labor training program the FEP (later renamed MPED). The latter started with the objective of tackling the shortcomings of employability among the poor youth between 18 and 24 years of age by offering vocational training and internships The former was created with the objective of decreasing the levels of maternal and infant mortality and chronic malnutrition among children below 2 years of age through incentivizing the use of health care services among pregnant women and children below 2 years of age. UDAPE (2015) and Apella, Blanco, and Landa (2015) analyzed the impact evaluation of both programs and found causal effects of each program on the pursued objectives on the beneficiaries. This annex presents an economic analysis for both programs using as inputs the results achieved in the impact evaluations. Additionally, the annex includes the calculation of the cost-benefit or cost-efficiency ratio. Juana Azurduy Mother-Child Grant 19 The BJA was created in 2009 with the objective of decreasing the levels of maternal and infant mortality and chronic malnutrition among children younger than two years old. The target population comprises all pregnant women and children below 2 years of age who do not have formal health care coverage. The stipend is a cash incentive of Bs 320 that is given to the pregnant woman for attending four prenatal checkups Bs 50 for each one a delivery attended by health care personnel, and one postnatal checkup (Bs 120). For children below 2 years of age, the program delivers a transfer of Bs 125 for each of the two-month development checkups of the child. In total, the mother receives an incentive of Bs 1,820. The program comprises a demand incentive for health care services and covers all of the municipalities in the country. According to the impact evaluation carried out by UDAPE (2015), the program had an impact on the indicators of low birth weight, perinatal mortality, and anemia, expressed in terms of DALYs prevented. In this regard, the evaluation of the effectiveness of the program is based on the calculation of cost-effectiveness, given the nature of the output indicators of the program. The calculation of the economic value of the DALY involves a number of strong assumptions about the working life of children. Therefore, to determine the effectiveness of the program, we use the cost-effectiveness analysis based on the parameters defined by the WHO. However, to estimate some measure of cost-benefit ratio, we assume an economic value for each DALY gained equal to the GDP per capita as a proxy. 20 In this sense, we can obtain the 18 This annex was prepared by Ignacio Apella and Gastón Blanco. 19 This section is based on UDAPE (2015). 20 See Brown (2008) for more details. 30

46 economic benefits of the project through the product of the quantity of the DALYs prevented and the GDP per capita. The calculation of the cost-effectiveness and cost-benefit ratio implies the following steps. First, we use the WHO statistics associated with the burden of disease attributable to premature birth, low birth weight, anemia, and the estimated prevalence of each disease in the Survey of Health and Nutrition Evaluation (2012). The DALYs are calculated for each low birth weight, each birth survival, and every child without anemia in Bolivia. Then, BJA benefits are estimated in terms of DALYs prevented taking as input the estimated reduction in the incidence of low birth weight, mortality at birth, and anemia, from the impact evaluation study. This information allows us to capture this gain through DALYs. Finally, a monetary cost per DALY prevented is calculated by dividing the total cost of the program and the estimated benefits measured in DALYs. This value can be compared to the one proposed by the CHOICE Program of the WHO, to evaluate the relative effectiveness of an intervention. CHOICE uses the GDP per capita as an available indicator to derive three categories of cost-effectiveness: highly cost-effective (less than the GDP per capita); cost-effective (between one and three GDPs per capita); and not cost-effective (more than three GDPs per capita). The Bolivian GDP per capita was US$2,480 in Table 3.1 shows the total cost of the program and the benefits achieved through DALYs. Table 3.1. BJA Program: Parameters and the Cost-Effectiveness and Cost-Benefit Ratio Component Value Total Cost Bs) 244,832,524 Total Cost (US$) 35,431,624 DALYs prevented by reducing the low birth weight 12,451 DALYs prevented by reducing perinatal mortality 34,963 DALYS prevented by reducing anemia 98 Total DALYs 47,512 GDP per capita (Bs) 17,261 GDP per capital (US$) 2,480 DALYs in $BO 820,104,632 DALYs in US$ 117,829,760 Cost-effectiveness per year (Cost per DALY prevented) ($BO) 5,190 Cost-effectiveness per year (Cost per DALY prevented) (US$) 745 Cost-benefit ratio per year Source: Author s elaboration based on UDAPE (2015). The WHO estimates for Bolivia are 81,275 DALYs associated with low birth weight and preterm birth, 39,500 DALYs with anemia, and 237,900 DALYs for neonatal morbidities associated with mortality, underweight babies, or premature pregnancy. 31

47 These values are used to calculate the profits generated by the program on perinatal mortality. According to the population projections of the National Institute of Statistics (INE), we consider a total of 291,158 births and 1,529,689 children aged 0 to 6 years in Bolivia in Furthermore, and according to the WHO, we consider an annual national incidence of low birth weight of 8.2 percent, an incidence of anemia in children of 52 percent, and a perinatal death rate of 19 per 1,000 live births. With this information, it is estimated that approximately 3.40 DALYs are lost for every case of low birth weight in Bolivia, DALYs for each case of anemia, and 43 DALYs for each perinatal death. The program's impacts with regard to DALYs are (a) the number of low birth weight cases prevented associated with the implementation of the BJA in urban areas is equivalent to 3,670 every year; it implies 12,451 DALYs prevented; (b) the number of births with perinatal death avoided by the BJA is per year, equivalent to 34,963 DALYs prevented; and (c) the number of children without anemia in rural areas as a result of the BJA is 1,979, which represents DALYs prevented. The program's impact is equivalent to 47,512 DALYs prevented per year as a consequence of the implementation of the BJA Program. The total cost of the BJA Program is Bs 244,832,524 (US$ 35,431,624) per year, considering the period from April 2009 to October With these parameters, the social cost per DALY prevented is Bs 5,190 (US$745.7). Using the criteria of cost-effectiveness based on the methodology of the WHO CHOICE Program and assuming that the impacts are extrapolated to all the beneficiaries during the prenatal period, the BJA has a high cost- effectiveness. Finally, from an economic perspective, the cost-benefit ratio is 0.298, meaning that for every Bolivian peso invested for the project, Bs 3.35 was achieved through income earned by DALYs. Mi Primer Empleo Digno The main objective of the MPED is to increase labor force insertion and improve employability among poor youth. The target population is poor youth between 18 and 24 years of age who have at least completed primary education in the public system. The MPED was implemented as a pilot in five cities (El Alto, La Paz, Santa Cruz de la Sierra, Cochabamba, and Montero) in 2009 and was expanded to Tarija and Sucre in 2012, reaching a total of 2,562 beneficiaries by mid The main results of impact evaluation analysis carried out by Apella, Blanco, and Landa (2015) are a positive causal effect on the working conditions of the project beneficiaries one year after receiving the training compared to a control group defined from the Bolivia Household Survey: (a) the likelihood to be employed one year after receiving the training was 19.1 percent higher among beneficiaries; (b) the likelihood to have a formal job was 10.2 percent higher among beneficiaries; and (c) the average wage among beneficiaries of the MPED was 56.3 percent higher than the average wage of the control group. The cost-benefit calculation of the program requires the estimation of the incremental social benefits generated by the intervention, the net social costs of the intervention (program costs), and the social discount rate to compare benefits and cost at present value. 32

48 To estimate the future cash flow of the program (incremental social benefits), it is assumed that this wage gain will remain throughout a beneficiary s economic active life (as the average age of beneficiaries is 21 years and the economic life in Bolivia is defined until 65 years; the period of economic activity for this calculation is 44 years). It is assumed that beneficiaries enter the labor market when they graduate from the program and remain employed for 44 years until their retirement. Also, we assume that the wage gains remain through their working life. These assumptions present a moderately optimistic scenario. 21 The program cost includes the financing of training and stipends for beneficiaries and the operating cost. The internal rate of return is the result of estimating the cash flow of the program considering the investment made by the program as costs and the benefits measured as the difference of wage between participants and a control group. Hence, the internal rate of return is calculated as the value of r obtained from the following equation, where is the wage gains produced by the program for participants at period t and is the cost of the program for participants at period t. 0. Alternatively, with the objective of having a better understanding of effectiveness of each invested Bolivian peso, it is possible to estimate the cost-benefit ratio. This ratio is represented by the following equation: where, C T t0 C t.δ t-1 B T t0 G t.n t.α.δ t-1, represents the average gain wage gain of each beneficiary of the program in each period t. represents the number of beneficiaries who completed the three stages of the program in each period t. is the percentage of the beneficiaries who are employed (91.6 percent). represents the total costs of the program in each time period t. is the discount factor equal to 1 1, where r is the discount rate. Table 3.2 shows the basic parameters of the program such as the number of beneficiaries who have graduated from it, the average age of the beneficiaries, and the number of years they will be in activity. Also, it presents the wage gain estimated in the impact evaluation, the employment 21 Two alternative scenarios could occur. One of these is in which the wage gap will converge to zero as the years pass. The second one implies that the wage gap is maintained over time but is equal to the wage gap between the control group and the supplemented group. In this document, we decided to present the mid-range scenario, knowing that the results could vary around it. 33

49 rate, and the total cost of the program per year. Figure 3.1 shows these gains by age group of the beneficiary. Table 3.2. Basic Parameters of the Program - MPED Variable Number of beneficiaries Beneficiaries average age (years) 21 Years in activity 44 Wage gains (in % of the average wage of the pesudocontrol group) Employment rate (%) Program s total cost (Bs, millions) Source: Apella, Blanco, and Landa (2015) and UDAPE. Figure 3.1. Wage Gain by Age (In Bs) 1,200 1, Source: Authors own elaboration based on Encuesta de Re-contacto y ECH (2011, 2014). Considering all these variables, the IERR implicit in this flow of costs and wage gains is 52 percent. Figure 3.2 shows the potential gain in annual wages during the lifetime of a representative worker who participates in the program as compared with one who does not participate, discounted by the estimated IERR. Overall, the gain is estimated at approximately Bs

50 600,000 Figure 3.2. Cumulative Wage Gain, Discounted by the IERR (52%) 500, , , , , Period Source: Authors own elaboration. Similarly, it is possible to calculate the present value of the financial flows of the program using three different social discount rates: 3 percent, 6 percent, and 12 percent. Table 3.3 shows the costs and the present values of the benefits of the program. Table 3.3. Cost-benefit Analysis - MPED Discount rate 3% 6% 12% Total Cost (in Bs.) 26,279,884 26,279,884 26,279,884 Present value of the benefit (in Bs.) 144,283,761 70,805,249 36,502,827 Cost/Benefit ratio Source: Authors our elaboration. According to these results we may suggest that the program is efficient in its implementation. Assuming different discount rates such as 3 percent, 6 percent, and 12 percent, the results show that for each Bolivian peso invested in each beneficiary, the program obtains a benefit, in wage gains, equal to Bs 5.49, Bs 2.69, and Bs, 1.39, respectively These values are the inverse of the cost-benefit ratio. 35

51 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team Members Names Title Unit Responsibility/ Specialty Lending Fabiola Altimari Montiel Senior Counsel LEGLE Legal Patricia Alvarez Senior Operations Officer GEDDR Operational Support Jorge C. Barrientos Consultant LCSHS - HIS Social Protection Consultant Miriam Cespedes Program Assistant GGODR Assistant Patricia De la Fuente Hoyes Sr Financial Management Specialist GGODR Financial Management Teddy Ernesto Landaeta Consultant LCCBO Consultant Lourdes Consuelo Linares Loza Sr Financial Management Specialist GGODR Financial Management Karla J. McEvoy Social Protection Specialist GSPDR Operational Support Dena Ringold Lead Economist GSPDR Social Protection Manuel Salazar Lead Social Protection Special GSPDR Task Team Leader Luis M. Schwarz Senior Finance Officer WFALA Finance Maria Alejandra Velasco Operations Analyst LCCBO Operational Support Aracelly G. Woodall Senior Program Assistant GTIDR Assistant Supervision/ICR Gastón Mariano Blanco Sr Social Protection Specialist GEDDR Task Team Leader Patricia Alvarez Senior Operations Officer GEDDR Operational Support Maria Lucy Giraldo Consultant GGODR Procurement Lourdes Consuelo Linares Loza Sr Financial Management Specialist GGODR Financial Management Dena Ringold Lead Economist GSPDR Task Team Leader Manuel Salazar Lead Social Protection Special GSPDR Task Team Leader Maria Concepcion Steta Gandara Sr Social Protection Specialist GSPDR Conditional Cash Transfers Maria Alejandra Velasco Operations Analyst LCCBO Operational Support Patricia De la Fuente Hoyes Sr Financial Management Specialist GSPDR Financial Management Marcela Ines Salvador Social Protection Specialist GSPDR Labor Training Julio Sanjines Gonzales Procurement Specialist GSPDR Procurement Patricia Velasco Project Assistant LCCBO Assistant Ignacio Apella Consultant - Economist GSP04 Economic Analysis Maria Ruth Llanos Consultant - Safeguards GSURR Social Safeguards Cecilia Zanetta Consultant - ICR Preparation GSP04 ICR Primary Author 36

52 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle US$, thousands No. of staff weeks (including travel and consultant costs) Lending FY FY Total: Supervision/ICR FY FY FY FY FY FY FY FY FY Total:

53 Annex 5. Beneficiary Survey Results Not Applicable 38

54 Annex 6. Stakeholder Workshop Report and Results Not Applicable 39

55 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR a. Summary of Borrower s ICR This section summarizes the full versions of the Implementing Agencies Completion Reports which were made available to the Bank between mid-september and end-october The full versions (in Spanish) can be found in the project files. Component 1 - Executive Summary, Final Report on the Bono Juana Azurduy Program, Ministry of Health 23 For the period , the planned targets have been met as established in the process and impact evaluations, the results of which show a difference in coverage of pregnant women and boys and girls under two years of age: 34 percent for women and 49.5 percent for boys and girls. However, there is evidence that the program has stimulated an increase in prenatal controls, especially in rural areas; has contributed to preventing low birth weight; and has led to institutional deliveries in health establishments and fewer ones at home with the presence of appropriate health personnel. In this regard, it is necessary to promote expanded coverage of pregnant women and develop a greater culture of participation in prenatal health controls, looking out for a healthier and safer life, guaranteeing the execution of the solution strategies. This is supplemented with the distribution of a universal health subsidy (a package of fortified food worth Bs 300 each, in a total of four deliveries starting in the fifth month of pregnancy), which began in October 2015 with a transfer of Bs 142,570,000 for the same year. This amount added to the Bs 67,499,783 for meeting joint responsibilities constitutes the investment budget for A prenatal lactation subsidy is needed to improve the quality of life of pregnant women in Bolivia and reduce the pathologies associated with maternal malnutrition, which has an effect on the maternal mortality indicators. The subsidy for pregnant women seeks to reduce anemia in this group, lower the risks of home birthing, and intervenes in the control of the puerperium and postnatal period to ensure the health of the mothers as well as the newborns less than seven days old. This is the period in which both mothers and newborns are at greatest risk of death. It is also necessary to guarantee the continuity of the technical and administrative quality of the personnel of the BJA Program, generating a healthy work environment with labor stability and the personnel s commitment and motivation. It is not enough to establish a government policy; its proper implementation must be monitored; the legal nature of the BJA Program should therefore be analyzed to ensure that the subsidy is efficiently implemented and that the program has the capacity to meet the technical, operational, administrative, and financial challenges. 23 A complete copy of the report submitted by the MoH (Informe Final del Programa Bono Juana Azurduy, Ministerio de Salud, Agosto 2015) can be found in the project files. 40

56 Component 2 - Executive Summary, My First Job with Dignity Program (MPED), Final Report, Ministry of Labor, Employment, and Social Welfare 24 The results achieved of the MPED Program between 2012 and June 2015 can be summarized as follows: Around 1,367 low-income youth were enrolled, 80 percent (1,095) of whom completed mid-level internships, found jobs, or started a business. The dropout rate in all the phases of the training process was 21 percent: 12 percent dropped out in phase 1 (development); 9 percent in phase 2 (consolidation); and 0 percent in phase 3 validation). The male dropout rate was higher than the female dropout rate. To benefit youth in the target group, contracts were signed for 46 training activities. Out of this, 44 were completed and only two were rescinded. Of all the training activities, two areas sewing/clothing industry and cooking/food accounted for 63 percent of the total (29 training activities). In the city of Santa Cruz, only one of the three planned training activities was completed (construction of housing units). The other two were not (sewing/clothing industry and metal mechanics). Most of the low-income youth in the program were women and mothers, accounting for 56 percent of the total beneficiaries. Financial execution of the amounts budgeted for the training activities for training and stipends was 90 percent and 80 percent, respectively. Financial execution of the total MPED Program with respect to the amount estimated is 80.2 percent and 98.6 percent of the amount disbursed. The amount of repayment of resources of the budget assigned to the MPED Program will be US$0.995 million of a budgeted amount of US$5.376 million. The process of the intervention model has been completed with 29 percent of the target group of poor young beneficiaries (1,095 of 3,800). Women and mothers in the program account for 56 percent of the young beneficiaries enrolled (768 of 1,367 enrolled); their percentage was 59 percent in phase 3 of validation or labor entry (645 of 1,095 beneficiaries). 24 A complete copy of the report submitted by the MoL (Programa Mi Primer Empleo Digno [MPED], Informe de Cierre, Septiembre de 2015, Ministerio de Trabajo, Empleo y Previsión Social, La Paz) can be found in the project files. 41

57 Of the 46 training activities, more than 52 percent were carried out in El Alto and Potosí, the cities with the highest poverty rates. Around 63 percent of the training activities were in the sewing/clothing industry and cooking/food. Performance of the Bank. The evaluation of the lending organization would be more complete if we knew all the competencies, functions, and responsibilities of the lending organization in this type of credit, so as not to confuse it with the evaluation of the Bank s personnel. Within the context of its freedom of action, influence, and competencies, the lending organization has assisted in the execution of the project through various mechanisms such as supervisory missions and timely no objections in response to the borrower s requests or requirements. It has provided timely guidance and advice to the National Project Coordination Unit and the Ministry of Labor in several procedures through direct consultations with officials of the lending organization both in the local office as well as with various experts in the Washington, DC office or other offices in the region, as required. The performance of the lending organization and, therefore, the project was affected by the real compression of the institutional environment in which the project had to be executed in the borrower s framework. It is contributing to one of the main Millennium Development Goals of reduction of extreme poverty and hunger, promoting gender equality, and empowerment of women, through results achieved by the MPED Program. Lessons Learned No project should be started (actually initiated) if some conditions are not present: (a) complete teams of human resources; (b) equipment and infrastructure; and (c) instruments for implementation of the intervention model (Operational Manual, Bidding document, training folders). There should be deadlines for implementation, established before starting the training activities. Everybody involved should be aware of these deadlines. There should be some additional counterpart conditions required of the recipient of the credit: (a) a counterpart of financial resources for the grace period after the project s termination and (b) a counterpart of human resources from the recipient (MTEPS). There is a need for attorneys trained in the procedures of the lending organization or with certification in these regulations. Specialized contracting attorneys are needed when there is prior review of the contracting processes. 42

58 Attorneys must be trained in administrative law. Specify that the recipient can do all ex post controls, but let the project go on. The roles must be clearly defined for the General Directorate of Employment and the National Project Coordination Unit, and other elements of the recipient that intervene in the contract process. There needs to be a mechanism to certify the training institutes or rate their performance. There must be a more rigorous mechanism to monitor the training institutes in the second and third phases. There needs to be a mechanism for online monitoring of the contract process with responsibilities and deadlines. The selection process for participants should be more rigorous from the psychosociometric standpoint, in addition to verification of the socioeconomic eligibility requirements of the beneficiaries. There should be crossed mechanisms for monitoring and supervision that provide direct feedback to the authorities, such as the Coordination of the National Office, to check the performance of the training institutes, operations officers, responsible parties in the departments, and youth. The project s UCN needs more autonomy in the management of the project, because that is called for in the legal framework of the loan agreement and instruments and the references on how the UCN operates with respect to objectives, goals, expenditure items, and eligibility of expenses. The project s team should be composed of better professionals and the salary scale must be competitive, not only equivalent to the salary scale of the recipient organization because the consultants do not have leave, bonuses, or health insurance and pay all their required deductions, which means that equivalence to the recipient s salary scale is not an accurate benchmark. Incentive and motivation mechanisms for the project staff should be created, such as administrative leave, sick leave, vacations after one year of service, training and updating programs, health insurance, and so on. There should be comprehensive project planning mechanisms covering both the technical aspects and the administrative aspects for supervision, monitoring, and meeting the project s goals. There should be an online mechanism of management indicators for decision making and review of the project execution. 43

59 The processes for academic certification, documentation, and continuous monitoring of these processes should be improved so that they are not the last step of training and certification, but an ongoing process. There should be a flexible mechanism and budgets for consultant services by product, using emerging technologies to improve project management in both the technical and administrative areas. Component 3 - Executive Summary, Final Report, Component 3, Cr Investing in Children and Youth Project Economic and Social Policy Analysis Unit UDAPE 25 Relevance of the development objectives with respect to current priorities. The reduction of malnutrition in Bolivia and the generation of jobs with dignity were included in the Development Plan; the project objectives were therefore consistent with the Plan. Currently, they are among the priorities established by the government and included in the Patriotic Agenda 2025 as well as in the plans, policies, and programs. The project objectives are, therefore, fully consistent with current social policy. In addition, having tools for monitoring the performance of the social programs and instruments that permit identifying individuals by their socioeconomic level (Registry of Beneficiaries) is extremely important for decision making by authorities to improve the effectiveness of public policies for the vulnerable, marginalized, and poor population. Sustainability of the results obtained. As mentioned above, the reduction of chronic malnutrition and the generation of jobs with dignity are priorities of Bolivia s current social policy. Consequently, in the context of the Patriotic Agenda 2025 and the development plans, it is expected that the results achieved so far will be consolidated and there will be further progress in both areas. The BJA Program and the MPED Program will continue. Moreover, the SIMPS is being implemented by the UDAPE, and the Registry of Beneficiaries was transferred in 2014 to the MDP for operation through the Unified Registry of Beneficiaries (RUB) of that ministry. Performance of the Bank and the borrower. For the duration and execution of the project, the Bank, as the financing organization, has met all the requests made by the UDAPE, making it possible to complete the various activities established in the annual operating plans and the goals established for Component 3 of the project. In addition, the UDAPE, as the executing agency, has complied with the administrative, financial, and fiduciary requirements established in the loan agreement. Conclusions The objectives agreed upon with IDA for Component 3 of the project have been achieved, with respect to the development and implementation of a System for Monitoring and Evaluation of Social Programs, the conduct of surveys to gather data for the Mother and Child and MPED Programs, the impact evaluation of the BJA Program, the study to monitor employability in the MPED Program, and the 25 A complete copy of the report submitted by the UDAPE can be found in the project files. 44

60 strengthening of the administrative and technical capacity to execute the monitoring instruments defined by the program. Interim unaudited financial reports have been submitted for all periods covered by the project. All expenditures and direct payments made with resources from the financing have been justified and documented. The only matter pending justification to IDA is the expenditure for payment of the External Financial Audit Around 99 percent of the resources assigned to Component 3 of the project have been disbursed and executed. IDA will be reimbursed for unspent resources in an approximate amount in currency of origin of SDR 7, (Seven thousand seven hundred twenty-two and 02/100 Special Drawing Rights), equivalent to US$10, (Ten thousand eight hundred seventy-one and 53/100 U.S. dollars) and the corresponding return of unexecuted balances in the project s account in Bolivian currency. For Component 3 of the project, 100 percent of the Procurement Plan approved by IDA has been executed. Lessons Learned Carrying out activities, studies, or products involving other public entities requires a high level of institutional coordination and the commitment of the officials of the entities involved. In the case of the health and nutrition survey to evaluate the impact of the BJA, the challenge was even greater: (a) interagency coordination at the technical level was required to agree on the content of the survey questions and other aspects of the study; (b) given that the survey was financed by three sources of funding the Bank, the IADB, and the MoH conducting three separate surveys was required; it was necessary to hire a consultant firm to supervise, monitor, and guarantee the consistency and quality of the surveys; and (c) it was necessary to coordinate the time of the contracts of the survey firms and the field operation because each of the organizations and entities that financed the surveys had their own procurement regulations. It is important to have interagency agreements when the implementation and execution of a specific program requires the involvement of officials of other public entities; these agreements help considerably with effective implementation. In the case of the System for Monitoring Social Programs and the design of the Registry of Beneficiaries, the UDAPE signed interagency agreements with the various public entities involved in those systems. When a project involves various co-executors and when their activities are related, it is important that the lending organization, in this case the Bank, assists in the 45

61 corresponding actions when there are problems with one of them that affect the activities or results of any of the other co-executors. This will require, when appropriate, that there be changes or updates in the operating manuals and the results called for in the agreements, to prevent possible failure to complete activities that would then not be executed in the framework of the loan agreement. With respect to fiduciary aspects, it is essential to have continuous interaction between the project management in the Bank and the executing agency, to make the various activities called for in the annual operating plans and procurement plans viable and flexible. b. Borrower s Comments on Draft ICR The Borrower s comments were received on December 16, 2015, which were incorporated in the ICR. 46

62 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders Not Applicable 47

63 Annex 9. List of Supporting Documents Altimari Montiel, F Financing Agreement, C4396-BO Conformed. Washington, DC: World Bank. Apella, I., and Blanco, G Análisis de efectividad de los mecanismos de focalización utilizados en programas de protección social en Bolivia, Manuscript, May Apella, I., G. Blanco, and F. Landa Monitoreo de la situación ocupacional de los beneficiarios del Programa Mi Primer Empleo Digno en Bolivia, Manuscript, June Harbitz, M., and M. Tamargo El significado de la identidad legal en situaciones de pobreza y exclusión social, June 2009, IADB, Washington, DC: World Bank. Montiel, M Amendment to Financing Agreement for Credit 4396-BO Conformed. Washington, DC: World Bank. Segura Labadia, E Official Documents - Second Amendment to Financing Agreement, C4396-BO. Washington, DC: World Bank. second-amendment-financing-agreement-c4396-bo. UDAPE (Unidad de Análisis de Políticas Sociales y Económicas) Evaluación de Impacto del Programa de Salud Materno Infantil Bono Juana Azurduy, La Paz: UDAPE. UDAPE and CIES Internacional Evaluación de Procesos del Programa Bono Juana Azurduy, La Paz: UDAPE. Vezza, E Escaneo de Políticas y Meta-Análisis: Juventud y Políticas de Empleo en América Latina; La Plata, Argentina: CEDLAS. World Bank. 2008a. Bolivia - Enhancing Human Capital of Children and Youth Project. Project Appraisal Document, Report No.: Document Date: FEB 07, 2008, Washington, DC: World Bank b. Bolivia - Investing in Children and Youth Project: Indigenous Peoples Plan, Report No. IPP259, JAN 24, 2008, volumes 1, 2, Washington, DC: World Bank

64 Bolivia - Enhancing Human Capital of Children and Youth Project. Restructuring Project Paper: Main Report, Project Paper Report No.: Document Date: JAN 13, 2011, Washington, DC: World Bank Bolivia - Improving Employability and Labor Income of Youth Project. Project Appraisal Document, Washington, DC; World Bank Group Bolivia - BO: Investing in Children and Youth: P Implementation Status Results Report: Sequence 1 through 13, Washington, DC: World Bank Group. Stakeholders Consulted as part of the ICR Mission Ministry of Health - Bono Juana Azurduy Program Rosa Suáres, Program Coordinator, BJA Isabel Dávalos, Former Program Coordinator, BJA Giovana Imaña, Former Program Coordinator, BJA Gaby Ayoroa, Coordinator UGESPRO/BM Sergio Rodríguez, Former Coordinator UGESPRO/BM Maria Frances Beltrán, Former Coordinator, National Council on Nutrition Ministry of Labor - MPED Program Edgar Altamirano, Director, Employment Directorate Juan José Blanco, Former Director, Employment Directorate Victor Hugo Ayala, Program Coordinator, MPED David Yujra, Former Program Coordinator, MPED Juan Ibañez, Former Program Coordinator, MPED Ministry of Development Planning, UDAPE Roland Pardo, Sub-Director, Social Policies World Bank ICR Team Gastón Mariano Blanco, Task Team Leader Patricia Alvarez, Project Team Ignacio Apella, Project Team 49

65 MAP 50