Document of The World Bank

Transcription

1 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Document of The World Bank IMPLEMENTATION COMPLETION REPORT (PPFI-Q0940 IDA-32440) ON AN ADAPTABLE PROGRAM CREDIT IN THE AMOUNT OF 17.8 MILLION TO THE REPUBLIC OF BOLIVIA FOR A HEALTH SECTOR REFORM PROJECT (APL-I) June 9, 2004 Human Development Department LC6 (Bolivia, Ecuador, Perú and Venezuela) Country Management Unit Latin America and the Caribbean Region Report No: BO

2 CURRENCY EQUIVALENTS (Exchange Rate Effective April 21, 2004) Currency Unit = Boliviano (Bs) Bs 1.00 = US$ US$ 1.00 = Bs FISCAL YEAR January 1 to December 31 APL CAS CDF DFID DHS FIS FPS HIPC IDA IDB IEC ICR IMCI INE MAR MBP MDGs MIS MOH PA PAI PAHO PCU PROISS SBS SEDES SNMN SNIS SUMI UDAPE ABBREVIATIONS AND ACRONYMS Adaptable Program Loan Country Assistance Strategy Comprehensive Development Framework Department for International Development Demography and Health Survey (Encuesta Nacional de Demografía y Salud) Social Investment Fund (Fondo de Inversión Social) Social Productive Investment Funf (Fondo de Inversión Productiva y Social) Highly Indebted Poor Countries International Development Association Inter-American Development Bank Information, Education and Communication Implementation Completion Report Integrated Management of Childhood Illness (Atención Integral de las Enfermedades Prevalentes de la Infancia) National Institute of Statistics (Instituto Nacional de Estadística) Resource Allocation Mechanism (Mecanismo de Asignación de Recursos) Mother-Baby Package (Paquete Materno-Neonatal) Millenium Development Goals Management Information System Ministry of Health (Ministerio de Salud y Deportes) Performance Agreements Expanded Program of Immunization (Programa Ampliado de Inmunización) Pan American Health Organization Project Coordination Unit Integrated Health Development Project (Proyecto Integrado de Servicios de Salud) Basic Health Insurance (Seguro Básico de Salud) Regional Health Administration (Servicio Departamental de Salud) National Maternal-Child Insurance (Seguro Nacional de Maternidad y Niñez) National Health Information System (Sistema Nacional de Información en Salud) Maternal and Infants National Insurance (Seguro Universal Materno-Infantil) Unit for the Analysis of Economic and Social Policies (Unidad de Análisis de Políticas Económicas y Sociales) Vice President: Country Director: Sector Director: Sector Manager: Task Team Leader: David de Ferranti Marcelo Giugale Ana-María Arriagada Evangeline Javier Juan Pablo Uribe

3 BOLIVIA HEALTH SECTOR REFORM PROJECT CONTENTS Page No. 1. Project Data Principal Performance Ratings Assessment of Development Objective and Design, and of Quality at Entry Achievement of Objective and Outputs Major Factors Affecting Implementation and Outcome Sustainability Bank and Borrower Performance Lessons Learned Partner Comments Additional Information 23. Annex 1. Key Performance Indicators/Log Frame Matrix 25. Annex 2. Project Costs and Financing 26. Annex 3. Economic Costs and Benefits 29. Annex 4. Bank Inputs 30. Annex 5. Ratings for Achievement of Objectives/Outputs of Components 33. Annex 6. Ratings of Bank and Borrower Performance 34. Annex 7. List of Supporting Documents 35. Annex 8. Borrower s Comments 36. Annex 9. Executive Summary of the Borrower s Final Project Evaluation 40.

4 Project ID: P Team Leader: Juan Pablo Uribe Project Name: BO- HEALTH REFORM-APL I TL Unit: LCSHH ICR Type: Core ICR Report Date: June 9, Project Data Name: BO- HEALTH REFORM-APL I L/C/TF Number: PPFI-Q0940; IDA Country/Department: BOLIVIA Region: Latin America and the Caribbean Region Sector/subsector: Health (74%); Sub-national government administration (12%); Compulsory health finance (8%); Other social services (3%); Central government administration (3%) Theme: Child health (P); Population and reproductive health (P); Health system performance (P); Participation and civic engagement (P); Public expenditure, financial management and procurement (S) KEY DATES Original Revised/Actual PCD: 07/30/1998 Effective: 12/13/1998 Appraisal: 00/00/0000 MTR: Approval: 06/15/1999 Closing: 12/31/ /31/2003 Borrower/Implementing Agency: Other Partners: THE REPUBLIC OF BOLIVIA/THE MINISTRY OF HEALTH STAFF Current At Appraisal Vice President: David de Ferranti Shahid Javed Burki Country Manager/Director: Marcelo Giugale Isabel Guerrero Sector Manager: Evangeline Javier Xavier Coll Team Leader at ICR: Juan Pablo Uribe ICR Primary Author: Juan Pablo Uribe; Sati Achath 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcome: Sustainability: Institutional Development Impact: Bank Performance: Borrower Performance: S HL H S S QAG (if available) Quality at Entry: Project at Risk at Any Time: No ICR S

5 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 Original Objective: The original objective of the Phase I of the Adjustable Program Loan (APL) under the Health Reform Program in Bolivia (the Project) was to reduce the infant and maternal mortality rates. During this phase, the strategies used were: (i) to increase coverage and quality of health services, and to empower communities to improve their health status; and (ii) to strengthen local capacity to respond to health needs. The objective was clear, important to the country s social development, and realistic in scale and scope. It was also timely and appropriate to the needs of the Borrower, considering that Bolivia had high absolute levels of infant mortality and, relative to its per capita income level, was one of the countries in the Latin America and Caribbean region with the worst overall infant and child mortality rates. The Project was also responsive to the needs of the Borrower. For example, it supported the implementation of legislation that strengthened a new system for targeted health sector financing towards priority interventions, in line with the sector s decentralized framework. Specifically, it supported: (i) the decree which fine-tuned the structure and role of the Departmental Health Authorities and re-created Health Districts, within the context set by the 1994 legislation that decentralized the health sector (Ley de Participación Popular and Ley de Decentralización Administrativa); and (ii) the l999 decree of the Basic Health Insurance (Seguro Básico de Salud, SBS) that expanded the coverage of the National Maternal-Child Insurance (Seguro Nacional de Maternidad y Ninez, SNMN) and improved its payment mechanisms. The Project was consistent with Bolivia s National Action Plan the basis of Bolivia s Country Assistance Strategy (CAS), discussed by the Board on June 16, 1998 which had poverty reduction as the centerpiece of the Governments development program up to the year The strategy for attaining this ultimate goal was to implement actions within a framework of four pillars: opportunity, equity, institutional strengthening and dignity. In particular, the Project was expected to contribute to strengthening two of the CAS pillars: (i) equity, by improving access to and quality of health services and by reorienting and improving the quality of public health expenditures (through better targeting via the SBS); and (ii) opportunity, by introducing reforms that would improve the return of investments in the creation of human capital. The health actions under the equity pillar complemented activities in education, water and sanitation, and rural productivity described in the CAS. The Project was also expected to directly contribute to improving one of the four variables that constituted the CAS s Core Benchmarks (the infant mortality rate), and would indirectly contribute to the other three core benchmarks (malnutrition, poverty and the poverty gap). The project design reflected lessons learned both in process and in contents from international interventions in countries with high mortality rates, from IDA s experience with the Bolivian health sector (under the Integrated Health Development Project -Proyecto Integrado de Servicios Salud, PROISS) and from the experience of other projects that were supporting the public health sector. In process, major lessons learned that were taken into account included: (i) ownership of reforms and commitment by national governments are key for success; (ii) countries with a weak capacity need support for program management and procurement; (iii) projects should be simple to administer and to supervise, even more when the institutional capacity is low; (iv) project objectives should be well defined and should be clear to all involved; (v) there should be early agreement on and dissemination of results-based indicators to track the project s performance; and (vi) successful implementation requires, among others, client orientation, flexible management, and decentralization of decision making, financial resources and procurement capabilities within clear operational plans with strong lines of communication. Project design also - 2 -

6 acknowledged lessons on contents, including: (i) effective immunization programs can have a significant impact on child mortality rates, especially when the initial rate is very high, and (ii) other cost-effective interventions, such as the Integrated Management of Childhood Illnesses (IMCI, Atención Integral de las Enfermedades Prevalentes en la Infancia) and the Mother-Baby Package (MBP, Paquete Materno-Neonatal) also can make an important contribution to improving the health status of children and mothers. The Project envisioned a series of social benefits that justified implementation on its own merits. These included: (i) reduced likelihood of death and illness among children under five and mothers using public health services, through the national implementation of the Expanded Program of Immunization (PAI), the IMCI and the MBP; (ii) additional benefits from the PAI to approximately one third of the population living in areas of high incidence of yellow fever and hepatitis B; (iii) health benefits to the population of five departments that included about 80 percent of all mothers and children in the country, through specific municipal health subprojects; (iv) strengthened local government capacity to respond to health needs through increased effectiveness and efficiency of decentralized health service financing and delivery; and (v) more effective use of public and (international) donor funds. In the long-term, welfare would improve as a direct and indirect result of the reduction of mortality and burden of disease. In addition, project benefits would be concentrated among the poor as (a) project interventions aimed to expand services already available to the better-off (hence, the expansion would go primarily to those not yet served), and (b) municipal subprojects would be targeted using a poverty index. The Project was not very complex in terms of implementation, even though its design included a number of innovative features. It was not overly demanding either, because its structure was fairly simple and the Borrower had the corresponding technical and administrative capacity. Initial challenges involving actors outside the direct control of the government included getting agreements from the nine Departmental Health Authorities to sign the Performance Agreements with the Ministry of Health and from each of the 311 municipalities to earmark resources to finance the recurrent costs of the public health insurance (SBS). Both challenges were met successfully and properly maintained through project implementation. [At the Project s initiation, there were 311 municipalities in the country. Between 1999 and 2003, an additional 13 municipalities were created. These new municipalities were also brought into the same sector agreements.] 3.2 Revised Objective: Not applicable. 3.3 Original Components: The Project consisted of three components, all related to achieving the Project s objective. The Ministry of Health (MOH) had sufficient technical, administrative and financial capacity for the successful implementation of these components. Furthermore, in light of the substantial policy dialogue that was taking place with the Government, the MOH was very much focused on the project objective. Following is a description of each one of the Project s components. Component I. Coverage and quality improvements of health services and empowerment of communities (US$36.6 million: 83.2 percent of the total project cost). (i). Support the implementation of a new medium term plan for immunizations that will: (a) create the capacity to develop and implement immunization policies; (b) strengthen health services to improve vaccination coverage and introduce new vaccines; and (c) strengthen the information and surveillance - 3 -

7 systems for communicable diseases. (ii). Implement new strategies for the IMCI and the MBP through: (a) strengthening program management to plan, coordinate and supervise activities; (b) development of norms, protocols and supervision instruments; (c) training of regional and local level staff; and (d) use of conventional and new mechanisms to enhance two-way communications with indigenous populations. (iii). Provide resources to finance demand-driven local subprojects to strengthen, inter alia, the quality of maternal and child services, the development of social communication activities to strengthen the information, voice and empowerment of beneficiaries, and the implementation of new instruments of health sector management. Component II. Strengthening local capacity to respond to health needs (US$5 million: 11.4 percent of the total project cost). (i). Support the implementation of the SBS through: (a) the creation of a unit to administer and modernize the system; (b) developing and managing the corresponding information system; and (c) providing technical assistance to municipalities. (ii). Strengthen the development of the Health Management Information System (MIS) by: (a) supporting the design and implementation of new modules related to the programs supported by the reform; and (b) invigorating the analysis and use of MIS reports at the local level. (iii). Strengthen the capacity and accountability of the health districts to manage and supervise the implementation of project activities. This included the introduction of performance agreements that set targets for key health indicators and assigned specific responsibilities and resources to achieve them. Component III. Coordination, monitoring and evaluation (US$2.4 million: 5.4 percent of the total project cost). Finance the establishment and operation within the MOH of a management structure appropriate for the coordination of all project activities and for monitoring and evaluation of project implementation in accordance with the pre-defined performance benchmarks. 3.4 Revised Components: Not applicable. 3.5 Quality at Entry: There was no official assessment of the quality at entry by the Quality Assurance Group (QAG). The Implementation Completion Report (ICR) deems the quality at entry to be satisfactory and the Project as well conceived. As mentioned above, the project objective was consistent with the CAS and with the Government s priorities and met the critical needs of Bolivia s health sector. With the technical support from the MOH, the quality of project design was adequate to meet its overriding objective. In addition, the project design incorporated a number of innovative features into the Credit Agreement, including the following: - 4 -

8 Benchmarks. The Project s Development Credit Agreement introduced an innovation to emphasize a results-oriented focus through implementation: it set out annual targets for eight performance benchmarks designed to measure the outcomes of key, project-supported health interventions aimed at reducing the infant and maternal mortality rates. Project success was defined in relation with annually reaching an agreed number of the targets set for these eight performance indicators. In case the Borrower did not reach these targets in at least five out of the eight benchmarks any given year, the Bank could take appropriate remedial measures, including the right to suspend disbursements and request a restructuring of the Project. Performance Agreements. Starting in 1999, the Project supported the implementation of annual Performance Agreements (PA), setting targets on a series of process and outcome indicators for each of Bolivia s nine departments. These PAs are signed by the MOH and the Regional Health Director and are evaluated semi-annually by the MOH, with technical support from the Project. Investment Mechanisms for the Health Sector. A Resource Allocation Mechanism (Mecanismo de Asignación de Recursos, MAR) was launched as a new investment scheme focusing on actions to reduce maternal and infant mortality. This was the instrument used to implement Project Component I-(iii). Its novel features, in comparison to previous investment schemes in the sector, were: (i) resource allocation was based on a weighted formula that took into account local human development indices (70 percent) and population (30 percent), thus prioritizing poorer municipalities; (ii) subproject definition involved the participation of all stakeholders (the community, the health sector and the municipality) in diagnostic assessments and in the setting of local priorities; and (iii) resource allocation was directed not only for strengthening the supply side, but also to subprojects designed to increase the demand for services. Project preparation and design involved extensive stakeholder consultation and incorporation of lessons learned from other projects in the health sector. Primary beneficiaries were consulted through focus groups; satisfaction surveys were done among indigenous women in areas of high child mortality; and communities, municipalities, departments, donors and NGOs were also consulted in numerous meetings. Around 20 meetings were held in La Paz and participating departments. In these meetings, the objectives and proposed procedures of the Project were discussed. These extensive stakeholder consultations and the participatory process followed for project preparation substantially contributed to enhancing quality and readiness at entry. The Project recognized and took into account the risk factors that could affect implementation. These risks and the appropriate measures incorporated into the project design to mitigate them included: (i) deterioration of the macroeconomic and political situation that could compromise resources for the health sector or decrease the government s presence in the sector. To mitigate this risk, the design of the SBS was based on the earmarking of resources at the municipal level, thus protecting its financing from national economic and political instability; (ii) insufficient coordination among Bolivian institutions to achieve a holistic approach to health. Here, project design envisioned the SBS as an overarching coordination framework into which all other national programs could feed, and promoted the PA as an instrument for better coordination in planning, prioritization and results-monitoring among the various government levels; and (iii) significant delays in national, departmental or municipal counterpart funds required for the investment subprojects, hampering execution of this component. To mitigate this risk, the MAR instrument anticipated careful planning of subproject implementation and fund flows, including required co-financing from the sub-national levels. During preparation, alternative project designs were considered: - 5 -

9 (i) Lending Instrument. Both the Bank and the Government considered different lending options. As the needs were large, while the option of an adjustment operation to introduce new reforms was also explored, the option of a traditional investment operation was favored by the Government. An APL was preferred because it was recognized that improving infant and maternal mortality would take time and that the health sector would have to adapt to changes occurring throughout the rest of the government (e.g. decentralization). In this context, an APL offered greater flexibility for implementation. In addition, the immediate reforms sought by the Project required the willing participation of municipalities and, while adjustment operations create incentives for the Central Government, they do not provide such for the participation of local governments. Such incentives could be incorporated in an APL, specifically through the MAR investments which were to give preferential access to reformers. (ii) Front-loading of reforms. In addition to the issues covered by the APL, the health sector in Bolivia presented other challenges, in particular those related to the operation of its social security entities and to human resources management. The team considered the option of including policy activities to address these areas, thus front-loading them on the reform agenda. However, this option was rejected for two reasons. First, the Project already contained crucial reforms to address maternal and child mortality (for example, establishment of the SBS, IMCI and MBP, and introduction of PA) and overloading the Project could distract attention and effort from them. Second, reforms in social security and human resources areas are second generation reforms; it was concluded that they would be better dealt with under future phases of the program. 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: Satisfactory. The Project succeeded in introducing an ambitious set of policies and interventions to improve the performance of the public health services. In particular, it introduced policies and instruments conducive to better infant and maternal health status, including the expanded infant and maternal public insurance scheme, the performance agreements, prioritized investment subprojects via MAR and the strengthening of cost-effective health interventions (including the PAI, IMCI and MBP). Likewise, it increased the participation of municipalities in health care delivery throughout the country, by earmarking to the SBS a significant fraction of the resources transferred from the national government to the municipal authorities and making explicit the responsibilities of the latter in health. As a result of this effort, as shown in Section 4.1.B, important intermediate and outcome indicators for infant and maternal health experienced significant improvements, most exceeding the targets initially defined. Overall, the Project contributed to significant advances in reducing the infant mortality rate in the country from its baseline of 67 deaths per 1,000 live births (1998 DHS) to the APL-I target set for 2001 at 60 deaths per 1,000 live births. Furthermore, preliminary results from the 2003 DHS financed by the ongoing APL-II Health Reform Project suggest that an even greater improvement took place, as the infant mortality rate is now estimated at 54 per 1,000. A national 2001 post-census survey on maternal mortality financed by the Project estimated a maternal mortality rate of 234 deaths per 100,000 live births in 2001, compared with the 1994 DHS estimate of 390 deaths per 100,000. However, these two estimates come from different methodologies and are not directly comparable. Therefore, final data from the 2003 DHS (which included a maternal mortality module) must be used for analyzing the trend. [The 1998 DHS, unfortunately, did not include a maternal mortality module that would allow for better trend analysis.] - 6 -

10 Major project outcomes and achievements are as follows: (A). Basic Health Insurance and Other Core Policies. The Project succeeded in establishing the SBS, a public financing scheme for priority health interventions for infants, pregnant women and women in childbearing ages. The SBS expanded and improved the National Maternal and Child Insurance program (SNMN) created in The SBS was a system by which municipalities earmarked 5.4 percent of the tax revenues transferred to them by the Central Government to finance 90 cost-effective priority interventions for the targeted group. This scheme was particularly important in reducing the economic barrier of access to health services, considered to be highly relevant in Bolivia, where only about percent of the population has access to social security or private health insurance. The majority of the population, mainly the poor and rural inhabitants, traditionally had to pay user fees for health services, even in public facilities. Major features of the SBS included: (i) participation of all of the country s municipalities; (ii) development of standard care protocols for interventions included in the benefits package; (iii) strengthening of administrative and financial control systems; (iv) articulation with non-public providers; (v) emphasis on indigenous issues; and (vi) implementation of focused actions to improve quality of maternal and child services. In addition, the SBS provided an overarching scheme under which different national programs (e.g. the PAI) and service delivery strategies (e.g. IMCI and MBP) were standardized and implemented. The SBS was regulated by the MOH, which established the priority interventions to be financed, set the rate of reimbursement to cover recurrent costs, and managed the information system. The contribution of the SBS in increasing coverage of priority maternal and child services is reflected in the increased coverage observed in core indicators, such as skilled birth attendance, fourth prenatal control, immunization coverage, and ambulatory child visits, as detailed in the next section. In addition, the SBS achieved significant institutional progress, as reflected by the fact that, in late 2002, the incoming national government decided to expand this policy and launch the Universal Maternal and Child Health Insurance (Seguro Universal Materno-Infantil, SUMI), this time through the enactment of a national law. This change also implied a gradual increase in the resources earmarked for the insurance at the municipal level, to reach up to 10 percent of the transfers at the end of a four-year period. At the same time, the management of this public insurance initiative was transferred to the MOH line staff, shifting (in 2003) the Project s contribution towards areas of operational urgency and technical assistance. The APL-II has taken this relation to the next step that will ensure that project support is focused on strategic areas of technical assistance key to improve the design and implementation of the SUMI, including monitoring and evaluation, and financial control. As a complementary action to the SBS, the Project also supported the nation-wide implementation of Performance Agreements as a new management instrument to align efforts at all government levels towards key health outcomes. The implementation of the MAR investment mechanism for promoting maternal and child interventions in five selected departments gave additional means and incentives for the municipalities to achieve the targets agreed upon at the local level. Both instruments (the performance agreements and the MAR investments) are being continued through the APL-II

11 (B). Key Performance Indicators Table 1 presents the eight key health indicators and their corresponding annual targets defined to monitor progress in achieving the Project s overall objective. In six out of the eight indicators, the Project exceeded the targets every single year between 1999 and 2002, and for another indicator only failed to meet the target in one year. Overall, there was substantial progress in the coverage of priority health interventions and over compliance with project targets. Table 1 The Project s Monitoring Indicators and Targets Achieved Coverage (Target) Percentage of births attended by Achieved Coverage 42% 47% 52% 54% 54% trained health personnel APL-1 Targets 36% 38% 40% 43% 46% 2. Women with complete cycle of Achieved Coverage 30% 31% 33% 36% 34% antenatal care visits APL-1 Targets 28% 30% 32% 36% 40% 3. Early neonatal hospital mortality Achieved Coverage (per 1,000 live births) APL-1 Targets Number of pneumonia cases in Achieved Coverage 81,462 91, , , ,244 children under 5, attended in health APL-1 Targets services 5. Number of acute diarrhea diseases Achieved Coverage 337, , , ,697 in children under 5, attended in APL-1 Targets health services 6. Third doses of DPT/Hib/Hepatitis Achieved Coverage 77% 85% 89% 92% 100% B vaccines APL-1 Targets 75% 82% 65% 75% 85% 7. Number of municipalities with Achieved Coverage DPT/Hib/Hepatitis B vaccine APL-1 Targets coverage below 80 percent 8. National financing of vaccines (in Achieved Coverage millions of US $) APL-1 Targets Source: The Project Coordination Unit (MOH) based on different information systems. (i) Coverage of births attended by trained health personnel. With a baseline of 36 percent for , the Project set a target of 43 percent for 2001, based on the information provided by the National Health Information System (Sistema Nacional de Información en Salud, SNIS). The initial trend in this indicator was one of the most successful: by end-2000, coverage had already reached 52 percent, exceeding the targets for 2001 and Progress was maintained at 54 percent between 2001 and 2002, as suggested by administrative data. The 2003 DHS preliminary results ratify this level of attainment for the period, estimating skilled birth attendance in the country at 55 percent. (ii) Complete prenatal care attendance. This indicator measures the number of women who, after having a first prenatal contact, go on to complete the recommended prenatal control scheme (minimum of 4 visits). It is regarded as an important measure of quality of care because it enables risk identification and reduction of complications associated with deliveries, as well as guaranteeing that mothers have access to family planning information and nutrition education (provided that consultations follow the respective protocols). The baseline for this indicator was low (28 percent). Targets were met every year until 2001, when the percentage of complete prenatal controls reached 36 percent. However, it then dropped slightly to 34 percent in This trend is consistent with that observed for skilled birth attendance, in that there was fast progress between 1998 and 2001, and the attained level was maintained thereafter

12 (iii) Early neonatal hospital mortality. This variable measures the number of deaths during the first seven days of life per 1,000 live births reported in all secondary and tertiary level hospitals in the country. The reporting system did not exist prior to the Project and was created to monitor progress in this variable. The mortality baseline was high, and because no major direct interventions were planned in high complexity hospitals during this program s phase, a modest target was established. This indicator was included to pave the way for the APL-II by providing incentives for the government to monitor this variable. The added attention paid off, as hospitals themselves rapidly reacted by establishing simple but important improvements in the management of newborns. Targets were met every year from 1998 to (iv) Number of child pneumonia cases attended in health services. Pneumonia is an important cause of child mortality in Bolivia. Monitoring this indicator is thus key to assess progress towards reducing child mortality. It was set in absolute numbers (instead of a percentage) to avoid controversies about the actual number of children with pneumonia (which would have served as the denominator but is difficult to estimate). However, the chosen definition also has limitations, as the total number of pneumonias may vary due to factors outside the control of the health sector. Following a slow start in 1999, the MOH boosted efforts to improve this indicator by raising the rate of reimbursements paid by the SBS for attending pneumonia cases, improving the availability of antibiotics, and training personnel in the IMCI strategy. Targets were met every year from 1998 to The number of child pneumonia cases attended increased from 81,462 in 1998 to 128,244 in (v) Number of child diarrhea cases attended in health services. As with pneumonia, acute diarrhea is an important determinant of morbidity and mortality among children under five, accounting for almost 15 percent of child mortality in Bolivia. Targets here were met every year from 1998 to The number of cases attended in health services increased from 337,123 in 1998 to over 450,000 in 2000, dropping again after that. As with the pneumonia indicator, interpretation needs to be done carefully. While this drop may be due to a decrease in coverage, it could also result from an improvement in living conditions and education of mothers, and thus evidence a decrease in the incidence of diarrhea diseases. This decrease over time is a desirable result in view of reducing infant mortality. But, given that this indicator does not enable to measure progress adequately, it was decided in 2002 to replace it with the percent of children under 5 who have received three doses of iron. This commitment was formalized as part of the APL-II performance indicators. (vi) Immunization coverage. The immunization coverage, based on administrative data from the SNIS, increased significantly from a baseline of 77 percent in 1998 to almost 100 percent in 2002, reflecting an apparent success of the PAI program. Targets were met every year from 1998 to In addition, in the year 2000 with support from the Project, the DPT vaccine was replaced at the national level by the pentavalent vaccine (including DPT, Hepatitis B and Haemophilus influenza type B), which is why new targets were set at lower levels starting that year. The change in vaccines did not appear to set back national coverage rates. Results from the 2003 DHS, although not comparable with the SNIS administrative data, also suggest a significant increase in immunization coverage rates in the country during the Project s period, increasing from 48.6 percent in 1998 to 71.5 percent in (vii) Number of municipalities with third-dose DPT coverage less than 80 percent. This is an important indicator of equity in immunization coverage and complements the previous one. Even when average national coverage levels are high, these may hide pockets of very low coverage which could be the source of outbreaks. This indicator measures the success of the PAI in eliminating these pockets of low coverage by reaching out to areas that, generally, have low access to health services and high poverty levels. In this indicator, targets were met in all years since 1999 and, by 2002, only 53 municipalities (mostly with very - 9 -

13 low populations) remained with DPT coverage levels below 80 percent. (viii) Financing of vaccines by the Borrower. This indicator was set to measure the national Government s effort in sustaining stable financing of the PAI. The national financing of vaccines only reached the set goal in In all other years, financing from the national treasury fell short from the target (by 76, 57 and 82 percent of programmed budget, respectively) and vaccines were mainly financed by international donors, with irregular timing and a potential negative effect on the Program. With the support of the APL-II and other Bank adjustment operations, alternative strategies are being explored to resolve this problem of sustainable and regular funding for PAI. 4.2 Outputs by components: Component I. Coverage and quality improvements of the health services and empowerment of communities. (a). Support the implementation of a new medium term plan for immunizations. The Project supported the strengthening of the PAI, in close coordination with the Pan American Health Organization (PAHO). This included support for the introduction of the new pentavalent vaccine and vaccination campaigns, prioritized investments in equipment and infrastructure for the cold chain and storage of vaccines, and training for monitoring and evaluation. Between 1999 and 2001, total funding from the Project for PAI reached approximately US$2.5 million. A parallel, complementary improvement in the surveillance and information systems for communicable diseases in the country (known as the epidemiological shield ) was launched by the MOH with support from the Inter American Development Bank (IDB). (b). Implement new strategies for the Integrated Management of Childhood Illness and for the Mother-Baby Package. The Project supported the design and implementation of norms and protocols for the IMCI (developed by PAHO and UNICEF) and the MBP strategies. Training of health personnel on these protocols was provided, as well as management instruments for supervising progress made. Specific activities supported by the Project under this objective included: adaptation of clinical and consumer-oriented texts in accordance with the epidemiologic and cultural reality of the country, to be used as training material for implementation of the IMCI; creation of 17 IMCI training centers, decentralizing staff training throughout the country; training of 1,784 healthcare staff, most of them from primary health care centers, to improve their clinical skills and quality of services delivered to children under 5; and support for and organization of an interagency committee for the IMCI initiative (including bilaterals and other international financing agencies), responsible for the unification of criteria and actions in this field. (c). Provide resources to finance demand-driven local subprojects to strengthen, inter alia, the quality of maternal and child services, the development of social communication activities to strengthen the information, voice and empowerment of beneficiaries, and the implementation of new instruments of health sector management. The MAR investment mechanism co-financed medical equipment, infrastructure and training in 311 subprojects in the five participating departments (La Paz, Oruro, Cochabamba, Chuquisaca and Santa Cruz, representing over 80 percent of the total population). Required counterpart funds by participating municipalities ranged between 20 to 35 percent of the total cost. A low level of interest by municipal governments in training led to a reassignment of funds to the equipment and infrastructure categories. Complementing this investment effort at the local level, the Project (a) helped the MOH design a planning

14 instrument for sector investments; and (b) supported efforts to promote an intercultural approach in health, including the design of the Pregnant Women s Rights Charter ( Carta de Derechos de la Mujer Embarazada ), the adaptation of specific features of the SBS to respond to indigenous groups, and the signing of agreements with the two leading indigenous organizations to facilitate their participation in the SBS and other project activities. Achievement of this component is satisfactory. Component II. Strengthening local capacity to respond to health needs. (a). Supporting the implementation of the SBS. The Project was determinant in the design and implementation of the SBS, which was later on transformed into the SUMI and now reaches roughly one third of the country s population. This collaboration included provision of technical assistance to participating municipalities in their role in implementing the insurance, design of contract mechanisms to be used with NGOs and other private providers, and support for the conformation of a central management unit with much needed information systems. See section 4.1 for more details on the progress made. (b). Strengthening the development of the management information system. The Project supported individual actions to improve the SNIS through the purchase of hardware and software, the design of internet links and funding for the creation and functioning of information analysis committees at the local level. Leadership in the development of the health sector information systems was taken by an IDB loan focused on the SNIS, but progress here was slower than expected. As a result, the SNIS today is still in need of further reforms and improvement. (c). Strengthening the capacity and accountability of the health districts to manage and supervise the implementation of project activities. The health reform in Bolivia during the nineties was mainly based on the pillars of decentralization and equity, seeking to improve quality and expand coverage of health services to the poor and rural areas. Since 1998, the Project supported the signing of annual performance agreements in all nine departments. These PAs are based on process and outcome indicators. The Project also supported bi-annual evaluations of progress made in these agreements. These regular evaluations were complemented in 2001 by an overall evaluation led by PAHO, that contributed to strengthening this instrument. In the coming years, the MOH expects to use PA at the municipal level. To complement this results-oriented effort at the departmental level, the Project supported the evaluation of second and third-level public hospitals during 2001 and This evaluation led to pilot experiences with performance agreements with third level hospitals in the municipalities of Santa Cruz and El Alto. Achievement of this component is satisfactory. Component III. Coordination, monitoring and evaluation. This component financed the functioning of the Project Coordination Unit (PCU), including the purchase of equipment, materials and supplies for its day to day activities. In addition, it financed a 2001 post-census national survey on maternal mortality (through an inter-institutional agreement between the National Institute of Statistic (INE) and the MOH), which provided preliminary estimates on the progress made in the reduction of maternal mortality (see Section 4.1). Likewise, it supported the realization of Health National Accounts for Bolivia. Other detailed actions supported here included the following: the PCU procured six audits of the Project Financial Statements, including the Project Preparation

15 Fund (PPF). These financial audits provided feedback on the Project s financial management processes which led to the implementation of improvement plans and the strengthening of internal controls; a procurement audit was contracted in The results identified opportunities to strengthen the procurement unit and improve the maintenance and control of relevant documentation; and specific evaluations were conducted in some core technical areas, including (i) an evaluation of the SBS, (ii) special audits of the Compensatory Health Fund of the SBS in different municipalities, and (iii) medical audits for specific cases in second-level hospitals. Finally, the PCU also focused on staff capacity building throughout project implementation. Training of staff contributed to the design of health care policies in relevant areas such as payment systems and contractual arrangements, decentralization, cost analysis, public insurance and health economics (including health accounts). Achievement of this component is satisfactory. 4.3 Net Present Value/Economic rate of return: Not Applicable. 4.4 Financial rate of return: Not Applicable. 4.5 Institutional development impact: High. The Project resulted in a high institutional development impact, as demonstrated by the following achievements: The Project was instrumental in establishing the SBS, which was later expanded and transformed into SUMI by the incoming administration. The development and launching of this public insurance scheme has had a sustained impact on the role municipalities play in the health sector and on public providers. In addition, the establishment of an explicit package of services free-of-charge for the population increased the awareness of the public on health issues and on their rights to access health services. Increased access to basic maternal and child services followed. The implementation of PAs changed the logic of interaction and planning in the health sector between the national level and the departments, introducing a results-based focus that has started to replace the traditional sector emphasis on inputs. The departments agreed on important objectives for annual programs, on measuring key indicators related to health outcomes, and on monitoring progress made in that direction. This effort, sustained throughout the Project s cycle, had never been done before in the country and is soon expected to reach the municipal level. The Project significantly strengthened the PAI, which was a key component as evidenced by the fact that three out of the eight performance indicators were related to immunization. Furthermore, the partnership among the Government, the World Bank and PAHO created a momentum for improving the policy environment and effectiveness of Bolivia s national immunization program. Evidence of this is that tools and methodologies of Bolivia s PAI are now being utilized to improve

16 national immunization programs at the global level. [The Global Alliance for Vaccines and Immunization (GAVI) has incorporated several tools and methodologies developed by PAHO together with the Bolivia PAI into its guidelines for funding requests for its Children s Vaccine Fund. These include the methodology for evaluating a national immunization program; annual and five-year action plans; inter-agency coordinating committees; rapid monitoring of vaccination coverage rates; supervision plans at the local level, and the establishment of rapid responses to emergencies.] The financial sustainability of the PAI remains a challenge that needs to be addressed in order to avoid backtracking in coverage rates in the second phase of the program. This issue is being considered through other ongoing operations in the country, reflecting the awareness level it has already gained. The Project helped to build capacity within the MOH by integrating technical staff in the PCU with different line units in the MOH. Initially, the PCU functioned as an independent unit. Later on, it was integrated into the MOH s General Direction for Health Services, leading to better coordination among project staff and the MOH s other units. For example, the Project created the Quality Observatory for quality of health service delivery, which strengthened the MOH s capacity in this area. Similarly, a team of health economists in the PCU continuously collaborated with staff in other units of the MOH, providing support to the consolidation of national health accounts and the inclusion of a stronger economic perspective in the decision-making process in the health sector. 5. Major Factors Affecting Implementation and Outcome 5.1 Factors outside the control of government or implementing agency: Donor coordination. Initially, the international donors highly relevant in Bolivia were skeptical of the SBS and the Performance Agreements. However, after one year of implementation, they were convinced with the benefits of these policies and the ownership from government. As a result, the Project received significant support and assistance from them. For example, the Department for International Devleopment (DFID) and the Canadian Cooperation planned their projects within the framework of the MOH policies supported by the Project and in close coordination with the PCU. The excellent coordination between PAHO and the Bank in the immunization program was another positive factor which helped project implementation. While the Bank relied on PAHO for technical know-how and on-the-ground presence, PAHO relied on the Bank for the policy dialogue with the government on the financial aspects of the immunization program within a stringent fiscal and economic scenario. This partnership contributed to secure financing of vaccines, despite constraints in availability of national treasury funds. Political unrest in The implementation of the Project progressed relatively well until the end of During its last year of implementation (2003), intense political and social unrest in the country culminated in October with the toppling of the national government. As can be expected, this affected the Project s implementation dynamics. In particular, conflicting messages regarding central government policies (e.g. decentralization and the public health insurance), intense political patronage and high turnover of staff, and serious fiscal constraints at the national and sub-national levels affected the PCU s technical and managerial capacities, its articulation with the MOH, and its implementation pace. 5.2 Factors generally subject to government control:

17 Collaboration with the government. For most of the Project s time-span, good collaboration between the Government and the Bank enabled the PCU to hire high-quality professionals available in the country, and also to prevent the use of project finance as a slush fund by the political parties. Frequent change of administration and political interference with managerial decisions. During the four-year implementation period, the national administration changed four times and the Minister of Health changed five times. High rotation of staff took place at the MOH, the PCU, and the sub-national levels, especially during the last part of the Project s implementation. On different occasions, a highly politicized environment placed project activities, especially the MAR infrastructure component, at risk of being politicized and derailed. The transition from the FIS to the Fondo de Inversion Productiva y Social (FPS), with internal changes in management and structure on the FIS side, further delayed implementation of this component. The PCU was also weakened because of this interference and turnover (especially during 2003), and remedial actions needed to be identified and agreed between the Borrower and the Bank. Economic downturn. The social unrest of February and October 2003 contributed to the deterioration of the macroeconomic and political conditions in the country, which in turn led to cuts in the operational budget of the MOH. The slowness in the provision of municipal counterpart funds to subprojects approved under the MAR component, compounded sometimes by slowness on the FPS side, weakened the ability of firms to provide guarantees for contracts on a timely basis. However, most Project activities had been implemented by that time. The ongoing effects of this economic, fiscal and political scenario is being addressed under the APL-II. Counterpart funding. The financing of the PAI was frequently affected by lack of national counterpart funding. In addition, the MOH had a significant debt pending with the vaccines fund administered by PAHO. Because of this, in July 2003, Bolivia was not able to order vaccines and syringes, causing a stock deficit for yellow fever, DPT and BCG, and a shortage of polio and pentavalent vaccines that placed coverage rates at risk, and required additional Bank support. The implementation of the MAR component also faced delays, mainly due to lack of adequate counterpart funds for the subprojects from departments and municipalities. Lack of capacity at the local and regional levels. Some of the Regional Health Administrations (SEDES) were institutionally weak, including few technical staff and a limited budget (especially in the case of the departments of Beni and Pando). This hampered implementation and weakened monitoring of policies at the local level. The PCU had to intervene to fill in these gaps, appointing regional personnel and increasing direct supervision at the local level in these places, raising administrative costs but preserving management capacity over project implementation. 5.3 Factors generally subject to implementing agency control: The implementation of the "Sistema Integrado de Gestión y Modernización Administrativa" (SIGMA), applied by the central government throughout the public administration since December 2002, led to delays in project execution, mainly due to pending issues in the installation of the financing system in the PCU and the implementation of rotatory funds at the regional (departmental) level. However, as with other negative factors affecting project implementation, the effects on this APL-I were minor (as project activities were mostly being concluded by then) and corrective actions are being discussed within the APL-II implementation. 5.4 Costs and financing:

18 The total cost of the Project, as estimated in the Project Appraisal Document (PAD), was US$44 million, out of which US$25 million (56.8 percent) was funded through the Bank s credit and the remaining US$19 million (43.2 percent) through the National Treasury and local (municipal) funds. About US$1 million of the committed funds (2.5 percent of the total cost and 4 percent of the Credit) remained undisbursed at the time of writing this ICR and are being cancelled. 6. Sustainability 6.1 Rationale for sustainability rating: Highly Likely. The sustainability of the Project is rated as highly likely, given the progress made in institutionalizing most of the policies supported through this APL-I. The Government is committed to continue its support to the sector both financially and in important policy reforms through the ongoing implementation of a second phase of the program. Despite four changes of national government, the infant and maternal health issues remain a top priority in the government s agenda. Some of the policy reforms supported by the Project, so far also backed by the incoming administration, seem sufficiently strong to resist any backtracking attempts. Such is the case, for example, of the public health insurance, now an integral feature of the Bolivian health system, with the population aware of and demanding their right to free, effective maternal and child services. Considering the current relevance of the SUMI and the fact that with enhanced HIPC resources municipalities are likely to receive approximately 50 percent more funds over the next 15 years than what they are receiving today, continued financing for the public insurance is also likely. Lastly, the APL-II, that started implementation during the last year of the APL-I, aims at the same development objectives and includes in its design components that give continuity and further expand the APL-I, thus strengthening its sustainability. Performance Agreements will require close vigilance from all interested parties. The clear message from the international donors community on the need to focus on results will contribute not only to its sustainability but also to its strengthening. This emphasis is compounded with the government s expression of interest in attaining the Millenium Development Goals (MDGs) and using concrete targets to monitor progress in the framework of its Poverty Reduction Strategy. A recently conformed multisectoral MDG Committee, led by the Unit for the Analysis of Economic and Social Policies (UDAPE), most likely will build on the health performance agreements experience to further monitor results in other social sectors. On immunizations, the infrastructure and managerial capacity of the PAI has been strengthened based on best practices and receives ongoing technical support from PAHO, improving its sustainability. The financing aspect of PAI within the ongoing fiscal and economic crisis is being addressed both through the second phase of the health program and the policy dialogue involving other Bank adjustment operations (e.g. through the proposed Social Sectors Programmatic Structural Adjustment Credit). 6.2 Transition arrangement to regular operations: Section 6.1 explains the transition arrangement to regular operations. In addition, there has been a careful transition from the APL-I to the APL-II, that not only allows for continuity of core policies but further strengthens their sustainability within regular health sector operations. Such is the case of the PAI (defined as a national priority program with recurrent costs to be financed through a mix of internal funds and international donations), the SUMI (elevated to the category of national law, including all municipalities and defining a gradual increase in financing), and the PAs (now being discussed to be implemented at the municipal levels). In addition, the APL-II will continue monitoring progress using the same indicators of

19 the first phase, with only one minor change identified so far that is, the replacement of the indicator on child diarrhea cases attended for one on the intake of iron sulfate among children. 7. Bank and Borrower Performance Bank 7.1 Lending: Satisfactory. The Bank s performance in the identification, preparation, and appraisal of the Project was satisfactory. The identification process focused on critical gaps and opportunities for interventions in Bolivia s health sector, by (i) identifying and targeting interventions towards the main causes of maternal and child mortality; and (ii) building on previous policies (e.g. SNMN and the decentralization law) and improving them. The Project s consistency with the Government s development priorities and the Bank s CAS was assured. In addition, the Bank acknowledged sociological aspects and local issues while designing the project. With a team comprised of proper skill mix, it brought in state-of-the-art expertise into project design, providing for flexibility and responsiveness to local needs. During preparation and appraisal, the Bank took into account the adequacy of project design and all major relevant aspects, such as technical, financial, economic, and institutional (including procurement and financial management). In addition, during appraisal, the Bank assessed the project s risks and benefits. The Bank had a consistently good working relationship with the Borrower during preparation and appraisal. Extensive stakeholder consultations at community, municipal, departmental and national levels was highly productive, and initiated a process of ownership that proved highly valuable at the implementation stage. 7.2 Supervision: Satisfactory. The Bank s performance during the implementation of the Project was satisfactory. Sufficient budget and staff resources were allocated and the Project was adequately supervised and closely monitored. Over the four years of project implementation, there were seven supervision missions, with an average of about two missions per year. This frequency was increased during the year 2003, which was particularly difficult given the political and economic context, through parallel missions related to other projects in the country where discussions of critical health issues were also included. The Bank s client relationship was cordial and productive, and the Bank continued to have a high quality policy dialogue with the Government, which was demonstrated during implementation. Review teams included specialists in public health, human development, health economics, financial management, and procurement. The Bank s partnership with donors, especially with PAHO, was highly productive. External consultants were used to provide support to specific aspects of project components, including the strengthening of primary health care, the design and implementation of the public insurance and the adequacy of health services to indigenous peoples. Participation of national, departmental and municipal representatives in the review missions enhanced Bank-Borrower bonding, better understanding and expeditious resolution of implementation bottlenecks. This also fostered mutual learning on institutional and field issues, and strengthened the implementing agencies in technical matters and Bank procedures. Aide-Memoires were regularly prepared, signed and transmitted, flagging outstanding issues and underscoring benchmarks for actions. These alerted the Government and the implementing agencies to problems with project execution and facilitated remedies in a timely manner, in conformity with Bank procedures. The Project Status Reports realistically rated the performance of the Project both in terms of achievement of development objectives and project implementation. Whenever delays in implementation occurred, the Bank s task team was able to define concrete steps and a timetable for putting the Project

20 back on track and pace. The Bank paid sufficient attention to the Project s likely development impact. The quality of advice and the follow-up on agreed actions were adequate. Credit covenants and remedies were enforced effectively. During the early stages of project implementation, the Bank conducted workshops for training of Borrower s staff in the areas of financial management, disbursements and procurement. With the decentralization of Bank functions to the Resident Mission in Bolivia, the Bank was able to provide better response and follow-up to the Borrower. The local support proved to be highly effective in improving the rhythm of response to the client, other interested stakeholders (including the international donor community) and project implementation. Local staff worked closely with the Government and the implementing agencies at all levels, and provided them with extensive assistance in the areas of technical contents, financial management and procurement. Changes of Bank staff responsible for supervising the Project did not seem to affect project implementation. 7.3 Overall Bank performance: Satisfactory. Overall, the Bank performance was satisfactory during Project preparation, appraisal and implementation. Borrower 7.4 Preparation: Satisfactory. The Borrower s performance in the preparation of the Project was satisfactory. The Borrower displayed the required level of commitment to project objectives and covered the adequacy of design on all major aspects, such as technical, financial, economic, institutional, environmental and sociological (including stakeholder consultations and commitment). The Borrower gave high priority to the Project in the context of the CAS discussions; for example, it approved two supreme decrees setting the legal stage needed for project implementation. The MOH had put together a strong team for the implementation of the Project. The Ministry of Finance had made a firm commitment to increase financing for vaccines and to open a budget line to make this financing transparent. The FIS had also assigned strong counterparts and had shown flexibility to deal with project preparation needs. Health authorities from five departments had shown great interest to participate in the Project by organizing workshops and preparing project implementation plans. Three of the departmental authorities had also expressed their support to the Project in written correspondence, already assigning counterpart funds for the 1999 fiscal year. The government officials and staff of the implementing agencies at the central, regional, and local levels worked closely with the Bank s project team on a continual basis, with full cooperation and enthusiasm. 7.5 Government implementation performance: Satisfactory. The Government s implementation performance was satisfactory. It consistently maintained its commitment throughout the implementation of the Project. The MOH, responsible for the overall implementation, was responsive to take corrective measures and effective in dealing with outstanding operational issues as implementation progressed. Appropriate levels of review and approval were usually in place, financial accountability and follow-up was observed, and expenditures were duly authorized before they were incurred. Project documentation was maintained properly for periodic review. Annual external audits gave favorable conclusions on project management. The latest part of project implementation, especially in 2003, was hampered by different factors

21 Counterpart funding problems affected the financing of vaccines. The economic constraints were such that the national government was not able to hold appropriate budget provisions. Throughout 2003, political interference with technical and managerial decisions within the Project also became an issue that had to be addressed. The PCU personnel changed, as did some of the key policies supported by the Project (the SBS was converted into the SUMI, and the health districts were transformed into Local Health Boards), eventually leading to implementation problems and/or delays. Finally, the political events of October 2003 added further obstacles to project implementation, mostly affecting the second phase of the reform program. 7.6 Implementing Agency: Satisfactory. The performance of the PCU was, overall, satisfactory. It was well organized and effective in dealing with procurement, disbursement, progress reports, and monitoring of indicators, and in maintaining proper records of the Project. It was receptive to the Bank s advice and highly collaborative with respect to meeting demanding benchmarks and deadlines. As pointed out above, during the last year of project implementation, it suffered the weakening effects of political interference and high instability, both on the administrative and technical areas. Performance of the FIS/FPS in the implementation of the MAR infrastructure component was partially satisfactory. The FIS arrangement for channeling prioritized investments for infant and maternal services at the municipal level confronted important internal and external obstacles, in particular during the period. External issues mainly originated in the structure and incentives of the National Compensation Policy. Internal difficulties included high levels of staff rotation, inadequate planning and programming of key activities, and slow processing of the demand-driven requests. These issues affected the Project s implementation pace and created, later on, problems in the preparation of withdrawal applications. Externally, the MAR mechanism was mainly confronted by problems with municipal counter funds, further delaying many of the works, and political interference. Delays in project implementation resulting from the above factors led to an extension of one year of the Project closing date (2003) to complete the MAR infrastructure and equipment component. Fortunately, most of the constraints were surpassed during this last year of implementation, and the MAR component was finally able to execute over 95 percent of its assigned budget. 7.7 Overall Borrower performance: Satisfactory. The overall performance of the Borrower was satisfactory. 8. Lessons Learned Developing a results-oriented culture. The use of well-defined indicators linked to expected outcomes and agreed ex-ante with the Borrower serves an important dual purpose. First, it helps improve project design and implementation by ensuring focus and aligning interventions towards the achievement of a set of clear benchmarks. Second, it helps the PCU, local governments and the MOH to center their policy dialogue on specific actions for achieving these pre-agreed results. The APL-I suggests that, for the successful introduction of such a results-oriented culture, several conditions need to be in place: (i) Policy instruments are needed to bring the nationally agreed targets to the operational level. This is where the PAs played an important role in this project. Their use ensured that the regions aligned their health priorities with the MOH. The APL-I experience suggests that

22 indicators need to be: (a) only a few to avoid loss of focus, (b) monitored using a consistent methodology, and (c) available on a periodic basis to allow for implementation of corrective actions. (ii) Incentives. The APL-I showed that the implementation of the PA must be accompanied by sufficient incentives at the national and regional levels to comply with the set targets and sustain the level of effort and interest. At the national level, one such incentive was making these benchmarks legally binding under the Credit Agreement, with a provision for remedial measures such as suspending disbursements in case the Borrower did not meet the targets. This innovative feature created the right environment and incentive for the national government and PCU team to focus on meeting the targets and achieving the desired outcomes. This setup could be further improved by developing an appropriate formula and remedial measures to avoid the possibility that one or more of the indicators are not consistently met, even though the Borrower has consistently met five out of the eight indicators, as required under the Credit Agreement. Likewise, at the regional level, the Project s experience suggests that the absence of such an incentive (for the departments) could explain the observed weak compliance with the targets set in some specific regions. (iii)the means to reach the targets. The APL-I suggests that the health sector must have the appropriate means to reach the agreed upon targets. For the APL-I, these included the SBS, the PAI, the IMCI and the MBP. These set of policies served health service providers as the right instruments to increase demand and coverage with quality, according to the targets set in the performance agreements. Taking into account the cultural barriers created by ethnic diversity. Cultural diversity remains one of the greatest challenges for improved health among the poor in Bolivia. The Project began to bridge the cultural barrier with the use of specific activities (e.g. focus groups on maternal health services with indigenous women, dialogue with indigenous organizations, and pilot experiences culturally adapting services to indigenous needs). These activities need to be developed in greater depth and their scope increased to progress further in bridging the cultural gap. Developing partnerships. While reaching a strong consensus with donors was difficult in the initial stages, ultimately this partnership turned out to be very useful for the success of the Project. For example, the national immunization program was successfully revamped due to a strong partnership established with PAHO. PAHO provided technical support for the Government and the Bank emphasized the financial and institutional aspects. This partnership was sustained during the whole period of project preparation and implementation. Similar efforts could strengthen impact in other project areas and are worth being started and secured as much as possible in the initial phases of project discussion and design. A focused and gradual approach. The Project focused on a few priority activities and emphasized capacity building, recognizing the multiplicity of needs and the limits of institutional response converging in the health sector. This approach allowed to avoid overburdening the limited capacity of participating institutions, while addressing a national top priority such as infant and maternal mortality. Being an APL, gradual complexity may be built into the agenda under the second phase of the Program, for example developing innovative supply and demand side arrangements that complement the public insurance scheme in overcoming barriers to access, or reaching for better articulation with the social security

23 Avoiding system failures. Although focusing on relevant outcomes such as reduced infant mortality and maternal mortality is important, it is equally important to acknowledge and avoid system failures. For example, the Project should ensure that, while making progress in meeting outcome targets, its progress on infant and maternal health status cannot be fully de-linked from the need for institutional strengthening or from more broad system improvements such as accountability, information and monitoring systems, and financial flows. 9. Partner Comments (a) Borrower/implementing agency: Following is the translation of the comments by the Borrower to this ICR (May 11, 2004). The original comments in Spanish are also included in Annex 8. In addition, the Borrower did an Evaluation Report at Project closure shared with the Bank during the ICR preparation stage (March 15, 2003). An executive summary of this Report is presented in Annex 9. The complete final report of this evaluation is available upon request. Borrower s Comments - Translation Ministry of Health and Sports Bolivia La Paz, May 11, 2004 CITE:MSD/0589/004 Mr. Juan Pablo Uribe Senior Health Specialist Latin America and the Caribbean Regional Office The World Bank Washington, D.C. Dear Sir: In response to your letter dated April 20 regarding the Implementation Completion Report (ICR) on the Health Reform Project (Phase I), Credit 3244-BO, I am pleased to submit the comments you have requested. For the first type of comments, related to the text and content of the document, the Project Management has already sent observations to the person in charge of health at the Resident Mission in Bolivia, in order to include the suggested changes and supplementary information in the report. With regard to the second set of comments, related to the opinions expressed on the general performance of the Project, I must express my agreement with the appraisal and ratings given on the execution of Credit 3244-BO, based on the design, approval, and

24 implementation activities of this credit, both from the Ministry of Health and the Project s Coordination Unit, as well as from the World Bank s project team and the Resident Mission and its support team. However, after conducting a review of the document and the assertions and comments reflected in it, it is necessary to point out the following: When the Health Reform Project was negotiated, it was implied that its financing would be aimed at generating changes in the country s health system; however, when we were informed of the scope of the Project s components, for example, the implementation of actions aimed at reducing maternal and child mortality, we noted that these take into account only part of the activities of the health system and, although the intention is to implement reform processes, these actions will only affect a few components. One of the lessons learned and suggestion to be taken into account in future projects, is that the name of the Project should denote its specific scope, as to avoid complications during its implementation. An analysis of the original components of the Project indicates that one of the actions of Component I was the empowerment of communities. The results included in the ICR do indeed reflect that some actions were implemented, such as the design of the Pregnant Women s Rights Charter or, in the area of interculturality, the implementation of the Basic Native and Indigenous Insurance, which was revoked as a result of the SUMI, and the incorporation of two large indigenous organization groups in the discussion of health issues. It would be important to undertake more aggressive actions to effectively involve the communities on health-related decision-making, being this an action that has been left pending for the second phase of the Project. One of the major achievements of the first component was generating awareness in the country to use national resources to purchase vaccines and syringes for the Expanded Program of Immunizations (PAI). The results in the first APL did not meet the expectations since the established goals were reached only for one year during project implementation. However, this issue is being permanently considered in the annual national budget approvals done by the Ministry of Finance. The inclusion of an article in a law requiring the Social Security system to transfer resources for preventive health programs, including the PAI, was a far-reaching measure. The decision on how to manage and use these funds lies within the sector, thus ensuring that a significant percentage of these resources be transferred to the PAI. One of the activities of the Project s second component strengthening the capacity to respond to health needs was to support the National Health Information System (SNIS). Our understanding is that the objective of the Project was to strengthen the health districts to adequately manage the information, in conjunction with the development of national actions financed by a second loan approved for the health sector via the Inter-American Development Bank. After four years of project implementation, given that these second loan faced implementation problems, the activities carried out by the World Bank credit did not

25 meet expectations, implying important investments with unsatisfactory results. As a lesson learned for our Ministry is that of not committing complementary actions from two different financing sources, if there is no assurance that the implementation timetables will be parallel. One of the important achievements of the second component is the implementation of the performance agreements between the Ministry and the Departmental Health Services (SEDES). The decentralization processes underway in the country requires that we be more specific in this type of instrument, moving from an instrument to verify operational activities within the framework of the structure and production of services to becoming a financial evaluation instrument. For this, it is important that the second APL include actions leading to the allocation of national and regional resources from public funds and demand international aid agencies to report on the resources allocated to each of the Departments. While analyzing the indicators used to evaluate the Project s achievements, we question their pertinence and the development of these indicators based on the general objective of the Project. Six of the eight project indicators are related to the productivity of the health system, one measures the capacity of the PAI in reaching key municipalities with respect to equity, and one reflects the country s willingness to finance health activities. There is no indicator regarding the empowerment of the communities, an important issue that has yet to be addressed. With regard to the construction of the indicators, we believe these should be reviewed. For example, the second indicator on the complete cycle of antenatal care reflects only the return of women meeting a cycle of four office visits over a denominator (number of first visits) which we are sure does not reflect the reality given the distortions in the information system, thus the results of this indicator will always be low. The same applies to the third indicator, related to early hospital neonatal mortality. On the other hand, those indicators on the number of pneumonia cases or the supply of iron sulfate may perhaps not be measuring the system s efficiency, since, in the first place, given the effect of the pentavalent vaccine, we should start by measuring the decrease in the prevalence and, in the second, the intake or use of the iron and not only its distribution. One comment from the Minister s office to the Project refers to the relevance of the investments made through the FPS. The National Cofinancing Policy and the procedures approved by the DUF for the FPS leave out the social ministries from participating in the decision-making on the investments made through this agency. This is not a problem derived directly from the Project, since it goes beyond the actions that might be executed by it, nevertheless the consequences of this do have an impact on the funds allocated to investments with resources from Credit 3244-BO and now from Credit 3541-BO. In analyzing the participation of other actors, such as process financiers or donors, one achievement that should be highlighted is the partnership with the Pan American Health Organization (PAHO), British Cooperation (DFID) and Canadian Cooperation (CIDA)

26 However, we must state that, unfortunately, relations with other institutions such as the Inter-American Development Bank (IDB), European Union (UE), Agency for International Development (USAID), other aid agencies from The Netherlands, Belgium and Japan, and agencies such as UNICEF, UNFPA, PMA, among others, have been nonexistent or infrequent, despite the existence of the Inter-Agency Committee. We hope that with these comments we are contributing to the final evaluation of the Project and stated the perception of this Ministry with regard to a project that was completed during the previous administration. We reiterate our most cordial salutations. (S) Fernando S. Antezama Aranibar Minister of Health and Sports (b) Cofinanciers: Not Applicable. (c) Other partners (NGOs/private sector): Not Applicable. 10. Additional Information A. The Bank s ICR Team consisted of the following members: Juan Pablo Uribe (Senior Health Specialist, Task Team Leader) Patricia Álvarez (Operations Officer) Nicole Schwab (Economist, Junior Professional Associate) Sati Achath (Consultant) Mary Dowling (Program Assistant) Patricia Bernedo (Program Assistant) Mónica Claros (Team Assistant) B. List of Task Team Leaders of the Project in chronological order: Daniel Cotlear Juan Pablo Uribe C. List of people interviewed for the ICR: Evangeline Javier (Sector Manager, WB) Daniel Cotlear (Country Sector Leader, WB) John Lincoln Newman (Country Manager, WB) Rudolf V. Van Puymbroeck (Adviser) Juan Pablo Uribe (Task Team Leader, WB) Nicole Schwab (Junior Professional Associate, WB; ex-pcu technical staff) Fernando Lavadenz (Health Specialist, WB; ex-pcu Director)

27 Patricia Álvarez (Operations Officer, WB) Gina Tambini (Area Manager, Family and Community Health PAHO; ex-pai staff) Cristian Pereira (current PCU Director and ex-technical Director of the PCU in Bolivia) Marina Cárdenas (DFID health staff in Bolivia; ex-pcu technical staff)

28 Annex 1. Key Performance Indicators/Log Frame Matrix Key Performance Indicators Outcome/Impact Indicators: Achieved coverage (target) Percentage of births attended by Achieved Coverage 42% 47% 52% 54% 54% trained health personnel APL-1 Targets 36% 38% 40% 43% 46% 2. Women with complete cycle of Achieved Coverage 30% 31% 33% 36% 34% antenatal care visits APL-1 Targets 28% 30% 32% 36% 40% 3. Early neonatal hospital mortality Achieved Coverage (per 1,000 live births) APL-1 Targets Number of pneumonia cases in Achieved Coverage children under 5, attended in health services APL-1 Targets Number of acute diarrhea diseases Achieved Coverage in children under 5, attended in health services APL-1 Targets Third dose of DPT/Hib/Hepatitis Achieved Coverage 77% 85% 89% 92% 100% B vaccine APL-1 Targets 75% 82% 65% 75%75% 85% 7. Number of municipalities with Achieved Coverage DPT/Hib/Hepatitis B vaccine coverage below 80 percent APL-1 Targets National financing of vaccines (in Achieved Coverage millions of US $) APL-1 Targets Output Indicators: Indicator Projected in SAR/PAD Actual/Latest Estimate Not applicable

29 Annex 2. Project Costs and Financing Annex 2a Project Costs by Components (in US$ million equivalent) Project Component Appraisal Estimate Actual/Latest Estimate* Percentage of Appraisal Component I. Coverage and quality improvements of the health services and empowerment of communities Component II. Strengthening local capacity to respond to health needs Component III. Coordination, Monitoring and Evaluation % % % Total % * Total costs, including both external and local financing

30 Expenditure Categories 1 Works Annex 2b Project Costs by Procurement Arrangements (in US$ million equivalent)* Procurement Method Appraisal Estimate Procurement Method Actual/Latest Estimate * ICB NCB Other NBF Total ICB NCB Other NBF Total Goods Services Miscellaneous Health ** 9.0 Subprojects Incremental Administrative Cost Refunding of cost-preparation advance Unallocated Total * Includes external funding (Credit) ** Includes various procurement modalities used under the MAR-I subcomponent. Note: NBF = Not Bank Financed (includes elements provided under parallel cofinancing procedures, consultants under trust funds, any reserved procurement and any other miscellaneous items). The procurement arrangements for items listed under Other and details of the items listed as NBF need to be explained in footnotes to the table

31 Annex 2c Project Financing by Component (in US$ million equivalent) Component Appraisal Estimate Actual/Latest Estimate Percentage of Appraisal Bank Government Bank Government Bank Government Component % 18.7% I. Coverage and quality improvement s of the health services and empowerme nt of communities Component % 118.6% II. Strengthenin g local capacity to respond to health needs Component III. Coordination, Monitoring and Evaluation % 100% TOTAL % 22.8%

32 Annex 3. Economic Costs and Benefits Annex 3 Cost Benefit Analysis (indicate currency, units and base year) Present Value of Flows Economic Analysis Financial Analysis Appraisal Latest Estimates Appraisal Latest Estimates Benefits Costs Net Benefits IRR/NPV NOT APPLICABLE

33 Annex 4. Bank Inputs (a) Missions: Stage of Project Cycle No. of Persons and Specialty (e.g. 2 Economists, 1 FMS, etc.) Month/Year Count Specialty Identification/Preparation 06/15/-06/19/98 6 SECTOR LIDER (1) INVESTMENT MECANISMS (1) BASICAL INSURANCE (1) INF. AND INSTITUTIONAL (1) PHARMACEUTICAL SPEC. (1) TASK TEAM LEADER (1) 09/07-09/11/ HEALTH SPECIALIST (1) EPIDEMIOLOGIST (1) INVESTMENT AND PROJECT ORG. SPECIALIST (1) TASK TEAM LEADER (1) 11/16/-11/24/ TASK MANAGER (1) IMMUNIZATIONS SPECIALIST (1) INVESTMT SYSTEMS SPEC. (1) HEALTH FINANCE SPEC. (1) INSTITUTIONAL SPEC. (1) IMCI SPECILIAST (1) FINANCIAL ASSESSMT. (1) Performance Rating Implementation Progress Development Objective Appraisal/Negotiation 01/28-29/ /09/ /12/ /19/1999 Supervision 12 RES REPRESENTATIVE (1); SECTOR LEADER; (1) CONSULTANT (Nutrition) (1) PROCUREMENT SPECIALIST (1) TASK TEAM LEADER (1) LAWYER (1) LEAD PROC. SPECIALIST (1) SENIOR FIN. MANAGMT (1) IMPL. AND INVEST. FUND SPECIALIST (CONSULTANT) (1) DISBURSEMENT OFFICER (1) 08/03/ SR. ECONOMIST/TM (1); S S

34 MONITORING AND EVALUAT (1); FINANCIAL SPECIALIST (1); MAR (1); PAHO, CHILD/MOTHER HEA (1); PAHO, IMMUNIZATIONS SP (1) 12/21/ /04/ /26/ HEALTH SPECIALIST (1); MAR (1) TASK TEAM LEADER (1) CONSULTANT (1) FINANCIAL MANAGEMENT (1) IMPLEMENTATION SPECIALIST (1) 03/03/2001 TASK TEAM LEADER (1) ECONOMIST (1) 08/24/2001 NUTRITIONIST (1) CONSULTANT (1) 11/23/2001 PROCUREMENT SPECIALIST 04/19/ MISSION LEADER (1); MAR - NUTRITION (1); EXTENSA (2); VACCINES (1) 08/30/ TASK MANAGER (1); PROJECT OFFICER (1); FINANC. MANAG. ANALYST (1); HEALTH SPECIALIST (2) 02/14/ TASK MANAGER (1); TASK MANAGER (NEW) (1); OPERATIONS OFFICER (1); SR. SOCIAL SCIENTIST (1); JPA (1); HEALTH SPECIALIST (1); FINAN. MANANGEMENT (1); PROC. ASSISTANT (1); CONSULTANT (3) 06/20/ TASK TEAM LEADER (1) HEALTH ECONOMIST (1) OPERATIONS OFFICER (1) HEALTH SPECIALIST CONSULTANT (1) PROCUREMENT SPECIALIST (1) PROCUREMENT ASSISTANT (1) FINANCIAL MANAGEMENT (1) 11/13/ TASK TEAM LEADER (1) OPERATIONS OFFICER (1) HEALTH ECONOMIST (1) HD SPECIALIST (1) HEALTH SPECIALIST (1) PROCUREMENT SPECIALIST HS HS HS S HS HS HS S

35 (1) FINANCIAL MANAGEMENT (1) ICR 03/01/04-04/30/ TASK TEAM LEADER (1) CONSULTANT (1) (b) Staff: Stage of Project Cycle No. Staff weeks Identification/Preparation 17 Appraisal/Negotiation 12 Supervision 40 ICR 4 Total 73 Actual/Latest Estimate US$ ('000)

36 Annex 5. Ratings for Achievement of Objectives/Outputs of Components (H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable) Rating Macro policies H SU M N NA Sector Policies H SU M N NA Physical H SU M N NA Financial H SU M N NA Institutional Development H SU M N NA Environmental H SU M N NA Social Poverty Reduction H SU M N NA Gender H SU M N NA Other (Please specify) H SU M N NA Private sector development H SU M N NA Public sector management H SU M N NA Other (Please specify) H SU M N NA

37 Annex 6. Ratings of Bank and Borrower Performance (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory) 6.1 Bank performance Rating Lending HS S U HU Supervision HS S U HU Overall HS S U HU 6.2 Borrower performance Rating Preparation HS S U HU Government implementation performance HS S U HU Implementation agency performance HS S U HU Overall HS S U HU

38 Annex 7. List of Supporting Documents 1. Aide Memoires, Back-to-Office Reports, and Project Status Reports. 2. Project Progress Reports. 3. Consultant Study Reports financed under the Project. 4. Borrower s comments to the ICR, dated May 11, 2004 (included as Annex 8). 5. Borrower s Project Evaluation Report dated March 15, 2004, and its corresponding executive summary (included as Annex 9). 6. Project Appraisal Document for Bolivia Health Sector Reform Project dated March 31, 1999 (Report No: BO)

39 Additional Annex 8. Borrower s Comments Original

40 - 37 -

41 - 38 -