Validation Report. Indonesia: Second Decentralized Health Services Project. Independent Evaluation Department

Transcription

1 Validation Report Reference Number: PVR-428 Project Number: Loan Numbers: 2074 and 2075 November 2015 Indonesia: Second Decentralized Health Services Project Independent Evaluation Department

2 ABBREVIATIONS ADB Asian Development Bank BKKBN Badan Kependudukan dan Keluarga Berencana Nasional or National Family Planning Coordinating Board DMF design and monitoring framework EIRR economic internal rate of return MCH maternal and child health MDGs Millennium Development Goals MOH Ministry of Health PCR project completion report NOTE In this report, $ refers to US dollars. Key Words adb, asian development bank, bkkbn, child health, decentralized health service, family planning, indonesia, maternal and child health, mch, mdg, millennium development goal, validation The guidelines formally adopted by the Independent Evaluation Department (IED) on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of IED management, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, IED does not intend to make any judgments as to the legal or other status of any territory or area.

3 PROJECT BASIC DATA Project Number: PCR Circulation Date: Sep 2014 Loan Numbers: 2074 and 2075 PCR Validation Date: Nov 2015 Project Name: Second Decentralized Health Services Project Country: Indonesia Approved ($ million) Actual ($ million) Sector: Health Total Project Costs: ADB Financing: ($ million) ADF: 2075: OCR: 2074: Loan: Borrower: Beneficiaries: Others: Cofinancier: Total Cofinancing: Approval Date: 19 Dec 2003 Effectiveness Date: 15 Mar Mar 2005 Signing Date: 16 Dec 2004 Closing Date: 31 Dec Dec 2013 Project Officers: Initial Reviewers: Quality Reviewer: J. Jeugmans Y. Shiroishi K. Saleh B. Lochmann G. Servais B. Lochmann K. Hardjanti, Consultant M.J. Dimayuga, Senior Evaluation Officer E. Gozali, Principal Evaluation Specialist, IED1 Location: Peer Reviewer: Director: From: Dec 2003 May 2004 Feb 2005 Dec 2008 Jun 2010 Nov 2013 M. Vijayaraghavan, Senior Evaluation Specialist, IED2 W. Kolkma, IED1 To: Apr 2004 Jan 2005 Nov 2008 May 2010 Nov 2013 Dec 2013 ADB = Asian Development Bank; ADF = Asian Development Fund; IED1 = Independent Evaluation Department, Division 1; IED2 = Independent Evaluation Department, Division 2; OCR = ordinary capital resources; PCR = project completion report. A. Rationale I. PROJECT DESCRIPTION 1. Indonesia has made substantial progress in improving the health status of women and children, albeit with substantial disparities across provinces. There was a concern that as a result of the decentralization reforms, Indonesia s progress in reaching the Millennium Development Goals (MDGs) on maternal and child health (MCH) would become slow or even at the risk of being reversed. Thus, in 2003, the government requested the Asian Development Bank (ADB) to finance the Second Decentralized Health Services Project 1 to help improve the health and nutritional status of vulnerable segments of the population, particularly women and 1 ADB Report and Recommendation of the President to the Board of Directors: Proposed Loan for the Government of Indonesia for the Second Decentralized Health Services Project. Manila.

4 2 children, in eight provinces. 2 Since the decentralization reforms in the early 2000s, district governments have been responsible in managing the country s health system. B. Expected Impact 2. The project s envisaged impact was improved health status of the population, especially the poor and vulnerable groups. The targets were (i) maternal mortality ratio reduced from 262 per 100,000 live births in 2005 to 175 per 100,000 live births by 2010, and (ii) infant mortality rate reduced from 35 per 1,000 live births in 2005 to 26 per 1,000 live births by C. Objectives or Expected Outcome 3. The project s expected outcome was improved health services in eight provinces and 90 districts and cities (see footnote 2). It aimed to achieve (i) improved capacity at the regional level to ensure the provision of quality health and family planning services; (ii) improved equitability, quality, and financial sustainability of local health services; and (iii) improved capacity of the Ministry of Health (MOH) and the National Family Planning Coordinating Board or the Badan Kependudukan dan Keluarga Berencana Nasional (BKKBN) to support the regional government s delivery of services. The outcome targets were (i) increased measles immunization coverage from 72.8% to 85.0%, (ii) increased contraceptive prevalence rate by 10.0%, (iii) increased number of births attended by skilled personnel from 50.0% to 53.4%, (iv) decreased unmet need for contraception from 9.1% to 8.0%, (v) increased number of children weighed at the community health posts from 30.0% to 60.0%, and (vi) a 20.0% increase in the number of obstetric complications managed by health facilities. D. Outputs 4. Originally, the envisaged project outputs were organized into three areas: (i) strengthened regional capacity to ensure the provision of health and family planning; (ii) more equitable, higher quality, and more sustainable local health services; and (iii) strengthened capacity to support decentralized health services. At midterm (2008), the outputs were regrouped into five areas to incorporate activities under the desa siaga (alert village) program and to better demarcate outputs by agencies and/or entities responsible for their deliveries. The five areas were (i) strengthened MCH services (five indicators), (ii) enhanced community participation for improved MCH (desa siaga block grants) (three indicators), (iii) revitalized family planning services (eight indicators), (iv) improved district capacity in managing decentralization (three indicators), and (v) strengthened central government capacity to support decentralized health services (two indicators). E. Provision of Inputs 5. The total project cost at appraisal was $143.0 million $73.2 million in foreign exchange cost and $69.8 million equivalent in local currency cost, financed by two ADB loans: (i) $64.8 million from ADB s ordinary capital resources, and (ii) a concessionary loan of $35.2 million from ADB s Asian Development Fund. The two ADB loans were to cover all of the foreign exchange costs (51.2% of the total project cost) and 38.4% of the project s local currency cost (18.8% of the total project cost).the government was to provide $43.0 million equivalent (61.6% of local currency cost and 30.0% of the total project cost). Project funds were provided as transfer 2 The eight provinces are South Sumatra, Bangka Belitung, Central Kalimantan, South Kalimantan, South Sulawesi, West Nusa Tenggara, East Nusa Tenggara, and Gorontalo. A ninth province, West Sulawesi, was subsequently added to the project.

5 3 grants to the subnational governments since the project was expected to benefit the poor and would focus on nonrevenue-generating health services. Participating local governments were expected to contribute to the project costs in accordance with their fiscal capacity, which did not occur. 6. The actual total project cost at completion was $102.9 million (28.0% lower than estimated at appraisal) $23.7 million of foreign exchange cost and $79.2 million of local costs. The total loan amount was $84.3 million (15.7% lower than at appraisal) and the government contribution was $18.6 million (56.7% lower than at appraisal). Due to exchange rate fluctuations of the special drawing right against the dollar, the total loan value increased to $102.8 million before taking into account loan cancellations. The government requested two partial cancellations: (i) $13.1 million in August 2008, and (ii) $3.0 million in April The cancellation brought the total loan amount to $86.7 million, of which $84.3 million was disbursed. The government requested ADB for a project extension of 2 years to support the rehabilitation of provincial health infrastructure in the province, mainly because of the damage in the health sector infrastructure due to the powerful earthquake in West Sumatra. The total damage in the health sector was estimated at $83.2 million and reconstruction needs were estimated at $77.6 million. F. Implementation Arrangements 7. The executing agency was the MOH. At the national level, the project was coordinated by a central project implementation unit in the MOH and a subunit in BKKBN. Provincial project implementation units supported and coordinated district and city implementation units. Health committees, represented by government services, professional associations, and civil society were set up to advise the regional project directors and to ensure participation and transparency in decision making. In the districts and cities, regular government staff worked full time on project implementation. As local capacity increased with project support, regional governments replaced contractual staff with civil servants. The BKKBN was decentralized in 2004, and family planning was merged with other local government offices, such as health and social welfare. Thus, about 50% of BKKBN staff was assigned to other services. The 732 block grants made to villages under the desa siaga program component were administered by 732 village forums. During project implementation, the number of project provinces increased from 8 to 10, including West Sumatra, to support the earthquake rehabilitation. II. EVALUATION OF PERFORMANCE AND RATINGS A. Relevance of Design and Formulation 8. The project completion report (PCR) 3 rated the project relevant. It was considered highly relevant at appraisal, as the design was created when Indonesia was just starting to conduct a major restructuring of the health system as a result of decentralization which shifted the responsibility for most health and family planning services to the district level. However, it mentioned that in the transition to a decentralized health system and family planning program, the provision of basic health services and the achievement of the health-related MDGs became unclear. The project s rationale also indicated a deep concern that as a result of the government s decentralization reforms, progress in reaching the target of the MDGs on MCH would become sluggish or even reversed (para. 1), the validation is of the opinion that this 3 ADB Completion Report. Second Decentralized Health Services Project in Indonesia. Manila.

6 4 should have been anticipated when designing the project. There was also a lack of professional and managerial skills at the provincial level, which emerged as a big challenge. 9. Nonetheless, the project was designed to support comprehensive reform and capacitybuilding efforts of MOH and BKKBN within the decentralization process. It was in line with the government s National Medium-Term Development Plan ( ) that emphasized health and poverty reduction, and the government s long-term strategy for the health sector. 4 It was anticipated that successful decentralization would increase access to better quality care, particularly for vulnerable groups. The project design incorporated lessons from the previous ADB-financed Decentralized Health Services Project 5 and aimed to expand the geographic coverage with the new project. It was relevant for improving health services in the targeted geographic areas, especially in the context of decentralization. The project was consistent with the ADB country strategy and program that prioritized reducing poverty and regional inequalities and promoting human development, 6 and with the ADB health sector policy that underlines access to effective, cost-efficient, and affordable basic health services. 7 Therefore, this validation also assesses the project relevant. B. Effectiveness in Achieving Project Outcome and Outputs 10. The PCR rated the project less effective. The PCR discussed four key outcome indicators that examined data from the Indonesia Demographic and Health Surveys (PCR, Appendix 9). First, the target to increase measles immunization coverage from 71.8% (2003) to 85.0% (2012) was partly achieved. In the project areas, the attainment was 77.9%. 8 Second, the target to increase contraceptive prevalence rate by 10% was achieved. In the project provinces, this rate rose from 49.6% to 55.2%, while in non-project provinces the increase was only 58.0% to 59.4%. Third, the goal to decrease the unmet need for contraception for married women to 8.0% was unsuccessful. Instead of decreasing, the indicator increased from 10.5% to 12.5% between 2005 and A fourth indicator the percentage of obstetric deliveries attended by skilled providers increased from 56.5% to 74.9% between 2003 and 2012 in project areas. The PCR (para. 32) recognized this as a significant increase but still somewhat short of the intended target of 80.0% as more time was needed for the provinces to make improvements. As discussed in para. 3, there were two other outcome indicators and these were not discussed in the PCR assessment. The fifth indicator was an increase in children weighed at community health centers. The targeted increase was only partly achieved at 68.0%, as compared to the target of 78.0%. On the sixth indicator percentage of obstetric complications managed by health facilities the target (20.0% increase) was exceeded as the level rose from a baseline of 3.0% to 57.1%. In all, two of six outcome targets were achieved and/or exceeded, three have progressed and were partly achieved, while one was not achieved. 11. While the targeted activities and outputs (21 indicators) were largely met (PCR, Appendix 1), 9 there was a shortfall with respect to the achievement of the project s expected 4 Ministry of Health Development Plan Towards Healthy Indonesia. Jakarta. 5 ADB Report and Recommendation of the President to the Board of Directors: Proposed Loan for the Government of Indonesia for the Decentralized Health Services Project. Manila. 6 ADB Country Strategy and Program: Indonesia, Manila. 7 ADB Policy for the Health Sector. Manila. 8 The slow increase in immunization coverage was mainly because (i) between 2006 and 2010, the government s budget allocation for MCH decreased; (ii) during the second phase of the project, significant resources were transferred to block grants for desa siaga, which did not include immunization; and (iii) it was constrained by limited road access and health facilities in remote areas, such as in parts of eastern Indonesia. 9 The PCR (Appendix 9) reported that progress in outcome indicators was not significantly different between project and non-project provinces (measles vaccination rate, birth attended by skilled providers, and infant mortality).

7 5 outcome. The PCR reported that improvements in sector indicators vary widely among districts, with progress in the eastern part of Indonesia lagging behind. There is also an ongoing concern that the expanded health services in the project provinces had not effectively reached the poor (PCR, Appendix 9). This validation recognizes the progress made in key sector indicators although it views some of the targets as actually too ambitious. Given the shortfall in actual achieved outcomes, this validation assesses the project less than effective. C. Efficiency of Resource Use in Achieving Outcome and Outputs 12. The PCR rated the project less efficient. This assessment was made based on the process efficiency of project implementation (PCR, paras ). During the first 3 years, the project experienced delays in releasing counterpart funds. There was a high staff turnover at the district level and there were gaps in government policy for staff incentives at the peripheral level. The original project design was in line with Ministerial Decree KMK35/2003, specifying that onlending was allowed only for regional projects that generate revenue. Hence, arrangements were made for financial sharing where the amount of project grants to districts and cities were linked to their fiscal capacity, with the local governments providing counterpart funds to finance the balance of project costs. Unfortunately, there was no clear implementation guidance, and local governments were not prepared to receive loans and provide sufficient counterpart funds. Efforts to correct these problems were undertaken during , but only 10% of project funds was disbursed after 48% of the project period had elapsed. 13. Following the midterm review, major changes were made in implementation arrangements and progress was expedited. This validation notes, however, that substantial detailed work was done at project completion to reestimate the project economic internal rate of return (EIRR) (PCR, Appendix 10). This analysis was incorrectly considered as a part of the sustainability assessment instead of efficiency (PCR, para. 37). The reestimated project EIRR was 31% compared to 37% at appraisal. With the additional West Sumatra subproject, the total reestimated project EIRR was 33%. This validation considers the economic analysis approach sound, utilizing conservative estimates of expected benefits (gains in disability-adjusted life years or DALY). 10 This validation, however, notes that the assumed start of economic benefits was somewhat optimistic. Adjusting for this, the project still yields robust EIRRs between 16% 20%. Given the aforementioned, this validation assesses the project efficient. D. Preliminary Assessment of Sustainability 14. The PCR rated the project likely sustainable. A preliminary assessment of the project s sustainability, including a review of recent trends in public health expenditure at all levels, and an assessment of the fiscal impact of project investments at the district level (where most investments were targeted), found that Indonesia s public health expenditure, especially at the district level, has increased. 11 However, districts have minimal effective control over their budgets. While the PCR mentioned that the projected fiscal burden is likely manageable in most districts, inequality in public health expenditure among districts could result in higher fiscal Follow-up discussions with the operations department suggested that this may be attributed to similar sector support provided to non-project districts and/or provinces, especially under the desa siaga programs supported by other external partners. Non-project provinces include those supported by the first Decentralized Health Services Project (Loan 1810-INO) of ADB. 10 The economic analysis used the approach utilized to evaluate the first phase project (see ADB Performance Evaluation Report. Indonesia: Decentralized Health Services Project. Manila). 11 Districts have an unequal share of health expenditure and in public health expenditure per capita.

8 6 burden in some districts. For the preliminary assessment of the project s sustainability, this validation assesses it to be likely sustainable. E. Impact 15. The project s expected impact, as indicated in the design and monitoring framework (DMF) of the PCR, was improved health status of the population, especially of the poor and vulnerable. However, the PCR did not provide a specific rating for impact. It only presented descriptions of the project s contributions, such as (i) empowered the community for an improved MCH; (ii) developed human resource for the health staff (particularly at the district and health center levels); (iii) developed health infrastructure, including in West Sumatra; and (iv) the desa siaga program component to have benefited those with higher incomes, while the training of midwives in the project s first phase benefited the poor. As discussed in para. 10, the progress in achieving the project s outcome took more time to realize and impact may be more evident several years after the project s completion, given the project s rather complex nature. More sustained work is needed for benefits to reach less developed and/or accessible areas and the lower-income strata in the target communities. Given the contributions of the project, as stated in items (i) to (iv) above, this validation assesses its impact significant, although this is a borderline case. III. OTHER PERFORMANCE ASSESSMENTS A. Performance of the Borrower and Executing Agency 16. The PCR rated the performance of the borrower satisfactory. The difficulty and substantial delay in the timely release of project counterpart funds during the first phase ( ) was caused by the transition phase toward a decentralized system, which delayed project implementation. At midterm review, this situation was resolved by redesigning the project. Thus, during the project s second phase ( ), counterpart funds were released and received in a timely manner. In redesigning the project, the executing agency worked closely with ADB to ensure that the country s priorities in attaining the MDGs could be reached. Nevertheless, the performance of the executing agency could have been better if its administrative personnel did not change often, and if the interaction between the MOH and BKKBN was closer. The project s decentralization ambitions were not matched by the capacities at the provinces and district levels. This issue should have been identified in the project preparation stage. In view of the lack of planning and prolonged implementation period, this validation considers the performance less than satisfactory. B. Performance of the Asian Development Bank 17. The performance of ADB was rated satisfactory by the PCR. During project implementation, ADB fielded 13 review missions, 2 inception review missions, 1 midterm review mission, 4 special review missions, and 1 project completion review mission. However, similar to the executing agency, ADB had a high turnover of project officers, where five different staff implemented the project. Nonetheless, ADB continued to provide technical support and guidance throughout project implementation. When it was essential to redesign the project after the midterm review, ADB worked closely with the MOH in (i) aligning the project to government priorities, particularly in its effort to achieve the MDGs related with MCH; (ii) adjusting the project scope; and (iii) reallocating the resources. ADB responded proactively to the adversity caused by the 2009 earthquake in West Sumatra. This validation considers the performance of ADB satisfactory.

9 7 IV. OVERALL ASSESSMENT, LESSONS, AND RECOMMENDATIONS A. Overall Assessment and Ratings 18. The PCR s overall rating of the project was partly successful relevant, less effective, less efficient, and likely sustainable. The project did not fully achieve its output and outcome indicators. In addition, due to the changes made in the midterm review, the project could not comprehensively support government decentralization efforts mainly because the MOH was cautious in decentralizing responsibilities. However, in contrast to the PCR rating, this validation considers the project efficient as it has a high reestimated EIRR, despite slow implementation progress in the initial years. Moreover, the project was successful in providing a disaster response by reconstructing and equipping health facilities in West Sumatra caused by the 2009 earthquake. Given these, this validation assesses the project successful. Overall Ratings Criteria PCR IED Review Reason for Disagreement and/or Comments Relevance Relevant Relevant Effectiveness in achieving outcome and outputs Efficiency in achieving outcome and outputs Preliminary assessment of sustainability Overall assessment Less effective Less than effective Less efficient Efficient Despite the slow initial implementation, the project benefits were large enough to outweigh the project costs as reflected in a sufficiently robust EIRR (paras ). Likely Likely sustainable sustainable Partly Successful successful Impact Not rated Significant The project did not fully meet its outcome targets, hence, this is a borderline rating between moderate and significant (para. 15). Borrower and executing agency Satisfactory Less than satisfactory There was lack of planning and project design was ambitious, given the lack of capacity at the province and district levels; implementation was delayed during the early stages due to delays in the release of counterpart funding (para. 16). Performance of ADB Satisfactory Satisfactory Quality of PCR Less than satisfactory Refer to para. 22. ADB = Asian Development Bank, EIRR = economic internal rate of return, IED = Independent Evaluation Department, PCR = project completion report. Note: From May 2012, IED views the PCR rating terminology of "partly" or "less" as equivalent to "less than" and uses this terminology for its own rating categories to improve clarity. Source: ADB Independent Evaluation Department. B. Lessons 19. The PCR derived two main lessons from the project. Firstly, it would have been better if the project resource was not geographically spread too thinly and did not cover too many

10 8 activities. Increased effectiveness could be achieved by prioritizing investments at different locations based on health needs and capacities. Secondly, the government s fiscal capacity to finance capacity development in the health sector remains limited. As such, it is important to ensure that investments are made in capacity building and in health infrastructure. This validation concurs with these lessons. The project experience also reaffirms the common understanding that a high turnover of project staff adversely affects project performance. C. Recommendations for Follow-Up 20. This validation agrees with the following four recommendations from the PCR. First, the project would have benefited from an independent impact evaluation of the desa siaga program, which would have allowed adjustments in succeeding interventions, and would have helped the government assess the feasibility of scaling up the program nationally. Second, while a large number of health professionals and staff were trained under the project, the quality of this training was less clear given the lack of thorough evaluation. Determining future skills gaps and addressing deployment practices will require frequent skills audits to ensure that workforce strategies are effectively implemented. Third, the original project design emphasized the need for more public private partnerships in health care delivery. Given the major shift to private providers for obstetric deliveries in recent years, it is essential to include the private sector in national target programs that address MCH. Fourth, greater synergy is required between health projects of this nature and government national poverty reduction and social protection programs. Further, inclusion of MCH into the poverty reduction initiative could better reduce child and maternal mortality. V. OTHER CONSIDERATIONS AND FOLLOW-UP A. Monitoring and Evaluation Design, Implementation, and Utilization 21. The project s monitoring and evaluation system and other related data collection were established. Throughout project implementation, overall policy guidance was ensured and review missions and meetings were held. Important meetings with in-depth discussions were conducted, especially when needed to decide on urgent issues, such as during the midterm review when adjustments to the project design were considered crucial and a redesigning of the project was undertaken. Results from the project s surveys and those done by development partners were used as information and data source. The project is commended for guiding its implementation on the basis of outcome progress tracking using the data from the Indonesia Demographic and Health Survey. This allowed a reevaluation of the project s economic and financial analysis. B. Comments on Project Completion Report Quality 22. Overall, the quality of the PCR is rated less than satisfactory. The PCR was prepared in line with its guidelines. However, it has issues in maintaining internal consistency in its reporting. For example, when assessing the project s effectiveness in achieving its outcomes, the outcome indicators mentioned in the PCR s main text were not consistent with those in the its DMF (PCR, Appendix 1). In turn, the DMF outcome data differed from the more in-depth and/or accurate data presented in Appendix 9. Moreover, the discussion of the EIRR was placed under sustainability when it should be under the efficiency section. However, the PCR is commended for its in-depth outcome analyses in Appendix 9, which presented not only baseline and post-completion health outcome levels but also results in control provinces. It was candid in providing analysis and description of the project s strengths and weaknesses and used data and

11 9 information collected from its own and other surveys. It included useful lessons that were derived from the project and gave sound recommendations for follow-up actions. C. Data Sources for Validation 23. Data sources for this PCR validation were taken from the PCR, report and recommendation of the President, and loan review mission reports. D. Recommendation for Independent Evaluation Department Follow-Up 24. A performance evaluation report may be carried out if this supports a broader evaluation (e.g., on health and/or social sectors or on Indonesia). The evaluation should be done after 3 years from project completion when enough time has elapsed to allow an assessment of the project s impact and to reexamine the project s effectiveness and sustainability.