IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-H0430 IDA-H2060 IDA-H3840) GRANTS IN THE AMOUNT OF SDR 76.9 MILLION (US$ MILLION EQUIVALENT)

Transcription

1 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Document of The World Bank Report No: ICR IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-H0430 IDA-H2060 IDA-H3840) ON GRANTS IN THE AMOUNT OF SDR 76.9 MILLION (US$ MILLION EQUIVALENT) TO THE ISLAMIC REPUBLIC OF AFGHANISTAN FOR A HEALTH SECTOR EMERGENCY RECONSTRUCTION AND DEVELOPMENT PROJECT March 31, 2010 Human Development Sector Afghanistan and Bhutan Country Department South Asia Region

2 CURRENCY EQUIVALENTS (Exchange Rate Effective February 16, 2010) Currency Unit = Afghani (AFN) AFN 1.00 = US$ US$ 1.00 = AFN FISCAL YEAR March 21 March 20 ABBREVIATIONS AND ACRONYMS ARI Acute Respiratory Infection ISN Interim Strategy Note ARTF Afghanistan Reconstruction Trust MDG Millennium Development Goal Fund BHC Basic Health Center MICS Multiple Indicator Cluster Survey BPHS Basic Package of Health Services MOH Ministry of Health CAS Country Assistance Strategy MOPH Ministry of Public Health CHC Comprehensive Health Center NA Not Available CHW Community Health Worker NGO Non-Governmental Organization DALY Disability Adjusted Life Year NRVA National Risk and Vulnerability Assessment DH District Hospital OPV3 Oral Polio Vaccine (3 Doses) DPT3 Diptheria, Pertussis and Tetanus PHD Provincial Health Director vaccine (3 Doses) EPI Expanded Program on Immunization PPA Performance-Based Partnership Agreement EU European Union PRR Priority Reform and Restructuring EPHS Essential Package of Hospital SDR Special Drawing Rights Services FMR Financial Management Report TA Technical Assistance GCMU Grants and Contracts Management TSS Transitional Support Strategy Unit HMIS Health Management Information UNICEF United Nations Children s Fund System IBRD International Bank for UNFPA United Nations Population Fund Reconstruction and Development IDA International Development Association USAID United States Agency for International Development IEC Information, Education and US$ United States Dollar Communication IMCI Integrated Management of VCT Voluntary Counseling and Testing Childhood Illness ISR Implementation Status Report WHO World Health Organization i

3 Vice President: Isabel Guerrero Country Director: Nicholas Krafft Sector Manager: Julie McLaughlin Project Team Leader: Emanuele Capobianco ICR Team Leader/Author: Patrick Mullen ii

4 Afghanistan Health Sector Emergency Rehabilitation and Development Project CONTENTS Data Sheet... i 1. Project Context, Development Objectives and Design Key Factors Affecting Implementation and Outcomes Assessment of Outcomes Assessment of Risk to Development Outcome (Rating: Moderate) Assessment of Bank and Borrower Performance Lessons Learned Comments on Issues Raised by Borrower/Implementing Agencies/Partners Annex 1. Project Costs and Financing Annex 2. Outputs by Component Annex 3. Economic and Financial Analysis Annex 4. Bank Lending and Implementation Support/Supervision Processes Annex 5. Beneficiary Survey Results Annex 6. Stakeholder Workshop Report and Results Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Annex 9. List of Supporting Documents Annex 10. Map iii

5 A. Basic Information Country: Afghanistan Project Name: Afghanistan Health Sector Emergency Reha Project ID: P L/C/TF Number(s): IDA-H0430,IDA- H2060,IDA-H3840,TF ICR Date: 03/31/2010 ICR Type: Core ICR Lending Instrument: ERL Borrower: Original Total Commitment: Revised Amount: XDR 76.5M Environmental Category: B Implementing Agencies: Ministry of Public Health Cofinanciers and Other External Partners: GOVERNMENT OF AFGHANISTAN XDR 43.7M Disbursed Amount: XDR 76.5M B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 05/13/2002 Effectiveness: 07/17/ /17/2003 Appraisal: 01/22/2003 Restructuring(s): 11/28/2006 Approval: 06/05/2003 Mid-term Review: 05/18/2005 Closing: 03/31/ /30/2009 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Borrower Performance: Satisfactory Moderate Satisfactory Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Highly Satisfactory Government: Satisfactory Quality of Supervision: Satisfactory Implementing Agency/Agencies: Highly Satisfactory Overall Bank Performance: Satisfactory Overall Borrower Performance: Satisfactory i

6 C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Performance (if any) Potential Problem Project No at any time (Yes/No): Problem Project at any time (Yes/No): DO rating before Closing/Inactive status: No Satisfactory Quality at Entry (QEA): Quality of Supervision (QSA): Satisfactory None Rating D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration Health Non-compulsory health finance 1 1 Water supply 1 1 Theme Code (as % of total Bank financing) Child health Gender Health system performance Population and reproductive health Tuberculosis E. Bank Staff Positions At ICR At Approval Vice President: Isabel M. Guerrero Praful C. Patel Country Director: Nicholas J. Krafft Alastair J. McKechnie Sector Manager: Julie McLaughlin Anabela Abreu Project Team Leader: Emanuele Capobianco Benjamin P. Loevinsohn ICR Team Leader: ICR Primary Author: Patrick M. Mullen Patrick M. Mullen F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The Project Development Objective (PDO) is to: (i) assist the MOH to achieve its stated goals of reducing the rates of infant and child mortality, maternal mortality, child ii

7 malnutrition, and fertility through expanding delivery of the Basic Package of Health Services (BPHS) and increasing equity in the delivery of services; (ii) strengthen the MOH's stewardship over the sector including a greater role in health care financing, coordination of partners, and overseeing the work of NGOs; and (iii) build the capacity of Afghan health workers to provide and manage health services. Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s) Indicator Indicator 1 : Value quantitative or Qualitative) Original Target Values (from Baseline Value approval documents) Under-5 mortality rate (per 1,000 live births) 257 (national) (2000 Interagency Working Group) Formally Revised Target Values 20% reduction 20% reduction Actual Value Achieved at Completion or Target Years 161 (national) ( NRVA with reference period end-2004) Date achieved 07/01/ /31/ /30/ /01/2004 Comments (incl. % achievement) Not included in ISRs. Reduction of 37% from baseline means that 187% of the target was achieved by end-2004 (reference period of the household survey estimate). Indicator 2 : Maternal mortality ratio (per 100,000 live births) Value quantitative or Qualitative) 1,900 (national) (2000 WHO, UNICEF, UNFPA) 10% reduction 10% reduction Not available Date achieved 07/01/ /31/ /30/ /30/2009 Comments Not included in ISRs. Project documents use 1,600/100,000 as the baseline, (incl. % citing World Development Indicators. achievement) Contraceptive Prevalence Rate - % of women years currently using a Indicator 3 : family planning method [modern] Value +10 percentage 13% (rural) (2007- quantitative or 5.1% (rural) (2003 MICS) points from 15% 08 NRVA) Qualitative) baseline Date achieved 07/01/ /31/ /30/ /01/2008 Comments Baseline updated from NA. Improvement of 155% from baseline. 87% of target (incl. % achieved. achievement) Indicator 4 : Treatment success rate among TB cases detected (cohort analysis) Value quantitative or Qualitative) 80% (national) (2003 HMIS) 80% 85% 89% (national) (2008 HMIS) Date achieved 12/31/ /31/ /30/ /31/2008 Comments Baseline updated from NA. Improvement of 11% from baseline. 105% of target iii

8 (incl. % achievement) Indicator 5 : Value quantitative or Qualitative) achieved. Proportion of children 6 to 59 months that have received Vitamin A supplement within last 6 months 90% (rural) (2003 MICS) 90% 90% 68% (rural) ( NRVA) Date achieved 07/01/ /31/ /30/ /01/2008 Baseline updated from 70%. Reduction of 24% from baseline. 76% of target Comments achieved. (2005 coverage was 44.8% (rural), after which Additional Financing (incl. % included support of mass immunization campaigns. Coverage rose to 68% achievement) (rural) in ) Indicator 6 : DPT3 coverage among children months. Value +35 percentage 39% (rural) (2007- quantitative or 20% (rural) (2003 MICS) points from 55% 08 NRVA) Qualitative) baseline Date achieved 07/01/ /31/ /30/ /01/2008 Comments Baseline updated from NA. Improvement of 95% from baseline. 71% of target (incl. % achieved. achievement) Indicator 7 : Provider knowledge score (% correct answers) on test administered by 3rd party Value quantitative or Qualitative) 54% (national) (2004 Balanced Scorecard health facility survey) Significant improvement 70% 69% (national) (2007 Balanced Scorecard health facility survey) Date achieved 07/01/ /31/ /30/ /01/2007 Comments (incl. % Improvement of 28% from baseline. 99% of target achieved. achievement) Indicator 8 : % of health facilities with women health workers Value 25% (national) ( % (national) quantitative or 80% 80% MSH/REACH) (2008 HMIS) Qualitative) Date achieved 01/01/ /31/ /30/ /31/2008 Comments (incl. % achievement) Indicator 9 : Value quantitative or Qualitative) Baseline updated from 60%. Improvement of 232% from baseline. 104% of target achieved. Number of consultations per person per year 0.23 (national) (2003 HMIS) Not specified (national) (2008 HMIS) Date achieved 12/31/ /31/ /30/ /31/2008 Comments Baseline updated from NA. Improvement of 335% from baseline. 100% of (incl. % target achieved. achievement) iv

9 (b) Intermediate Outcome Indicator(s) Indicator Indicator 1 : Value (quantitative or Qualitative) Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Coverage of antenatal care -% of all pregnant women receiving at least one antenatal care visit +30 percentage 30% (rural) (2007-5% (rural) (2003 MICS) points from 35% 08 NRVA) baseline Date achieved 07/01/ /31/ /30/ /01/2008 Comments Baseline updated from NA. Improvement of 500% from baseline. 86% of target (incl. % achieved. achievement) Indicator 2 : Proportion of pregnant women receiving at least two doses of tetanus toxoid Value (quantitative or Qualitative) 31.5% (rural) (2003 MICS) +30 percentage points from baseline 60% 34% (rural) ( NRVA) Date achieved 07/01/ /31/ /30/ /01/2008 Baseline updated from NA. Improvement of 8% from baseline. 57% of target Comments achieved. (2006 coverage was 28.3% (rural), after which Additional Financing (incl. % included support of mass immunization campaigns. Coverage rose to 34% achievement) (rural) in ) Indicator 3 : Proportion of births attended by skilled attendants (excluding trained TBAs) Value (quantitative or Qualitative) 6% (rural) (2003 MICS) +10 percentage points from baseline 16% 15% (rural) ( NRVA) Date achieved 07/01/ /31/ /30/ /01/2008 Comments Baseline updated from NA. Improvement of 150% from baseline. 94% of target (incl. % achieved. achievement) TB case detection rate (number of sputum positive cases detected as % of target Indicator 4 : based on estimated prevalence, ie. case-finding) Value (quantitative or Qualitative) 23% (national) (2003 HMIS) 50% 50% 79% (2008 HMIS) Date achieved 12/31/ /31/ /30/ /31/2008 Comments Baseline updated from NA. Improvement of 243% from baseline. 158% of (incl. % target achieved. achievement) Indicator 5 : % of children 6-18 months who received breast milk and appropriate complementary food in the last 24 hours Value +30 percentage (quantitative or Qualitative) 68% (rural) (2003 MICS) points from baseline 75% Not available Date achieved 07/01/ /31/ /30/ /30/2009 Comments Baseline updated from NA. Using a different denominator, the 2006 v

10 (incl. % achievement) Afghanistan Health Survey found that 28.4% of children aged 6-9 months in rural areas received complementary foods, while the NRVA found a rate of 39% in rural areas. % of children 0-6 months who were exclusively breastfed in the last 24 hours Indicator 6 : Value +30 percentage (quantitative 82% (rural) (2003 MICS) points from 90% or Qualitative) baseline Date achieved 07/01/ /31/ /30/ /01/2008 Comments Baseline updated from NA. Reduction of 4% from baseline. 88% of target (incl. % achieved. achievement) Indicator 7 : Value (quantitative or Qualitative) 79% (rural) ( NRVA) Proportion of parents able to spontaneously name the danger signs of diarrhea and ARI and the appropriate response percentage percentage 48% (rural) (2003 MICS) points from Not available points from baseline baseline Date achieved 07/01/ /31/ /30/ /30/2009 Comments (incl. % Baseline updated from NA. achievement) Indicator 8 : Measles immunization coverage among children months Value +35 percentage 54% (rural) (2007- (quantitative 76% (rural) (2003 MICS) points from 85% 08 NRVA) or Qualitative) baseline Date achieved 07/01/ /31/ /30/ /01/2008 Comments (incl. % achievement) Indicator 9 : Value (quantitative or Qualitative) Baseline updated from NA. Reduction of 29% from baseline. 64% of target achieved. (2005 coverage was 52.8% (rural), after which Additional Financing included support of mass immunization campaigns. Coverage remained steady at 54% (rural) in ) Score out of 100 on the Balanced Scorecard which summarizes quality of care in BHCs and CHCs 50 (national) (2004 Balanced Scorecard health facility survey) Not specified (national) (2008 Balanced Scorecard health facility survey) Date achieved 07/01/ /31/ /30/ /01/2008 Comments Project documents indicate baseline of 44.8 but survey reports specify 50. (incl. % Improvement of 44% from baseline. 131% of target achieved. achievement) Decrease in variation between clusters in HHS in terms of coverage of basic Indicator 10 : services (ie. reduction in variance) Value (quantitative or Qualitative) Not available 50% decrease from baseline 50% decrease from baseline Not available Date achieved 01/01/ /31/ /30/ /30/2009 Comments Not included in ISRs. (incl. % vi

11 achievement) Indicator 11 : Number of CHWs per 1,500 population submitting monthly reports Value (quantitative or Qualitative) Not available Not available 0.7 (Specified in ISRs) 0.97 (project areas) (2008 HMIS) Date achieved 01/01/ /31/ /30/ /31/2008 Comments (incl. % 139% of target achieved. achievement) G. Ratings of Project Performance in ISRs No. Date ISR Archived DO IP Actual Disbursements (USD millions) 1 11/22/2003 Satisfactory Satisfactory /07/2004 Satisfactory Highly Satisfactory /16/2004 Satisfactory Highly Satisfactory /13/2005 Highly Satisfactory Highly Satisfactory /15/2005 Highly Satisfactory Highly Satisfactory /14/2006 Highly Satisfactory Highly Satisfactory /18/2006 Highly Satisfactory Satisfactory /07/2007 Highly Satisfactory Satisfactory /02/2007 Highly Satisfactory Satisfactory /21/2008 Highly Satisfactory Satisfactory /05/2008 Highly Satisfactory Satisfactory /23/2009 Satisfactory Satisfactory H. Restructuring (if any) Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO Amount Disbursed at Restructuring in USD millions 11/28/2006 HS S IP Reason for Restructuring & Key Changes Made vii

12 I. Disbursement Profile viii

13 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. At the time of appraisal in 2003, with a population estimated at 24.4 million 1 and GDP per capita of US$ 186, 2 Afghanistan was among the poorest countries in the world, emerging from decades of conflict. Following the fall of the Taliban regime in 2001, Afghanistan was in the midst of post-conflict reconstruction and transition with substantial security and development support from the international community. Insecurity was rising in parts of the country. Consistent with its poverty and long experience of conflict, Afghanistan s health situation was similarly among the poorest in the world. Under-five mortality was estimated at 257 per 1,000, 3 while the maternal mortality ratio was estimated at 1,900 per 100,000 live births, 4 both rates among the highest in the world. 2. At the time of appraisal of this project, health services were largely provided by nongovernmental organizations (NGOs) financed under humanitarian programs, while in many rural areas there were no modern health services at all. There were large deficiencies and inequalities in infrastructure and human resources, while NGO programs were uncoordinated, without any common policy framework or technical standards. Along with daunting geographic barriers to access, women s access to health care was restricted by cultural factors and a lack of female health providers. During , with the support of a series of joint donor missions, the Ministry of Public Health (MOPH) under the new transitional government assessed needs, established strategic directions and put in place the policy framework for development of health services. 3. A flurry of policy development during included the National Health Policy, the Basic Package of Health Services (BPHS) and the National Salary Policy. Drawing on experiences with NGO contracting in other countries (notably Cambodia), the World Bank took the lead in advocating a strategy whereby the MOPH would concentrate on its stewardship role while contracting out service delivery to NGOs. The development programs of the three major donors in the sector the World Bank, the United States Agency for International Development (USAID) and the European Union (EU) were developed and implemented on that basis. 5 1 United States Census Bureau (2010), International Data Base, available at 2 International Monetary Fund (IMF) (2010), IMF Data Mapper, available at 3 UNICEF, WHO, World Bank and UN Population Division (2007), Levels and Trends of Child Mortality in 2006: Estimates developed by the Inter-agency Group for Child Mortality Estimation, New York. 4 WHO, UNICEF and UNFPA (2004), Maternal mortality in 2000: Estimates Developed by WHO, UNICEF and UNFPA, Geneva. 5 A detailed description of this policy process in can be found in Strong, L., Wali, A. and Sondorp, E. (2005) Health Policy in Afghanistan: Two Years of Rapid Change, A Review of the Process from 2001 to 2003, London School of Hygiene and Tropical Medicine, London. 1

14 4. The rationale for Bank assistance was provided by the Bank s strong engagement in reconstruction in Afghanistan and its support for the country s 2002 National Development Framework, which includes health and nutrition programs supporting one of the framework s three pillars. The Bank s 2003 Transitional Support Strategy (TSS) stated that the Bank would assist the government in its broad priorities across sectors, including development of basic social services. One of the five overall objectives of the Bank s 2003 TSS was to improve basic social services to the population. However, the TSS lists the health sector as an area of consolidation suggesting Bank disengagement once the health project then under preparation had successfully established the performance-based contracting system. Nevertheless, the document stated that this would be reviewed in light of implementation experience in the context of preparation of a full Country Assistance Strategy (CAS). 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 5. The Project Development Objective (PDO) is to: (i) assist the MOH to achieve its stated goals of reducing the rates of infant and child mortality, maternal mortality, child malnutrition, and fertility through expanding delivery of the Basic Package of Health Services (BPHS) and increasing equity in the delivery of services; (ii) strengthen the MOH's stewardship over the sector including a greater role in health care financing, coordination of partners, and overseeing the work of NGOs; and (iii) build the capacity of Afghan health workers to provide and manage health services. 6 Key indicators are provided above in the Datasheet. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 6. The PDO was not changed. Key indicators were slightly modified and baseline estimates were provided in the Additional Financing documents. Following are changes made to the PDO indicators: Health outcome indicators were either not included in ISRs (under-five and maternal mortality) or dropped from the Results Matrix in the Additional Financing documents (infant mortality and child malnutrition). This is due to difficulties in measurement and the fact that there is a well-established scientific basis for considering that utilization of relevant basic health services (which make up the bulk of project indicators) will result in improved health outcomes. Full immunization coverage was replaced by DPT3 coverage as a PDO indicator (DPT3 coverage was previously an Intermediate Outcome indicator) due to the fact that they essentially reflect the same thing: access to routine immunization services. An indicator of knowledge possessed by MOPH managers was replaced with an indicator of health provider knowledge as reflection of the capacity development aspect of the PDO. 6 Note that the Ministry s name changed to Ministry of Public Health (MOPH) in This document uses MOPH consistently for the sake of clarity. 2

15 The per capita annual consultation rate was upgraded to a PDO indicator from an Intermediate Outcome indicator because it is a key measurement of overall service delivery, reflecting the core objective of the project. 1.4 Main Beneficiaries 7. The primary target group was meant to be a rural population of approximately 6 million (for details see Table 3 in Annex 2). In addition, the project provided support to national immunization campaigns, benefiting the country s children, as well as monitoring and evaluation of health services nationwide. The MOPH benefited from institutional development supported by the project. 1.5 Original Components (as approved) 8. Components of the original project (total estimated grant financing of US$ 59.6 million in 2003) were as follows. Component 1. Expand the delivery of the basic package of health services (US$ 46.5 million). This was to finance contracts (called Performance-based partnership agreements or PPAs) with NGOs to deliver the BPHS in seven provinces. A Strengthening Mechanism was to support delivery of the BPHS to a further three provinces by the MOPH. Component 2. Equipping the rural health infrastructure (US$ 3.7 million). Anticipating the planned construction of 500 primary health care facilities by USAID, this component was to support equipment and furniture for Comprehensive and Basic Health Centers (CHCs and BHCs). Also to be financed were basic surgical equipment for district and provincial hospitals. Component 3. Pilot tests of important innovations (US$ 1.0 million). This was to finance a pilot program to test the impact on diarrhea incidence of a household-level safe water system compared to improved water supply. Also to be supported was an evaluation of different health financing models, notably user fees and a community contribution scheme. Component 4. Capacity building and training (US$ 6.7 million). This component was to provide information and communications technology and transport to provincial health administrations in the provinces supported by the project. Technical assistance and training was to be provided to MOH at the central and provincial levels to strengthen management capacity. The component was to support baseline and follow-up household surveys and regular health facility assessments to be done by a contracted third party, as well as improved technical supervision of health facilities by provincial and central MOH staff. 1.6 Revised Components 9. Project components were modified as follows by the first (US$ 30 million in 2006) and second (US$ 20 million in 2008) Additional Financing. 3

16 Component 1. Expand the delivery of the basic package of health services (total US$ 82.2 million). The first Additional Financing expanded support to delivery of the BPHS to fully cover an additional province as well as to eight new clusters of under-served populations. The second Additional Financing was to also experiment with pilot strategies to support services in insecure areas, particularly Helmand province. The first and second Additional Financing maintained support to the NGO contracts and MOPH Strengthening Mechanism until project closing in mid A total of 12 service delivery contracts with NGOs were in place by the end of the project. Component 2. Equipping the rural health infrastructure. (total US$ 1.1 million) Due to financing of medical equipment by another donor and the fact that necessary equipment in project areas was purchased by the NGOs and the Strengthening Mechanism under Component 1, the majority of the funding was reallocated to the NGO contracts under Component 1 and used for the establishment of sub-center health facilities and mobile outreach activities. Component 3. Pilot tests of important innovations (total US$ 1.0 million). There was no change to this component. Component 4. Capacity building and training (total US$ 10.2 million). The first Additional Financing included support to ongoing activities, notably monitoring and evaluation and technical assistance to MOPH management, and added support to a series of annual hospital performance assessments, an assessment of capacity needs of MOPH managers, and training of community midwives and community health workers (CHWs). Component 5. Polio Eradication and Measles/Neo-Natal Tetanus Control (total US$ 8.0 million). The first Additional Financing added support for national immunization campaigns against polio, measles, and neo-natal tetanus, including Vitamin A distribution. 1.7 Other significant changes 10. The project s implementation period was extended to a total of six years, from mid-2003 to mid Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 11. Preparation. A number of features of the preparation process positively affected project outcomes. Embedded in sector strategy development. Preparation of the project was intertwined with support and technical dialogue on the post-conflict reconstruction of Afghanistan during , in particular through a series of joint donor missions co-led by the World Bank. The most important strategic choice was whether to pursue development of a largely government-run health system or to contract out service delivery to NGOs. The Bank emerged as the strongest champion of the latter option and this shaped the IDA-financed project as well as the development projects supported by other major donors, notably USAID and the EU. 4

17 Drew on experience in other post-conflict situations. Experience with public-private partnerships for the delivery of health services, notably in Cambodia, was widely shared and discussed with partners and MOPH. Strategy owned by government. The debate on implementation strategy was contentious and the Bank was accused by some partners of pushing the NGO contracting strategy too strongly. However, MOPH came down clearly for the option of contracting-out service delivery. The MOH was effectively in the driver s seat from this moment onwards, pushing consistently for the implementation of the BPHS and subcontracting service delivery to NGOs. 7 Based on sector policy framework. The technical content of the project was determined by parallel development of sector policies, in particular definition and costing of the BPHS and specification of a National Salary Policy. Established implementation capacity. The importance of early support to development of MOPH implementation capacity, particularly establishment of the GCMU during project preparation, cannot be overstated in order to achieve rapid start-up in a low-capacity post-conflict setting. 12. Design. The core design of the project was appropriate to the country and sector context and essential to project outcomes. This was to contract NGOs to deliver a basic package of health services in rural areas. At the same time, MOPH and the Bank team agreed to implement a hybrid Strengthening Mechanism strategy, whereby services would be delivered directly by the MOPH in three provinces using reformed public sector management procedures, supported by technical assistance. This was appropriate in that it provided a necessary compromise between the vision of a government-run health service and the contracting strategy as well as an opportunity to compare results between government-run and contracted-out services. 13. Risks. The project addressed a variety of risks which can be grouped into the following. Overall security and political risks. Mitigation measures were (i) the NGO contracting strategy, (ii) community engagement in support to and supervision of health facilities, and (iii) additional support for interventions in highly insecure areas. Socio-economic and cultural determinants of project outcomes. It was recognized that project outcomes are dependent on a range of socio-economic and cultural factors, notably poverty, education and gender. The main mitigation measures were to address gender barriers to access to care by focusing on increasing the number of women health providers and extending services geographically, as well as to monitor access to services by the poor through surveys of patients and households. Opposition to the contracting strategy. Opposition to the contracting strategy is cited by project documents as an important risk and the response is to support the creation of an evidence base to inform the debate. More broadly, project support of the Strengthening Mechanism provided an alternative option both for purposes of 7 Strong, Wali and Sondorp (2005). 5

18 comparison as well as mitigation of the risk that no alternative strategy would be available to replace the NGO contracts should they be stopped. Capacity constraints. A major mitigation measure was to strengthen the capacity of MOPH through support for the GCMU. Strong technical oversight of the NGOs by the GCMU in turn mitigated the risks of the capacity constraints on the NGO side. 14. A 2003 Quality Assurance Group (QAG) review rated Quality at Entry as Satisfactory. 2.2 Implementation 15. The following issues that arose during implementation negatively affected project performance. Delay in MOPH Strengthening Mechanism. Implementation of the Strengthening Mechanism was delayed as MOPH instead proceeded with a bidding process to contract NGOs like in the other provinces. Because the bid prices received were judged to be too high, MOPH returned to the Strengthening Mechanism strategy. The late start initially led to poorer performance in the Strengthening Mechanism provinces compared to the provinces supported by NGO contracts, but this difference in results on average disappeared as implementation proceeded. Rising insecurity. Insecurity increased substantially during the course of project implementation, affecting operations, supervision, and monitoring and evaluation. NGOs adapted in various ways, including modifying movements of staff and supplies, stopping the use of easily-identifiable vehicles, obscuring their association with government, and strengthening relationships with local communities. The second Additional Financing includes support for innovative interventions in insecure areas such as conditional transfers to households (designed to bring people to health services to reduce the need for outreach to isolated areas). 16. The following adaptations during implementation positively contributed to project outcomes. Increased emphasis on reaching communities with basic services. The results of monitoring and evaluation, particularly a 2006 household survey financed by the project, indicated that utilization of key services remained low, likely due to barriers to access (discussed in Section 3.2 below) to the facility-based services that had been much improved by the project. Project management and the Bank team increasingly emphasized the importance of getting services out to remote communities as well as improving demand for services. This led to the establishment of a new category of health facility (sub-centers) closer to communities, as well as the use of mobile teams to provide curative and preventive services. Additional financing also added support to mass immunization campaigns to improve coverage. NGO innovations. Although the technical standards that the NGOs were required to meet were quite detailed, there was room for innovation to achieve progress on the key indicators. The main examples of such innovations introduced by the NGOs (and later encouraged by MOPH project-wide) were the use of sub-centers and mobile 6

19 teams to bring services closer to remote communities. Another example is the hiring by several NGOs of female health workers from Tajikistan in order to make progress towards targets for female staff of health facilities. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 17. Design. Effective monitoring and evaluation was essential to the overall strategy of the project and substantial resources (approximately US$ 10 million, or about 10% of total funding) were devoted to it. Data collection was not confined to the project areas but provided provincelevel estimates nationwide to permit MOPH and donors to monitor progress in the sector as a whole. A prominent US university, partnered with a research institute from the region, was contracted to design and implement the monitoring and evaluation system. High-quality annual facility surveys produced indices to measure various aspects of service provision and quality. An overall index (called the Balanced Scorecard ) was constructed in order to provide one measure to summarize each province s performance. This design turned out to be very useful by providing easy to compare (over time and between provinces) measures of performance in various areas based on more detailed data in specific areas. A clearer strategy on household surveys should have been developed from the outset. For baseline data, the project intended to rely on a Multiple Indicator Cluster Survey (MICS) done in 2003, but this turned out to have problems with the sampling frame (requiring later re-analysis). For follow-up data, the project initially intended to rely on a multi-sectoral survey, the National Risk and Vulnerability Assessment (NRVA), but questions about the quality of the health module led it to finance a dedicated survey of rural areas in Despite this experience, MOPH decided to rely on a subsequent NRVA in to provide trend information on household level indicators. Final estimates from this survey became available in 2009 to allow assessment of key indicators that required household level survey data. Despite the regular availability and quality of the facility surveys and the problems and delays associated with household surveys, key indicators for the project were largely service utilization indicators that required household surveys for reliable measurement. Targets were in many cases considered to be ambitious (for example, by a QAG Quality at Entry review in 2003) and in some cases were not met despite nevertheless impressive progress. PDO indicators and Intermediate Outcome indicators reflected the main aspects of the PDO: (i) improvement in health outcomes; (ii) expansion of basic service delivery; and (iii) development of capacity. Most project indicators focused on the core objective of the project, to expand basic services. With regard to health outcomes, there were some relevant PDO indicators included, but well-known difficulties in measurement and attribution do not allow an assessment of trends, except in the case of child mortality. Rather than including health outcome indicators, it would have been better to rely on health service utilization indicators to reflect the health outcome-related PDO elements given the well-established scientific basis for assuming that higher utilization of these basic interventions will result in better health outcomes. With regard to the capacity-building aspect of the PDO, several relevant 7

20 indicators, such as health care provider knowledge, were measured by the annual facility surveys. At the same time, for regular monitoring, MOPH made effective use of the health management information system (HMIS) and a standardized supervision checklist. 18. Implementation. Overall, the various data sources for monitoring and evaluation were effectively implemented. Five annual health facility surveys were done, which provided regular and comparable data to monitor trends and compare provinces. The contracted university applied its technical expertise to analyze and reanalyze the MICS and NRVA data, mitigating their shortcomings, and providing estimates for key project indicators. The HMIS and supervision systems improved over time with substantial support from USAID. The contracted university has functioned as a genuine partner, responding to numerous impromptu requests (not included in their terms of reference) such as analysis of community health worker effectiveness, a study of hospital quality of care, a drug quality assessment, and development of a proposal for a demographic surveillance system. 19. Utilization. The project support to monitoring and evaluation, particularly the nationwide annual facility surveys, has contributed to development of a data and results management culture at the different levels of the health system. This is discussed further in Section 3.1 below. 2.4 Safeguard and Fiduciary Compliance 20. Safeguards. The project was prepared under OP 8.50 Emergency Recovery Assistance and was classified as environmental category B, and safeguards classification S2. An environmental and medical waste management plan was developed and disclosed. Annual facility surveys provided regular data on medical waste management. In 2004, in the provinces supported by the project, an average of 46% of health facilities were properly disposing of sharps (ie. needles, lances, scalpels). By 2007, this had increased to 90%, although the proportion fell back in 2008 to 79%, due to a dramatic drop in this indicator in Balkh province. 8 Project performance in this area exceeded the national average. Nationwide, in % of facilities were properly disposing of sharps, rising to 73% in Procurement and the NGO contracts. How procurement of the NGOs was handled was a key factor in the performance of this project. These procurements were complex and could therefore have entailed a lengthy procurement process, but in this case the first three NGO contracts were signed quickly, by October 2003, only three months after grant effectiveness. The remaining NGO contracts, as well as the contract for monitoring and evaluation (another complex procurement), were signed early The structure of the contracts, focused on results rather than inputs (discussed further below in Section 6), was a crucial factor in the performance of the NGOs and the project as a whole. 8 Data were collected in Helmand province only in 2004 due to insecurity in subsequent years. The 2008 estimate did not include Farah province due to insecurity. 8

21 22. Procurement and supply of inputs. An important feature of the design of the NGO contracts was that resources and inputs were managed under the NGOs own procedures. There was neither centralized procurement of inputs nor any requirement that the NGOs follow Bank procurement procedures. Between 2004 and 2008, a drug availability index (calculated from the proportion of health facilities with stocks of five essential drugs at the time of the survey) rose from 60 to 91 in provinces supported by the project. This was consistently better than the national average, which rose from 68 to 84 (Figure 1). It may be noted that a significant proportion of other provinces relied on centrally-procured inputs. A study that tested samples of drugs from health facilities in 2007 concluded that there was no significant problem with drug quality. Figure 1. Drug availability index Project province average National average Among project provinces, data are not available for Helmand nor for Farah Source is Afghanistan Health Sector Balanced Scorecard Reports, In contrast to the smooth drug supply assured by the NGOs, the MOPH Strengthening Mechanism, which was required to follow government procedures and Bank guidelines, experienced significant delays in one procurement of pharmaceuticals (although other procurements were completed satisfactorily). The Bank declared misprocurement because of repeated extensions of the bid validity period, lack of clarity on the quantities of drugs to be procured, and poor coordination between the different government entities involved in the procurement process. It should be noted that no goods were procured and no amount of the grant was cancelled. A 10-point action plan, including recruitment of a procurement consultant by MOPH, was agreed to strengthen procurement management by the project. By project closing this action plan had not yet been completely implemented. Overall, split responsibilities for procurement within MOPH and between MOPH and a central agency for procurement facilitation led to delays. 24. Financial management and disbursement. Overall, considering that there were no financial management systems in place when the project became effective, financial management capacity has been built steadily over the period of implementation. The Ministry of Finance was responsible for financial management and disbursement while MOPH also maintained 9

22 disbursement records. A manual system was used until 2005 by GCMU when a computerized information system was adopted government-wide. Supporting documents were maintained satisfactorily for all payments. Based on the final audit report, it appears that the difficulties the MOPH experienced early during project implementation had a negative impact on the management of the local project accounts. Financial reporting was poor initially but improved considerably over time. Annual audited financial statements were submitted regularly, although later than the mandated submission deadline. In 2009, a number of small transactions for rehabilitation and remuneration of MOPH staff (totaling less than USD 70,000) were declared by the Bank team to be ineligible expenditures because relevant categories were not included in the grant agreement. (Even though such expenditures were done by the NGOs and Strengthening Mechanism with no problems under Category 1). More importantly, shortcomings in MOPH s project budgeting and monitoring led to an over-commitment to the NGO contracts of over US$ 12 million that was eventually met by retroactive financing provided under the successor project. 2.5 Post-completion Operation/Next Phase 25. A follow-up operation, the Strengthening Health Activities for the Rural Poor (SHARP) project, was approved in 2009 with IDA financing of US$ 30 million and significant cofinancing from several trust funds. The new project replicates very closely the objectives and implementation strategy of the preceding project and indeed provides significant retroactive financing (over US$ 12 million) for previous extensions to the NGO and monitoring and evaluation contracts. During preparation, there was some discussion of expanding the MOPH Strengthening Mechanism but MOPH and the Bank team chose not to do so because of potential capacity constraints and the challenges posed by rising insecurity. 26. The large retroactive financing of the NGO contracts indicates that there was a significant gap (of about 12 months) between the end of available financing under the first project and effectiveness of the follow-up project. This was due to delays in advance procurement of the new NGO contracts. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation (Rating: Satisfactory) 27. Alignment with development strategy (Rating: Highly Satisfactory). With its focus on improving basic services for maternal and child health and nutrition, as well as control of communicable diseases, the project directly supports Afghanistan s efforts to attain the Millennium Development Goals (MDGs). The 2008 Afghanistan National Development Strategy ( ), prepared over a period of two years with extensive consultation, has three pillars, including Economic and Social Development, with the following objective: Reduce poverty, ensure sustainable development through a private-sector-led market economy, improve human development indicators, and make significant progress towards the Millennium Development Goals (MDGs). The government s Health and Nutrition Sector Strategy is centered on increasing access to basic services and specifies that MOPH will continue to strengthen its stewardship, monitoring and evaluation and regulatory functions while continuing the effective partnership with non-governmental organizations. 10

23 28. The Bank s 2009 Interim Strategy Note (ISN) ( ) supports the national development strategy with a focus on three pillars. The first pillar is, Building the capacity of the state and its accountability to its citizens, including continued support to national programs in education and health, with the objective of continuing to build the credibility of the state by maintaining delivery of essential services to the population. The successor health project (SHARP) is included in the ISN 2009 program. This close alignment with the Bank s current country strategy can be contrasted with its 2003 strategy which envisioned withdrawal from the health sector contingent on a review of implementation experience. The success of the health project has evidently contributed to maintaining the health sector as an area of focus of the Bank s support. 29. Objectives (Rating: Satisfactory). That the successor operation is almost identical to this project is testament to the fact that it remains highly relevant to current priorities. Afghanistan s epidemiological profile is one of a poor population with low life expectancy (estimated at 43 years), high fertility (total fertility rate of 6.27), and a morbidity and mortality burden dominated by communicable diseases in conjunction with poor nutrition and poor maternal health outcomes. Although the PDO could have perhaps been formulated to be more focused on expansion of service delivery (ie. by not including health outcomes that are difficult to measure), the core objective of expanding primary health care services to address the main burden of disease is highly relevant. Other objectives relating to improving equity and developing capacity similarly continue to be highly relevant. 30. Design (Rating: Satisfactory). The core strategy of the project to achieve its objectives is to expand access to basic health services that have been shown to be effective in addressing the main causes of morbidity and mortality in low-income countries. The BPHS (Component 1) includes immunization, antenatal, delivery and post-natal care, nutrition services and prevention and treatment of communicable diseases, including acute respiratory infection, diarrhea, malaria and TB. The project also supported immunization campaigns to rapidly increase coverage of high-impact interventions (Component 5). These were interventions proven to contribute to health outcomes. 31. At the same time, there were questions raised since preparation about whether mental health and disability services should be included in the BPHS and thus supported by the project. On the one hand, there was evidence that these were significant health issues among the population. On the other hand, other health problems had a more immediate impact on maternal and child mortality, were more amenable to cost-effective interventions, and required immediate attention in a highly-challenging context. A similar debate arose over support to hospital services. In 2005, services addressing mental health and disability were included in the BPHS and the successor project (SHARP) included support to an assessment of hospital requirements. Given the still low coverage rates of key services such as qualified delivery care, the argument for focusing support on proven high-impact interventions remains strong. Nevertheless, the magnitude of the indirect impacts of psychological distress and physical disability on physical health and poverty status may well be significant but they are unknown. 11

24 32. Support to strengthening core government stewardship and management capacity (Component 4), with heavy investment in monitoring and evaluation, similarly remains highly relevant in a situation where the core functions of the state continue to require development. Operations research on technical and policy issues (Component 3) remains relevant to a situation like Afghanistan, where government needs to take decisions on issues (like diarrhea prevention and health financing strategies) that are highly dependent on contextual factors and for which evidence is needed. 33. Implementation (Rating: Highly Satisfactory). The implementation strategy remains highly relevant to a situation where public-private partnership for service delivery allows limited government capacity to be focused on stewardship functions. At the same time, the MOPH Strengthening Mechanism allowed government to build capacity to directly manage service delivery under reformed procedures, providing an alternative model. Rising insecurity is the main contextual change over the project implementation period, and the NGO contracting strategy remains adapted to the situation. 3.2 Achievement of Project Development Objectives (Rating: Satisfactory) 34. Project development objectives. The PDO described in Section F of the Data Sheet contains the following main elements. i. Improve health outcomes (infant and child mortality, maternal mortality, child malnutrition, and fertility); ii. Expand delivery of and access to a basic package of health services; iii. Support national immunization campaigns; iv. Focus on the under-served rural population and improve equity; v. Develop the capacity of government for stewardship of the sector; and vi. Develop the capacity of Afghan health workers to manage and deliver services. 35. This section describes trends in selected PDO indicators that reflect the above elements of the PDO. Additional PDO indicators and Intermediate Outcome indicators are described in Annex 2. i. Improve health outcomes (Rating: Satisfactory) 36. Health outcomes. Under-five and maternal mortality were specified as PDO indicators although not included in the ISRs. Such health outcome indicators are generally no longer used as project key indicators due to widely-known difficulties in measurement and attribution. It is better to rely on indicators of utilization of health services that are known to have an impact on these outcomes. Measurement difficulties are evident here. A model-based estimate at baseline (in 2000) for under-five mortality is widely-accepted 257 per 1,000 live births. The 2006 Afghanistan Health Survey estimated under-five mortality in rural areas at 191 per 1,000 live births, with a reference period of The NRVA estimated under-five mortality nationwide at 161 per 1,000, with a reference period end Insofar as the baseline estimate of 257 is valid, this would represent a substantial improvement (40% of baseline) over a period 12

25 that overlaps with the first part of project implementation. 9 Similarly, the model-based estimate for infant mortality at baseline (in 2000) was 165 per 1,000 live births while the NRVA estimated a rate of With regard to the maternal mortality ratio, a model-based estimate for 2000 is 1,900 per 100,000 births. However, data for follow-up estimates are not available. The PDO also mentions child malnutrition and fertility. There are no nationally-representative household survey data on child malnutrition. The NRVA estimated a total fertility rate (TFR) of 6.3 nationally and 6.5 in rural areas, but no reliable baseline estimate is available. 10 However, analysis of data from that survey indicates a modest decline in fertility in Afghanistan in recent years. Several project indicators measure service utilization and behaviors that are known to have an impact these health outcomes. These indicators (ie. antenatal and delivery care, vitamin A supplementation, breastfeeding behavior and contraceptive prevalence) are discussed below and in Annex 2. The available data on health outcomes and for the most part large improvements in coverage of services that are known to improve health outcomes justify a rating of Satisfactory for this element of the PDO. ii. Expand access to basic health services (Rating: Highly Satisfactory) 38. Household surveys provide the gold-standard in developing countries for estimates of service utilization indicators, while HMIS collects data from health facilities on service provision that, combined with estimated catchment populations, provide regular reporting of service coverage. The annual facility surveys supported by the project are considered to be of high quality and provide data on service provision and quality. Combined, these sources of information provide a comprehensive picture of how access to basic services changed during the project implementation period, both nationwide and in the provinces supported by the project. Details on survey methodologies are provided in Annex 2. It should be emphasized that when assessing trends and comparing project areas to the country as a whole, the pattern of insecurity that changed over time represented an important confounder. 39. Child health care. The proportion of children aged months who were fully immunized was included as a PDO indicator but replaced in the 2006 Additional Financing document with the proportion who had received DPT3 vaccination. Like the proportion of fully immunized children, DPT3 coverage is considered an indicator of the performance of routine immunization services, but can be more frequently estimated. Household survey data to measure trends between 2003 and 2008 are available for these indicators. Full immunization coverage in rural areas increased from 15.4% in 2003 to 33.2% in 2008 in the provinces supported by the project. Nationally, in rural areas full immunization coverage increased from 11.2% to 33.3%, representing an increase of 22 percentage points. With regard to DPT3 coverage, the baseline 9 Detailed discussions of methodological issues behind estimation of child mortality from the available survey data can be found in 2006 Afghanistan Health Survey report, the NRVA report, and UNICEF (2006) Best Estimates of Social Indicators for Children in Afghanistan, , Kabul. 10 The 2003 MICS provided a TFR estimate but there are questions about its sampling frame and the re-analysis of the data done by the project (applying an updated sampling frame) did not include re-estimating the TFR. 13

26 was specified at 20%, which is the estimate for rural areas from the 2003 MICS data. 11 The 2006 Afghanistan Health Survey found coverage of 34.6% in rural areas and the NRVA found 39%, indicating significant improvement in comparison to Reproductive and maternal health care. The PDO indicators include the contraceptive prevalence rate, specifically the proportion of women years currently using a modern family planning method. The project s baseline was 5.1%, which is the estimate for rural areas from the 2003 MICS. Province-level household survey data are available to provide good information on trends (Figure 2). In provinces supported by the project, the 2003 MICS found that 7.8% of women of reproductive age were currently using a modern contraceptive method. This increased to 9.9% in 2005 (NRVA) and 10.7% in 2008 (NRVA). Nationally, in rural areas, the improvement was more dramatic, rising from 5.1% in 2003 to 15.5% in 2006 and then leveling off at 13.0% in Figure 2. Modern contraception prevalence in rural areas (% women aged currently using a modern method) and Delivery care by a qualified provider in rural areas (% deliveries) 18.0 Modern contraceptive prevalence 20.0 Qualified delivery care Project provinces National average (rural) Project provinces National average (rural) MICS 2005 NRVA 2006 Afgh Health Survey NRVA MICS 2005 NRVA 2006 Afgh Health Survey NRVA 41. Although not included as a PDO indicator but as an Intermediate Outcome indicator, it is important to examine trends in utilization of qualified delivery care as this is a core service supported by the project and good household survey data are available to confidently assess trends. The baseline for qualified delivery care coverage was 6% in rural areas (2003 MICS) and the project fixed a target of 16%. Figure 2 illustrates significant improvement in coverage, as in rural areas, by 2006, 18.9% of deliveries were by qualified personnel (Afghanistan Health Survey), a proportion which leveled off to 15.0% in 2008 (NRVA). In provinces supported by 11 Overall estimates for rural areas from the 2003 MICS given in the text are from a reanalysis of the data using an updated sampling frame and are the median value of the provincial-level estimates for rural areas. 14

27 the project, qualified delivery care rose from 7.9% in 2003 (MICS) to 9.8% in 2005 (NRVA) and 15.6% in 2008 (NRVA). Even though utilization remains low by international standards, coverage of qualified delivery care tripled since General curative care and disease control. The annual per capita number of outpatient consultations is measured by the HMIS, based on reporting by public sector (ie. government and NGO) facilities. The baseline fixed by the project was 0.23, with a target of 1.0. Nationally (ie. including both urban and rural areas), this statistic rose to 0.6 in 2006, 0.7 in 2007 and 1.0 in In provinces supported by the project, the average reported outpatient consultation rate in 2008 was 1.2, ranging from 0.8 in Helmand to 1.7 in Kapisa. 43. The project s support to improved tuberculosis control is important in that the disease constitutes a very large burden in Afghanistan and control requires quality facility-based services. Afghanistan is considered a high-burden country, with in 2007 an estimated 168 new cases per 100,000 population and 8,200 deaths (WHO, 2009). The treatment success rate rose from 80% reported in 2003 to 89% reported in 2008, which prevented a substantial number of deaths. The annual facility surveys also showed substantial improvements in indicators of TB services, as described in Annex Overall health service provision. Part of the project s success in improving health coverage involved support for substantial physical expansion of health services. Nationally, in 2002, WHO estimated that 496 health facilities were delivering basic health services, while the number in 2008 delivering the BPHS was 1,170. In the project provinces, these figures were 168 in 2003 and 385 in Survey data show that service utilization drops rapidly as travel time to the nearest health facility increases. Figure 3. Balanced Scorecard indicator of health service provision (scale 0-100) Project provinces National Among project provinces, data are not available for Helmand nor for Farah Source is Afghanistan Health Sector Balanced Scorecard Reports,

28 45. The synthesis index (on a scale of 0-100) of the results of the annual facility surveys, called the Balanced Scorecard, is specified as an Intermediate Outcome indicator, but should be emphasized as an indicator of overall project performance. It is the core measure used by the government and the NGOs to measure performance of the health system at the province level, and it provides a measure from a reliable source of the effectiveness of project support to provision of the BPHS. Nationally, from a baseline of 49.6 in 2004, this score increased to 71.7 by Provinces supported by the project showed similar improvement (Figure 3). 46. Leveling-off in progress and the impact of insecurity. Along with general improvements in the indicators, other patterns are evident. For many indicators, after rapid improvement between and 2006, there seems to have been a leveling-off in While the facility surveys often showed a continuing straight-line increase in indicators, the household surveys suggested stabilization or some decline in utilization indicators between 2006 and This is partly due to differences in the sampling frame between the 2006 and household surveys. 12 When this is accounted for, the declines observed since 2006 largely disappear with the exception of contraceptive prevalence. However, the rapid rate of increase observed does not reappear, indicating a leveling off in these indicators since Although this may reflect the law of diminishing returns, likely more important is the increasing insecurity in many parts of Afghanistan. For example, the correlation coefficient between a three-point provinciallevel scale of insecurity (with a higher score meaning higher insecurity) and percentage improvement between 2004 and 2008 in full immunization coverage is quite significant at (ie. increased insecurity is correlated with lower improvements in coverage). A similar pattern is seen between the insecurity score and improvement in the Balanced Scorecard index, with a correlation coefficient of (source is author s estimates). 47. Nevertheless, the project s support for innovative approaches in contexts of high insecurity showed results, as for example, utilization of basic services rose substantially (see graph in Annex 2) after the contracted NGO in Helmand province introduced: conditional cash transfers, performance-based incentives for CHWs and health facility staff, establishment of additional sub-centers and monitoring and supervision by local authorities and community members. 48. Despite improvements, still low coverage levels. At the same time, it should also be noted that in many cases health service utilization started from a very low baseline and is still very low by regional and international standards an example is coverage of qualified delivery care which despite a 150% increase since 2004, was still only 15% in This reflects persistent and difficult-to-address geographical and cultural barriers that are difficult to address by improved facility-based services. 49. Overall, there was substantial expansion of basic health services due to the project, even though some of the over-ambitious targets that it set for itself were not met. 12 The NRVA covered several provinces that were not sampled by the 2006 Afghanistan Health Survey due to insecurity, specifically Kandahar, Helmand, Zabul, Nuristan and Uruzgan. 16

29 iii. Support national immunization campaigns (Rating: Moderately Satisfactory) 50. Immunization to prevent polio, measles and neonatal tetanus can be delivered by mass campaigns, along with vitamin A supplementation. Figure 4 describes trends in coverage of these interventions, showing high coverage in rural areas of measles immunization and vitamin A supplementation due to recent national campaigns. By 2005, coverage levels had dropped, and the 2006 Additional Financing included substantial support to national immunization campaigns. Coverage levels then improved in An apparent decline between 2006 and for some of these indicators is largely due to different sampling frames (due to insecurity) for the two household surveys. Overall, in coverage of polio and tetanus toxoid vaccination was higher than the 2003 baseline, while coverage of measles vaccination and vitamin A was lower. However, compared to 2005, coverage of measles vaccination and vitamin A supplementation improved during the period that the project allocated substantial resources to mass campaigns. Figure 4. Coverage of interventions delivered by national campaigns (rural areas) polio (OPV3) measles tetanus toxoid (2 doses) vitamin A 2003 MICS 2005 NRVA 2006 Afgh Health Survey NRVA iv. Target under-served rural populations and improve equity (Rating: Satisfactory) 51. An important element of the project objectives was to address two major barriers to access to health care in the country: geographic and gender. The project targeted support exclusively to rural areas. Table 1 provides household survey estimates of coverage of three key health services in urban and rural areas in 2003 and It shows that although the absolute 17

30 percentage point gap between urban and rural areas remained similar over time, improvements in rural areas occurred at a much faster rate relative to baseline. 52. The project also focused on improving cultural accessibility of health services to women by encouraging health facilities to have female staff and by training community midwives. The proportion of health facilities with female staff was a PDO indicator, with a baseline of 24.8% (from a assessment supported by USAID) and a target of 80%. In 2008, according to the HMIS, 83% of facilities delivering the BPHS were staffed by at least one female health worker. The annual facility surveys collected data on the gender distribution of new outpatients, finding that overall, in 2004, 54.7% of new outpatients were female and this increased to 60.8% in Project provinces showed similar improvement, increasing from 51.8% in 2004 to 59.5% in (Figure 5). Table 1. Trends in urban and rural areas in three health service utilization indicators, full immunization modern contraceptive prevalence qualified delivery care urban urban % urban increase % 19% 80% rural rural % rural increase % 155% 150% Sources are 2003 MICS and NRVA. Figure 5. Females as % of new outpatients (BPHS facilities) Project provinces 47.0 National Among project provinces, data are not available for Helmand nor for Farah Source is Afghanistan Health Sector Balanced Scorecard Reports,

31 53. Demand-side gender barriers remain significant. Household survey data provide some insight into gender-related barriers to access. Perhaps unexpectedly, given the importance of gender-related barriers, the 2006 Afghanistan Health Survey found that the reasons for not seeking care mostly cost and distance did not differ between men and women. Similarly, the NRVA found that 49% of women and girls reported that cost was a reason for not seeking care, and 47% reported distance. Only 6% cited a lack of female health personnel, 5% cited a lack of someone to accompany them, and 5% mentioned traditional constraints such as lack of permission from the husband or family. However, it is evident that gender barriers are intertwined with cost and distance barriers in that women and girls are often required to be accompanied by a male family member in order to access health care, thereby doubling travel, food and lodging costs. Decision-making on these issues usually resides with the male head of household. The NRVA found that in 80% of households the male head of household was the decision-maker with regard to spending for health care for his spouse. This proportion was 77% with regard to health spending for a child. The large progress in improving supply-side factors (ie. female health workers) will need to be followed by strategies to address such complex demand-side factors. v. Develop capacity for stewardship of the sector (Rating: Highly Satisfactory) 54. Development of core capacity for stewardship. Overall, the project supported the effective implementation of national strategies and policies under the stewardship of the MOPH. At project appraisal, most health services were provided by NGOs in an uncoordinated way, each following their own technical strategies and standards. By the end of the project, the government was clearly in charge of setting these strategies and standards and (in the case of the Bankfinanced project) directly managed service-delivery contracts with NGOs. 55. Contract management capacity. The project supported the creation and development of GCMU as a core capacity within MOPH to manage NGO contracts. The BPHS and National Salary Policy in particular provided strong a technical framework to shape definition of the NGO contracts, budgets and workplans (as well as those of the MOPH Strengthening Mechanism). These were quite detailed and (along with monitoring of performance indicators) provided a basis for supervision and management of the contracts by GCMU. During the course of implementation, the capacity and track-record of the GCMU consultants led MOPH to make use of them for tasks not directly related to project implementation. In particular, they were used to provide just-in-time response (such as doing analyses or drafting policy documents) and for continuous support to some departments of the Ministry. Similarly, the other main donors who were financing NGO service delivery contracts came to support GCMU and closely involve it in implementation of their projects. USAID is now also moving towards channeling its funding through GCMU rather than managing NGO contracts directly. 56. Role of provincial health administrations. In contrast to GCMU, the role of the provincial health administrations was more ambiguous, and questions can be raised about whether the project could have done more to support this level. The provincial health administrations contributed to technical oversight of the NGO contracts but did not have significant management responsibilities. To an extent, this situation reflected the centralization vs. decentralization debate within government and among donors in Afghanistan. At the same 19

32 time, project management (and the Bank team) wanted to avoid the risk of unproductive or political interference in NGO operations by provincial authorities. The project supported the creation of Provincial Coordination Committees and provided some material support (such as communications equipment), but in contrast to other donor-funded programs, the project did not provide technical assistance. Because project-supported provinces did as well or better than the rest of the country with regard to service delivery and utilization indicators, it seems that this made little difference to the core project objective, but it may have been a shortcoming with regard to the capacity-building objective. 57. Data and results management culture. The project s strong support to monitoring and evaluation has been a crucial contribution to the development of a data and results culture of focus on indicators, targets and results. Health facilities track monthly evolution in key coverage indicators, comparing performance to targets, and the same is done at higher levels by NGOs, provincial health directors, GCMU, MOPH Strengthening Mechanism, and indeed the Minister of Health. The regular information available from the Balanced Scorecard facility survey and the HMIS influences management and policy decisions; for example, identifying poorlyperforming provinces for increased scrutiny and support, or revealing the need to bring services closer to communities through sub-centers. The NGO contracts emphasized key performance indicators and targets which were regularly monitored by MOPH during implementation. Poorly-performing provinces and NGOs were quickly identified. In one case (Nimroz province), the NGO contract was terminated by GCMU and a replacement was contracted. In other cases (Badghis and Farah provinces), GCMU successfully pushed the NGOs to improve performance and intensified field supervision. In addition, on the basis of the results of regular facility surveys, performance bonuses were paid to the best-performing NGOs (and also to high performing MOPH Strengthening Mechanism provinces that were evaluated along with the NGO-supported provinces). 58. Innovations to influence policy. The project as a whole rapidly scaled-up a number of innovations, most importantly the performance-based NGO contracting strategy and annual facility surveys to measure results. At the same tijme, Component 3 supported large-scale pilot programs and randomized impact evaluations to assess specific interventions and strategies. One study randomly assigned villages (totaling 1,500 households) into groups receiving a point-ofuse safe-water product ( Clorin ), hygiene education, improved tubewells, a combination of all three approaches, and a control group. Baseline and follow-up household surveys were done over the one-year period of implementation. The 2007 analysis found that only the combination of interventions led to a decline in diarrhea morbidity, as the relative risk of new cases during the peak summer diarrhea season was decreased by 39%. The main point of the pilot was to test the effectiveness of the point-of-use product and the study s policy impact was that the government did not adopt this particular intervention as a priority. 59. A pilot to test health financing approaches divided a total of 55 health facilities into a group where a community-based health insurance strategy (community health fund) was implemented, a group where user fees were introduced, a group where services were provided free-of-charge, and a control group which continued with whatever cost-recovery mechanism was in place at baseline. Baseline and follow-up facility and household surveys in the catchment areas were done. In the group implementing the community health fund, an evaluation after one 20

33 year of implementation found that the median enrolment rate among catchment populations was only 6% and that there was no measureable effect on household-out-of pocket payments for health services, although utilization increased among enrolled households. The community health funds were thus discontinued in due to low enrolment. In the group applying user fees, the study found that user fees represented only 3.4% of total facility operating costs, and that effects on changes in out-of-pocket expenditures and utilization were difficult to discern, seeming to be largely dependent on the systems in place at baseline. With regard to policy impact, the experience with the community health funds led to the removal of community insurance mechanisms as a policy option. More importantly, the lack of clear benefits or costs of user fees measured by this study contributed to the government s decision in 2008 to abolish user fees entirely for BPHS services. Overall, the pilot programs provided a very strong evidence based for policy decisions in the two areas of intervention. vi. Develop capacity for managing and delivering services (Rating: Satisfactory) 60. The project s support to the MOPH Strengthening Mechanism also represented a significant contribution to building government capacity, focused on direct service delivery in three provinces. MOPH at the central and provincial levels rapidly developed their capacity to manage 1,200 health workers and necessary inputs, although as discussed above, there were problems with drug procurement. After a late start, performance in the three provinces on average became comparable to the NGO-assisted provinces as measured by the Balanced Scorecard, although household survey data indicates that one of the three (Kapisa) has lagged in recent years. The Strengthening Mechanism contributed to the government s broader public sector reform agenda, making the MOPH the largest user of reformed personnel selection and management procedures, called Priority Reform and Restructuring (PRR). In the follow-up operation, the government has chosen not to expand the Strengthening Mechanism, but it remains an alternative model to the NGO contracting strategy. In fact, government officials indicate that the element of competition with the other NGO-contracted provinces remains important to continued performance and accountability of the Strengthening Mechanism. Figure 6. Provider knowledge score in BPHS facilities (% correct answers) Project provinces 10.0 National Among project provinces, data are not available for Helmand nor for Farah Source is Afghanistan Health Sector Balanced Scorecard Reports,

34 61. The NGO contracts and the MOPH Strengthening Mechanism included in-service training of health workers as part of the program to improve delivery of the BPHS as well as a focused effort to train community midwives. One of the PDO indicators is a score reflecting provider knowledge which is measured by the annual facility surveys. The score is the percentage of questions answered correctly by health workers, and the 2004 baseline is 53.5%, with the target fixed at 70%. 13 Figure 6 illustrates that overall, from a baseline of 53.5% in 2004, the provider knowledge score rapidly increased to 68.4% in 2005 and stabilized to 68.7% in In project-supported provinces, starting from the same baseline, the score reached 74.2% in Efficiency (Rating: Highly Satisfactory) 62. Technical efficacy (Rating: Highly Satisfactory). In developing the BPHS, the MOPH and partners applied what was called a Public Health-Based Decision Framework, which for each intervention assessed: (i) its impact on Afghanistan s disease burden; (ii) its proven effectiveness; (iii) the extent to which it could be scaled-up; (iv) its affordability in the long term; and (v) equity in access to the service. In other words, the BPHS was assessed on the basis of proven efficacy and cost, but also context-specific factors relating to disease burden, feasibility, resource constraints and equity. With regard to technical efficacy, the services included in the BPHS are consistent with international best-practice based on available evidence. As an example, Table 7 in Annex 3 shows that the BPHS delivers most of the interventions that have been identified as the most effective in preventing child mortality in developing countries. 63. Cost-effectiveness (Rating: Highly Satisfactory). Similarly, the cost-effectiveness of the services that are delivered as part of the BPHS has been demonstrated by numerous studies, reviewed for example by the 1993 World Development Report and Disease Control Priorities in Developing Countries. Estimated cost-effectiveness ratios for the interventions supported by the project cost under US$ 100 per Disability-Adjusted Life Year (DALY) and thus considered bargains. For example, full immunization is estimated at US$ 15 per DALY, IMCI at US$ 40, antenatal and delivery care at US$ 40, and family planning at US$ Actual cost and impact (Rating: Satisfactory). In the Afghanistan context, the project demonstrated that large improvements in basic health service provision and utilization can be achieved at an annual cost of on average US$ 3-4 per capita (see Table 3 in Annex 2), which can be compared to estimates of US$ per capita annually required to provide a package of services that will achieving significant progress towards the health-related MDGs. The 2002 estimate of US$ 4.50 required to deliver the BPHS 17 turned out to be sufficient to achieve the 13 Project documents state the baseline as 51%. 14 The methodology to measure provider knowledge was changed in 2008, adding clinical case scenarios to knowledge questions. 15 World Bank (2000) Investing in Best Buys: A Review of the Health, Nutrition and Population Portfolio, FY , Washington. 16 Jha, P. et al. (2002) Improving the Health of the Global Poor, Science 295: Newbrander, W., Yoder R., and Debevoise, A.B. (2007) Rebuilding health systems in post-conflict countries: estimating the costs of basic services, International Journal of Health Planning and Management 22:

35 gains described in the previous section. Partly in an effort to refine these cost estimates on the basis of actual experience, a study of the costs of 21 NGO contracts financed by USAID 18 found that that it was difficult to predict costs on the basis of security level, remoteness or even service utilization rates. This meant that fixed costs represented a significant proportion of total costs. One implication was that per capita budgeting remained a sensible approach for planning and determining NGO contract amounts. Another implication was that province-level contracts (as mostly done by the Bank-financed project) were likely more efficient than contracts to support lower aggregates of population. A third implication was that the efficiency of these fixed costs may be enhanced across the sector by reducing transaction costs through harmonized strategies. This was also achieved to a great extent in Afghanistan through the strong policy framework and the harmonized approach of the three major donors. 65. Finally, it should be noted total public sector health spending (from international and domestic sources), at around US$ 250 million annually, represents just 4% of total public spending in Afghanistan, 19 but the improvements in indicators described above suggest that it has had a disproportionate impact. 3.4 Justification of Overall Outcome Rating (Rating: Satisfactory) 66. Table 2 provides a summary of scores on different aspects of project outcomes, using a four-point scale and weights assigned to reflect the relative importance of different factors. The end result is an Outcome rating of Satisfactory. 67. The project was found to be highly relevant in terms of its alignment with development strategy. It was substantially relevant in terms of its objectives, which were well-suited to the challenges faced by the country but sometimes over-ambitious. The project was found to be substantially relevant in terms of its design, which focused on expanding delivery of basic interventions that address the main burden of disease in the country, although there were questions raised about whether there should have been support to mental health and disability services. The project s implementation strategy, based performance-based contracting of NGOs combined with strengthening of MOH capacity, was highly relevant to the situation. 68. With regard to Efficacy, available data on health outcomes (specifically child mortality) show substantial improvement, although data limitations prevent a definitive judgment. The project set itself indicators and targets mostly relating to its core objective of expansion of delivery and utilization of the BPHS in rural areas of Afghanistan. In some cases, project targets were ambitious for the difficult context of Afghanistan. High quality data are available to assess trends in service utilization during the project implementation period. With the exception of several interventions amenable to delivery by mass campaigns, coverage indicators (including PDO indicators) showed substantial improvement in both project areas (where the project 18 Ameli, O. and Newbrander, W. (2008) Contracting for health services: effects of utilization and quality on the costs of the Basic Package of Health Services in Afghanistan, Bulletin of the World Health Organization 86: World Bank (2009) Building on Early Gains: Challenges and Options for Afghanistan s Health and Nutrition Sector, Washington. 23

36 focused support on rural areas) and overall in rural areas of Afghanistan. The project supported almost a third of the provinces in the country and substantially contributed to setting the technical and implementation framework put in place by the government and the other major donors country-wide. The project objective relating to equity was substantially achieved although improvements in supply-side factors will need to be followed by strategies to address complex demand-side barriers. The project was highly successful in achieving its objectives in terms of system development and capacity building. Table 2. Project outcome evaluation scores Weight Score Weighted Score Relevance Alignment Objectives Design Implementation Efficacy Improve health outcomes Expand access to basic health services Support national immunization campaigns Target under-served rural populations and improve equity Develop capacity for stewardship of the sector Develop capacity for managing and delivering services Efficiency Technical efficacy Cost-effectiveness Total cost Total = Highly Unsatisfactory, 2 = Unsatisfactory, 3 = Moderately Unsatisfactory, 4 = Moderately Satisfactory, 5 = Satisfactory, 6 = Highly Satisfactory 69. In terms of Efficiency, the technical efficacy and cost-effectiveness of the interventions supported by the project are well-founded in the public health evidence base, while the actual costs of delivery of these services were found to be low, with substantial impact in terms of rapid expansion of coverage. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 70. The project s impact on urban-rural and gender equity is discussed above. The project s support for pilot programs in health financing contributed to a government policy decision to remove user fees with the intention of increasing access by the poorest. The 2006 Afghanistan Health Survey collected data on care-seeking and health expenditures by socio-economic status. Details are provided in Annex 3. Overall, the poorer were more likely to seek care at a public 24

37 sector facility, and public sector facilities on average cost less. The project s support to public sector provision of basic services was in this manner pro-poor. 71. Civil society development. The project contributed to development of civil society in Afghanistan in a general fashion by encouraging public-private partnerships for service delivery that maintained the strong role of non-state actors in provision of a core public service. (b) Institutional Change/Strengthening 72. As institutional strengthening was one of the elements of the PDO, the project s impact is discussed above. (c) Other Unintended Outcomes and Impacts (positive or negative) 73. No significant outcomes other than those discussed above have been identified. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 74. The annual health facility surveys financed by the project collected data on patient satisfaction. Overall, the patient satisfaction index remained stable at a high level (81.8 in 2004 and 80.7 in 2008). The NRVA found that in rural areas, improving health services was the third and second highest priorities respectively for male and female shuras (representative bodies). Similarly, a 2009 national opinion survey by the Asia Foundation found that healthcare was ranked fourth among the most important development issues (behind roads, power and water). This survey also found that a large proportion of the population seems to have a positive perception of the government s provision of health services, as 63% said it was doing a very or somewhat good job in this area. Further details are provided in Annex Assessment of Risk to Development Outcome (Rating: Moderate) 75. The main development outcome of the project is increased access by the rural population to basic health services. The most important potential risks to this outcome were identified as: (i) deterioration in the security situation; and (ii) insufficient financing to sustain service delivery. Deterioration in the security situation is a largely exogenous risk unless government success in delivering basic services to the population has an impact on the conflict. Although the health services have been affected in the most insecure areas, the NGO contracting strategy has shown its resilience to the deteriorating security situation. With regard to the risk of insufficient financing to maintain services, the successor project has addressed this in the immediate term and has also made progress in diversifying funding sources. 76. Possible change in government commitment to the implementation strategy, specifically NGO contracting, was a risk insofar as this strategy was instrumental in achieving project outcomes. However, establishment and continued support for the MOPH Strengthening Mechanism provides some mitigation to this risk, while the reality of capacity limitations will continue to present a severe challenge to proposals for substantial expansion of direct government provision of health services. 25

38 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry (Rating: Highly Satisfactory) 77. A 2003 QAG Quality at Entry review concluded with a Satisfactory rating. The review found that the project had a sensible design to meet the needs of a difficult context. Its core objective was to meet immediate needs but it also tried out approaches for longer-term development of the system. It technical and implementation strategies were developed as part of a joint approach with other donors. It was ready for implementation at Board approval, including having technical guidelines in place, tender documents prepared, an environmental framework in place and fiduciary management arrangements. However, the panel questioned whether the project could be implemented in the three year timeframe and found that some indicators were ambitious. It also suggested that the project could have included support to mental health and disability on a pilot basis and could have elaborated more on medium-term development of the health system. 78. In retrospect, the positive assessments of the QAG review have been borne out, while the questions raised have largely been answered, justifying an upgrading of the Quality at Entry rating. The positive aspects of project preparation and design are discussed in Section 2.1 above. With regard to the QAG panel s discussion of potential weaknesses, its prediction that the implementation period of three years would be insufficient was belied by the need for Additional Financing by end Targets set by the project were indeed ambitious, but most of them were substantially met. The panel suggested that the project expand its focus to include other health problems as well as further health systems development work. In practice, the tight focus on the project on expanding delivery of a relatively limited package of high-impact interventions was one of its major strengths. At the same time, it included substantial support to health systems issues (ie. the health financing and water/hygiene pilots and the MOPH Strengthening Mechanism). Further expansion of its scope would not have been advisable. (b) Quality of Supervision (Rating: Satisfactory) 79. There were only minor shortcomings in the proactive identification of opportunities and resolution of threats by the Bank team during supervision. Following is a summary assessment of the various aspects of quality of supervision. Focus on Development Impact. After initial attention to process issues (ie. contracting of the NGOs) during project start-up, the Bank team quickly turned to a strong focus on project outcomes as measured by key indicators. Aides-memoire emphasize trends in key indicators and supervision missions focus on the latest available data from HMIS, the annual facility surveys, and household surveys. Adaptations during project implementation were founded on analysis of quantitative data on outcomes, leading for example to greater emphasis on community-based services. Very late in the project implementation period, this focus on project 26

39 outcomes seems to have been diffused somewhat and aides-memoire came to be dominated by discussion of process and fiduciary issues. Supervision of Fiduciary and Safeguard Aspects. Rapid start-up of the project was supported by effective and timely review and support to fiduciary and safeguards processes, notably the initial contracting of the NGOs and monitoring and evaluation organization. A key contribution to project performance was the innovative design of the NGO contracts. The Bank team enlisted the support of Bank management to intervene with MOPH and the Ministry of Finance effectively intervened with Bank management and the government to address periodic delays in disbursement to the NGOs. Role in Ensuring Sustainable Financing. In communication to management, notably through ISRs, the team flagged early on the need to sustain and identify diversified sources of financing for the NGO contracts. The team was successful in preparing two Additional Financings and a successor project to achieve this that included alternative financing sources. Nevertheless, a gap between the first and second projects led to significant retroactive financing of the contracts by the successor project. (c) Justification of Rating for Overall Bank Performance (Rating: Satisfactory) 80. Successful project implementation reflects Highly Satisfactory quality at entry, while some shortcomings, particularly the fact that a long gap was allowed between the first and second projects, indicate Satisfactory performance with regard to supervision. The combination leads to an overall rating of Satisfactory for Bank performance. 5.2 Borrower Performance (a) Government Performance (Rating: Satisfactory) 81. Overall, when considering the baseline situation, where there was hardly a functioning government, progress in establishing a health administration that provides effective oversight on the sector is impressive. A minor shortcoming in government performance is related to the effects of centralization of fiduciary functions in the Ministry of Finance, which led to communication and coordination problems that contributed to disbursement and procurement delays. There was strong government commitment to and ownership of project objectives (i.e., achieving progress towards the MDGs by expanding the BPHS) and a supportive policy environment was put in place. This included a strong technical framework, and, very importantly, the opening of space for innovations, notably the GCMU, NGO contracting strategy and adaptations to administrative procedures required by the MOPH Strengthening Mechanism. (b) Implementing Agency or Agencies Performance (Rating: Highly Satisfactory) 82. There were no substantial shortcomings in implementing agency performance. MOPH showed strong commitment to the project objectives and implementation strategy and ensured effective stakeholder consultation and coordination, such as in the development of key technical policies such as the BPHS. It put in place the required capacity and ensured effective 27

40 management of the NGO contracts and Strengthening Mechanism, including strong use of data on performance from the monitoring and evaluation system. It established effective and close working relationships with project implementing partners. MOPH maintained the necessary implementation capacity and structures to ensure continuity between the first and second projects. Along with the involved NGOs, MOPH contributed to smooth transitions on the ground in cases where the NGO changed following the second project s contracting process. 83. As mentioned previously, during the last phase of the project implementation period, MOPH overcommitted to NGO contracts, requiring significant retroactive financing from the successor health project. From the point of view of fiduciary management, this can be interpreted as a shortcoming in planning and budgeting. From the point of view of the imperative to keep services running and achieve results on the ground, this can be seen as (audacious) risk-taking with the risk paying off, as the necessary financing ultimately became available. (If the Bank financing had not become available, it is not inconceivable that funding for these contracts could have been identified from other sources). From a point of view that prioritizes results over process, in the end, this over-commitment does not detract from the Highly Satisfactory evaluation of MOPH s performance. (c) Justification of Rating for Overall Borrower Performance (Rating: Satisfactory) 84. The combination of Satisfactory government performance and Highly Satisfactory implementing agency performance leads to an overall rating of Satisfactory. 6. Lessons Learned 85. Effectiveness of NGO contracting strategy in a difficult context. The project demonstrated that contracting of NGOs to deliver basic health services can be successful in improving service coverage in the most difficult of circumstances. The structure of the contracts was an important factor. The NGO contracts were creative adaptations of the standard Bank lump-sum contract, bringing them in practice very close to being output-based: the focus of contract management was on outputs in terms of improved service delivery, even though payments were not directly linked to such outputs. GCMU s supervision and control over the NGOs was largely focused on technical outputs rather than on oversight or management of inputs. The balance between providing autonomy to NGOs to achieve results in the way that they see fit and maintaining effective supervision and control is very difficult to achieve. In this case, a number of elements provided government with confidence that it could hold the NGOs accountable for results without close control over resources and inputs. These were the emphasis on performance indicators in the contracts, a strong technical and policy framework, strong capacity in the GCMU, and regular availability of reliable data on service provision and quality from the annual facility surveys. Further contributing to government s comfort level, GCMU closely shaped and monitored NGO expenditures but through a budget process rather than through a reimbursement process that would entail administrative controls and delays. That is, the NGOs were required to develop and follow a very detailed budget and report quarterly on expenditures against that budget but payments to the NGOs were not tied to these expenditures. 86. Important to ensuring accountability was the province-level targeting of the NGO contracts, whereby each NGO was held accountable for results that were measured on a 28

41 province-wide basis by the annual facility survey. Further strengthening the NGOs accountability for results was the fact that health service personnel were employees of the NGOs and not the government, enabling large NGO autonomy in their management (as long as technical and remuneration standards set by the government were met). Similarly, NGO control over procurement ensured that, being responsible for all inputs, they could be held accountable for results. The usual arguments for centralized procurement are that it results in cost-savings through economies of scale and ensures quality, particularly in the case of drugs. However, centralized procurement carries great risk in that any delays in the process will affect the entire project and entire beneficiary population, whereas delays experienced by one NGO only affect part of the project beneficiaries. The economy of scale argument ignores the costs of this risk and indeed the costs to the beneficiary population of unavailable or delayed inputs, particularly drugs. Quantitative data on drug availability and quality described previously indicate that assigning responsibility to the NGOs for procurement of necessary inputs was a successful strategy. 87. Government capacity and implementation mechanisms. An important contributor to the success of the project was the early establishment of core management capacity within MOPH. The GCMU was composed of competitively-selected Afghan consultants who were remunerated at rates that were competitive with what was offered by NGOs, UN organizations and other international organizations. This attracted highly-qualified professionals into government. However, objections were raised based on: (i) a perception that the GCMU was equivalent to a project implementation unit (PIU), operating parallel to or outside the Ministry; and (ii) the view that remuneration levels were unsustainable. These considerations relate to the structure and financing of the public sector, while the project adopted an implementation strategy that was focused on the best way to achieve project outcomes. In this context, with massive international aid coming into Afghanistan, much of it bypassing government and directly implemented by international firms and organizations, it is difficult to rectify the challenge of seeming to handicap the public sector by preventing it from paying competitive remuneration to attract the best-skilled staff. 88. Importance of monitoring and evaluation to a results-based implementation strategy. This project underlined the large benefits that can be had from investment in a well-functioning monitoring and evaluation strategy. Factors that were important to the success of the monitoring and evaluation strategy included closely support by the Bank team on technical issues, clarity of the tasks described in the ToRs (ie. specified number of health facility and household surveys), the fact that the Request for Proposals was attractive to well-qualified institutions, and effective management of the contract by MOPH. Comparison with experience in other countries also suggests the importance of the fact that the contracted organization was a university whose primary motivation was not profit. High quality staff from the ranks of graduate students were deployed, the organization had a large reputational stake in ensuring high quality according to international best practice, it went beyond its terms of reference to pursue deeper analysis (often with the self-interested aim of producing peer-reviewed publications), and it established an excellent working relationship with MOPH by being highly responsive to pressing requests for data collection and analyses. 29

42 89. Contracting vs. government-managed service delivery. Even though health services in few upper-income countries are provided directly by government, the default position for health sector development in lower-income countries is often a vision of expansion of government-run health services. The alternative public-private partnership strategy adopted by this project met substantial resistance from some partners, including some NGOs, and is often viewed as a transitional measure until it is possible for government to take over management of the services. Whether government will continue in the medium to long term to focus its capacity on acting as the financer and steward of the sector rather than as direct service provider is an open question. Further strengthening of the role of local NGOs in service delivery may be important to how this question is resolved. Also important will be the extent to which improved basic health services contributes to establishing the legitimacy of the state in the eyes of the population regardless of who provides the services. 90. The need for predictable financing of basic service delivery. The investment project paradigm is poorly adapted to supporting the delivery of basic social services in countries such as Afghanistan that require ongoing financing of recurrent costs. Investment projects are basically designed to make one-off investments that are intended to produce returns after the end of the project. A number of features of the Bank s traditional investment project framework stem from this, such as short project implementation periods and the difficulties in financing of civil servant remuneration (on the assumption that the government should be able to handle such recurrent costs after the investment is completed). In this case, the investment is in human capital and the main strategy is not one-off capital investment in equipment or infrastructure but financing of the recurrent costs of delivering basic health services, notably personnel and drugs. The government and the Bank team must go through contortions to make the investment project instrument meet this need. For example, the team processed several Additional Financing proposals and a successor project in order to meet the needs over what can be considered a relatively short period of around eight years. It is abundantly clear that continued international support to financing basic services in Afghanistan will be needed for the foreseeable future and predictable financing instruments are needed that are adapted to continuous support for the recurrent costs of health service delivery. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies MOPH s evaluation of the project is presented in Annex 7. (b) Cofinanciers Not applicable. (c) Other partners and stakeholders 91. Representatives of donors and NGOs were interviewed for this evaluation. They expressed strong support for a harmonized strategy of ensuring and expanding delivery of the BPHS through NGO contracting paired with effective contract management by GCMU. Other donors are supporting GCMU and moving towards more closely involving it in NGO contract 30

43 management under their programs. The Bank-financed project s support to monitoring and evaluation has provided the government, donors and NGOs with an important tool the Balanced Scorecard to supervise the sector. Donors contrasted the support of their programs to the provincial health administrations to the more limited support of the Bank-financed project. Although NGOs emphasized their close coordination with the provincial health administrations they also described challenges of these relationships. The NGOs indicated that the flexibility of their contracts under the Bank-financed project allowed them to introduce innovations and confront the challenges of insecurity. Insecurity is a major challenge, and the NGOs also cited disbursement delays as an important problem. It was suggested that there should be greater focus on the community level, including developing strategies for more effective support to community health workers and community health education. (See Annex 8) 31

44 Annex 1. Project Costs and Financing (a) Project Cost by Component (in US$ Million equivalent) Appraisal Estimate including Additional Components Financing (US$ millions) Actual/Latest Estimate from Borrower (US$ millions) Percentage of Appraisal Expand the delivery of the basic package of health services % Equipping the rural health infrastructure % Pilot tests of important innovations % Capacity building and training % Polio eradication and measles/neonatal tetanus control % Physical Contingencies Total Baseline Cost Price Contingencies 2.3 Total Project Costs % Total Financing Required % Note: Total original project cost was US$ 59.6 million. The first Additional Financing (2006) was US$ 30 million and the second Additional Financing (2008) was US$ 20 million. Actual disbursement by component reported by the Borrower, totaling US$ million, differs from the total of US$ million provided by the Bank system due to exchange rate fluctuations. (b) Financing Source of Funds Type of Cofinancing Appraisal Estimate (US$ million) Actual/Latest Estimate (US$ million) Percentage of Appraisal Borrower % IDA H0430 IDA Grant % IDA H2060 IDA Grant % IDA H3840 IDA Grant % Total % 32

45 Annex 2. Outputs by Component 1. This Annex provides details on Intermediate Outcome Indicators as well as some PDO indicators that were not discussed in the main text. 2. Data sources. Especially considering the context, the main data sources can be considered of good to high quality. The design and instruments of the five annual facility surveys financed by the project were based on international best practice. Each round sampled a total of over 600 health facilities across the country, with over 20 sampled in most provinces, providing sufficient statistical power for most estimates. The sample size for the 2003 MICS was 20,806 households but there were questions raised about the sampling frame used, which dated from The data were reanalyzed by the project s monitoring and evaluation contractor applying an updated sampling frame. The estimates presented here are from this reanalysis. The 2005 NRVA was a multi-sectoral survey that contained a health module. It used the updated sampling frame and sampled 30,822 households. The 2006 Afghanistan Health Survey, implemented by the project s monitoring and evaluation contractor, sampled 8,278 households in 29 of the 34 provinces. The provinces of Kandahar, Helmand, Zabul, Uruzgan and Nuristan were excluded due to insecurity. The NRVA sampled 20,576 households in all provinces. Four of the 396 selected districts were not visited due to insecurity. Component 1. Expand the delivery of the basic package of health services (actual cost US$ million) 3. This Component was the core of the project, accounting for the bulk of financing and focused on the basic objective of expanding provision of basic health services. Table 3 provides population figures for the areas supported by the project under this Component as well as total and annual per capita support. 4. Health outcomes. Child and maternal mortality are PDO indicators but not included in ISRs, likely because it is not possible to measure short-term trends. Child mortality is discussed in the main text. The baseline for the maternal mortality ratio used by project documents is 1,600 per 100,000 births, citing World Development Indicators, and the project target is stated as a 10% reduction. An inter-agency working group using available data and models of known correlates of maternal mortality estimated a ratio of 1,900 for 2000, while a similar estimate for 2005 was 1,800. The range of uncertainty for such estimates is so large and the data on which they are based so poor, that it is impossible to use the maternal mortality ratio as a project indicator. A rigorous 2002 study by UNICEF and the Centers for Disease Control (CDC) focused on four districts, estimating an overall ratio of 1,600. There are little or no data to inform an assessment of trends subsequent to The government intends to implement a large-scale survey in the near future to enable an estimate of maternal mortality. 5. Child health care. Immunization and vitamin A coverage indicators are described in the main text. Several Intermediate Outcome indicators relate to appropriate breastfeeding, important to infant and child health and nutrition. Appropriate breastfeeding involves exclusive breastfeeding from 0 to 6 months and the introduction of appropriate complementary foods after 6 months. One indicator is the proportion of children 0-6 months who were exclusively 33

46 breastfed in the last 24 hours. The baseline from the 2003 MICS is 82% for rural areas, with the project target fixed at 90%. The 2006 Afghanistan Health Survey found that 83% of children 0-5 months were exclusively breastfed, while the NRVA found a prevalence of 79% in rural areas. However, the survey probed deeper on this issue and found that a significant proportion of mothers were giving supplementary liquids to their infants. When this is taken in account, the rate of exclusive breastfeeding drops to 35% in rural areas. The other indicator is the proportion of children 6-18 months who received breast milk and appropriate complementary food in the last 24 hours. This was measured by the 2003 MICS at 67.8% in rural areas, but was not measured with the same denominator by subsequent surveys. The 2006 Afghanistan Health Survey found that 28.4% of children aged 6-9 months in rural areas received complementary foods, while the NRVA found a rate of 39% in rural areas. Table 3. Beneficiary populations with total and annual per capita support Population Disbursed (USD) Contract period (mos) Annual per capita support during contract period (US$) Annual per capita support during project period (71 mos) (US$) NGO contracts 4,759,441 78,292, Badghis 420,400 5,270, Balkh 949,600 16,417, Farah 343,400 8,778, Helmand 756,400 10,219, Nimroz 138,500 3,099, Samangan 318,500 8,116, Saripul 474,800 8,262, Wardak 448,700 8,598, Badakhshan 81, , Baghlan 176,700 1,987, Khost 203,200 2,517, Paktika 164,220 1,476, Paktika 102,800 1,225, Paktya 115, , Ghazni 65, , MOPH Strengthening Mechanism 1,240,148 15,253, Kabul (rural) 133,200 1,521, Kapisa 366,606 4,720, Panjshir 144,314 1,832, Parwan 596,028 7,178, Overall 5,999,589 93,546, Source is MOPH. 6. An Intermediate Outcome indicator is the proportion of parents able to spontaneously name the danger signs of diarrhea and ARI and the appropriate response. The estimate of 48% for rural areas from the 2003 MICS is for the proportion of parents able to name at least one 34

47 danger sign of diarrhea and ARI. Subsequent household surveys did not collect data on this indicator. 92. Reproductive and maternal health care. Contraceptive prevalence (PDO indicator) and qualified delivery care coverage (Intermediate Outcome indicator) are described in the main text. In addition, antenatal care coverage was an Intermediate Outcome indicator, with the baseline fixed at the 2003 MICS estimate for rural areas of 4.6% and the target at 35%. Figure 7 illustrates that in the provinces supported by the project, the proportion of pregnant women who received antenatal care rose dramatically from 5.6% in 2003 (MICS) to 12.6% in 2005 (NRVA) and 28.2% in 2008 (NRVA). In rural areas nationwide, the improvement was similar, as 4.6% of pregnant women received antenatal care in 2003 (MICS), rising to 32.3% in 2006 (Afghanistan Health Survey) and stabilizing or slightly declining to 30.0% in 2008 (NRVA). Figure 7. Antenatal care by a qualified provider in rural areas (% of births) Project provinces National average (rural) 2003 MICS 2005 NRVA 2006 Afgh Health Survey NRVA 7. General curative care. The outpatient consultation rate (PDO indicator) is described in the main text. In addition, the annual facility surveys provide data on the proportion of basic health centers (BHCs) that had over 750 outpatient consultations per month. Figure 8 shows that the data collected by the facility surveys is consistent with the HMIS-reported rise in outpatient consultations. In the project provinces, for example, 23.2% of BHCs had over 750 outpatient consultations per month in 2004 and this steadily rose to 82.9% in

48 Figure 8. Basic Health Centers with over 750 outpatient consultations per month (% BPHS facilities) Project provinces National Among project provinces, data are not available for Helmand nor for Farah Source is Afghanistan Health Sector Balanced Scorecard Reports, Figure 9. Indicators of TB detection & treatment service provision (% BPHS facilities) TB register TB clinical guidelines TB smear lab capacity (CHC & DH) Source is Afghanistan Health Sector Balanced Scorecard Reports, Disease control. Data collected by the annual facility surveys provide indicators of provision of TB services (Figure 9). Nationally, the proportion of facilities that had a TB register (indicating they are providing TB treatment) rose from 18.3% in 2004 to 65.0% in The proportion of facilities that had clinical guidelines on TB diagnosis and treatment rose from 18.6% to 76.6%. The proportion of Comprehensive Health Centers (CHCs) and District Hospitals (DHs) that had laboratory capacity to do TB smears for case detection rose from 20.5% to 91.4%. 9. Addressing insecurity. Figure 10 shows the evolution of key indicators in Helmand province after introduction of innovative approaches such as conditional cash transfers and health provider incentives as strategies to mitigate the impact of insecurity. 36

49 Figure 10. Changes in selected indicators after introduction of strategies to mitigate insecurity in Helmand province Component 2. Equipping the rural health infrastructure (actual cost US$ 0.1 million) 10. Because the necessary equipment was financed by another donor and by the NGOs and Strengthening Mechanism under Component 1, this component was dropped by the first Additional Financing and the funds (US$ 2.6 million) reallocated to Component 1. Component 3. Pilot tests of important innovations (actual cost US$ 1.0 million) 11. The water/hygiene pilot randomly assigned villages (totaling 1,500 households) into groups receiving a point-of-use safe-water product ( Clorin ), hygiene education, improved tubewells, a combination of all three approaches, and a control group. Baseline and follow-up household surveys were done over the one-year period of implementation. The 2007 analysis found that only the combination of interventions led to a decline in diarrhea morbidity, as the relative risk of new cases during the peak summer diarrhea season was decreased by 39%. As a result, the government did not emphasize any single intervention as a priority. 12. The health financing pilot divided a total of 55 health facilities in ten provinces into a group where a community-based health insurance strategy (community health fund) was implemented, a group where user fees were introduced, a group where services were provided free-of-charge, and a control group which continued with whatever cost-recovery mechanism was in place at baseline. Baseline and follow-up facility and household surveys in the catchment areas were done, while the study also made use of HMIS, financing reports, qualitative 37

50 interviews, and other sources of information. Implementation started in In the group implementing the community health fund, an evaluation after a year of implementation found that the median enrolment rate among catchment populations was only 6% and that there was no measureable effect on household-out-of pocket payments for health services, although utilization increased among enrolled households. The intervention was discontinued in 2006 due to low enrolment. In the group applying user fees, the 2007 evaluation found that user fees represented only 3.4% of total facility operating costs, and that effects on changes in out-of-pocket expenditures and utilization were difficult to discern, seeming to be largely dependent on the systems in place at baseline. The lack of clear benefits or costs of user fees measured by this study contributed to the government s decision in 2008 to abolish user fees entirely for BPHS services. Component 4. Capacity building and training (actual cost US$ 5.4 million) 13. A relevant Intermediate Outcome indicator is the number of community health workers (CHWs) per 1,500 population submitting monthly reports. A baseline was not specified but a target of 0.7 was indicated in ISRs. HMIS data indicate that the rate was 0.97 in projectsupported provinces in Component 5. Polio Eradication and Measles/Neo-Natal Tetanus Control (actual cost US$ 7.9 million) 14. Component 5 of the project provided support to national immunization campaigns against polio, measles, and neo-natal tetanus, including vitamin A distribution. Figure 4 in the main text provides trends in coverage levels of these interventions. 38

51 Annex 3. Economic and Financial Analysis 1. Efficacy. Table 5 describes a number of child health interventions and their efficacy against common causes of morbidity and mortality in developing countries. It has been estimated that if these interventions were universally available, 63% of child deaths would be prevented in the 42 countries that account for 90% of deaths among under-five children. The table indicates whether the interventions are included in the BPHS and at what level of health facility. All except possibly a component of antenatal care (antenatal steroids) 20 and prevention of mother-to-child transmission of HIV/AIDS are included in the BPHS. Other components of the BPHS, reproductive and maternal health services as well as disease control and general curative services, are based on international evidence for efficacy. Table 4. Cost-effectiveness of BPHS components (US$ per DALY) BPHS component US$/DALY Maternal and newborn health Antenatal and delivery care incl. routine maternity care 125 Postpartum care Family planning Care of the newborn Child health and immunization Expanded Program on Immunization (EPI) 16 Integrated Management of Childhood Illness (IMCI) incl. ARI treatment (community) 140 incl. ARI treatment (PHC facility) 28 Public Nutrition Prevention of malnutrition incl. breastfeeding promotion 8-11 incl. vitamin A supplementation 6-12 Assessment of malnutrition incl. growth monitoring 8-11 Treatment of malnutrition Communicable disease treatment and control Control of tuberculosis 5-50 * Control of malaria 24 Control of HIV (IEC, VCT, counseling) 10 ** * DOTS only ** VCT only Source is various studies cited by Jamison et al. (eds.) (2008) Disease Control Priorities in Developing Countries, Second Edition, World Bank and Oxford University Press, Washington and New York. Estimates for South Asia are provided when available. Otherwise, the estimates are for low-income countries. Estimates are in 2002 US$. 20 It is not clear from the available information whether this specific intervention is part of standard protocols for delivery of the BPHS. 39

52 2. Efficiency. A standard measure of efficiency in the health sector is the cost-effectiveness ratio. The cost per disability-adjusted life year (DALY) is commonly used, with DALYs defined as the years of life saved by the intervention, accounting for disability level. Because of their low cost and high efficacy in addressing the major causes of child mortality, the neonatal, child and maternal health interventions included in the BPHS are very cost-effective. Table 4 provides cost-effectiveness ratio estimates for components of the BPHS, indicating that the package is highly cost-effective. 3. Table 3 above provides the annual per capita costs of support to beneficiary populations. On average, during their contract periods, the NGOs provided US$ 4.03 per capita support to basic health services. This ranged from US$ 2.42 to During the entire project period, average annual per capita support was US$ 2.78, which by any measure is an extremely low level of expenditure that has nevertheless led to substantial improvements in service coverage. The comparison underlines the efficiency of the project, but also suggests that greater resource levels may be needed to sustain the rate of improvement. 4. Equity. The 2006 Afghanistan Health Survey collected data from households on health care seeking and out-of-pocket expenditures by socio-economic status. The poorest quintile were less likely to seek care when ill; 35% did not seek care, compared to 12% among the highest quintile. Among the poorest quintile, the most common reasons for not seeking care related to cost of transport or care (42%) or distance or lack of transport (41%). Of those who sought care, the poorer were more likely to first go to a public sector facility. Among the poorest quintile, 30% went to a public health clinic and 19% to a district or provincial hospital, compared to 23% and 13% respectively in the highest quintile. The better-off were more likely to go to a private provider. In total, 47% among the poorest quintile went to some kind of a private provider, compared to 61% among the highest quintile. At the same time, the cost of private primary health care providers was higher, as the median out-of-pocket cost (including fees, tests, drugs, transport, food and lodging) for those who went to a public health clinic was 150 Afganis, compared to 600 for a private doctor or clinic. The median cost for those who went to a district or provincial hospital was 500 Afghanis. Overall, the data suggest that the poor use public sector primary health care more than the better-off, thereby incurring lower costs than if they went to private providers. 5. Cost. As discussed in the main text, significant expansion of coverage of basic health services has been achieved at an annual per capita cost of US$ 4 to 5. This can be compared to estimates of the incremental cost for achieving high coverage of basic interventions low-income countries that are provided in Table 6. The Afghanistan cost is comparable to the estimated cost in the first years of expansion of coverage. 21 This, along with the still low coverage levels of many interventions in Afghanistan, suggests that spending will need to increase in future years in order to further expand coverage. 21 The BPHS does not include HIV/AIDS treatment, which is supported by other programs. 40

53 Table 5. BPHS delivery of highly-effective interventions to reduce child mortality Proven efficacy BPHS delivery level diarrhea pneumonia measles malaria HIV/AIDS birth asphyxia preterm delivery neonatal tetanus neonatal sepsis Health Post BHC CHC DH Preventive interventions breastfeeding x x x x x x x (promotion) insecticide-treated materials x x x x x x complementary feeding x x x x x x x x (promotion) water, sanitation, hygiene x x x x x (promotion) Hib vaccine x x x x zinc x x x x x x vitamin A x x x x x x x antenatal steroids x newborn temperature management x x x x tetanus toxoid x x x x x nevirapine and replacement feeding x antibiotics for premature rupture of membranes x x x x x clean delivery x x x x x measles vaccine x x x x antimalarial intermittent preventive treatment in pregnancy x x x x x Treatment interventions oral rehydration therapy x x x x x antibiotics x x x x x x x antimalarials x x x x x newborn resuscitation x x x x zinc x x x x x vitamin A x x x x x Source for efficacy information is Jones, G. et al. (2003) How many child deaths can we prevent this year? Lancet 362:

54 6. Fiscal space. Total public spending on health in is estimated at US$ 278 million, which represents 4% of total public spending. Only 27% of the national budget comes from domestic revenue sources as the remainder is financed by donors. Figure 11 shows that spending on security represents over 40% of public spending. Overall, these patterns suggest that further increases in public sector spending on health will need to come from international sources, likely through reallocation from security expenditures as security (hopefully) improves (even though most security spending is poorly fungible). Table 6. Estimated per capita annual incremental costs of achieving high coverage of basic health services by 2015 in low-income countries (2002 US$) low high low high Maternity-related conditions Child immunization Treatment of childhood illnesses Malaria prevention Malaria treatment Tuberculosis treatment HIV prevention HIV/AIDS care Highly-active anti-retroviral therapy (HAART) All interventions All interventions except HIV/AIDS care & HAART Source is Jha, P. et al. (2002) Improving the Health of the Global Poor, Science 295: Figure 11. Public spending in by sector Other 3% Private sector development 3% Agriculture 10% Social protection 2% Education 9% Governance 5% Health 4% Infrastructure and national resources 23% Security 41% 42

55 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Lending Names Title Unit Responsibility/ Specialty Supervision/ICR Arun Kumar Kolsur Procurement Specialist SARPS Procurement Asila Wardak Jamal Consultant SASDI Social Celine Ferre Young Professional YPP Operations Deepal Fernando Senior Procurement Specialist SARPS Procurement Emanuele Capobianco Senior Health Specialist SASHN Health Ghulam Dastagir Sayed Public Health Specialist SASHN Health Hasib Karimzada Program Assistant SACAF Operations Henri Aka Procurement Specialist SASHD Operations (ICR) Kavitha Viswanathan E T Consultant SASHD Health Kayhan Natiq Consultant SASHD Health Kees Kostermans Lead Public Health Specialist SASHN Health Kenneth Okpara Senior Financial Management Specialist SARFM Financial management Mohammad Arif Rasuli Senior Environmental Spec. SASDI Environment Mohammad Tawab Hashemi E T Consultant SASHD Health Muhammad Wali Ahmadzai Financial Management Analyst SARFM Disbursement Nagaraju Duthaluri Senior Procurement Specialist SARPS Procurement Patrick Mullen Senior Health Specialist AFTHE Health (ICR) Rahimullah Wardak Procurement Specialist SARPS Procurement Silvia Albert Program Assistant SASHD Operations Tekabe Ayalew Belay Senior Economist (Health) SASHN Health (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle US$ Thousands (including No. of staff weeks travel and consultant costs) Lending ,555 Supervision/ICR ,493 Total 415 1,146,049 43

56 Annex 5. Beneficiary Survey Results 1. Household surveys find that improved health services remain a priority for the population. The NRVA found that in rural areas, improving health services for men and women was the third highest priority for male shuras (representative bodies) (behind improved irrigation and drinking water supply), and the second highest priority for female shuras (behind drinking water). In addition, for female shuras, an additional category improved health services for women was specified as the fourth highest priority. Similarly, a 2009 national opinion survey by the Asia Foundation found that healthcare was ranked fourth among the most important development issues (behind roads, power and water). 2. However, the 2009 opinion survey found that expectations are high for health services to improve, as 75% expected the availability of clinics and hospitals to be much or somewhat better in the next year, and 74% expected the availability of medicines to similarly improve. A large proportion of the population also seems to have a positive perception of the government s provision of health services, as 63% said it was doing a very or somewhat good job in this area. The comparison with other areas in Table 7 indicates that government performance in providing social services (health and education) is perceived to be significantly better than in other areas, with perhaps implications for broader discussions about the need for consolidating the legitimacy and support for the Afghan state in a situation of civil conflict. Table 7. Perceptions of the performance of the national government in specific areas, 2009 very good job somewhat good job somewhat bad job very bad job Education Healthcare system Creating job opportunities International relations Economic development Fighting corruption Security Source is The Asia Foundation (2009) Afghanistan in 2009: A Survey of the Afghan People. 44

57 Annex 6. Stakeholder Workshop Report and Results Not applicable. 45

58 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Following is the text of MOPH s evaluation of the project: Assessment of the project s rationale and objective Afghanistan s health status after the Taliban was one of the worst among the poor countries with alarming indicators as: Under 5 mortality 257/1000 live births Infant mortality 165/1000 live birth Maternal mortality 1600/100,000 live births Acute malnutrition 6-12% Iron deficiency anemia 71% in pregnant women TB 70,000 new cases/year Life expectancy 42.6 (CDC/UNICEF/MOH 2002, MOH 2004, UNICEF MICS 2003) The above stated health profile of Afghanistan after the Taliban s fall determined the rationale for the project: Provision of an essential minimum primary health care package was necessary to meet the basic health needs of the population, in particular mothers and children. Project objectives: a. Reducing infant and child mortality, maternal mortality, child malnutrition and increasing access to reproductive health through the expansion of basic health care services. b. Enhancing MOPH s institutional role in the health sector c. Building MOPH capacity to better provide and manage health services The selected objectives and rationale were appropriate to the health situation of Afghanistan: The project objectives addressed the most important causes of maternal and child mortality and morbidity. They focused on the problems of the most vulnerable groups such as children and women. Priority was given to rural and remote areas Equity considerations were important and the project focused on the needs of the poor. Achievement of the objectives could contribute to solve other health-related problems. The objectives were affordable, practical, acceptable and measurable. Assessment of project design 46

59 After the fall of the Taliban, Afghanistan required rapid improvement of access through expansion of basic health services, with special focus on vulnerable and high risk groups (mothers and children), focus on communicable and vaccine-preventable diseases. The program should have an affordable cost, and be manageable by a minimum number of health workers. It should be community based and consider the lack of capacity in the government. The design of the HSERDP project fully considered the socio-economic, political, and cultural context and prioritized the health needs of the country. The project was design was adapted to realities such as resource limitations, and scarcity of professional and skilled health workers. The project design reflected the lack of capacity in the government and the fact that about 80% of health services were delivered by NGOs. To address the country s health problems and achieve the objectives, the Ministry of Health with the technical support of stakeholders (WHO, UNICEF, UNFPA, WB, EC, USAID, JICA and others) designed the basic package of health services as a national strategy. The Basic Package of Health Services was completed in April The BPHS consists of the following components: Maternal and new born health, Child health and Immunization, Public Nutrition, Communicable diseases, Mental Health, Disability care, Supply of essential drugs. The important decision in the design of the project was the idea of contracting the health services which was introduced by WB on the basis of findings from a pilot project from Cambodia. The Performance based Partnership Agreement (PPA) implementation strategy was accepted by MoPH. The provision of the basic package of health services to all 8 provinces (Balkh, Samangan, Saripul, Wardak, Helmand, Farah, Badghis and Nimroz) was contracted to NGOs, and the Ministry of Health focused on its stewardship role. The provision of BPHS for 3 provinces (Panjsher, Kapisa and Parwan) was assured by MoPH directly through the Strengthening Mechanism. Bottlenecks and lessons learned during implementation The Ministry of Public Health and different implementing NGOs have raised few issues regarding the project. The most important are: Delay in payments: the implementing agencies faced this problem often and especially in the last 2 years of the project. All parties were responsible for delays (NGO, MoPH, MOF and WB) but mostly it happened in the Ministry of Finance. The project didn t have any budget for construction of health facilities which affecte the service quality and raised communities complaints. In the WBsupported project in some provinces we have up to 60% health facilities located in private houses while the situation was much better in EC and USAID-supported provinces because of having standard buildings. Delay in submission of third party evaluation report (Balanced Score Card, BSC). It was taking about one year to be released, while sooner release of BSC would allow for more timely changes. Excessively complicated procurement processes with involvement of three parties (MoPH, WB and MoF). 47

60 Division of procurement responsibilities within MOPH. While the procurement of services and contract management was the responsibility of GCMU, goods procurement was the responsibility of the Administration Directorate and recruitment of individual consultants was the responsibility of the Human Resources directorate. This fragmentation caused delays. Ambiguity of Provincial Health Directors role. In some provinces their interference was too high while in some others their contribution was low. Interference of local authorities in project implementation. Outdated government procedures. Poor coordination among ministries and departments. Monitoring difficulties especially in insecure areas. The main lessons learned from the HSERDP are: the most important reason for the success of the project is the contracting out approach with a performance based partnership and using a lump sum type of contract, establishment of GCMU with qualified staff, GCMU s technical support to other departments of MoPH, monitoring and evaluation by a third party, managerial and financial autonomy of implementing agencies. Lack of budget for construction of health facilities and less focus on capacity-building of Provincial Health Offices (PHOs) had some negative impacts and caused community complaints. Project outcomes: The project has significantly improved the health status of the Afghan people. The following table is a brief comparison of national health indicators between the start and end of the project MICS 2005 NRVA 2006 AHS 2008 NRVA Contraceptive Prevalence Rate Estimate (%) Proportion of Pregnant Women At least One ANC Skill Birth Attendance BCG vaccine coverage Polio Vaccine coverage Measles vaccine coverage Full immunization DPT3 Coverage Under 5 Y Mortality rate/1000 live births Under 1 Y Mortality rate /1000 live birth

61 Evaluation of the Borrower s performance Generally we can assess the government performance and contribution as satisfactory. Despite of problems and difficulties, the government was committed to successfully manage the project. The existing problems with government especially regarding procurement and financial management, were mentioned above. Evaluation of the performance of the Bank The WB contribution in the preparation of the project was great. Actually the contracting idea and performance based partnership was introduced and encouraged by WB experts. The WB performance was high and WB colleagues and missions always were very cooperative. Our only comment relates to training and capacity building we think that the WB could arrange more training to build the capacity for MoPH particularly in the fields of finance and procurement. The MoPH requests that WB kindly consider this issue in the future. Sustainability of the program We did our best to make the program sustainable through efficiency, effectiveness and minimizing the basic package of health services. 4 USD per capita is very low and should be affordable even for a poor government like Afghanistan. On the other hand, we improved community awareness and convinced them regarding their basic need for the BPHS. MoPH is also trying to attract the attention of the international donor community to BPHS and is fund-raising through different forums. 49

62 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders 1. Representatives of the other major donors in the sector, the EU and USAID, were interviewed for this evaluation. The following points were made: There is strong support for a harmonized strategy of ensuring and expanding delivery of the BPHS through an NGO contracting strategy. The Bankfinanced project is recognized for its strong contribution to development and implementation of this strategy. The Bank-financed project is also recognized for its support to the creation and development of GCMU. USAID has provided substantial support to GCMU in recent years and is moving towards channeling its funds through government so that GCMU manages the NGO contracts directly, as it does under the Bank-financed project. The EU program, which continues to directly contract NGOs, is also finding ways for MOPH to get more closely involved in their management. The Bank-financed project s support to monitoring and evaluation has provided the government and donors with an important tool the Balanced Scorecard to supervise the sector. The EU s recent evaluation of its program made use of the Balanced Scorecard results in addition to other information. USAID emphasized its strong support to the HMIS and routine supervision as very important to MOPH s oversight. Donors contrasted the support of their programs to the provincial health administrations to the more limited support of the Bank-financed project. The importance of the Provincial Health Directors (PHD) to management of the system was emphasized, as it was suggested that in some cases the NGOs may have too much autonomy for important decisions. There was also concern that the provincial health administrations do not have a clear institutional home within MOPH. Insecurity is the main challenge facing the sector and there is a correlation between the most insecure areas and other socio-economic factors that affect health outcomes. The NGOs are adapted to dealing with insecurity with their flexibility and close ties to the community. 2. A number of NGOs that were contracted under the Bank-financed project were also interviewed and the following issues were discussed. The importance of the BPHS as the backbone of the health system was emphasized. NGOs cited as important achievements the establishment of BPHS services, training of community midwives, and support to community health workers. The flexibility of the NGO contracts under the Bank-financed project was emphasized, allowing them to confront the challenges of insecurity, for example by reallocating budget to hardship allowances, and to introduce 50

63 innovations, such as paying for the transport costs of patients referrals rather than replacing ambulances that were destroyed in the conflict. The strong role of the GCMU was cited as a success factor, as well as the Balanced Scorecard and HMIS system that supported a focus on results. The NGOs also emphasized their coordination with the provincial health administration, including monthly meetings and joint supervision, but also cited their relationships with the provincial administrations as an important challenge. Personnel management issues as well as competing demands between the provincial and central levels were given as causes of friction. Insecurity presented the most serious challenge, and the NGOs have developed a variety of strategies, most importantly strengthening links with community elders to prevent kidnappings and disruption of drug supply. Supervision by the NGO and MOPH is greatly curtailed by insecurity. The NGOs also indicated that disbursement delays have presented a major challenge, related to the gap between the first and second Bank-financed projects. The NGOs suggested that there should be greater focus on the community level, including developing strategies for more effective support to community health workers and community health education. They perceive as constraining current government rules on the required educational level of community midwives and remuneration of community health workers. 51

64 Annex 9. List of Supporting Documents Government and Project Documents Central Statistics Office, Islamic Republic of Afghanistan (2009) National Risk and Vulnerability Assessment 2007/8: A profile of Afghanistan, Kabul. Central Statistics Office, Islamic Republic of Afghanistan, and UNICEF (2003), Multiple Indicator Cluster Survey 2003, Afghanistan. Moving Beyond 2 Decades of War: Progress of Provinces, Kabul. Islamic Republic of Afghanistan (2008) Afghanistan National Development Strategy ( ), Kabul. Islamic Republic of Afghanistan (2008) Health & Nutrition Sector Strategy (2007/ /13), Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2004) Afghanistan Health Sector Balanced Scorecard: National and Provincial Results, Round 1 (2004), Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2005) A Basic Package of Health Services for Afghanistan, 2005/1384, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2005) A Basic Package of Health Services for Afghanistan, 2009/1388, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2005), National Salary Policy For Non- Governmental Organizations and Ministry of Health Strengthening Mechanism Working in the Afghan Health Sector, Revised Version of Original Policy, October 2005, Salary Policy Working Group, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2005) Afghanistan Health Sector Balanced Scorecard: National and Provincial Results, Round Two 2005, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2006) 2005 National Risk and Vulnerability Assessment: Analysis of Priority Health Service Delivery Indicators, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2006) Afghanistan Multiple Indicator Cluster Survey 2003: A Re-analysis of Critical Health Service Delivery Indicators, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2006) Afghanistan Health Sector Balanced Scorecard: National and Provincial Results, Round Three 2006, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2006) Afghanistan Health Survey 2006: Estimates of Priority Health Indicators for Rural Afghanistan, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2007) Safe Water Systems Project, Afghanistan, Final Report 2007, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2007) Afghanistan Health Sector Balanced Scorecard 2007, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2007) Operations Research Study on Community Health Worker Performance in Afghanistan: Findings , Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2007) Drug Quality Assessment Study, Afghanistan 2007, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2008) 2007 Afghanistan Hospital Assessment: National, Provincial and Kabul Results, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2008) Afghanistan Health Sector Balanced Scorecard 2008, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2008) Final Evaluation Report on Health Financing Pilots: The Effects of User Fees vs. Free Services on Primary Care in Afghanistan, Kabul. Ministry of Public Health, Islamic Republic of Afghanistan (2009) HMIS Data Overview: 1387, Presentation. 52

65 Ministry of Public Health, Islamic Republic of Afghanistan (2009) Afghanistan HMIS Indicators in 1387, Presentation. Ministry of Public Health, Islamic Republic of Afghanistan (2009) HMIS Data Overview: 1388, Presentation. World Bank Documents World Bank (2000) Investing in Best Buys: A Review of the Health, Nutrition and Population Portfolio, FY , Washington. World Bank (2002) Second Joint Donor Mission to Afghanistan on the Health, Nutrition and Population Sector: Aide-Memoire. World Bank (2002) Transitional Support Strategy: Afghanistan, Report No AF, Washington. World Bank (2003) Afghanistan: Health Sector Emergency (PO78324) - Final Quality at Entry Assessment (QEA6), Quality Assurance Group, Washington. World Bank (2003) Technical Annex for a Proposed Grant of SDR 43.7 Million (US$ 59.6 Million Equivalent) to the Transitional Islamic State of Afghanistan for a Health Sector Emergency Reconstruction and Development Project, Report No. T-7552-AF, Washington. World Bank ( ), Aides-memoire. World Bank ( ), Implementation Status Reports. World Bank (2006) Additional Grant Document: International Development Association Proposed Additional Grant of SDR 21.0 Million (US$ 30 Million Equivalent) to the Islamic Republic of Afghanistan for the Health Sector Emergency Reconstruction and Development Project, Report No AF, Washington. World Bank (2008) Project Paper on a Proposed Additional Financing (Grant) in the Amount of SDR 12.2 Million (US$ 20 Million Equivalent) to the Islamic Republic of Afghanistan for a Second Additional Financing for Health Sector Emergency Reconstruction and Development Project, Report No AF, Washington. World Bank (2009) Building on Early Gains: Challenges and Options for Afghanistan s Health and Nutrition Sector, Washington. World Bank (2009) International Development Association and International Finance Corporation Interim Strategy Note for Islamic Republic of Afghanistan for the Period FY09-FY11, Report No AF. World Bank (2009) Proposed Project Paper on a Proposed Grant in the Amount of SDR 19.9 Million (US$ 30 Million Equivalent) to the Islamic Republic of Afghanistan for a Strengthening Health Activities for the Rural Poor Project, Report No AF, Washington. World Bank and Islamic Republic of Afghanistan (2006), Agreement Amending Development Grant Agreement (Health Sector Emergency Reconstruction and Development Project) between Islamic Republic of Afghanistan and International Development Association, Grant No. H2060-AF, Washington. World Bank and Islamic Republic of Afghanistan (2008), Financing Agreement (Second Additional Financing for Health Sector Emergency Reconstruction and Development Project and Amendment to the Development Credit Agreement) between Islamic Republic of Afghanistan and International Development Association, Grant No. H3840-AF, Washington. World Bank and Transitional Islamic State of Afghanistan (2003) Development Grant Agreement (Health Sector Emergency Reconstruction and Development Project) between Transitional Islamic State of Afghanistan and International Development Association, Grant No. H0430-AF, Washington. Other Sources Ameli, O. and Newbrander, W. (2008) Contracting for health services: Effects of utilization and quality on the costs of the Basic Package of Health Services in Afghanistan, Bulletin of the World Health Organization 86 (12):

66 Ameli, O. and Newbrander, W. (2008) Contracting for health services: effects of utilization and quality on the costs of the Basic Package of Health Services in Afghanistan, Bulletin of the World Health Organization 86: Arur, A., Peters, D., Hansen, P., Mashkoor, M.A., Steinhardt, L.C. and Burnham, G. (2009) Contracting for health and curative care use in Afghanistan between 2004 and 2005, Health Policy and Planning 25: Campbell O.M.R. (2006) Strategies for reducing maternal mortality: getting on with what works, Lancet 368: Darmstadt, G.L. et al. (2005) Evidence-based cost-effective interventions: how many newborn babies can we save? Lancet 365: International Monetary Fund (IMF) (2010), IMF Data Mapper, available at Jha, P., Mills, A., Hanson, K., Kumaranayake, L., Conteh, L., Kurowski, C., Nguyen, S.N., Cruz, V.O., Ranson, K., Vaz, L.M.E., Yu, S., Morton, O. and Sachs, J.D. (2002) Improving the Health of the Global Poor, Science 295: Jones, G., Steketee, R.W., Black, R.E., Bhutta, Z.A., Morris, S.S. and the Bellagio Child Survival Study Group (2003) How many child deaths can we prevent this year? Lancet 362: Loevinsohn, B. and Sayed, G.D. (2008) Lessons from the Health Sector in Afghanistan: How Progress Can be Made in Challenging Circumstances, Journal of the American Medical Association 300 (6): Newbrander, W., Yoder R., and Debevoise, A.B. (2007) Rebuilding health systems in post-conflict countries: estimating the costs of basic services, International Journal of Health Planning and Management 22: Palmer, N., Strong, L., Wali, A. and Sondorp, E. (2006) Contracting out health services in fragile states, British Medical Journal 332: Peters, D.H., Noor, A.A., Singh, L.P., Kakar, F.K., Hansen, P.M. and Burnham, G. (2007) A balanced scorecard for health services in Afghanistan, Bulletin of the World Health Organization 85 (2): Rao, K.D., Waters, H. Steinhardt, L., Alam, S., Hansen, P. and Naeem. A.J. (2009) An experiment with community health funds in Afghanistan, Health Policy and Planning 24: Sabri, B., Siddiqi, S., Ahmed, A.M., Kakar, F.K. (2007) Towards sustainable delivery of health services in Afghanistan: Options for the future, Bulletin of the World Health Organization 85 (9): Strong, L., Wali, A. and Sondorp, E. (2005) Health Policy in Afghanistan: Two Years of Rapid Change, A Review of the Process from 2001 to 2003, London School of Hygiene and Tropical Medicine, London. The Asia Foundation (2009) Afghanistan in 2009: A Survey of the Afghan People, New York. UNICEF (2006) Best Estimates of Social Indicators for Children in Afghanistan, , Kabul. UNICEF, WHO, World Bank and UN Population Division (2007), Levels and Trends of Child Mortality in 2006: Estimates developed by the Inter-agency Group for Child Mortality Estimation, New York. United States Census Bureau (2010), International Data Base, available at USAID (2008) Evaluation of the Performance-Based Partnership Grants Project in Afghanistan, Washington. Waldman, R., Strong, L. and Wali, A. (2006) Afghanistan s Health System Since 2001: Condition Improved, Prognosis Cautiously Optimistic, Afghanistan Research and Evaluation Unit, Kabul. 54

67 WHO (2009) Global Tuberculosis Report, Geneva. WHO, UNICEF and UNFPA (2004), Maternal mortality in 2000: Estimates Developed by WHO, UNICEF and UNFPA, Geneva. 55

68 IBRD TAJI KI S TAN UZB E KI S TAN AFGHANISTAN TURKMENISTAN CHINA JAWZJAN KUNDUZ BADAKHSHAN TAKHAR BALKH SAMANGAN R YA B FA N BAGHLA SARIPUL NURISTAN GHOR N T HERA MA KABUL KABUL WA R D A K KUNAR GH PARWAN M YA N BA LA SA PI KA D G H I S BA NANGARHAR LOGAR P A K T I A G H A Z N I U R U Z G A N KOWST PAKISTAN H FA R A PAKTIKA ISLAMIC REPUBLIC OF IRAN BUL ZA AFGHANISTAN SUPPORT TO BASIC PACKAGE OF HEALTH SERVICES (BPHS) WORLD BANK-FINANCED (Performance-based Partnership Agreements with NGOs) HILMAND WORLD BANK-FINANCED (MOPH Strengthening Mechanism) R KANDAHA Z N I M R O USAID-FINANCED EUROPEAN UNION-FINANCED OTHER OR NO MAJOR DONOR Kilometers 100 Miles NATIONAL CAPITAL DISTRICT BOUNDARIES* PROVINCE BOUNDARIES* PA KISTA N This map was produced by the Map Design Unit of The World Bank. The boundaries, colors, denominations and any other information shown on this map do not imply, on the part of The World Bank Group, any judgment on the legal status of any territory, or any endorsement or acceptance of such boundaries. INTERNATIONAL BOUNDARIES *Administrative divisions based on 2003 data. SOURCE: Islamic Republic of Afghanistan, Ministry of Public Health. APRIL 2010