IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-H3930) ON A CREDIT IN THE AMOUNT OF SDR 24.3 MILLION (US$ 40.0 MILLION EQUIVALENT) TO THE

Transcription

1 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Document of The World Bank Report No: ICR IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-H3930) ON A CREDIT IN THE AMOUNT OF SDR 24.3 MILLION (US$ 40.0 MILLION EQUIVALENT) TO THE REPUBLIC OF CONGO FOR A CONGO HEALTH SECTOR SERVICES DEVELOPMENT PROJECT December 30, 2014 Health, Nutrition and Population Global Practice (GHNDR) Country Department (AFCC2) Africa Region

2 CURRENCY EQUIVALENTS Exchange Rate Effective December 1, 2014 Currency Unit = XAF XAF538 = US$1 US$ 1.45 = SDR 1 FISCAL YEAR January 1 December 31 ABBREVIATIONS AND ACRONYMS ACT Artemisinin-based Combination Therapy AFD Agence Française de Développement (French Development Agency) ART Antiretroviral Treatment BCG Bacillus Calmette Guérin BWMP Biomedical Waste Management Plan CBA Cost-Benefit Analysis CER Cost-Effectiveness Ratio CDMT Cadre de Dépenses à Moyen Terme (Medium-Term Expenditure Framework) CHW Community Health Worker CNLP Comité National de Lutte contre la Pauvreté (National Committee to Fight Malaria) COMEG Congolaise de Médicaments Esssentiels et de Génériques (Congolese Company of Essential Generic Medicines) CORDAID Catholic Organization for Relief and Development Aid CPS Country Partnership Strategy CSI Centre de Santé Integré (Integrated Health Center) CSS Circo-conscription Socio Sanitaire (Health District) DALY Disability-Adjusted Life Year DCA Development Credit Agreement DDS Direction Départementale de la Santé (Regional Health Directorate) DEP Direction des Etudes et de la Planification (Direction of Planning) DHS Demographic and Health Survey DO Development Objective DOTS Directly Observed Treatment, Short-course coverage DPHLM Directorate of Pharmacies, Laboratories and Medicines DTP Diphtheria, Pertussis, and Tetanus EEA External Evaluation Agency EPOS EPOS Health Management

3 ESMF EU FM GDP GOC HIV/AIDS HMIS HR HRH ICR IDA IMCI IOI IP IPPF IPT ISN ISR ITN LLITNs M&E MCH MOHP MSASF MTEF MTR MU MS NGO NHA NHDS NPV OI PAD PBF PCU PDO PDSS PER PHC PNDS Environmental and Social Management Framework European Union Financial management Gross Domestic Product Government of Congo Human Immunodeficiency Virus-Acquired Immunodeficiency Syndrome Health Management Information System Human Resources Human Resources for Health Implementation Completion Report International Development Association Integrated Management of Childhood Illnesses Intermediate Outcome Indicator Indigenous Peoples Indigenous Peoples Planning Framework Intermittent Preventive Treatment Interim Strategy Note Implementation Status and Results Report Insecticide-Treated Bed Nets Long-Lasting Insecticide-Treated Bed Nets Monitoring and Evaluation Maternal and Child Health Ministry of Health and Population Ministère de la Santé, des Affaires Sociales et de la Famille (Ministry of Health, Social Services and Family Welfare) Medium-Term Expenditure Framework Mid Term Review Moderately Unsatisfactory Moderately Satisfactory Non-Governmental Organizations National Health Accounts National Health Development Strategy Net Present Value Objective Indicator Project Appraisal Document Performance Based Financing Project Coordination Unit Project Development Objective Programme de Développement des Services de Santé (Heath Service Development Project) Public Expenditure Review Primary Health Care Plan National de Développement de la Santé (National Health Development Plan)

4 PNLP PLVSS PPA PRSP PSE QER PBF ROC RF RPF SDR TA TB TFR TSS TTL UN UNICEF US VCT VSAT WB Programme National de Lutte contre le Paludisme (National Malaria Control Program) Projet de Lutte contre le VIH/SIDA et de Santé (HIV/AIDS Control and Health Project Performance Purchasing Agency Poverty Reduction Strategy Paper Paquet de Services de Santé Essentiels (Package of Essential Health Services) Quality Enhancement Review Results Based Financing Republic of Congo Results Framework Resettlement Policy Framework Special Drawing Rights Technical Assistance Tuberculosis Total Fertility Rate Transitional Support Strategy Task Team Leader United Nations United Nations Children s Fund United States Voluntary Counseling and Testing Very-small-aperture terminal World Bank Vice President: Makhtar Diop Acting Country Director: Jan Walliser Practice Manager: Trina Haque Project Team Leader: Hadia Nazem Samaha ICR Team Leader: Paul Jacob Robyn

5 Republic of Congo CONGO HEALTH SECTOR SERVICES DEVELOPMENT PROJECT TABLE OF CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design Key Factors Affecting Implementation and Outcomes Assessment of Outcomes Assessment of Risk to Development Outcome Assessment of Bank and Borrower Performance Lessons Learned Comments on Issues Raised by Borrower/Implementing Agencies/Partners Annex 1. Project Costs and Financing Annex 2. Outputs by Component Annex 3. Economic and Financial Analysis Annex 4. Bank Lending and Implementation Support/Supervision Processes Annex 5. Beneficiary Survey Results Annex 6. Original and Revised Results Frameworks Annex 7. Stakeholder Workshop Report and Results Annex 8. Summary of Borrower's ICR and/or Comments on Draft ICR Annex 9. Comments of Cofinanciers and Other Partners/Stakeholders Annex 10. List of Supporting Documents MAP

6

7 A. Basic Information Country: Congo, Republic of Project Name: CG Rep. Health Sector Services Development Project ID: P L/C/TF Number(s): IDA-H3930 ICR Date: 12/24/2014 ICR Type: Core ICR Lending Instrument: SIL Borrower: Original Total Commitment: Revised Amount: XDR 24.30M Environmental Category: B Implementing Agencies: Ministry of Health and Population Cofinanciers and Other External Partners: B. Key Dates REPUBLIC OF CONGO XDR 24.30M Disbursed Amount: XDR 24.26M Process Date Process Original Date Revised / Actual Date(s) Concept Review: 11/08/2007 Effectiveness: 02/02/ /15/2010 Appraisal: 03/18/2008 Restructuring(s): 03/02/ /17/ /20/2014 Approval: 05/29/2008 Mid-term Review: 04/21/ /23/2011 Closing: 05/29/ /30/2014 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Borrower Performance: Moderately Satisfactory Moderate Moderately Satisfactory Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Unsatisfactory Government: Moderately Unsatisfactory Quality of Supervision: Moderately Satisfactory Implementing Agency/Agencies: Moderately Satisfactory Overall Bank Overall Borrower Moderately Satisfactory Performance: Performance: Moderately Satisfactory

8 C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Performance (if any) Potential Problem Project at any time (Yes/No): Problem Project at any time (Yes/No): DO rating before Closing/Inactive status: Yes Yes Moderately Satisfactory Quality at Entry (QEA): Quality of Supervision (QSA): None None Rating D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration Health Sub-national government administration 7 7 Theme Code (as % of total Bank financing) Child health Health system performance Malaria Population and reproductive health E. Bank Staff Positions At ICR At Approval Vice President: Makhtar Diop Obiageli Katryn Ezekwesili Country Director: Jan Walliser Marie Francoise Marie-Nelly Practice Manager/Manager: Trina S. Haque Lynne D. Sherburne-Benz Project Team Leader: Hadia Nazem Samaha Khama Odera Rogo ICR Team Leader: ICR Primary Author: Paul Jacob Robyn Paul Jacob Robyn

9 F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The development objective of the project is to support the strengthening of the health system in order to effectively combat the major communicable diseases and improve access to quality services for women, children and other vulnerable groups. Revised Project Development Objectives (as approved by original approving authority) The PDOs were not revised. (a) PDO Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Indicator 1 : % children under five years of age who slept under an ITN the previous night Value quantitative or Qualitative) Date achieved 12/30/ /31/ /30/2014 Comments Core indicator in PAD. Achieved. The indicator attained 135% of the target (incl. % value. achievement) Indicator 2 : Number of outpatient consultations for children under five years of age Value visits visits ( visits per quantitative or (0.625 visits per visits per capita per year) capita per year Qualitative) capita per year) Date achieved 01/01/ /31/ /30/2014 Comments (incl. % achievement) New indicator at restructuring. Achieved. The indicator attained 313% of the target value. The original target value was underestimated and in fact below the baseline value when converting to visits per capita per year. Indicator 3 : % Children fully immunized for Penta3 (DTP, Hep. B, Hemovirus) Value quantitative or Qualitative) 65% 70% 72% Date achieved 05/30/ /31/ /30/2014 Comments (incl. % achievement) Core indicator in PAD. Achieved. The indicator attained 103% of the target value. Indicator 4 : Children receiving a dose of vitamin A (%) Value quantitative or Qualitative) 66% 70% 78% Date achieved 05/30/ /31/ /30/2014 Comments (incl. % Core indicator in PAD. Achieved. The indicator attained 111% of the target value.

10 achievement) Indicator 5 : Percent births attended by skilled health personnel Value quantitative or Qualitative) 86% 90% 93.6% Date achieved 05/30/ /31/ /30/2014 Comments Core indicator in PAD. Achieved. The indicator attained 104% of the target (incl. % value. achievement) Indicator 6 : Number of persons tested of HIV ( Niari, Pool et Plateaux) Value quantitative or Qualitative) Date achieved 12/31/ /31/ /30/2014 Comments New indicator at restructuring. Achieved. The indicator attained 106% of the (incl. % target value. All department-level targets were met. achievement) Indicator 7 : Pregnant women receiving ART Value quantitative or Qualitative) Date achieved 12/31/ /31/ /30/2014 Comments (incl. % achievement) New indicator at restructuring. Not achieved. The indicator attained 9.7% of the original target value. The target value was largely overestimated due to an overestimate of health facilities offering ART services in the targeted zone. Indicator 8 : Number of persons under treatment Value quantitative or Qualitative) Date achieved 12/31/ /31/ /30/2014 Comments (incl. % achievement) New indicator at restructuring. Achieved. The indicator attained 161% of the original target value. The majority of the people under treatment were in Niari department, with coverage less successful in Pool and Plateaux. (b) Intermediate Outcome Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Indicator 1 : Long lasting insecticide-treated malaria nets distributed (number) Value (quantitative or Qualitative) Date achieved 05/30/ /31/ /30/2014 Comments (incl. % achievement) Core indicator in PAD. Achieved. The indicator attained 159% of the original target value. The bednets were distributed in the six original target departments and Brazzaville.

11 Indicator 2 : Health facilities renovated and equipped (number) Value (quantitative or Qualitative) Date achieved 05/30/ /31/ /30/2014 Comments (incl. % achievement) Core indicator in PAD. Achieved. The indicator attained 100% of the original target value with 32 health facilities being renovated and equipped with updated medical equipment. Indicator 3 : Survey use of bed nets - 2 surveys in 6 departments Value (quantitative or Qualitative) 67% % Date achieved 06/30/ /31/ /30/2014 Comments (incl. % achievement) New indicator at restructuring. No target ever defined. Only endline survey conducted. The survey found positive results - there was a 22% increase in coverage from baseline. Indicator 4 : Survey Household health (mini DHS) Value One survey Two surveys (quantitative No survey completed (0) completed (1) completed (2) or Qualitative) Date achieved 06/30/ /31/ /30/2014 Comments (incl. % achievement) New indicator at restructuring. Achieved. The indicator met its objective as only one survey was planned but two were completed (baseline and endline of impact evaluation. Indicator 5 : Health facility survey Value (quantitative or Qualitative) No survey completed One survey completed (1) Two surveys completed (2) Date achieved 06/30/ /31/ /30/2014 Comments (incl. % achievement) New indicator at restructuring. Achieved. The indicator attained 200% of the original target value as only one survey was planned but two were completed (baseline and endline of impact evaluation. Indicator 6 : Survey health status indigenous population OP 4.10 Value One survey (quantitative No survey completed completed (1) or Qualitative) One survey completed (1) Date achieved 06/30/ /31/ /30/2014 Comments New indicator at restructuring. Achieved. The survey on the health status (incl. % indigenous population was completed. achievement) Indicator 7 : Payments made to PBF pilot health facilities in 3 departments Value (quantitative or Qualitative) 0% 90% 93% Date achieved 06/30/ /31/ /30/2014 Comments (incl. % achievement) New indicator at restructuring. Achieved. The indicator achieved 103% of the original target value as 93% of payments have been made to PBF facilities as of May 2014.

12 Indicator 8 : Value (quantitative or Qualitative) New indicator at restructuring. Develop a national strategy for operational research and carry out 3 research studies 0 Report completed by December 2013 No report completed Date achieved 06/30/ /31/ /30/2014 Comments New indicator at restructuring. Not achieved. This activity was not conducted (incl. % due to budgetary deficits (fluctuating exchange rate between US$ and SDR). achievement) G. Ratings of Project Performance in ISRs No. Date ISR Archived DO IP Actual Disbursements (USD millions) 1 06/28/2008 Satisfactory Satisfactory /31/2008 Satisfactory Satisfactory /28/2009 Satisfactory Satisfactory /29/2009 Satisfactory Satisfactory /29/2009 Moderately Satisfactory Moderately Unsatisfactory /28/2010 Unsatisfactory Unsatisfactory /26/2011 Moderately Moderately Unsatisfactory Unsatisfactory /05/2011 Moderately Unsatisfactory Moderately Satisfactory /11/2012 Moderately Satisfactory Satisfactory /26/2012 Moderately Satisfactory Satisfactory /26/2013 Satisfactory Moderately Satisfactory /30/2013 Satisfactory Satisfactory /02/2014 Moderately Satisfactory Moderately Satisfactory H. Restructuring (if any) Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO IP Amount Disbursed at Restructuring in USD millions Reason for Restructuring & Key Changes Made 03/02/2012 N MU MS Change in project components 12/17/2013 N S MS Reallocation of funds + Extension of closing date 02/20/2014 N S S Reallocation of funds

13 I. Disbursement Profile

14

15 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Country Context. At the time of project preparation, the Republic of Congo (Congo) was recovering from a decade of civil war that left the majority of the citizens destitute and the country s infrastructure in shambles. Congo s social indicators were rather poor compared to countries with comparable income. Although the economic situation in the country had recently improved and Congo was recently classified as a lower-middleincome country, 50 percent of the population still lived in poverty, and the country remained IDA (International Development Agency) eligible. 2. Sectoral Context. Congo s population of 3.9 million was growing at an annual rate of 2.8 percent, and the estimated total fertility rate (TFR) of 6.3 was one of the highest in Sub-Saharan Africa. The average life expectancy was only 49 years with no significant increases for both sexes since Sixty to ninety percent of the morbidity in Congo was due to infectious diseases and maternal and child illnesses. In 2006, 55 percent of outpatient consultations in public hospitals in Congo were due to malaria alone. The infant mortality rate of 75 deaths per 1,000 live births and maternal mortality ratio 781 deaths per 100,000 live births were elevated for a country with significant resources. The HIV/AIDS prevalence rate of around 4.3 percent had remained stagnant for some years, although the tuberculosis (TB) incidence was rising over time. 3. The weaknesses in Congo s health system and the poor outcomes were adequately diagnosed in the Plan National de Développement Sanitaire (PNDS - National Health Strategy) that was prepared with support from technical agencies, as well as local and external partners and approved in The PNDS - while underlining the continued dominance of communicable diseases in Congo and the resurgence of malaria, TB and cholera as endemic problems also highlighted the following key problems: (i) high levels of malnutrition; (ii) gaps in leadership and organization of the sector at all levels in addition to absence of accountability; (iii) inappropriate management of human resources; (iv) poor quality of services (including recurrent stock outs of drugs and supplies); (v) poor infrastructure and lack of basic maintenance; (vi) weak monitoring and evaluation (M&E) with no feedback loops; and (vii) very weak regional and district health systems without community participation. 4. Paradoxically, the 2005 Demographic and Health Survey (DHS) found that the use of health services in Congo was relatively high. For example, the proportion of women delivering in the presence of skilled birth attendants was over 80 percent, indicating that the high maternal mortality could be indicative of poor quality of services. The skewed distribution of facilities was another important factor contributing to poor health outcomes in Congo, with over half of the health facilities being in the two major urban areas (Brazzaville and Pointe Noire). The civil war destroyed Congo s fragile health infrastructure and public health services and recovery had been slow. 5. Consequently, the government, in collaboration with national stakeholders and its external partners, embarked on the preparation of a new comprehensive PNDS for the period. Apart from IDA, other external donors had committed to support the 1

16 sector during the duration of the PNDS included the European Union (EU), AFD (Agence Française de Développement (French Development Agency) and the United Nations (UN) agencies. It focused on the need to improve the health status of the population in general and that of women, children and other vulnerable/marginalized groups in particular. The plan sought to strengthen primary health care through the district health system and community approach (Bamako initiative). The PNDS recommended a revision in approach, focusing on strengthening the performance of the health systems at all levels and developing integrated health programs to deliver a well-defined package of essential health services (paquet de services essentiels - PSE). This would be done within the decentralized approach already adopted for the public sector and by creating an enabling environment for private sector participation. This would require revision of the institutional framework in order to enhance communication and accountability at the central level and boost stewardship capacity for policy formulation, planning, coordination and support of the regional and district health management teams. 6. Country Partnership Strategy (CPS) and Rationale for Bank Involvement. The Government of Congo had initiated several activities that were indicative of a strong commitment to reforms. They included: a) satisfactory implementation of the Transitional Support Strategy (TSS, ) comprising the rehabilitation of basic small infrastructure, primary education and combating HIV/AIDS; b) elaboration of the Interim Strategy Note (ISN, 2007) that had two major themes: improving governance and strengthening access to service delivery for the poor. The World Bank supported the Republic of Congo through the CPS, which was approved in May The 2009 CPS, aligned with the full Poverty Reduction Strategy Paper (PRSP) approved by the Congolese Government in April 2008 envisioned a transition from projects to a more programmatic approach by strengthening Congo s fiduciary capacities and institutional arrangements for service delivery. 7. Recognizing the leading role the World Bank had in the health sector given its considerable experience in Africa, the Government of Congo sought IDA s leadership in stimulating and coordinating dialogue between authorities and stakeholders on health sector reforms. The Bank was also a key advisor in the preparation of the PRSP and an advocate. 1.2 Original Project Development Objectives (PDO) and Key Indicators 8. The Project Development Objective of the project was to support the strengthening of the health system in order to effectively combat the major communicable diseases and improve access to quality services for women, children and other vulnerable groups. Within the comprehensive sector development program of the PNDS, the IDA supported project would focus primarily on strengthening sector stewardship, fiduciary systems and monitoring and evaluation (Component 1); human resources (Component 2); infrastructure assessment, mapping and comprehensive rehabilitation plan, including limited construction (Component 3), and the delivery of a package of essential services, with emphasis on Malaria, HIV/AIDS and maternal and child health (MCH), pharmaceuticals management, community participation/indigenous peoples and equity (Component 4). 2

17 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 9. There was no revision to the PDO. The 2012 restructuring introduced substantial revisions to the Results Framework (RF) and key project indicators. Prior to restructuring, the project had a total of 37 indicators. At the time of the 2011 Mid-Term Review (MTR), it was highlighted that: many of the indicators were difficult to measure; the RF was burdensome and unrealistic to maintain given the weaknesses of the M&E and health management information system in Congo. For example, during the MTR, 24 of the intermediate outcome indicators had no data available at all. As such, at restructuring, the RF was substantially simplified, reducing the overall number of indicators, dropping the indicators that were challenging to measure, and introducing indicators better linked to the PDO and which would be able to be adequately measured through the PBF pilot and other newly introduced activities. 1.4 Main Beneficiaries 10. The main beneficiaries of the project were women, children and other vulnerable groups such as the Autochthone people in Congo. 11. Other beneficiaries included: (i) the Ministry of Health and Population (MOHP), which would benefit from support in collecting and analyzing data, developing health reforms (with a focus on decentralization), strengthening management of human resources for health, and defining a package of essential health services; and (ii) public and private health facilities, which would benefit from rehabilitation and equipment upgrades. 1.5 Original Components 12. Component 1: Strengthening leadership capacities in managing a functioning and decentralized health system (US$10.2 million: US$8.87 million IDA, US$1.33 million counterpart funding). This component would strengthen management and leadership capacities at all levels within the government s decentralization program. The process will involve strengthening capacities in planning and managing sector operations, working within a framework of partnership and stakeholder coordination under the following three sub-components. 13. Sub-component 1.1: Capacity building for leadership and effective management of a decentralized health system. Situation analyses would be done and the recommendations would be used to inform appropriate institutional reforms in the sector through creation of a new organogram. Specific attention was to be placed on capacity building for planning and management of the sector at all levels as well as strengthening the coordination of all stakeholders. 14. Sub-component 1.2: Strengthening fiduciary systems. A thorough assessment of the current financial and procurement systems identifying gaps was to be undertaken. Additional analytical works, including National Health Accounts, were to lay grounds for a Mid-Term Expenditure Framework (MTEF). The project would use the information to strengthen operational capacities for transparent financial management (FM) and procurement. Systems would be put in place to document all resources at all levels for equitable, efficient and a sustainable allocation of resources. Both internal and external 3

18 audit functions would be developed. In addition, studies would be conducted to identify an appropriate health sector financing system and policy for the Congo. Ministry procurement capabilities at the central, regional and district levels would also be strengthened. 15. Sub-component 1.3: Strengthening Monitoring and Evaluation. Through this subcomponent, a M&E system strategy and plan for the sector was to be developed and implemented to improve the availability and use of needed information by actors at multiple levels of the system. An operational plan to refine and support the scale-up of a national health management information system was to be developed and implemented. Epidemiologic surveillance would be strengthened, for epidemic-prone diseases and for supporting sentinel sites activities for tracking illnesses, insecticide-resistance, treatment efficacy and pharmaceutical co-vigilance, among others. Appropriate planning for periodic household and facility-based surveys would be ensured, and a health system operational research plan developed and implemented. Finally, support would be provided for periodic updates of service delivery maps. 16. Component 2: Institution of an efficient and effective system for managing health sector human resources (US$0.80 million: US$0.69 million IDA, US$0.11 million counterpart funding). A baseline human resources for health (HRH) database would be used to establish a separate HRH Directorate in the central MOHP with specific human resource management functions and the technical capacity to create position descriptions and build a repository of technical fields and professions employed throughout the health sector. The Directorate was to organize a multi-sector consultative framework for working with ministries of education which are responsible for training medical and paramedical staff to serve in the MOHP. Under this component, incentives were to be created to motivate staff in all areas. The project aimed to design and set up a mediumterm development plan for HRH. 17. Component 3: Rehabilitation and equipment of health facilities (US$2.30 million: US$2.00 million IDA, US$0.30 million counterpart funding). IDA would support the mapping and priority and norms/standards-setting exercise and assist the government in ensuring rational expenditure of their resources through phased rehabilitation of facilities, with special attention to the primary care level. 18. Sub-component 3.1: Infrastructure rehabilitation, maintenance and construction. Based on the health infrastructure inventory report, the project was to support the adoption of a rational comprehensive infrastructure rehabilitation plan and establishment of a standardized regular maintenance calendar for buildings and equipment. 19. Sub-component 3.2: Equipment standardization and maintenance. A full evaluation of the equipment situation in Congo was to be undertaken with IDA support in order to establish national equipment standards and norms and rational phased procurement plans. 20. Component 4: Improvement of access to a package of quality essential health services (PSE) (US$23.90 million: US$20.80 million IDA, US$3.10 million counterpart funding). This component, whose objective was to strengthen the organization and delivery of essential health services in the country, was comprised of four sub-components. 21. Sub-component 4.1: Define and provide a Package of Essential Services (PSE). 4

19 The PSE had already been defined with a clear content of services for children (Integrated Management of Childhood Illnesses - IMCI), mothers, adolescents to combat major communicable and non-communicable diseases. Maternal and child health outcomes and malaria were to receive special attention. IDA funds would be used to support the elaboration of norms, treatment protocols and service organization and training of staff at all levels of the health system. 22. Sub-component 4.2: Strengthen the procurement and efficient management of essential medicines and medical supplies. Operational capacity of the central medical store (Congolaise de Médicaments Esssentiels et Génériques - COMEG) was to be strengthened in order to undertake the expanded role of procurement of all pharmaceuticals and supplies according to the financing modalities in the agreement already signed between the Bank, EU and the Government of Congo. The Directorate of Pharmacies, Laboratories and Medicines (DPHLM) was also to be strengthened and new structures created to collaborate with the COMEG in monitoring pharmaceuticals supplies and stocks management. Quality control procedures would be established under this sub-component and qualified staff at the regional health directorates (Direction Départementale de la Santé - DDS) will be trained to enforce them. 23. Sub-component 4.3: Empower communities. This sub-component aimed to empower communities in their roles as co-managers of health services. It would entail building of professional capacity within the Ministry of Health, Social Services and Family Welfare (Ministère de la Santé, des Affaires Sociales et de la Famille MSASF, which was later renamed Ministry of Health and Population - MOHP) and decentralized levels to coordinate government-community partnerships and strengthen community participation in the management and delivery of health services and enhancing effective synergy between PSE and other determinants of health (water, environmental sanitation, vector management). 24. Sub-component 4.4: Promote equitable access to quality health services for all. Activities were to include an analysis of constraints to accessing services among the most vulnerable and poor segments of the population, and the establishment of solidarity measures to support basic needs of the poor. 1.6 Revised Components 25. During the March 2012 restructuring, components or sub-components were revised as follows: Table 1: Summary of key changes to project components, 2012 restructuring Components as in the PAD Change % disbursed as of 11/30/2011 Components in Restructuring Paper Rationale for change 5

20 Component 1: Strengthening leadership capacities of MSASF in managing a functioning and decentralized health system. 1.1 Capacity building for leadership and effective management of a decentralized health system 1.2 Strengthening fiduciary systems 1.3 Strengthening Monitoring and Evaluation Component 2: Development and implementation of an efficient and effective system for managing human resources for health Component 3: Rehabilitation and equipment of health facilities 3.1 Infrastructure rehabilitation, maintenance and construction. 3.2 Equipment standardization and maintenance Component 4: Improvement of access to a package of quality essential health services 4.1 Define and provide a Package of Essential Services (PSE) 4.2 Strengthen the procurement and efficient management of essential medicines and medical supplies 4.3 Empower communities 4.4 Promote equitable access to quality health services for all Modified Partially kept in new Component 1 Dropped Moved to Component 3 Remaining activities moved to new Component 2 Modified Moved to Component 2 Moved to Component 2 Partially moved to Component 1 Partially moved to Component 1 Dropped Dropped Partially moved to Component 2 39% Component 1: AccessComponent 1 was found to be (US$4.0 million to a package of high very ambitious with limited out of US$10.20priority health design details and had little million) services impact on the health system and on provision of essential services. The indicators in the M&E subcomponent were unrealistic and difficult to measure. Some activities were now incorporated in the new Component 1 (strengthening decentralization) and Component 3 (M&E). The sub-component on application of an appropriate FM system and medium term expenditure framework, in order to enhance the financing of the sector, was also dropped. 62.5% Component 2: (US$0.5 million Support to provision out of US$0.8 of priority health million) services 191% (US$4.4 million out of US$2.3 million) 30% (US$7.1 million out of US$23.9 million) Component 3: Monitoring and Evaluation and Project Management Mostly completed at time of restructuring. The completed and remaining activities were merged into the new component 2. Would become Component 2 to provide support to new Component 1and include rehabilitating and equipping of PHC centers covered by PBF, and safe disposal of biomedical waste in these facilities. Although having the largest amount of financing, Component 4 had the lowest disbursement ratio. Activities under Component 4 were narrowed and put under the new Component 1 and delivered through a PBF pilot. The Indigenous Peoples subcomponent activities were restricted to Indigenous Peoples health needs assessment under Component 2. The sub-component on improving the efficient procurement and management of essential medicines and medical supplies was dropped. 26. Component 1: Access to a package of high priority health services (US$

21 million). This component aimed to address the three health problems that account for the largest share of the burden of disease in the Congo: malaria, maternal and child mortality, and interrupting the transmission of mother to child of the HIV virus. It also included a Performance-Based Financing (PBF) pilot. The option to include a PBF pilot in the restructured project was chosen as an innovative strategy to improve health service delivery in targeted areas of the country. The rationale for selecting departments for the pilot (Pool, Plateau and Nairi) was based on the poor health outcomes of the populations in these areas, the poor quality of health services at facilities, geographical accessibility of the population and health facilities, and the financial envelope available for the pilot. 27. Malaria control sub-component (US$25,000): The project was to finance prevention activities, mainly Long-Lasting Insecticide-Treated Bed Nets (LLITNs) and artemisinin-based combination therapy (ACT). 28. Performance-Based Financing sub-component (US$3 million). The project was expected to implement a PBF pilot in three departments (Plateaux, Niari, Pool) over a twoyear period. The PBF pilot would involve the provision of a package of essential health services by contracted health facilities, who would receive performance bonuses based on the quantity and quality of verified health services. 29. Component 2: Support to provision of priority health services (US$5.045 million). This component aimed to provide the tools enabling the effective delivery of the high impact health services defined in Component 1 through the renovation and equipment of physical facilities, retraining of health workers and the implementation of the biomedical waste management plan in the health centers rehabilitated under the project covered by the PBF zones. Component 2 was also to support the finalization of the analysis of the data collected under the human resources management component from the previous component 2, which included the health workforce census as well as the health facilities master plan. 30. Component 3: Monitoring and Evaluation and Project Management (US$3.230 million). Component 3 was a new component to strengthen the Government s ability to track performance of its health sector, make adjustments, when needed, and determine whether activities undertaken were resulting in the anticipated health benefits. It also was intended to continue the technical assistance to the MOHP in implementing the project through the financing of a full-time project coordination unit. This component also included an assessment of the utilization of bed nets and a study to assess the health-seeking behavior and health needs of the indigenous people (Autochthone). 1.7 Other significant changes 31. The implementation arrangements were changed through a shift from ministerial management of the project to the creation of a dedicated project coordination unit. Prior to the creation of the project coordination unit, the Minister of Health and Population was the official coordinator of the project and was expected to manage the day to day activities of the project. In reality, the Minister could not fully assume the roles and responsibilities of a full-time project coordinator, and as such, the day-to-day activities were not effectively managed, thus leading to delays in activity implementation, disbursement etc. The intention of creating a full-time coordination unit was to improve efficiency and 7

22 effectiveness of project implementation. As such, qualified staff was hired, including M&E experts, fiduciary and procurement specialists, and public health specialists. The project coordination unit was attached to the Cabinet of the MOHP. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 32. Project preparation. The project identification mission was conducted in September 2007 and pre-appraisal mission in December During the Decision Meeting (March 17, 2008) the team was authorized to appraise, although several recommendations were noteworthy. First, given the status of the health sector it was felt that a strategic comprehensive approach was most appropriate. The project support would be largely for institutional reforms and capacity building (Component 1) and service delivery with special attention to malaria, maternal and child health and community-level actions. Second, the activities financed under Component 2 (human resources) were to support planned activities to reinforce human resources for health that were to be financed by the government and AFD, such as the preparation of the human resources for health strategy and support the Human Resources (HR) Directorate. Lastly, IDA support to infrastructure would be limited to facility mapping and prioritization of very limited rehabilitation in exceptional circumstances. The project appraisal mission was conducted from March 18-29, 2008 immediately after the Decision meeting. 33. The project was prepared in a timely manner and the PAD contained comprehensive and high quality background analysis on the projects that the health sector faced and systemic weaknesses that led to poor health service delivery outcomes. Through the four project components included in the PAD, the project was designed to address both systemic organizational challenges at the central level, as well as the need to increase decentralized decision-making for improved organization of health care at the periphery level. 34. On the other hand, the risk assessment may not have underlined enough gaps in leadership and organizational capacity that would later contribute to the limited progress achieved during the first phase of the project. While the institutional arrangements called for management of the national project by key staff within the Government, many of these key staff were rotated and changed during the first year of the project. As such, general knowledge, ownership and commitment among key ministerial staff were inadequate, as was their technical capacity to take on the procedural requirements of implementing a large and complex operation. The first few ISRs (Implementation Status and Results Report) of the project do not underline these challenges, and it was not until late 2009 that such issues were brought to management attention. It was not until early 2010 that concrete actions were taken to improve project performance and reorient key actions to achieve results. 35. Quality at Entry. No Quality Enhancement Review (QER) was conducted. 2.2 Implementation 36. The project was approved on May 29, 2008 and signed on July 18, However, it was only effective on February 2,

23 37. The effectiveness conditions for the project were as follows: (1) Establishment of a Procurement Unit within the MSASF: recruitment of international procurement expert and national procurement staff and provide initial training for national procurement; (2) Establishment in the MSASF of the financial management system including (i) the employment of the international FM expert and the Financial Director and the Treasurer, (ii) the adoption of the FM Procedures manual; (iii) the assignment of two internal auditors coming from the Inspection Générale des Finances to work in MSASF, and assumption of their functions; (3) Adoption of new organizational structures for the MSASF, establishing the new posts of Secretary General of MSASF, Director General positions, Head of Finance, Head of Planning and Research, Head of Human Resources and Head of Monitoring and Evaluation; and (4) Completion and adoption of the Project Implementation Manual satisfactory to IDA. 38. The four conditions of effectiveness were not fulfilled until January 2009, leading to a delay in the project becoming effective from the initial target date for effectiveness of September 30, As noted in the risk assessment during project preparation, government ownership and strong leadership were important elements to be in place to ensure the project s success. While efforts were made to move forward with implementation, a lack of stewardship from the ministry, coupled with a complex project design and a new approach to implementation arrangements for the country, progress was very slow during the initial project period. A replacement of key government actors that were involved in project preparation with new and less-informed staff also contributed to the delays. Table 2: Key dates in implementation of the PDSS project Date May 1, 2008 February 2, 2009 June 14-18, 2011 Late 2011 March 2, 2012 October 2012 December 17, 2013 February 20, 2014 December 2014 June 30, 2014 Key event Board approval of PDSS Effectiveness Mid-Term Review Creation of Project Coordination Unit Key Restructuring: introduction of Performance- Based Financing Launching of PBF in Pool, Plateau and Niari departments Restructuring: Extension of closing date Restructuring: Extension of closing date Board approval of PDSS II Closing of PDSS 39. While declared effective on Feb 2, 2009, project implementation did not begin until December 2009 (19 months after the board date). As a result, it was downgraded from Satisfactory to Moderately Unsatisfactory in December Given the substantial delays in implementation, the project should have been downgraded earlier than December In February and April 2010, two additional supervision missions were carried out but project implementation remained problematic. Overarching issues with disbursement 9

24 (below 6 percent), implementation, and rehabilitation remained. 40. While long effectiveness delays were common in the country, the key reasons for the delays and roadblocks were weak government capacity for project management, and insufficient human, material and financial resources. Regarding project management, the issues were numerous: weak coordination; limited understanding and ownership by the MOHP and development partners; administrative bottlenecks at the central level of the ministry (such as the changing of key staff in the ministry, limited skills and lack of training among administrative staff, and generally slow administrative procedures for implementation of activities); and limited incentives for the increased workload for staff tasked with project implementation. Regarding human, material and financial resources, insufficiencies abounded: continuous reshuffling of key staff; no provision of counterpart funds and only US$500,000 being provided as an advance for implementation, which may have not been sufficient to meet effectiveness conditions and move forward with project implementation. In addition, procurement and M&E plans were not produced on time due to staff turnover and limited training of key personnel. 41. The team began supporting the government by supervising the project in the field monthly and closely monitoring progress. In July 2010, important activities were launched such as: long lasting insecticide-treated bed nets had been procured, a contract for the DHS survey was signed and vehicles and equipment were being procured. The Minister of Health and Population remained the coordinator of the project but appointed a Deputy Coordinator (Coordinateur Délégué) on February This Deputy Coordinator reported to the minister and his functions were to coordinate and facilitate the implementation of the project with the technical departments and to coordinate the work of the international experts and local long term staff. The Ministry began to communicate to the staff at all levels within the Ministry about the project to enhance ownership. Table 3: Project design and implementation challenges prior to the Mid-Term Review, 2011 Challenges Example Low capacity for project management and implementation Weak leadership Changes in coordination: two coordinators and two delegate coordinators in first two years due to ministry staff turnover Limited understanding and ownership by the MOHP Insufficient dialogue/coordination with development partners development partners Lack of understanding and acceptance by central and departmental MOHP actors due to ministry staff turnover Administrative bottlenecks at the central level of Delay in creation of new organogram and related texts Inexistence of key governing documents (règlement intérieur) Delay in creating Direction of Administration and Human Resources and Direction of Organization and Evaluation 10

25 Delay in recruiting consultants Poor motivation Additional workload among key MOHP staff without additional compensation Poor working conditions for consultants (lack of office space, limited engagement from ministry staff, etc.) Insufficient human, material and financial resources Human resources Continuous reshuffling of key staff at central and departmental levels (for example, 80% of departmental health delegates that were trained in FM were replaced in two years) Financial resources No provision of counterpart financing Limited project preparation funds provided (US$ 500,000) Material resources Limited or no office space for project management, new directions and consultants Delays in preparing action plans and procurement plans Slow public procurement procedures Long delays in finalizing M&E plan and program Source: Rapport Final du Programme de Développement des Services de Santé ( ), Ministère de la Santé et de la Population 42. At the time of the May 2011 MTR, the project faced the following issues: (i) delay in implementation, (ii) slow disbursement, (iii) lack of data to assess progress made, (iv) poor M&E capacity, (v) no inventory of public facilities was available in order to move ahead with rehabilitation activities, and (vi) the package of essential services had still not been developed. 43. The MTR highlighted many of the growing problems with the initial design of the project. The key problems identified included: (i) limited governance capacity for management of the project, including buy-in and appropriation at all levels of the system; (ii) the project included a set of complex interventions that were dependent on high-level health system reforms (for which many had not seen much progress up to that point); and (iii) the implementation arrangements relied on a central-level Ministry of Public Health administration that was not accustomed to Bank operations and lacked the essential capacities and resources for project management. In addition, several key weaknesses of the project were highlighted: (i) inadequate staffing and lack of clarity about allocation of responsibilities, (ii) weak capacity, insufficiently trained staff, low motivation, and lack of authority in the coordination unit; (iii) insufficient knowledge and ownership of the project by some MOHP Directorates and provincial management; and (iv) an unmanageable RF with 37 indicators to measure. 44. Options discussed with the Government at the MTR in June 2011 included canceling part of the grant or formally restructuring the project together with a time extension of the closing date. However, the Minister of Health and the Country Director agreed on a series of actions that, if accomplished by October 2011 would lead the Bank to consider a restructuring of the Project and an extension of Closing Date rather than 11

26 canceling the project. These actions included: (i) distributing Long-Lasting Insecticide- Treated Bed Nets; (ii) engaging consultants to design a PBF pilot scheme; (iii) improving the functioning of the project management unit; (iv) decentralizing project activities and involvement of regional directors; and (v) a greater focus in the MOHP action plan on activities directly related to mother and child health. 45. The third option, to extend and restructure the project, was chosen, and the Bank recommended the following actions to remedy the observed challenges: (i) continue to support dialogue about civil service reforms and provide technical assistance if necessary; (ii) relocate the project coordination unit to a new space and provide adequate facilities, equipment, administrative support and budget to function and act independently; (iii) improve communication/advocacy about sector reforms and about the project s role to all levels of the Ministry; (iv) have technical staff in the field for service purchasing, training and monitoring agents before PBF pilot began; (v) distribute the bed nets by October 2011 with technical assistance provided by UNICEF (United Nations Children s Fund) and others as needed; (vi) discuss a revision of the RF and indicators prior to restructuring; and (vii) recruit two auditors by the MOHP to carry out auditing functions and ensure adequate control of the flow of funds from the central to departmental levels. As such, the project restructuring aimed to improve the project design by: (i) simplifying the project design by introducing high-impact activities with clear public health objectives; (ii) simplifying the RF and investing in the development of a strong M&E program, including the generation of reliable data through routine administrative data (through the Performance-Based Financing (PBF) intervention), punctual surveys, and an impact evaluation; and (iii) introducing a clearly-defined project coordination unit that would manage the day-to-day project activities. 46. These actions included: (i) distribution of the Long-Lasting Insecticide-Treated Bed Nets; (ii) engaging consultants to design the PBF pilot scheme; (iii) improving the functioning of the project management unit; (iv) decentralization of project activities and involvement of regional directors; and (v) a greater focus in the MOHP action plan on activities directly related to mother and child health. Given the considerable progress that has been made on these actions, a restructuring procedure was initiated in January By January 2012, the project implementation was progressing according to the action plan and disbursement was at 49 percent, up from 15 percent in March 2011 and 6 percent in November The project disbursed almost half of its funds in one year. Based on the progress made in one year, the accelerating rate of overall progress and the achievement of most of the goals agreed upon during the MTR, the project ratings were upgraded, the project was extended and restructured on March 2, During the restructuring several components and sub-components were modified to simplify the project. 48. In addition, the expected government counterpart financing of US$300,000 per year was never provided, leading to the cancellation of several planned activities. 49. After restructuring, implementation progressed more smoothly with only some minor delays in implementation. The PBF pilot began on schedule and was producing positive improvements in health service delivery, the rehabilitation of health centers was progressing, and investments under Component 3 were resulting in improved monitoring 12

27 and evaluation systems for the MOHP. During the May 2013 mission, the Bank and the counterpart agreed to a six month extension of the project closing date, leading to a new closing date of June 30, This extension provided sufficient time to complete all outstanding activities and allow for the PBF pilot to continue until implementation of the new Health System Support Project II which was expected to be effective on June 30, As the disbursement profile in Section I of the Data Sheets shows, the project s disbursements lagged behind the PAD s planned disbursements. In early 2012, only half of the US$40 million IDA grant had been disbursed and committed six months prior to the original closing date. Progress towards achieving the PDOs remained modest. While Component 4 (increasing access to a package of priority health services) was the main component of the project (in terms of financing envelope and), it remained the weakest performing activity and had the lowest disbursement percentage of all four components. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 51. The M&E system described in the PAD was comprehensive and appropriately responded to the weaknesses of the health sector information system in Congo. The subcomponents of the monitoring and evaluation system that was proposed included: i) routinely reported health management information, including facility-based information on service activities, illnesses and deaths, logistics management information related to tracking equipment and consumables (e.g. pharmaceuticals and bed nets), among others; ii) epidemiologic surveillance, particularly for epidemic-prone diseases and sentinel site activities (e.g. tracking illnesses/deaths, insecticide-resistance, treatment efficacy and pharmacovigilance); iii) periodic survey assessments, at both household and facility levels; and iv) health systems operations research, for example, to identify best practices. Most of these subcomponents were created and used during the project s lifetime, with the HMIS (Health Management Information System) and epidemiological surveillance systems being reinforced and annual statistical reports being produced, various surveys being conducted (health facility, household, DHS, maternal and child health, human resources for health, and indigenous peoples). Achievements in developing an operational research program were hindered by lack of project funds towards the end of the project. 52. The 37 indicators that were originally included in the project never had data collected for them (over 24 had no data at all by the MTR). The challenges in collecting information and estimating values for these indicators is highlighted in the first six ISRs and report from the 2011 Mid-Term Review and restructuring paper. Half of the PDO indicators were dropped at restructuring. As data generation and M&E systems were substantially strengthened at restructuring, data become more available and higher quality (including reinforcement of the HMIS system, generation of routine PBF data, several large-scale surveys and annual statistical reports, and an impact evaluation for the PBF pilot). 53. The PBF pilot included an evaluation component to contribute to the evidence on the approach s effectiveness in the Congolese context. To that effect, a quasi-experimental impact evaluation (before and after with non-randomized treatment and control groups) of the PBF pilot was designed and a small-scale household survey was planned at baseline and endline of the pilot in order to measure changes in outcomes of interest. The survey was carried out two years after the previous DHS with over-sampling in the PBF pilot 13

28 districts and matched comparison districts. 54. The M&E ratings in the Implementation Support Report (ISRs) varied over time, from Satisfactory during project effectiveness to Moderately Satisfactory in the first year, this was due mostly to the realization that there were far too many indicators for which data was not available and that a system was not yet in place to collect the data. During the December 2009 supervision mission, lack of progress on M&E was identified as an issue to be addressed, and it was noted that M&E activities were not being implemented due to the lack of capacity. By July 2010, the M&E rating were downgraded to Moderately Unsatisfactory. It became clearer as the project progressed that reliable data was not available and quality of information was a significant stumbling block to measuring results. Data used for project monitoring and the Results Framework in the post-restructuring phase were produced in a timely manner and of high quality as the majority of indicators were based on PBF data and thus verified and counter-verified. 55. M&E became an important component to tackle during the MTR and hence at restructuring, it became obvious that the existing M&E framework needed to be mainstreamed so as to ensure inclusions of indicators that could be monitored and measured during the years to come. Therefore, during restructuring, a revised RF with measureable indicators was developed. The new framework contained 16 versus 37 indicators with reliable baselines that were used to evaluate progress. Additionally, the project recognized the need for reliable data and for capacity building in this area. During restructuring a new component was developed that recognized the need to strengthen the Government s ability to track performance of its health sector, make adjustments when needed and determine whether activities undertaken are resulting in the anticipated health benefits. The new M&E component included support for the DHS as well as support to the Health Management Information System, strengthening epidemiological surveillance and initiating a new program of operational research. In the post restructuration phase, several key results were accomplished, including the production of HMIS tools, an annual statistical report, and thematic reports such as the DHS survey, a national health accounts report, and a report on the health needs of indigenous people. 56. After restructuration, indicators for the revised RF were being collected, submitted and validated on time. A full-time M&E expert was recruited for the project coordination unit, contributing to the quality and timeliness of data management. M&E ratings remained either MS (Moderately Satisfactory) or S (Satisfactory). 2.4 Safeguard and Fiduciary Compliance 57. Environment. The environment category of the project at the time of appraisal was B. The PAD identified weaknesses in the establishment and use of incinerators, the lack of adequate plans and/or internal management procedures (technical guidelines), the inadequacy of collection, storage and sorting of biomedical waste from household waste, and the lack of adequate and regularly-supplied individual protection equipment as issues to be addressed. Additionally, inadequate staff skills and behaviors in the biomedical waste management were also identified. The restructured project included technical assistance to implement the waste management plan at the health centers included in the PBF and those to be upgraded under Component 2. It also included provision of personal protection equipment, training of health staff, and distribution of information and awareness materials 14

29 at the health centers. 58. In the post-restructuring phase, the project conducted a diagnostic assessment of the health status and health needs of indigenous population and developed a targeted and realistic health plan to improve their health under Component 2. The Environmental and Social Management Framework (ESMF) was developed during project preparation and found acceptable. However, the national biomedical waste management plan (BWMP) for health facilities, which was part of the ESMF, was found to be too ambitious. During restructuring, it was decided that the project would only cover primary care facilities covered by PBF and those being rehabilitated under the project, but that the government had implementation responsibility for provincial and national hospitals and health centers not covered by the Project. Regarding resettlement, the assessment conducted during restructuring determined that the rehabilitation work on the health centers (the second phase) would not cause resettlement of people as the buildings already exist and will not be expanded, but will simply be upgraded and renovated within their existing footprint. Consequently, a Resettlement Policy Framework (RPF) was not needed as a result of the relatively minor adjustments at restructuring and the safeguards framework was not redisclosed. 59. Financial Management. In the initial project design prior to restructuring, the Direction of Financial resources, created in the context of new organizational structures of the Ministry of Health, coordinated all fiduciary aspects (budgeting, accounting, disbursement of funds and ensuring timely project auditing. Given the lack of training of personnel in accounting and FM and lack of experience with accounting, disbursement and auditing procedures for IDA-financed projects, fiduciary management was strengthened by recruiting international experts in FM, procurement, an internal auditor, financial controller and accountant. Prior to restructuring, ISR ratings for Financial Management varied between S and MS, with decreasing ratings over time. 60. In the post-restructuring phase, the project coordinating unit maintained sound FM system throughout the implementation of the project. The project staff that was hired had good experience in implementing IDA projects; having received several trainings on the use of World Bank fiduciary procedures as well as project software. Unaudited Interim Financial Reports were submitted on time, reviewed and found to be satisfactory. The external auditors issued clean audit reports and the management letter from the external auditors did not raise any major issues; there were no overdue audit reports and interim financial reports from the project. After restructuration, FM ratings began with MS and eventually improved to Satisfactory as actions agreed upon during supervision meetings were adhered to in a timely manner with satisfaction from the FM team. 61. The project had a FM manual which detailed out key internal control procedures from transaction initiation, review, approval recording and reporting. The manual was updated after the effectiveness as necessary as possible. There was a clear separation of duties within the FM unit. One Designated Bank account was opened in a commercial bank for the purposes of project implementation; the disbursement rate during the project closure was percent. There were no ineligible expenditures identified during supervision mission and external auditors. The overall FM system was rated as satisfactory during the final supervision missions. 62. Procurement. Prior to restructuring, ISR ratings for Procurement varied between 15

30 S and MS, with decreasing ratings over time. Post-restructuring, ISR ratings continued to improve and were rated Satisfactory towards the end of the project. All procurement plans were being executed as agreed upon and by following Bank guidelines. An in-depth review in ISR 11 highlighted some problems but they were quickly addressed by the team. Procurement plans were being prepared in time and submitted on time and large contracts such as the hiring of organizations related to the PBF pilot and evaluation, surveys and other operational activities were completed on time and without major delays or concerns. 63. Project Management. At the beginning of the project, technical assistance provided to implement the project has consisted of employing six international consultants and two national consultants. The international consultant in human resources for health had considerable trouble fulfilling his terms of reference due to (i) limited success in integrating into the MOHP; and (ii) limited engagement from ministerial counterparts. As a result, the consultant resigned and the post was not replaced. At the time of effectiveness two internal auditors were assigned to carry out auditing and financial control functions. However they resigned subsequently due to dissatisfaction with their working environment and two consultants were recruited and assigned to the project coordination unit to replace the auditors who left. These two national consultants were added to remediate the weaknesses in carrying out internal auditing functions and to ensure adequate control of the flow of funds going from central to departmental levels, i.e. an internal auditor and a financial controller. 2.5 Post-completion Operation/Next Phase 64. The PAD stated that the project would: (i) contribute to the PRSP objectives by supporting the MOHP s PNDS; (ii) establish planning and management capabilities at central regional, district, and health facility levels; and (iii) reinforce health service delivery for a package of essential maternal and child health services. Though the achievements related to reinforcement of management capacities at the central level remained modest at the time of project closing, most objectives, by project closing, had been achieved in a manner that provides a basis for the current new health project to being implemented (PDSS II). 65. The PBF pilot in the three departments produced significant results, and has paved the way for close-to-national scale up of PBF to improve utilization and quality of maternal and child health to over 80 percent of the population. The new health project is primarily financed by the Government (US$100 million counterpart financing), with the World Bank providing US$10 million in IDA and a US$10 million Health Results Innovation Trust Fund. 66. The Implementation Completion Report (ICR) mission s discussion of the postproject prospects identified challenges going ahead. First, while enthusiasm is extremely high at the department, district and health facility level, the lack of understanding and ownership of the PBF approach remains, however, limited by central-level actors of the MOHP. Second, as the new project is primarily financed by the Government, bottlenecks in making funds available may block the timely and effective disbursement for PBF payments. 16

31 3. Assessment of Outcomes 67. To facilitate the assessment of outcomes, the following discussion is linked both to the summary of ratings in Table 7 and to the specific results of the PDO indicators and Intermediate Outcome indicators in the Section F of the Data Sheet. 3.1 Relevance of Objectives, Design and Implementation 68. Relevance of the project objectives both before and after restructuring is rated Substantial. The objectives of the project were in line with diagnoses of the country s health sector and remained relevant to the country s national objectives as it supported the priorities of the PND. Project preparation involved: (i) significant sectoral research to identify the health system challenges in Congo and appropriate responses to these challenges; and (ii) extensive participation by the Government. As subsequently adapted to the more focused approach to improving health service delivery and efficiency in financing through PBF, these objectives remained relevant by continuing to address both the service delivery objective and the longer term policy development objectives (decentralization, HRH development, referral systems, incentive mechanisms). 69. After identification of the bottlenecks in achieving the project s development objectives, the 2012 restructuring aimed at introducing high-impact public health interventions and a robust M&E system. The project also fits within the framework of the Bank s policies, including the 2007 Interim Strategy Note, which supported the first ever PRSP (adopted in 2008). The PRSP aimed to: (i) improve governance and consolidating peace and security; (ii) promote growth and macroeconomic stability; (iii) enhance access to basic social services; (iv) improve the social environment and integration of vulnerable groups; and (v) combat HIV/AIDS. The subsequent CPS ( , ) also included specific objectives linked to strengthening the delivery of basic services through the use of Congo s large oil revenues. One of the key lessons learned from the CPS was the need to build the capacity of the public administration, including putting in place performance-based mechanisms. 70. Relevance of the project design and implementation both before and after restructuring is rated as Modest. The project principles remained relevant through the empowerment of MOHP services for the implementation of the project, and reliance on enhanced local capacity to manage the implementation of project initiatives and participate in the studies and evaluations. The components, focusing on improving stewardship of the health sector and the availability and quality of essential health services, with a focus on decentralization of decision-making, appropriately responded to the challenges that Congo s health sector was facing at the time of project design. In addition, the creation of a project coordination unit after the initial challenges in implementation was an important design adjustment to respond to the administrative weaknesses that were hindering implementation at the project onset. 71. However, based on the difficult early implementation experience, it was clear that (i) the design was too complex for the context with the large number of indicators being a sign of the complexity; (ii) there were too many small activities and those were not defined in operational terms (e.g. leadership strengthening ), thus the need now to focus on fewer 17

32 high impact priorities; (iii) the RF was overly complex with unrealistic indicators that were difficult to measure and relied on a weak information system and not enough attention was paid to M&E; and (iv) there was too much focus on workshops, training, and process measures that had little impact on the health system and the health status of the population. 72. The project restructuring simplified its design and focused on high impact activities that led to substantial improvements in overall implementation and results thanks to a simplified RF, strengthened M&E and revised implementation arrangements deemed more effective. The post restructuring redesign of the project greatly improved the relevance of the project design to best support high impact services to improve health status. To that effect, a strong focus was put in implementing a PBF pilot scheme (which was based on a feasibility study as to whether or not PBF schemes could be introduced in Congo), renovating and equipping health facilities and putting in place a robust framework for monitoring progress and evaluating outcomes including relevant national surveys. 73. Prior to restructuring, the objectives were overall in line with the existing political orientations and technical considerations, but the weaknesses in project design noted handicapped the translation of these orientations into effective actions. After restructuring, the project objectives and design were more substantially aligned. Considering the Substantial relevance of the Project Development Objectives and Modest relevance for the project design and implementation, the overall relevance is rated as Modest. 3.2 Achievement of Project Development Objectives 74. The project s efficacy rating is based on: (i) the project s results as summarized in section F of the Data Sheet; (ii) the quantitative results over the lifetime of the project for the Objective Indicator (OI), Intermediate Outcome Indicator (IOI) as presented in Table 4, and outputs in Annex 2; and the (iii) the project s weighted results as a proportion of actual disbursements before and after restructuring. The following table provides an overview of the results for the outcome and intermediate outcome targets in the data sheet and shows that 81% of the project s targets (13) were fully achieved, one was partly achieved and two were not achieved: Table 4: Summary of the achievement of project targets, Outcomes Intermediate outcomes Total Achievement No. % No. % No. % Achieved (95%+) 7 88% 6 75% 13 81% Partially achieved (36-0 0% 1 13% 1 6% 95%) Not achieved (0-36%) 1 13% 1 13% 2 13% Total 8 100% 8 100% % 75. Given the general lack of data on indicators in the pre-restructuring phase and significant strengthening of data generation in the post-restructuring phase as indicated in the M&E section of the ICR, the assessment related to the achievement of PDOs will largely focus on the indicators that were retained at restructuring and for which data is 18

33 available until the closing of the project (8 OIs and 8 IOIs). 76. Outcome Objective 1: Support the strengthening of the health system in order to effectively combat the major communicable diseases. 77. OI1: % children under five years of age who slept under an ITN the previous night (Core indicator in PAD). The DHS surveys in 2005 and showed that at a national level the percentage of children under five years of age slept under an ITN the previous night increased from 6% to 26.4%, although data for the DHS survey was collected during the PDSS ITN distribution campaign so does not register the complete results of the project s activities. The results of the 2012 bednets survey that covered six departments in the country showed that 80.9% of households had an ITN and coverage rates for 75.1 percent of children under 5 and 71.5 percent of pregnant women had slept under an ITN the previous night. With a baseline value set at 6% and a project target value of 60% coverage, achieving coverage of over 70% in the departments targeted by the mass distribution shows that the project was effective in improving coverage for use of bednets and in turn resulted in achievement of the outcome indicator. 78. IOI1: Long lasting insecticide-treated malaria nets distributed (number) (Core indicator from PAD). Over 700,000 bednets were distributed between October 2011 and March Given that the end project target was the distribution of 440,000 bednets, the target was largely met. 79. IOI3: Survey use of bet nets 2 surveys in 6 departments. Only one survey was conducted. No baseline survey was conducted and the endline survey was completed in 2012, twelve months after the 2011 mass distribution of bednets. Thus this indicator was only partially achieved. 80. In the post-restructuring phase, the project, under subcomponent 1.1 (Malaria Control) organized campaigns and distribution of long-lasting ITNs in 6 department (Pool, Plateaux, Sangha, Cuvette, Cuvette Ouest, and Likouala. To support these distribution campaigns, the project also organized media campaigns (radio and television) to educate the population on the benefits of utilizing long lasting insecticide treated bednets. 81. IOI2: Health facilities renovated and equipped (number) (Core indicator in PAD). In the pre-restructuring phase, the two sub-components of Component 3 supported rehabilitation, maintenance and construction of health facilities (3.1) and standardization of maintenance and equipment (3.2). The component intended to support the adoption of a rational comprehensive infrastructure rehabilitation plan and establishment of a standardized regular maintenance calendar for buildings and equipment. Counterpart funding was supposed to finance renovation of existing facilities and new construction undertaken only in exceptional cases. The Government of Congo and other partners were to provide the resources for setting up Maintenance workshops and to train bio-engineers to cover every region in the country. 82. The end project target was fully equipping and renovating 32 health facilities, which was achieved with 0 being completed in 2010, 11 being completed in 2011 and 22 in The majority of activities were completed in the post-restructuration phase. The PBF pilot also contributed to increasing the structural quality of health services in targeted facilities, including the availability of equipment and investments in renovation of these 19

34 facilities. Quality scores, based on an evaluation tool that uses over 200+ indicators to measure the structural quality of PBF facilities every three months, increased from 51 percent to 71 percent in Plateau Department, 41 percent to 62 percent in Pool, and 54 percent to 57 percent in Niari. 83. IOI4: Mini household health survey (mini-dhs) (New indicator introduced at restructuring). To measure the health status and health seeking behavior of the target populations in the three departments covered by the PBF pilot, a population-based household survey was to be conducted prior to the PBF rollout. The survey was completed by EPOS/Brandeis (contracted as the External Evaluation Agency for the pilot and impact evaluation) in November 2012 and again in April-May 2014 for the endline survey of the impact evaluation. As such, this intermediate outcome indicator was achieved. 84. IOI5: Health facility survey (New indicator introduced at restructuring). To measure the baseline performance and quality of care provided at health centers included in the pilot, the project aimed to conduct one health facility survey prior to the launching of the pilot. Two health facility baseline surveys were conducted, once in February-March 2012 by CORDAID (contracted to design and rollout the PBF pilot), and again in October- November 2012 by EPOS/Brandeis. In addition, an endline survey was completed by EPOS/Brandeis for the impact evaluation. As such, this intermediate outcome indicator was achieved. 85. IOI6: Survey on health status of indigenous population (New indicator introduced at restructuring). In order to better understand the health needs and health seeking behavior of indigenous populations in Congo and contribute to the development of an Indigenous Peoples Plan Framework, the project aimed to conduct a study on the health status of these populations. The survey was conducted in June 2013 and the report and indigenous peoples plan were disseminated and validated in November As such, this intermediate outcome indicator was achieved. 86. IOI7: Payments made to PBF pilot health facilities in 3 departments (New indicator introduced at restructuring). A key element of a well-performing PBF program is timely payment of subsidies to contracted providers, based on verification of reported results. As such, the projected set as an end target value that 90% of contracted health facilities would receive their payments on time. As of September 2013, 60% of facilities were receiving payments on time, increasing to 93% in May As such, the intermediate outcome indicator was achieved. 87. IOI8: Develop a national strategy for operational research and carry out 3 research studies (New indicator introduced at restructuring). The project aimed to initiate a national research strategy and finance activities related to the design and rollout of this strategy. This activity was not conducted due to budgetary deficits for the project as a result of fluctuating exchange rate between US$ and SDR. Despite this, other activities related to operational research were financed by the project, such as the national health map assessment and other thematic studies. 88. Outcome Objective 2: Improve access to quality services for women, children and other vulnerable groups 89. OI2: Number of outpatient consultations for children under five years of age 20

35 (New indicator introduced at restructuring). Data from the national health information system shows that approximately 194,826 under-5 consultations were conducted at a national level for an estimated coverage of visits per capita per year (based on an assumption that 20% of the 4.2 million population are children under 5). HMIS data shows that coverage increased over time, resulting in a coverage rate of visits per capita per year (394,061 visits) for outpatient consultations for children under 5 in Project activities that contributed to the increase in coverage for childhood outpatient consultations include the hiring of CORDAID/Memisa and the piloting of PBF in the departments of Pool, Plateau and Niari, the training of health cadres in PBF, increased monitoring and enhanced supervision and coaching of health care providers through PBF, and the increase in financial resources at the level of the health care provider through the PBF scheme. A total of 104 health facilities were contracted in the three pilot departments. Substantial improvements in the quantity and quality of health services were observed in the pilot areas, as detailed in Annex OI3: % Children fully immunized for Penta3 (DTP, Hep. B, Hemovirus) (Core indicator in PAD). The objectives of the project were to increase immunization coverage from 65 percent at baseline (both the national vaccination scheme in 2007 and the 2005 DHS survey report 65 percent coverage) to 70 percent coverage. Data from the national vaccination scheme shows that coverage increased to 80.9 percent in 2010, 87.7 percent in 2011, 84 percent in 2012 and 84 percent in The DHS survey also shows a substantial increase albeit slightly less, with coverage at the national level at 72 percent. 92. With an end target of 70%, both sources of data show that the project met the target for this outcome indicator and achieved its objective. Several activities contributed to the achievement. Several activities of the project contributed to this achievement. First, activities related to the support of decentralizing decision-making to the department- and district-levels of the health system provided opportunities to reinforcing logistics for the national immunization program, while the PBF pilot supported immunization activities at the provider and district levels by purchasing specific services related to immunization (the number of children fully vaccinated by a health center). 93. OI4: Children receiving a dose of vitamin A (%) (Core indicator in PAD). The objective also aimed to increase coverage for children receiving vitamin A supplementation to combat micronutrient deficiencies among children. For children, lack of vitamin A causes severe visual impairment and blindness, and significantly increases the risk of severe illness, and even death, from such common childhood infections as diarrhoeal disease and measles. At baseline the project reported coverage for this indicator at 66 percent (2005 DHS). According to the national immunization program, coverage increasing over time to 78.9 percent in 2010, percent in 2011, 78 percent in 2012 and 78 percent in The project had a target of 70 percent coverage for children receiving a dose of vitamin A. Statistics from national immunization program, whose activities include mass campaigns for the distribution of vitamin A among children, shows that coverage remained high throughout the duration of the project. In addition to activities financed under Component 4 related to the development and implementation of a package of essential health services, the PBF pilot also contributed to the increase in coverage over time with 21

36 vitamin A being a component of the essential package of services to be provided by primary care health centers. 95. OI5: Percent births attended by skilled health personnel (Core indicator in PAD). Increasing the percentage of women who deliveries are attended by a skilled health care provider was one of the main objectives of the project. Coverage increased substantially over the lifetime of the project. The 2005 DHS reported coverage at 86 percent (used as a baseline value for the project), while the DHS reported coverage at 93.6%. With a target value of 90%, the project met its end target value and even surpassed it. In addition to the rehabilitation of 32 health facilities to ensure that these facilities had high quality equipment and material for the provision of maternal and child health services, PBF statistics show that the number of births assisted by a professional health worker increased substantially in the three departments covered by the pilot (Niari, Pool and Plateau). During the first quarter of the PBF pilot (October-December 2012), 1,509 deliveries were purchased through the scheme, increasing to 2,169 in the last quarter of OI6: Number of persons tested of HIV (Niari, Pool and Plateaux) (New indicator introduced at restructuring). After restructuring, the project aimed to increase the number of people tested for HIV, introducing a new indicator to measure progress in the three departments included in the PBF pilot where subsidies were provided to facilities for the provision of voluntary counseling and testing. At baseline for the restructuring (2009/2010), the number of persons tested was 3,500 in Niari, 2,611 in Pool and 3,827 in Plateau. The target values for the outcome objective were to test 3,800 in Niari, 3,000 in Pool and 3,900 in Plateau, or 10,700 in total. The PBF pilot data reports that these targets were achieved, with a total of 11,246 persons being tested in the three departments between 2012 and OI7: Pregnant women receiving ART (New indicator introduced at restructuring). After restructuring, the project also aimed to increase the number of pregnant women receiving anti-retroviral treatment in the three departments in the PBF pilot. At baseline (2009 data) the numbers included 3 women in Niari, 6 in Pool and 11 in Plateau. The target values for these indicators were 255, 261 and 193, respectively. During the PBF pilot these numbers increased over time, from 17, 28 and 29 in 2011, respectively, to 21, 6 and 35 in , yet the targets were far from being met. 98. It was not until the health facility assessment survey was conducted that it was identified that not all the health centers contracted through PBF were in a position to offer these services. As such, the targets that were originally set were overestimated and the expected results for this indicator were not achieved. This led to inflation in the target values expected to be achieved by the closing of the project. Even though the target values were not met, it is worth noting that the number of women put on ART did increase between baseline and endline in the PBF zones. It can be hypothesized that the investments of the project in improving health service delivery in both the three departments and general health system strengthening activities at the national level contributed to these improvements. 99. OI8: Number of persons under treatment (New indicator introduced at restructuring). Finally, the project also aimed to improve coverage of HIV+ persons under treatment in the general population of the three departments in the PBF pilot zone. Baseline 22

37 numbers report a total of 712 people under treatment in the three departments in 2009, and the target value for the project was 2,216. Based on PBF data from 2012 and 2013, the target was met and surpassed, with a total of 3,571 people put under treatment While at the time of restructuring limited progress was made towards achieving the PDO indicators, after restructuring the project made good progress on all PDO indicators and even surpassed the target for the majority of indicators (see Table 4 and Table 5). The PBF pilot produced significant improvements in the utilization and quality of health services in the targeted areas. For example, the number of outpatient consultations (verified and purchased) increased from 16,460 in the period of October-December 2012 to 25,931 in the period of July-September 2013, and the average quality score increased from 35 percent at baseline (2012) to 56 percent by September 2013 (see Annex 3 for more information on the results of the PBF pilot). Table 5: PDO indicators that surpassed end project target values Project Outcome Indicators 1. % children under five years of age who slept under an ITN the previous night 2. Number of outpatient consultations for children under five years of age 3. % Children fully immunized for Penta3 (DTP, Hep. B, Hemovirus) 4. Children receiving a dose of vitamin A (%) 5.Percent births attended by skilled health personnel End Project Target End Project Results 60% 80.9% 0.2 visits per per capita per year visits per capita per year 70% 72% 70% 78% 90% 93.6% 101. It should be noted that a few indicators did not come close to meeting their targets. For example, the indicator pertaining to pregnant women receiving ART did not meet any of its targets in the three PBF departments Other achievements prior to restructuring. Although not measured by any of the OI or IOIs, the project implemented several activities that contributed to strengthening leadership within the Ministry of Health and Population. Prior to restructuring, under Component 1 the project: (i) facilitated the creation of functional procurement within the MOHP; (ii) the creation of administrative, financial and accounting manual, which is still applied today; (iii) managers at all levels of the system (central, departmental, and health district)) were trained in FM and procurement; (iv) the creation of two directions linked to M&E (the Direction of Organization and Evaluation and the Direction of Studies and Planning); and (v) the elaboration and validation of a National M&E Plan Under Component 2, the Direction of Human Resource Administration was created in An international expert was recruited to support this direction (but quit in 2011). It was identified that the new direction itself had a lack of human resources in terms of both 23

38 capacity and quantity, leading to limited effectiveness in its mission. The creation of special units for HR management at the Departmental level was not achieved, nor the national human resources for health policy. The majority of resources expected to be provided by AFD were never provided, also contributed to the limited achievements of this objective Under Component 3, a variety of equipment was provided to facilities across the country, including ambulances (5); general equipment provision; 20 vehicles for the central level; 14 for the department level, and 40 motorcycles for the CSS (Circo-conscription Socio Sanitaire - health district) level; very-small-aperture terminal (VSAT) and internet connection in all departments and central-level locations Finally under Component 4, results achieved: (i) validation of documents related to norms, standards, guides and protocols for the provision of the essential package of services; (ii) reorganization and strengthening of the Direction of Family Health; (iii) strengthening of child health services through routine and national campaigns for vaccinations; (iv) strengthening of maternal, neonatal and obstetrical care through the provision of essential equipment and materials to facilities providing these services; (v) strengthening of HIV/AIDS preventive and treatment care; (vi) strengthening of management of communicable diseases through updating and diffusion of various guides and protocols, distribution long-lasting insecticide treated bednets, extension of DOTS (Directly Observed Treatment, Short-course coverage), (vii) revision and updating of policies related to pharmaceutical management; and (viii) development and diffusion of policies related to community engagement Other achievements, post-restructuring. Under Component 2, Support to provision of priority health services, the activities related to the retention of health workers, which were to be financed by AFD were not achieved due to the inability to mobilize the resources committed by them. As such, only one activity, the finalization of the report on human resources for health, was completed. The activities related to strengthening community participation were also not implemented due to lack of financing for this component Under Component 3, Monitoring and Evaluation and Project Management, the project also produced new HMIS tools, the 2012 annual statistical report, the production of the national guide for integrated epidemiological surveillance, the evaluation of the National Health Development Strategy, the development of a database for the national health map, documents related to public-private partnerships, production of the national health accounts report, the DHS survey, and surveys related to urgent obstetric and neonatal care. Additional key project coordination unit staff were recruited, including a financial controller, an internal auditor, and a chief accountant Overall achievement result. Based on the project's overall results shown in the Data Sheet, the project s outputs presented in Annex 2, and the ICR guidelines (requiring separate outcome ratings weighted in proportion to the share of actual credit disbursements made in the periods before and after formal restructuring where key associated outcome targets have been formally revised), the following table assesses the project's overall efficacy rating is rated Modest before restructuring, Substantial after restructuring, and Substantial overall. Table 6: Combined overall project achievement ratings / Efficacy 24

39 Against Original PDO/Targets Against Revised PDO/Targets Considerations Rating value 3 5 Amount disbursed Overall Weight (% disbursed before/after restructuring) 40% 60% 100% Weighted value (1 x3) Final rating (rounded) Substantial 109. As indicated in Table 6 and Table 7, the restructuring and extension of the project contributed significantly to the achievement of the PDO. In addition, a number of the project s achievements are not captured in the results reported above, in particular the successful piloting of PBF in three departments leading to improved coverage and quality of health services and paving the way for a large-scale scale-up of PBF to be financed by the new health operation (PDSS II). 3.3 Efficiency 110. Technical efficiency. Annex 9 of the PAD based its economic and financial analysis on the proposed interventions by applying Cost-Benefit Analysis (CBA) by estimating the equivalent value of the benefits and costs to the society as a result of undertaking the project. The analysis was supported by an in-depth contextual analysis of health financing in Congo, including budget execution at all levels of the health system and health outcomes by socio-economic status. Based on this analysis, the project aimed to improve the allocative and technical efficiency of health financing by supporting the development and provision of a package of essential health services. To determine the economic viability of implementing the project, the project team conducted a CBA, which compared monetized benefits with costs and resulted in a positive Net Present Value (NPV) of $11,894,241,711. The results indicated that the project would be an efficient investment of resources, generating substantial benefits for the target population that outweighed the resources committed to the project. More information on this is provided in Annex The project resources have been allocated to maximize the welfare of the community. The project financed high impact interventions such as the distribution of LLITNs and provision of maternal and child health services in rural areas. Information and statistics on Congo s health sector are more readily available now due to the production of the annual health statistics report, several health facility survey reports, a health worker census report, and a report on the health needs of indigenous populations in Congo Given the resources used, the project implemented the most effective interventions by: (a) mass distribution of high-impact interventions such as LLTNs; (b) implementation of a PBF pilot in rural and underserved areas leading to substantial increases in the quality and utilization of maternal and child health services; and (c) strengthening the HMIS and data availability in the country for improved health sector planning and policy 25

40 development. LLTNs are one of the most cost-effective interventions in malaria prevention 1, and per capita budget for the Congo PBF pilot was less than US$3 per capita per year. Given that PBF programs make payments only after verification of performance, output-based investments in health service delivery are more likely to generate results more efficiently than traditional input-based financing methods where payment is not contingent on performance Performance Based Financing impact evaluation. A large-scale impact evaluation was conducted from to scientifically measure the effect of the Performance Based Financing intervention on key outcomes of interest, such as the coverage and quality of maternal and child health services. As the evaluation was completed just months prior to the closing of the project and provides a high quality, rigorous assessment of the efficiency and effectiveness of the project, the ICR team concluded that it would be more relevant to include results of the impact evaluation than to re-conduct a Cost-Benefit Analysis as was done in the PAD The evaluation was conducted in 5 departments (3 under PBF and 2 as control) and included baseline and endline health facility and household surveys. Difference-indifference analysis was applied to estimate the effects of the PBF program (see Annex 3 for details) The evaluation found statistically significant effects for several key indicators related to the quality and coverage of maternal and child health services. Using household survey data to estimate coverage, positive and significant effects were found for offering of HIV testing and receiving of HIV test results. Positive (but not significant) effects were found for increased utilization of certain maternal health services, such as: at least three prenatal consultations among pregnant women; increased coverage of post-natal care services; and use of family planning services. For child health, a positive (and significant) impact was found for coverage of child health services Positive effects were also found for child health services: children sleeping under long-lasting insecticide treated bednets and coverage of BCG (Bacillus Calmette Guérin, tuberculosis vaccine) among children aged months. The evaluation also found that children in the PBF zones were less likely to visit health facilities in case of illness PBF was also found to have positive effects on perceived quality of care. In particular, the impact evaluation showed that the perceived quality of care among patients using health facilities (beneficiaries) substantially improved in the PBF pilot zone when compared to the comparison zones. Statistically significant differences were found for: (i) increased provision of drugs; (ii) improved reception attitudes by health workers; (iii) improved hygiene of facilities; and overall quality Using data from health facility survey, the impact evaluation found substantial 1 A recent systematic review found the median financial cost of protecting one person for one year was $2.20 for insecticide-treated bednets (White et al., 2011). The study found that the median incremental cost effectiveness ratio per Disability Adjusted Life Year averted was $27 for bednets, $143 for indoor residual spraying, and $24 for intermittent preventive treatment of malaria. 26

41 increases in the provision of maternal and child health services such as institutional deliveries outpatient consultations; severely ill patients being referred from primary care to hospitals; children receiving Vitamin A supplements; and women receiving HIV testing The bonuses paid through PBF led to an increase in the quantity and quality of selected health services in health centers and district hospitals. Each additional unit of service delivered will lead to an improvement in the patient s health status. The maternal and child health services purchased through the PBF intervention are known to be cost effective interventions and for many of them, the evaluation found an increase in utilization and quality. For example, nutritional services for children aged 6 to 59 months have been found to have a cost-effectiveness ratio (CER - US$/DALY (Disability-Adjusted Life Year)) of US$41-43; a fully vaccinated child has a CER of US$296; and Integrated Management of Childhood Illness programs have been found to have a CER of US$218 (see Annex 3 for references) The evaluation also found that the financial contributions of the PBF subsidies to health facilities relative to subsidies and inputs provided by the Ministry of Health to these facilities was relatively small (ranging from 0.2 percent for hospitals to 13.5 percent for health centers), but still led to substantial gains in the coverage and quality of services. Complementary components of the PBF approach, such as coaching and training for improved facility management and social marketing, most likely also contributed to the significant improvements in health service delivery in PBF facilities The PBF intervention paid a total of US$1,883,259 in subsidies to health facilities and to decentralized regulatory structures (district and department health teams) over a 16 months period, to cover a target population of 767,215. This translates into an annual per capita investment in subsidies paid directly to health facilities of US$1.64 per capita per year. Relative to overall government spending in the health sector, at 1.7 percent of total GDP (Gross Domestic Product), and US$39 per capita per year National Health Accounts (NHA ), the PBF intervention can be seen as highly efficient when compared to results obtained through traditional government investments (as measured in the control areas of the impact evaluation) Details of the impact evaluation methodology and results are included in Annex Managerial efficiency. Managerially, the implementation arrangements during the project s initial phase, which sought to use internal government structures and methods with the Bank s fiduciary management procedures created leadership, coordination and disbursement problems from the beginning. In addition, the delay in implementation (2008), the slow start spending (2009), and the reshuffling of key ministerial staff included in the project development phase ( ) all contributed negatively to the project s efficiency. Finally, as can be seen in the volume of activities achieved since the restructuring of the project and the creation of a Project coordination unit, the revised implementation arrangements after project restructuring contributed significantly to the rate of disbursement and number of activities and objectives achieved Management efficiency. From a management perspective, the project had mixed results. During the pre-restructuration phase, the project dealt with systemic management and organizational weaknesses at the central level of the health system, leading to delays in implementation. While the Minister of Health was the project coordinator and central- 27

42 level directors key project team members, high turnover among these posts, coupled with a need for capacity reinforcement in project management, contributed to the limited achievement of results prior to restructuration The introduction of the project coordination unit and hiring of key technical staff (M&E, public health, procurement, financial management, etc.) to the unit that were dedicated to project implementation led to a dramatic improvement in managerial efficiency during the post-restructuration phase. This change in managerial efficiency can be observed in the level of activity that was maintained throughout the duration of the postrestructuration phase and the achievement of many of the projects indicator targets Based on the results of these dimensions of efficiency, the project s efficiency is rated at Modest before restructuring, Substantial after restructuring, and Substantial overall. 3.4 Justification of Overall Outcome Rating 127. Based on considerations of the various ratings criteria and indicators and disbursement prior to (40%) and after project restructuring (60%), Table 7 presents the overall outcome rating for the project as Moderately Unsatisfactory prior to restructuring, Satisfactory after restructuring, and Moderately Satisfactory overall. Table 7: Summary of project ratings Original project Restructured project Overall Rating criteria Relevance Modest Modest Modest Objectives Substantial Substantial Design/Implementation Modest Modest Efficacy Modest Substantial Substantial Efficiency Modest Substantial Substantial Overall MU S MS 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 128. The project contributed to the government s PRSP and targeted several of the poorer departments of the country with the piloting of the PBF scheme (Pool, Niari and Plateau). The PDOs were articulated around two groups known to be the most vulnerable: pregnant women and children under the age of five. The majority of the Project Outcome Indicators related to maternal and child health (PDO1, PDO2, PDO3, PDO4, PDO5) were all surpassed, resulting in an increase in coverage for key MCH services such as outpatient consultations for children, child immunization, vitamin A distribution, and skilled deliveries. By introducing interventions such as the distribution of bednets and PBF where indigenous people lived, the project contributing to improving the availability of essential preventive and curative health services. The project also produced a detailed assessment of indigenous peoples health-seeking behavior and health needs, which will assist the 28

43 Government in designing effective interventions for improving health outcomes among them. (b) Institutional Change/Strengthening 129. At the broadest level, the project achieved only limited success in advancing the reform agenda, particularly prior to restructuring where the major focus was using centrallevel institutional structures to implement reforms. In particular: (i) the project did not achieve the ambitious steps graduating toward a program approach, resulting in the creation of a project coordination unit anchored within the MOHP (leading to greater efficacy in attaining project goals); and (ii) the integration of project implementation into the Ministry s own structures or even the more limited step of transferring skills related to FM, procurement and human resource management was not fully achieved. Efforts to broaden the participation of technical and financial partners into the project achieved very limited success. The project steering committee did not meet regularly, and interviews during the ICR mission emphasized that Ministry leadership of the sector was insufficient The project achieved greater results in institutional strengthening after restructuring, with the introduction of the PBF pilot. The PBF pilot led to substantial reforms at the department, district and health service delivery level and paved the way for much larger institutional reforms that will be implemented in the new operation. The PBF approach is now fully-embedded in the MOHP s approach to improving results in health service delivery, supported by the substantial counterpart funding included in the new health operation (US$100 million). (c) Other Unintended Outcomes and Impacts (positive or negative) 131. While not a specific objective of the PBF pilot, the pilot experience has provided an opportunity for the Government to engage in a strategy for public-private partnership within the health sector. As both public and private health facilities were qualified to sign PBF contracts, the strategy has been found to be a mechanism to strengthen communication, coaching planning between the decentralized authorities of the Ministry of Health (districts and departments), and improve reporting by private health facilities. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 132. As an impact evaluation of the PBF pilot was conducted and results disseminated during the last month of project implementation during a stakeholder workshop, the results of the evaluation, including beneficiary perceptions of the PBF pilot, is presented in Annex 3. Overall, the evaluation found that the utilization and quality of essential maternal and child health services increased due to the PBF pilot, resulting in improved coverage of high quality services among the beneficiary population. 4. Assessment of Risk to Development Outcome Rating: Moderate 133. Several elements should contribute to reducing risks to the development outcomes; these include strategic measures for implementing a national health strategy focusing on results and value for money, oil revenues and available resources for the health sector, and improve management in human resources for health and information systems. The gains 29

44 the project accomplished in decentralizing decision making also reduce the risk to development outcomes. However, several risks identified in the PAD still exist: stewardship and ownership, continued administrative decentralization, and capacity for managing large-scale and complex operations While significant progress has been made and the successes of key interventions like the PBF pilot have led to large-scale reforms planned under the new health sector support project, PBF remains a new and technically challenging intervention that needs substantial implementation support over the short- to medium-term. The PBF pilot was implemented by an international NGO (non-governmental organization) with substantial experience in designing and implementing PBF pilots. The level of technical capacity and ownership of the MOHP to take over this promising reform and scale it up to cover the majority of the country (as planned in the new health operation) remains questionable. This is particularly important given the outcomes of the project design prior to restructuring, where a number of reform initiatives started under the project were abandoned (for example reforms linked to improve management of human resources for health). While the Directorate of Human Resources was created and remains functional, significant and continuous support will be needed Financially speaking, the risk to development outcome remains modest given the low cost per-capita of PBF (approximately US$3 per capita per year) and financial capacity of the Government of Congo. As such, the Government has committed US$100 million towards scaling up PBF to cover 84 percent of the population under the new Bank project. As such there seems substantial government commitment to scaling-up PBF, as was noted throughout the ICR interviews conducted by the ICR team. Despite limited understanding of the technical aspects of the approach by the majority of key informants interviewed, levels of enthusiasm were quite high for PBF. The opportunities for collaborations with other development partners for scaling-up PBF also remains to be seen. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Unsatisfactory 136. Preparation of the PDSS project was conducted relatively quickly and the PAD combined an abundance of ideas and specific activities for strengthening MCH services, backed by sound contextual analysis. The institutional frameworks were in-line with the approach to program implementation that were being introduced in the region at that time. Despite this, several important risks and challenges were not addressed and as such the institutional arrangements were not well-suited to the implementation of the project s initiatives The four conditions of effectiveness were not fulfilled until January 2009, leading to a delay in the project becoming effective from the initial target date for effectiveness of September 30, It was not until the fourth ISR in late 2009 and in early 2010, when supervision and implementation support activities were intensified, that concrete actions were taken to address (i) capacity, commitment and motivational challenges at the central level for implementation of the project, (ii) the overly-complex institutional reforms that 30

45 comprised certain components of the project (Component 1, 2 and 4), and (iii) the poorly designed Results Framework for measuring progress achievements The major consideration in rating quality at entry as unsatisfactory is that after one year of effectiveness (which was also delayed), little progress had been made in project implementation and severe weaknesses to the project design and implementation arrangements were identified. A Quality Enhancement Review, which was not conducted for the project, may have helped identify and address these issues. As such, challenges in implementation began immediately after Board approval and was not fully addressed until the Mid-Term Review several years later. (b) Quality of Supervision Rating: Moderately Satisfactory 139. At the beginning of the project, supervision mission were carried out on a regular basis. Although there were blockages in moving forward with implementation, the first four ISRs assess the project as Satisfactory for both DO and IP. Given the delays in effectiveness and then implementation, the early ISRs could have been used as an opportunity to inform management of challenges on the ground. These issues were not underlined in ISRs # In February and April 2010 two additional supervision missions were carried out but project implementation remained problematic; some progress had been achieved but the overarching issues with disbursement (below 6 percent), implementation and rehabilitation remained. After over a year with very little progress, the Bank team and Government felt the only way they would be able to see the project performance improve was to increase the level of Bank supervision and technical support to the project, by introducing an PBF pilot, investing more on M&E, strengthening the counterpart team and formalizing their role and ultimately restructuring the project during mid-term In retrospect, the Bank team could have been more proactive in addressing the initial problems by increasing the level of supervision and implementation support earlier on in the project, or conduct the MTR and eventual restructuration early on. The information provided in the first four ISRs was limited and did not point out the challenges the project was facing. It was not until ISR #5 in December 2009 that any implementation challenges were noted. The quality and comprehensiveness of ISRs began to improve after this point Once the problems were identified and the action plan was put in to place, the regular supervision missions allowed for the thorough treatment of the essential issues (technical, legal, fiduciary) by the Task Team Leader (TTL) and Task Team members, and the systematic review of actions taken on recommendations of the previous missions. With a couple of exceptions, from then on the supervision mission aides-mémoires were comprehensive, and the implementation status reports (ISR) were candid For the remainder of the project, supervision and implementation support were characterized by flexibility and proactivity as measures were planned early in the project to address the weaknesses in the institutional arrangements for project implementation and the insufficient data to monitor the project s progress. While project restructuring did occur somewhat late into the operation after substantial time without much progress in 31

46 implementation, the Action Plan that preceded the restructuring and subsequent restructuring allowed the project to substantially improve performance and led to an achievement of the majority of the project s objectives. After restructuring, each of the issues above was adequately addressed and the project went on to accomplish the majority of its objectives. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 144. Following ICR guidance indicating that, when ratings for the two dimensions are in different ranges (Unsatisfactory for quality at entry and Moderately Satisfactory for supervision), the rating for overall Bank performance depends on the outcome rating and is therefore rated Moderately Satisfactory. The rating is justified as while Bank performance for ensuring quality at entry was low and supervision during the first year could have been improved, the Bank s continuous technical assistance and field support from 2010 to 2014 contributed significantly to turning around a problem project to a successful project that paved the way for future health operations in the country. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Unsatisfactory 145. Significant ministerial changes and shortcomings in the institutional arrangements, weakened political and technical leadership for the project within the MOHP. The team of ministry directors that were appointed closely after project implementation began largely came from academic institutions and had little managerial experience. After the establishment of the new cabinet, the Minister took the lead to ensure better coordination, with a senior staff being appointed to coordinate the project. First, while the basic policies and directives for the sector were established in the PNDS, the daily operations to monitor implementation of the project were ineffective due to the preliminary institutional arrangements: (i) the role of project coordinator being filled by the Minister of Health and Population, who was unable to manage the day-to-day aspects of project management; (ii) a general lack of leadership and clear roles and responsibilities among the key government stakeholders at the central level; (iii) a poorly functioning steering committee; (iv) reform initiatives that were initiated but not completed; and (v) the lack of central-level support, which was particularly felt at departmental and district levels. On the other hand, through the piloting of PBF after restructuring, the project demonstrated that, by channeling a certain level of financial resources and providing autonomy at the operational level (district and health facility), highly satisfactory results can be obtained. The Government has recognized this and in turn the key feature of the new health operation is the scaling-up of PBF to close to national scale. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 146. Central-level Ministry management (pre-restructuring). Prior to the restructuring, the implementation approach aimed to use existing MOHP structures at the central level for coordination and execution of activities. As mentioned above, this 32

47 approach faced challenges (lack of leadership, lack of technical skills, high staff turnover, lacking clarity in roles and responsibilities. During the first few months of implementation with the newly formed project coordination unit (just before restructuration), the project coordination unit s performance was handicapped by the initial implementation arrangements of using central-level ministerial departments for project management. The modification of these arrangements, the recruitment of limited (but dedicated and competent) staff, and the enthusiasm of central and decentralized MOHP personnel for the PBF pilot contributed to the project s achievement of many of its expected results Project Coordination Unit (post-restructuring). In the post-restructuration phase the project coordination unit had responsibility for the overall implementation of the project; specific project implementation responsibilities were shared with the appropriate Ministry services for technical matters. The introduction of a Project coordination unit and effective project coordinator at the time of restructuring contributed significantly to the greater achievement of results during the post-restructuration phase. In the postrestructuration phase, the ISRs consistently rated project management, FM and procurement as satisfactory or moderately satisfactory. Assessment of implementation progress varied considerably, from moderately unsatisfactory to moderately satisfactory over the period, to moderately satisfactory or satisfactory starting in 2012 for the entire period of the restructured project. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 148. Following ICR guidance indicating that, when ratings for the two dimensions are in different ranges (moderately unsatisfactory for government performance and moderately satisfactory for the implementing agency performance), the rating for overall Borrower performance depends on the outcome rating and is therefore rated Moderately Satisfactory. 6. Lessons Learned 149. A preliminary political economy assessment may help identify appropriate institutional frameworks for project coordination. During the project preparation phase, a more in-depth political economy assessment could have potentially identified challenges in institutional capacity and the need to establish effective incentive frameworks for project management by the Ministry of Health. Weak government structures cannot lead reforms about improving governance and leadership. Relying on structures with limited capacity and a multitude of activities and priorities led to limited implementation success and achieving of results during the pre-restructuration phase. The high staff turnover (including the Minister of Health and Population), lack of full-time project coordinator, and weak management capacity (financial, procurement, HR) all contributed to the limited results prior to restructuration For large-scale, complex projects a dedicated team of professionals must be created for effective implementation. Having a dedicated project coordination unit can contribute to effectively implementing planned activities, having efficient management (financial, organizational, procurement) of project resources, and avoiding blockages caused by high ministerial staff turnover and limited motivation from civil servants who 33

48 wear many hats at one time. Projects should substantially invest in strengthening government counterparts and Bank teams should ensure that sufficient implementation support is provided at the onset of the project. In addition, when capacity is limited, it may be more prudent to begin with a simplified project design with modest objectives, and once capacity is strengthened, scope and complexity can be expanded Streamline resources to the beneficiaries. In situations where leadership and coordination remain weak at the central level, efficient and effective interventions that streamline resources to the decentralized level where services are provided to the target population should be introduced, such as Performance Based Financing Reliance on significant counterpart and development partner financing can impede achieving project results. The fact that co-financing from certain development partners for specific project activities was not realized (for example AFD and HRH activities under Component 2) led to limited progress and results achieved in certain areas. Equally important, the expected government counterpart financing of US$300,000 per year was never provided, leading to the cancellation of several planned activities. Future World Bank projects should ensure that co-financing is ensured before linking essential project components to non-bank financing sources (for example the substantial counterpart financing that is expected to be contributed in the new health operation Reinforce Monitoring and Evaluation systems through a two-pronged approach. Investing in the HMIS system through both strengthening routine data creation and management (annual statistical reports, data produced by PBF) and punctual largescale surveys (DHS, health worker survey, bednets survey, PBF impact evaluation surveys, etc.) can lead to higher quality data for tracking results and general system strengthening. Prior to restructuring, the RF was over-complex with approximately half the indicators remaining without data for the first two years. Through a restructuration that put emphasis on strengthening the M&E and health information systems and included an impact evaluation of an ambitious pilot project, the project was turned around into one with a realistic RF that relied on sound data and analysis while strengthening national systems. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies: 154. The task team and the Borrower reviewed and agreed on the results of the indicators reported in the Data Sheet. The Borrower has prepared a comprehensive final evaluation report in French. The Borrower s final evaluation report is available from the Project Files. (b) Co-financing: The majority of resources expected to be provided by AFD for strengthening human resources for health were never provided. (c) Other partners and stakeholders: Not applicable 34

49 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Appraisal Estimate at Estimate Components Restructuring Original credit (US$ millions) (US$ millions) 1. Strengthening leadership capacities in managing a functioning and decentralized health system 2. Institution of an efficient and effective system for managing health sector human resources 3. Rehabilitation and equipment of health facilities Percentage of Appraisal Restructuring (US$ millions) Actual/Latest Estimate (US$ millions) * % % % Improvement of access to a package of quality essential health % 4.0 services Physical contingencies 0.8 Price contingencies 2.0 Total Project Cost % 40 (b) Financing Source of Funds Type of Cofinancing Appraisal Estimate (USD millions) Actual/Latest Estimate (USD millions) Borrower IDA Grant Percentage of Appraisal 35

50 Annex 2. Outputs by Component Original Components Component 1: Strengthening leadership capacities in managing a functioning and decentralized health system Key achievements Sub-Component 1.1 Capacity building of leadership and management at all levels of the system Updating the organizational framework of the Ministry of Health Establishment of management bodies of the PDSS: inter-sectoral coordination committee and a development group for the health sector Setting up the framework for the program management at the departmental level Renovation and equipment of the office of Secretariat for coordinating the PDSS Recruitment, assignment and training of staff in accordance with defined job descriptions Technical support for the DEP (Direction des Etudes et de la Planification - Direction of Planning) Equipping of all planning units Recruitment of an expert in public health Formation of 7 cadres for national level planning and management Provision of vehicles at the central level and departmental levels to manage the CSSs in as part of Component 3 Supporting the function of management studies and planning Extension of the PDSS in all departments Routine supervision of DDS to CSS and CSS to health centers Sub-Component 1.2 Reinforcement of the fiduciary system Study the analysis of the situation of the FM of the health sector and the expenditures Launch of the national health accounts Organization of two training mission for the launching of national health accounts Restructuring of the steering committee and technical committee Recruitment of an international FM expert Recruitment of an international procurement expert Recruitment of an international financial controller Assignment of two delegates from Ministry of Finance to the Direction of Financial Resources to ensure the internal audits of accounts Put in place accounting tools and fiscal monitoring at all levels of the system (manual procedures, administrative, financial and accounting software and financial accounting systems) 36

51 Training staff in fiduciary management Organization of support of 2 missions for training in FM in each DDS (2010 and 2011) Production of an audit report for 2009 and two external audit reports in 2010 Nine financial reports transmitted to the World Bank Two Joint reviews with the World Bank Transmission of the financial report for the first trimester in 2011 Two technical supervisions being carried out by the World Bank Sub-Component 1.3 Strengthening Monitoring and Evaluation Recruitment of two international experts on M&E Analyses of the documentation of the health M&E system Rapid survey of the training capacity of the public health system to implement PDSS Development of a conceptual framework for M&E Evaluation of the situation of the HMIS Technical guidelines developed and updated for surveillance Reproduction of material for surveillance Dissemination of technical guides for surveillance Development of collection protocol Support for data collection Training of personnel responsible for data collection Component 2: Institution of an efficient and effective system for managing health sector human resources (HRH) Key achievements A situation analysis of the Human Resources for Health (conducted by INTRAHEALTH INTERNATIONAL) Creation of the Direction of Administration and Human Resources (Direction de l administration et des ressources humaines - DARH) in 2009 Allocation of a building at the Direction of Administration and Human Resources Staff in place as well as, office furniture and computer equipment Participation of the Director of Administration and Human Resources at the International Symposium on the Development of Competence in HRH held in Bangkok in November 2010 Assignment of 4 office managers and staff to support this component Component 3: Rehabilitation and equipment of health facilities Key Achievements Sub-Component 3.1 Infrastructure rehabilitation, maintenance and construction 37

52 Assessment of equipment based on 2 surveys conducted by INTRAHEALTH (2008) and MSP (2010) An inventory of health training Adoption of a national plan for rehabilitation of equipment and health infrastructure Staffing of all departmental and central directorates of the Ministry of Health and offices equipped with furniture and computer equipment Sub Component 3.2 Equipment standardization and maintenance Training of personnel for maintenance of infrastructure Training of personnel for maintenance of biomedical equipment Training of two officers from the Direction of Infrastructures, Equipment and Supplies to maintain biomedical equipment Component 4: Improvement of access to a package of quality essential health services (PSE) Key Achievements Sub-Component 4.1 Define and provide a Packet of Essential Services (PSE) Set up and offer the package of quality essential health services (PSE) Revision and updating of manuals and protocols of the PSE and standards for M&E of the service offered with a focus on delivery of quality services Adoption of standards and guidelines for the PSE Dissemination of the document outlining the PSE Strengthen the capacity of the staff at the integrated health center to implementation of the IMCI strategy Ensure the routine immunization and health information needed Ensure the activities in monitoring the development of the child Reinforce capacity for monitoring maternal and neonatal mortality Adopt a strategic framework to control cervical and breast cancer Hold biannual supervision meetings on the control of infectious diseases at the central level Reinforce the capacity of health staff of management of malaria cases Ensure accessibility and use of long lasting insecticide treated bednets in children under 5 years and pregnant women Ensure the preventive treatment of malaria for pregnant women Ensure the integration of the activities of management of TB through DOTS in 80 percent of integrated health centers identified for the delivery of the essential package of service Reinforce the capacity of health personnel for screening and diagnoses of Tuberculosis Train staff in research of these diseases 38

53 Organization of the treatment and care of HIV /TB coinfections for forms of multidrug resistant Tuberculosis by health facilities Assure the integration of prevention activities and the treatment and care for HIV/AIDS and support the integration of mental health within the minimum package of services in the integrated health centers Reinforce the capacity of the treatment and care of HIV/AIDS in each CSS Assure the integration of the treatment and care for schistosomiasis in the structure of the health system Organize the screening and mass treatment of children aged 5 to 15 years in areas with a prevalence> 20 percent (for schistosomiais) Ability to disseminate guidelines, and manage and control Preparedness for emergencies with sufficient protective materials, drugs and consumable inventory for response to major epidemics and natural catastrophes. Post Restructuring Sub-Component 1.1 Malaria Control Key Achievements Organize campaigns and distribution of Long Lasting Insecticide bednets in 6 department (Pool, Plateaux, Sangha, Cuvette, Cuvette Ouest, and Likouala) Organize media campaigns (radio and television) to educate the population on the benefits of utilizing long lasting insecticide treated bednets Sub-Component 1.2 Performance Based Financing Ensure financing for the activities of CORDAID Ensure financing for the activities of EPOS Recruit a consultant to train personnel at the DDS, CSS and other relevant health personnel on Performance Based Financing (PBF) Organize a training meeting for the DDSs, CSSs and other relevant health personnel to be trained on PBF Train 10 persons from the Health Ministry on PBF in Douala, and Cotonou in the international PBF course Organize a training on PBF in Brazzaville Develop an PBF manual Sub-Component 2.1 Rehabilitation and equipping of health facilities Key Achievements Rehabilitation of 32 Integrated Health Centers Equipping of 32 Integrated Health Centers with biomedical equipment 39

54 Organize the training in management of biomedical waste in the rehabilitated Integrated Health Centers Organize a meeting on sensitization to PBF Acquire 5 ambulances for the base hospitals Acquire 7 canoes for the target departments Acquire 7 motor boats for the selected departments Provide materials and equipment for mobile health clinics Sub-Component 3.1 Monitoring and Evaluation Train 395 health workers on how to work on the HMIS and fill out the proper information needed Duplicate the collection tools Provide the health infrastructure (DDS, CSS, hospital level and other operational levels) with the equipment, information and accessories necessary Provide the health infrastructure (DDS, CSS, hospital and other operational levels) the equipment for their offices Provide the health infrastructure (DDS, CSS, hospital and other operational levels) generators Ensure 24 missions from the central level to the DDS and CSS levels (12 missions were conducted in 2012) Sub-Component 3.2 Management of the Project Rehabilitation of the local PCU Equipping of the PCU with furniture for the offices Equipping the PCU with electronic equipment and accessories Equipping the PCU with an internet connection Providing the PCU with a generator Financing the activities of EPOS and CORDAID Ensuring that the consultants working in the PCU are properly remunerated Ensure the PCU has the support to provide the reports for supervision 40

55 Annex 3. Economic and Financial Analysis 1. Initial situation. Annex 9 of the PAD based its economic and financial analysis on the proposed interventions by applying Cost-Benefit Analysis (CBA) by estimating the equivalent value of the benefits and costs to the society as a result of undertaking the project. In order to determine whether the project is worthwhile, the project team conducted Cost-Benefit Analysis (CBA) by estimating the equivalent value of the benefits and costs to the society as a result of undertaking the project. A CBA is a procedure whereby costs and benefits of a project are identified, measured, and compared in monetary terms so as to generate net returns to a project s investment. 2. To determine the economic viability of implementing the project, the project team conducted a CBA, which compared monetized benefits with costs. Direct benefits are considered the benefit of avoiding or delaying the health care costs associated with treating diseases addressed in PDSS programs; and increased productivity from not being ill. To monetize project benefits, the project team characterized the costs of health care that can be delayed or avoided as a result of utilizing services supported by the project. The results indicated a positive Net Present Value (NPV) of $11,894,241,711, hence the project would generate the expected benefits. 3. Performance Based Financing impact evaluation. A large-scale impact evaluation was conducted from to scientifically measure the effect of the Performance Based Financing intervention on key outcomes of interest, such as the coverage and quality of maternal and child health services. As the evaluation was completed just months prior to the closing of the project and provides a high quality, rigorous assessment of the efficiency and effectiveness of the project, the ICR team concluded that it would be more relevant to include results of the impact evaluation than to re-conduct a Cost-Benefit Analysis as was done in the PAD. 4. The impact evaluation was conducted in 5 departments (Pool, Niari and Plateau where PBF was being implemented and Bouenza and Cuvette as control departments) and included baseline and endline health facility and household surveys. Difference-indifference analysis was applied to estimate the effects of the PBF program. The health facility survey sampled 73 health facilities across the five departments and visited the same health facilities for baseline and endline. The household survey sampled approximately 1350 households for each survey round through two-stage sampling. The same communities were visited for baseline and endline. 5. The evaluation found statistically significant effects for several key indicators related to the quality and coverage of maternal and child health services. Using household survey data to estimate coverage, positive and significant effects were found for offering of HIV testing (difference = +10.6%, p<0.01); and receiving of HIV test results (difference = +Positive (but not significant) effects were found for increased utilization of certain maternal health services, such as: at least three prenatal consultations among pregnant women (difference = +0.4%); increased coverage of post-natal care services (difference = +7.1%); and use of family planning services (difference = +6.8%). For child health, a positive (and significant) impact was found for coverage of child health services (difference = +10.2%, p<0.01). 41

56 6. Positive effects were also found for child health services: children sleeping under long-lasting insecticide treated bednets (difference = +3.3%, p<0.01) and coverage of BCG among children aged months (difference = +3.0%, p<0.001). The evaluation also found that children in the PBF zones were less likely to visit health facilities in case of illness (difference = -8.8%, p<0.001). 7. PBF was also found to have positive effects on perceived quality of care. In particular, the impact evaluation showed that the perceived quality of care among patients using health facilities (beneficiaries) substantially improved in the PBF pilot zone when compared to the comparison zones. Statistically significant differences were found for: (i) increased provision of drugs (difference = +14.2%, p<0.001); (ii) improved reception attitudes by health workers (difference = +10.8%, p<0.001); (iii) improved hygiene of facilities (difference = +11.1%, p<0.001); and overall quality (difference = +6.4%, p<0.05). 8. Using data from health facility survey, the impact evaluation found substantial increases in the provision of maternal and child health services such as institutional deliveries (difference = +3.1 services, p<0.001) outpatient consultations (difference = +24.7, p<0.001); severely ill patients being referred from primary care to hospitals (difference = +5.0, p<0.001); children receiving Vitamin A supplements (difference = +24.7, p<0.001); and women receiving HIV testing (p<0.001). 9. The bonuses paid through PBF led to an increase in the quantity and quality of selected health services in health centers and district hospitals. Each additional unit of service delivered will lead to an improvement in the patient s health status. The maternal and child health services purchased through the PBF intervention are known to be cost effective interventions and for many of them, the evaluation found an increase in utilization and quality. For example, nutritional services for children aged 6 to 59 months have been found to have a cost-effectiveness ratio (CER - US$/DALY) of US$ ; a fully vaccinated child has a CER of US$296 3 ; and Integrated Management of Childhood Illness programs have been found to have a CER of US$ The evaluation also found that the financial contributions of the PBF subsidies to health facilities relative to subsidies and inputs provided by the Ministry of Health to these facilities was relatively small (ranging from 0.2 percent for hospitals to 13.5 percent for health centers), but still led to substantial gains in the coverage and quality of services. Complementary components of the PBF approach, such as coaching and training for improved facility management and social marketing, most likely also contributed to the significant improvements in health service delivery in PBF facilities. 11. Financial analysis. The Economic and Financial Analysis included in the PAD noted that the direct benefits to be considered for this operation are the benefit of avoiding or delaying the health care costs associated with treating diseases addressed in PDSS programs, and increased productivity from not being ill. The Results Framework and impact evaluation of the PBF intervention showed that there were significant increases in 2 Jamison et al (2006). Disease Control Priorities in Developing Countries, p Ibid 4 DCPP (2008) Using Evidence About Best Buys to Advance Global Health 42

57 utilization of key maternal and child health services in the targeted areas at less than $2 per capita per year investment. The PBF intervention paid a total of US$1,883,259 in subsidies to health facilities and decentralized regulatory structures (district and department health teams) over a 16 month period, to cover a target population of 767,215. This translates into an annual per capita investment in subsidies paid directly to health facilities of US$1.64 per capita per year. Relative to overall government spending in the health sector, at 1.7 percent of total GDP, and US$39 per capita per year National Health Accounts (NHA ), the PBF intervention can be seen as highly efficient when compared to results obtained through traditional government investments (as measured in the control areas of the impact evaluation). 12. The project conducted other efficient activities and high-impact interventions, such as the mass distribution of insecticide-treated bednets. A recent systematic review found the median financial cost of protecting one person for one year was $2.20 for insecticidetreated bednets (White et al., 2011). The study found that the median incremental cost effectiveness ratio per Disability Adjusted Life Year averted was $27 for bednets, $143 for indoor residual spraying, and $24 for intermittent preventive treatment of malaria. 13. Process evaluation. The impact evaluation was accompanied by a process evaluation that capitalized on the lessons learned from the Congo PBF pilot. These lessons can be summarized as follows (Table 2A): Table 2A: Summary of process evaluation results, Republic of Congo PBF pilot Strengths and successes Reinforcement of health worker managerial and planning capacities (coaching and training of 162 health workers) Improved participation of key health sector actors in the planning, supervision and evaluation of health service delivery More regular and higher quality supervisions Improved access to health services (greater availability, reduced service fees, increased outreach activities, etc.) Greater financial and material resources for health facilities Substantial improvements in health service delivery Greater availability of health workers at health facilities Higher quality action plans at decentralized levels of the health system Improved management capacity at decentralized levels of the system Reinforcement of the role of community Weaknesses and challenges Insufficient integration of vertical programs into the package of services targeted by PBF Weak distribution networks for high quality essential medicines Insufficient and inefficient distribution of health workers in rural areas Insufficient and inefficient distribution of health facilities in rural areas 43

58 health committees in health facility planning and management 14. The process evaluation provided several recommendations for future PBF interventions in the Republic of Congo: 1. Strengthen health sector capacity building at all levels to ensure proper steering of PBF implementation by the Ministry of Health; 2. Ensure payments are made on time and that health facilities have sufficient autonomy to distribute bonuses based on performance; 3. When scaling-up, the pilot experience should contribute to adapting general PBF principles to the specific context of the Republic of Congo; 4. When scaling-up from the pilot phase, PBF should be integrated into the broader health care financing strategy in the Republic of Congo. 44

59 Table 3A: Summary of evaluation results, Republic of Congo PBF impact evaluation All Statistically significant Impact N Difference -in- Difference Positive Negative Positive Negative Maternal health Ratio of live births during 5 previous years (%) 2, % 1 Institutional deliveries 2, % 1 Cesareans 2, % 1 Any prenatal consultation (%) 2, % 1 At least 3 prenatal consultations (%) 2, % 1 Postnatal consultation (%) 2, % 1 Non-use of family planning methods 2, % 1 HIV/AIDS services Offered HIV tests (% of pregnant women) 2, %** 1 1 Use of HIV test (% of pregnant women offered) 1, % 1 Reception of HIV test (% of women tested) 1, %** 1 1 Child health services Children receiving nutritional consultations 2, % 1 Possession of bednets 3, % 1 Use of bednets among households with bednets 3, % 1 Utilization of child health services 1, %*** 1 1 Chidren aged months receiving BCG (%) % 1 Children aged 1-5 years receiving DPT3 (%) 2, %** 1 1 Quality of care Waiting time during last consultation (min) 2, Received drugs (yes=1, no=0) 2, %*** 1 1 Reception (% good reception) 2, %*** 1 1 Hygiene (% good hygiene) 2, %*** 1 1 Quality ((% good quality) 2, %* % 33% 29% 10% 45

60 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Lending Names Title Unit Responsibility/ Specialty Supervision/ICR Maurice Adoni Senior Procurement Specialist AFTPW Procurement Michael N. Azefor Sr Public Health Spec. AFTHE Public Health Monthe Bienvenu Biyoudi Senior Operations Officer AFCE1 Operations Nestor Coffi Country Manager AFMNE Country Operations Josyane E. Carmen Costa Program Assistant AFMCG Operations Nicolette K. DeWitt Lead Counsel LEGOP Legal Aissatou Diack Senior Health Specialist AFTHW Health Bourama Diaite Senior Procurement Specialist AFTPW Procurement Aissatou Diallo Senior Finance Officer CTRLA Disbursement Bella Lelouma Diallo Senior Financial Management GGODR Financial Specialist Management Adrien Arnoux Dozol Jr Professional Officer HDNHE Operations Astania Kamau Temporary MNSHD Operations Jean Jacques Frere Former TTL AFTHW TTL Gyuri Fritche Senior Health Specialist GHNDR Public Health Sariette Jippe Program Assistant GHNDR Operations Maud Juquois ETC GHNDR Health Economist Mohamed Ali Kamil Senior Health Specialist AFTHE Health Lombe Kasonde Operations Analyst HDNHE Operations Dieudonne Ndumbi Katende Information Analyst AFRIT Information Management Clement Tukeba Lessa Senior Procurement Specialist Kimpuni GGODR Procurement Amadou Konare Consultant LCSEN Safeguards Jean Charles Amon Kra Sr Financial Management Specialist AFTMW Financial Management Luc Lapointe HQ Consultant ST AFTPW Operations Mahamat Goadi Louani Senior Human Development AFTHW Human Specialist Development Josiane Maloueki Louzolo Team Assistant AFMCG Operations Clementine Maoungou Public Information Assistant AFRSC Communication Tazeem Mawji Consultant GHNDR Public Health Melisse Elizabeth Murray E T Consultant AFTHD Operations Hadia Samaha TTL GHNDR TTL 46

61 Social Yasmin Tayyab Senior Social Development Spec AFTCS Development Paul Jacob Robyn Health Specialist GHNDR Public Health (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY , Total: , Supervision/ICR FY , FY , FY , FY , FY , FY , Total: ,

62 Annex 5. Beneficiary Survey Results 1. As an impact evaluation was completed just before project closure, no beneficiary survey was conducted for the ICR. The impact evaluation found that perceived quality of care was substantially higher in PBF health facilities than comparison facilities. Annex 3 provides detailed results of the impact evaluation. 48

63 Annex 6. Original and Revised Results Frameworks Original project (from PAD): No. Indicators Baseline Target Component 1: Strengthening leadership capacities in managing a functioning and decentralized health system Sub-component 1.1: Strengthening leadership and management capacities at all levels of the decentralized system 1 Number of Steering Committee meetings held NA 12 2 Number of DDS/CSS who submitted a costed annual work plan NA 12 according to the guidelines by the deadline 3 % planned supervision missions conducted by NC/DDS/CSS to a 0 100% decentralized level Sub-component 1.2: Strengthening the financial management and procurement systems 4 % national budget allocation to health 6% 12% 5 % DDS and facilities receiving their budget in a timely manner NA - 6 % CSI/CSS/DDS with staff trained in budgeting and planning 0% 100% 7 % procurement plan activities completed on time 0% 100% Sub-component 1.3: Strengthening the monitoring and evaluation system (including routinely reported information, epidemiologic surveillance, periodic surveys, operations research) 8 % CSI/HR submitting activity reports on time 0% 80% 9 Number DDS who develop and submit trimesterly report % key indicators (to be specified) for which data is available at each 0% 85% level (CSS/DDS/NC) 11 % of EMONB and EMONC facilities conducting direct obstetric - 75% maternal and perinatal audits Component 2: Institution of an efficient and effective system for managing health sector human resources 12 % HR planned activities realized % facilities (by type) with HR according to norms (by type of personnel) - - Component 3: Rehabilitation and equipment of health facilities 14 % planned activities related to rehabilitation and equipment realized % of facilities (by type) meeting physical standards # facilities (by type) equipped according to norms for PSE Number of DDS that utilize the maintenance budget according to plan - 12 Component 4: Improvement of access to a package of quality essential health services Sub-component 4.1: Define and provide a PSE 18 Number of facilities offering PSE NA % births attended by skilled personnel 86% - 20 Number of new acceptors of modern contraceptive methods (by facility) NA - % pregnant women attending ANC services who receive full ANC NA - 21 package: (4+ visits, Iron, 2+ doses IPT, TT, etc) 49

64 % children under five with fever accessing an effective antimalarial 8.6% 50% 22 within 24 hours of onset of symptoms 23 % pregnant women placed on ART conforming to norms 60% 90% 24 Number of facilities offering VCT for PMTC Number of facilities implementing DOTS strategy TB cure rate 63% 85% 27 % households with at least two ITNs 4% 60% 28 % children under five who slept under an insecticide-treated bed net the 6% 60% previous night 29 % children 12 to 23 months of age who received DPT3 (urban/rural) 82.6% 90% (measured both through routine reporting and household surveys) 54.8% 90% 30 % children 6 to 59 months of age who received a Vitamin A supplement 71.0% - during the past six months (urban/rural) 60.8% - Sub-component 4.2: Strengthen the procurement and efficient management of essential medicines and medical supplies 31 % facilities with no stock outs of key medicines (to be specified) for TBD 90% more than two weeks during the period (3 months) Sub-component 4.3: Empower communities in their roles as co-managers of health service 32 % zones, rural quartiers and villages with at least one CHW trained and 0% 95% possessing a CHW Kit 33 % mothers/caretakers who know at least 3 danger signs that require NA - consulting a health facility 34 % children under five with diarrhea in the previous 2 weeks who 53.5% - received ORS or recommended home solution or increased liquids Sub-component 4.4: Promote equitable access to quality health services for all Number of CHWs of indigenous minority groups (pygmies) trained, given a Kit and providing care in the 3 DDS over the number planned % caretakers of children under five experiencing illness in the previous two weeks reporting cost as a barrier to seeking facility-based care % CSI/HR/CHU implementing a reorganized system for facilitating access to health care services by the poor 0% 85% NA - 0% 95% Restructured project (from Restructuring Paper): New or retained from Baseline Target No. Indicator PAD 1 % children under five years of age who slept under an ITN Core indicator 67% Maintain the previous night in PAD coverage 2 Number of outpatient consultations for children under five visits per New years of age capita per year 3 % Children fully immunized for Penta3 (DTP, Hep. B, Core indicator 65% 70% Hemovirus) in PAD 4 Children receiving a dose of vitamin A (%) Core indicator 66% 70% in PAD 5 Percent births attended by skilled health personnel Core indicator 86% 90% in PAD 6 Number of persons tested of HIV ( Niari, Pool et Plateaux) New

65 7 Pregnant women receiving ART New Number of persons under treatment New - - Intermediate Outcome indicators Long lasting insecticide-treated malaria nets distributed Core indicator 0 440,000 9 (number) in PAD Core indicator Health facilities renovated and equipped (number) in PAD Survey use of bed nets - 2 surveys in 6 departments New 67% TBD by 11 surveys 12 Survey Household health (mini DHS) New - Final report 13 Health Facility Survey New - Final report 14 Survey health status indigenous population OP 4.10 New No survey done Final report 15 Payments made to PBF pilot health facilities in 3 New 0% 90% departments 16 Develop a national strategy for operational research and New - Final report carry out 3 research studies 51

66 Annex 7. Stakeholder Workshop Report and Results 1. There was no stakeholder workshop held for this ICR. 52

67 Annex 8. Summary of Borrower's ICR and/or Comments on Draft ICR 1. Held from April 30 to May 31, 2014, this final evaluation mission of PDSS was entrusted to a mixed team of an international expert and a national expert with the objective to proceed with (i) an analysis of the relevance and performance of the project, (ii) a review of the results and impacts of actions taken, (iii) a summary of lessons learned in the program components and (iv) the suggestions and recommendations at the end of the setting work of PDSS. The methodology was based on a review of key documents, interviews with key informants at the central level health facility visits and coordination bodies in the departments of Brazzaville, Pool, Cuvette and Niari selected by stratification on the basis of their demographic, epidemiological and health coverage. The evaluation team reviewed - component by component - the degree of achievement of outcomes and impacts of actions taken by the PDSS, firstly, during the period from April 2008 to May 2012 (the "first period or period I ") and on the other hand, the period from June 2012 to June 2014 (the" second period or period II "). For each component, the analysis has consistently focused on achievements against expected results and discussed the constraints and challenges. 2. During the first period of the project, the PDSS project experienced multiple burdens with mixed results. The Bank's supervision mission conducted in June 2011 revealed a series of constraints, both structural and operational, to project implementation. These included an excessive delay in the implementation of planned activities; lethargic administrative procedures and mechanisms for planning, supervision, coordination and management; low estimated disbursement; lack of prioritization of activities and a lack of visible impact on access to care. 3. The mid-term review covering the period from February 2009 to May 2012 identified two major issues, namely (i) the low capacity for stewardship of the project (mainly due to weak leadership, low ownership actors and donors, a low level of motivation of staff involved and a low level of government structures and institutions,...) and (ii) insufficient human, material and financial resources. 4. Following the recommendations of the MTR, the Bank and Government decided to restructure the project to refocus and redefine the priorities, retaining only high-impact activities. To enable the project to better focus on priority areas, the number of program components was reduced from 4 to 3 components and the Results Framework was streamlined. The project management approach has been redesigned and experts were grouped in a "Project Coordination Unit (PCU)". 5. This restructuring allowed the project improve its performance and attain significant achievements, resulting in the launch of major rehabilitation of health facilities, the initiation of the Performance Based Financing pilot, and ultimately a substantial improvement in achieving project objectives. Despite the difficulties, the results obtained by the PDSS are satisfactory and constitute a major contribution to the national health system. 6. Overall, it appears that the PDSS has contributed significantly to: (i) strengthening the national health system (both central and decentralized levels), (ii) strengthening MOHP 53

68 directorates and health programs (especially those related to maternal and child health), (iii) strengthening the Community system (to a lesser extent), and (iv) through PBF motivating health care providers to provide quality care. 7. Two major factors are likely to contribute to the sustainability of the project s achievements. First, scaling-up PBF could help improve the quality of care and address the challenge of motivation. Second, ownership of the PDSS by the Congolese Government is shown by their contribution of US$100 million to the new health operation. 8. Overall, the Borrower ICR recommends: (i) strengthening oversight mechanisms of activities and coordination of partners, (ii) improving the volume, predictability and sustainability of financing, (iii) pursue capacity building at the decentralized level and strengthening human resources, and (iv) improve organizational planning of the national health map. 54

69 Annex 9. Comments of Cofinanciers and Other Partners/Stakeholders 1. None. 55

70 Annex 10. List of Supporting Documents Legal o PAD, Congo Health Sector Services Development Project (Report No: CG) (May 2008). o Restructuring Paper, Health Sector Services Development Project (Report No. No: CG) (February 2012). o Poverty Reduction Strategy Paper (PRSP), 2009 o Poverty Reduction Strategy Paper (PRSP), 2007 o Aides mémoires, and Implementation Status Reports Environmental and fiduciary o Cadre de planification en faveur des populations autochtones Managerial o Project implementation manual M&E o Rapport de la revue à mi-parcours du PDSS Brazzaville, juin 2011 o Superivsion/implementation reports, BTORs and Aide Memoires o Implementation/monitoring reports from Cordaid and EPOS Technical o Rapport Final Du Programme de Développement Des Services De Santé ( ). Juin, o Rapport d etat d avancement mi-parcours du programme de developpement des services de sante fevrier 2009-mai o Mise en oeuvre de l Approche de Financement Basé sur les Résultats dans les 3 départements pilotes (Plateaux, Pool et Niari) Par L Agence d Achat des Services de Santé AASS, (Powerpoint Presentation) o Evaluation finale du projet pilote de financement basé sur la performance (FBP) en République du Congo, Par EPOS/Brandeis, Mai 2014 (Powerpoint Presentation) o Evaluation finale du PDSS présentation des résultats préliminaires Par Dr Juma Kariburyo Mr.Amos Kalla (Powerpoint Presentation) o AASS/FBR/MSP/Rapport final d activités de la 2 ème phase du projet, avril 2014 o Aide mémoire mission de revue à mi-parcours du PDSS, 29 avril -08 mai 2013 o Annuaire des statistiques sanitaires 2012 du Congo, juillet 2013 o Document d évaluation du PDSS (version française et anglaise) o Document de Stratégie de Réduction de la Pauvreté (DSRP), Janvier 2007 o Document de Stratégie pour la Réduction de la Pauvreté Rapport d avancement annuel du FMI, Mars

71 o Document de Stratégique pour la Croissance, l Emploi et la Réduction de la Pauvreté (DSCRP), o Enquête de base sur l utilisation des services de santé, 2010 o Enquête Démographique et de Santé du Congo EDSC-II, o Etude sur les établissements sanitaires au Congo, PDSS 2010 o Plan National de Développement Sanitaire, o Plan National de Développement Sanitaire (PNDS), o Manuel d exécution de l achat des services de santé dans le cadre du FBR, Cordaid-Memisa, février 2012 o Rapport final d évaluation de la possession et l utilisation des moustiquaires imprégnées d insecticides à longue durée d action dans six départements en République du Congo, rapport final, juin 2013 o Rapport sur l état et les besoins des populations autochtones en matière de santé en République du Congo, PDSS

72 MAP 58