Document of The World Bank

Transcription

1 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Document of The World Bank FOR OFFICIAL UE ONLY IMPLEMENTATION COMPLETION REPORT (IDA PPFI-P9960) ON A CREDIT IN THE AMOUNT OF U$40 MILLION TO THE REPUBLIC OF NIGER FOR A HEALTH ECTOR DEVELOPMENT PROGRAM (HEALTH II) June 7, 2004 Report No: This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization.

2 CURRENCY EQUIVALENT (Exchange Rate Effective December 2003) Currency Unit = Franc CFA (XOF) XOF 525 = U$ 1 U$ 1 = XOF FICAL YEAR January 1 December 31 ABBREVIATION AND ACRONYM ADB African Development Bank AID Acquired Immune Deficiency yndrome CA Country Assistance trategy CDF Comprehensive Development Framework CFA Franc CFA (Communaute finaciere africaine) CHD Regional Hospital CI Centre de ante Integre DAC Development Assistance Committee DAF Directorate of Financial Affairs DCA Development Credit Agreement DD Departmental Health Directorate DEP Directorate of tudies and Planning DH Demographic and Health urvey DRP Regional Directorate Public Health DF/GP Directorate of Training and Personnel Management GDP Gross Domestic Product GTZ Deutsche Gesellschaft fuer Technische Zusammenarbeit HD District Hospital HIPC Heavily Indebted Poor Countries HNP Health, Nutrition, Population HQ Headquarters ICR Implementation Completion Report IDA International Development Association IMF International Monetary Fund MAP Multi-country AID Project MDGs Millennium Development Goals MOH, MOPH Ministry of Health M&E Monitoring and Evaluation MTR Mid-term Review ODA Oficial Development Assistance OECD Organisation for Economic Co-operation and Development ONPPC National Chemical and Pharmaceutical Products Bureau PAD Project Appraisal Document PD National Health ector Development Plan

3 PIU, PCU PMA PRC PRP PR QAG QER IP PA WAp TB, TBC TTL UNDP UNFPA WBI WHO, OM Project Implementation/Coordination Unit Minimum Package of ervices Poverty Reduction upport Credit Poverty Reduction trategy Paper Project tatus Report Quality Assurance Group Quality Enhancement Review ector Investment Program trategic Partnership with Africa ector-wide Approach Tuberculosis Task Team Leader United Nations Development Programme United Nations Population Fund World Bank Institute World Health Organization Vice President: Country Director ector Manager Task Team Leader/Task Manager: Callisto E. Madavo Pedro Alba Alexandre V. Abrantes Markus Repnik

4 NIGER HEALTH ECTOR DEVELOPMENT PROGRAM (HEALTH II) CONTENT Page No. 1. Project Data 1 2. Principal Performance Ratings 1 3. Assessment of Development Objective and Design, and of Quality at Entry 2 4. Achievement of Objective and Outputs 5 5. Major Factors Affecting Implementation and Outcome ustainability Bank and Borrower Performance Lessons Learned Partner Comments Additional Information 22 Annex 1. Key Performance Indicators/Log Frame Matrix 23 Annex 2. Project Costs and Financing 27 Annex 3. Economic Costs and Benefits 29 Annex 4. Bank Inputs 30 Annex 5. Ratings for Achievement of Objectives/Outputs of Components 34 Annex 6. Ratings of Bank and Borrower Performance 35 Annex 7. List of upporting Documents 36 Annex 8. Project Objectives, Components, and ubcomponents 38 Annex 9. Regional cope: Choice and Expansion of Districts 39 Annex 10. Insufficient Clarity of Objectives 40 Annex 11. Assessment of Project Outcome and Output Indicators 41 Annex 12. Niger Health ystem - A ystem in a Precarious ituation 42 Annex 13. Data upporting Assessment of Niger's Health ystem 43 Annex 14. Degree of Functioning of Health Infrastructures Visited During ICR Mission 47 Annex 15. Problems urrounding Health Care taff 48 Annex 16. Unsatisfactory Efficiency Infrastructure ubcomponents 49 Annex 17. Outputs and Activities Planned/Realized 51 Annex 18. Translated ummary Government Report 52 MAP() IBRD 27872

5 At the end of the day only results in the field count. We have an obligation to be more systematic about identifying what gets results, intensifying our support for partnerships that deliver results, and measuring and reporting on these results. James Wolfensohn The increased international focus on development results is of key importance if the MDGs are to be achieved by Countries need to focus on results in order to design better strategies and make better policy decisions. The World Bank and other development agencies need to focus on results in order to provide the support that makes the difference in the lives of people in client countries. Given this importance, the ICR tried to assess the project from a strong results perspective. It is acknowledged that applying today s knowledge and focus on results in judging a project designed in the mid 1990s might lead to unfavorable ICR ratings, despite the demonstrated level of efforts by the Government and the Bank.

6 Project ID: P Team Leader: Markus Repnik Project Name: Health ector Development Program TL Unit: OPCC ICR Type: Core ICR Report Date: June 7, Project Data Name: Health ector Development Program L/C/TF Number: IDA-29150; PPFI-P9960 Country/Department: NIGER Region: Africa Regional Office ector/subsector: Health (94%); Central government administration (6%) Theme: Health system performance (P); Decentralization (P); Other communicable diseases (); Participation and civic engagement () KEY DATE Original Revised/Actual PCD: 04/26/1994 Effective: 01/06/ /29/1997 Appraisal: 03/12/1996 MTR: 05/08/2000 Approval: 09/05/1996 Closing: 06/30/ /31/2003 Borrower/Implementing Agency: Other Partners: GOVERNMENT OF NIGER/MIN HEALTH TAFF Current At Appraisal Vice President: Callisto E. Madavo Jean-Louis arbib Country Director: Pedro Alba Theodore Ahlers ector Manager: Alexandre V. Abrantes Helena Ribe Team Leader at ICR: Djibrilla Karamoko Denise A. Vaillancourt ICR Primary Author: Markus Repnik; Remo Meloni 2. Principal Performance Ratings (H=Highly atisfactory, =atisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, U=ubstantial, M=Modest, N=Negligible) Outcome: ustainability: Institutional Development Impact: Bank Performance: Borrower Performance: U UN M U U QAG (if available) Quality at Entry: Project at Risk at Any Time: No ICR U

7 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 Original Objective: As stated in the Development Credit Agreement, signed in eptember 1996, the objective of the U $40 million IDA credit was to assist the borrower in improving the health condition of its population, through the improvement of (i) quality and coverage of basic health services (ii) access of the population to essential generic drugs; and (iii) efficiency of agencies operating in its health sector through strengthening management at central government and local levels, and through the participation of the private sector and the population in the delivery of health care services. The project was designed as a ector Investment Program (IP), intended to make a significant contribution to the National Health ector Development Plan (PD) (U $40 million to a planned total of U $275.5 million for the period). IDA assessed that quality and detail of the PD provided sufficient basis to opt for a IP approach. The PD was complemented by the national health sector policy that aimed at ensuring continued improvement of the population's health through (i) decentralization of basic health care services; (ii) increased efficiency of preventive health care; (iii) rationalization of health services management; (iv) introduction of partial cost recovery; (v) increased accessibility and service quality at all levels; and (vi) effective inter-sectoral collaboration to solve environmental problems. The national health sector policy and PD, with support from the project, intended to turn the health system around toward basic health care by empowering districts. The project was expected to have a special focus on the poor and vulnerable groups by (i) significantly improving the availability and affordability of essential generic drugs for the poor; and (ii) identifying districts for improving service coverage. 3.2 Revised Objective: The project objective was not revised. 3.3 Original Components: The project consisted of three components: (i) improving the quality and coverage of basic health services (79% of total planned costs); (ii) improving the access of the population to essential generic drugs (9% of total costs); and (iii) improving sector effectiveness and efficiency by building capacity and forging partnerships in support of health sector reform (12% of total costs). The project components had a national and/or regional scope. The regional project area was defined as five districts in two regions in the eastern part of the country, namely Diffa (districts of Diffa and Maine-oroa) and Zinder (districts of Goure, Tanout, and Mirriah). The DCA explicitly stated that additional districts might be added during project implementation. The relative importance of the three project components was not stated, but based on the original budget allocation, significant importance was given to the construction (rehabilitation, expansion, new construction) and purchase of equipment of the health care infrastructure within component one

8 Annex 8 provides a more detailed overview about project objectives, components and sub-components including the respective regional scope. 3.4 Revised Components: The project components were not revised, but major changes were made through a significant expansion of the regional project area. The scope of the project was expanded from five to 14 districts without increasing the project budget, enlarging the project area by more than 260% in terms of targeted population reaching about 40% of Niger s overall population. The expansion of the project area occurred in three phases: (i) In October 1996, one month after signing the Development Credit Agreement (DCA), two districts were added. These districts should have been supported by The Netherlands, but given their withdrawal after the coup d etat of January 1996, the districts were left without external support; (ii) In eptember 1997, it was agreed to add six additional districts. This decision was linked to the closure of IDA s population project (IDA Cr NIR) which left several district hospitals unfinished and unequipped (see section 7.2); (iii) In November 1998, IDA agreed to add Nguigmi as an additional district so that all districts of the Diffa region were covered by the project. These major revisions were not approved by the World Bank Board as the relatively broad DCA stated the possibility of the inclusion of additional districts. Annex 9 provides a more detailed overview about the choice and expansion of districts. 3.5 Quality at Entry: Unsatisfactory. At the time of project preparation, the Bank QER/QAG process was not established. The unsatisfactory ICR rating of quality at entry is based on the following: (i) inadequacy of project design despite the right IP vision; (ii) insufficient clarity and limited realism of objectives; (iii) insufficient focus on results; and (iv) insufficient level of readiness for implementation and assessment of risks and sustainability. (i) Design appropriateness The PD and the project had the right vision. Both put the right emphasis on the importance of a country-owned sector strategy, decentralization, the development of horizontal services, and donor coordination in a IP framework. At the time of appraisal, the project was in line with PD and CA priorities. Project design was also consistent with the Bank s safeguard policies. However, it can be concluded that the PD and the project design had important shortcomings: (i) despite the emphasis put on the flexibility needed to further develop the PD and IP approach, the project design concentrated on an overly detailed preset series of inputs, e.g. defining up front, all necessary training for health care staff to be conducted during implementation; (ii) the health sector reform was guided by "norms and standards," prepared by the MOH with IDA assistance during preparation. These norms included the size of health infrastructure, but the size of health centers appears to be too large (see project efficiency in section 4.1). The fact that the project followed these norms implied significant spending on over-expanding existing health infrastructure and limited the project's ability to regularly reconsider priorities during implementation; (iii) despite the objective to improve health care quality, an explicit definition of how increased quality would be measured was not made in the PD and PAD; (iv) the PD and project rightly envisioned to bring essential care close to the population through the development of decentralized and - 3 -

9 integrated district health services. But it appears that the interdependency of reforms at the district level and the central MOH level was overlooked as project design heavily focused on the district level reforms and not enough on the central level of MOH in an effort to address systemic country-wide sector bottlenecks. These design shortcomings also characterized, in part, other Bank HNP operations during the 1990s. (ii) Clarity and realism of objectives Insufficient clarity of objectives. The PAD did not define a project objective, but took the PD objectives, referred to as "program objectives." Yet, the PD objectives stated in the PAD did not fully correspond to the PD objectives stated in the national PD document. During project preparation, IDA agreed with the Government to cluster a set of PD and sector policy objectives together and to exclude other PD objectives. This re-definition and reduction of PD objectives for project purposes was not mentioned in the PAD. While the PAD did not have a project objective, the DCA and the Memorandum of the President had one, rightly reducing the project objective to a contribution to the PD. Yet the project objectives stated in these two documents were also not fully coherent. For example, according to the PAD, the objective concerning essential generic drugs was to improve their availability. The DCA stated that the objective was to improve access of the population to these drugs. Improving access requires different and more far-reaching activities than improving drug availability. Another example is the objective to support the Government in the coordination of aid in PD implementation. This objective was stated in the Memorandum of the President, but not stated in the PAD or in the DCA, and no explicit activities had been planned to achieve this objective. Annex 10 summarizes objectives stated in the PD, health sector policy, PAD, Memorandum of the President, and DCA. Limited realism concerning scope and speed of health sector reform in light of country implementation capacities. The PD and the project had very ambitious reform objectives concerning decentralization and strengthening of the district level - namely to reorient incentives and accountability so that higher levels of MOPH administration will be serving and supporting the peripheral levels rather than the reverse - These objectives, demanding a range of policy and institutional reforms involving a number of organizations at the national, regional, and district level, were expected to be achieved in a short period of time in a country with very limited capacities and which, in addition, experienced a coup d'etat during the project preparation phase. Project design included a significant amount of construction, a decision that probably did not sufficiently take into account capacity constraints in the construction sector and relatively limited internal fiduciary control systems in the country, according to WBI governance indicators. It appears that the complexity of human and institutional challenges to bring about fundamental health sector reform were somewhat underestimated and that the lessons learned from the ICR of the previous IDA health operation (Credit 1668-NIR), namely that objectives should be sufficiently modest and that institutional capacity is the most constraining factor, were not fully taken into account. (iii) Focus on results Given the differences among the PD and project objectives stated in different documents, it was not clear how best to manage project implementation from a results perspective. Given the fact that the DCA is the legally binding document, the objectives stated in the DCA were followed to assess the results of the project (see section 4.1). Limited appropriateness of chosen indicators. The choice of indicators reflected more the traditional focus on activities and outputs, and included only one correctly chosen outcome indicator, the utilization rate. For this indicator, targets were set and baseline data was available, but for many other indicators this was missing. Annex 11 provides an assessment of project outcome and output indicators stated in the PAD

10 Choice of the original project area did not fully correspond to the project s special focus on the poor and vulnerable groups, as the area's five districts were in two regions with a lower than average poverty rate as indicated by the 1996 poverty assessment. Hence, there appears to be a design contradiction between the project s stated special poverty focus and the criteria agreed to with the Government. These criteria, governing the choice of project-supported districts, did not consider the district s poverty aspects. (iv) Readiness for implementation and assessment of risks and sustainability Insufficient implementation readiness. The preparation of the project was a lengthy process (concept review in April 1994, approval in eptember 1996) and relatively well-funded (U $868,000 including trust fund resources, see Annex 4) supporting the Government in further developing the PD. However, questions regarding whether the project was ready to be implemented can be raised, given the fact that: (i) the project design included many conditions that partially contributed to the 10- month time lag between project approval and effectiveness; (ii) the PD and the project were designed under the assumption of significant donor contributions to finance PD implementation. But the project did not consider the trend since the early 1990s of reduced aid to Niger. Aid per-capita has decreased by about 50% over the last decade; (iii) the design did not adequately consider the implications of the coup d'etat 1996; and (iv) a comprehensive and in-depth capacity assessment, including financial management and procurement capacity of MOH directorates responsible for project implementation, was not undertaken, given the correct identification of significant sector capacity constraints. Inadequate assessment of sustainability. The economic analysis of the PD in the PAD calculated that by the end of the project all recurrent costs would be covered through national contributions. Given the significant amount of IDA-financed investments in infrastructure and equipment, within Niger's macro-economic context, this assumption was not realistic. 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: Unsatisfactory. Despite significant inputs (IDA provided about 1/3 of all sector aid over the previous seven years) and remarkable efforts, project outcome and achievement of objectives are rated unsatisfactory. The following methodology has been applied for the assessment: First, given that the project was designed as a contribution to the PD, an overview of the current health system in Niger, based on key determinants of health outcomes, has been prepared. It demonstrates that Niger s health system is in a precarious situation with alarmingly poor health outcomes. Life expectancy at birth is estimated at 45 years, and infant mortality is very high at an estimated 126 per 1,000 births. An estimated 265 out of every 1,000 children die before the age of five, and malnutrition under the age of five is increasing and affects 40% of all children. Annex 12 provides a more detailed visual overview and Annex 13 shows the supporting data indicating the changes over the previous years. econd, this contribution was assessed using the six determinants of health outcomes that reflect the project objectives and the only project outcome indicator (see Annex 11): (i) utilization rate of basic health services; (ii) coverage rate of services; (iii) access to essential generic drugs; (iv) quality of basic health services; (v) participation of the private sector and the population in the delivery of health care services; and (vi) management efficiency of health sector organizations. The assessment included the contribution in all 14 districts of the expanded project area as expansion decisions were taken in the early - 5 -

11 phases of project implementation. Third, as these six determinants of health outcomes were not clearly defined in the PD and the PAD, the following criteria (see figure) have been used for the assessment: Criteria to assess outcome / achievement of objectives Utilization CI new consultations per inhabitant per year ervice quality Personnel technical and socio-cultural competences: Able to establish trustful relations, belong to the community, accountable to the population, able to do correct diagnosis and treatment, permanently available Organization of care : Waiting time, team work, outreach to risk groups, integration of care Access to drugs Buildings & equipment: Basic functionality, socially adapted Referral system: CI & HD as integrated system, well functioning CIs & HDs prerequisite Coverage Potential geographic coverage Real geographic coverage: Personnel in place, no out of stock drug, infrastructure & equipment Real coverage from clients perspective: Financial & sociocultural accessibility Management Efficiency Priority setting in Resource constraint environment Addressing systemic health sector bottlenecks Health personnel management Public health expenditures Donor sector performance & contributions Participation Forms of participation: Financial contribution, contributions in work, management contributions, contracting out of services takeholder Groups: Private sector, NGO s CBO s, Individuals Access to drugs Availability of essential generic drugs Financial and sociocultural accessibility to essential generic drugs The assessment of the achievement of project outcome and objectives is made along these criteria and based on the data in Annex 1 and Annex 13, but also on empirical evidence the ICR team gained during an extended field mission including visits to more than 20 health centers and district hospitals in all regions of the project area. The chosen criteria reflect the strong ICR focus on results. The ICR, in its overall assessment, made the distinction between the demonstrated level of efforts of the Government and the Bank and the actual achievement of outcome and objectives. If other criteria, giving more weight to efforts and processes, were used, the rating might be different. (i) ignificant decrease in the utilization rate in most of the 14 project districts Criteria to assess outcome and achievement of objective: PD and the PAD did not explicitly define how utilization of basic health care services would be measured. The ICR follows the criterion introduced at the MTR, namely new consultations per inhabitant per year for district health centers. The utilization rate is the only demand-side criterion, hence it is the most important criterion to assess project outcome and achievement of objectives (see visual above summarizing the interplay of all the assessment criteria). Country-wide, utilization of basic health care services is still very low but increased from 18% to 24% between 1996 and 2002/2003. Although there seem to be some reliability questions concerning this data - 6 -

12 (possible double-counting in at least two districts), the overall trend is clear. Project contribution: The utilization rate in most of the 14 project-supported districts decreased significantly, on average by 38%, between 1996 and 2002/2003. In 13 out of the 14 districts, the utilization rate for curative care dropped, and for the only district that showed an increase (N guigmi) data does not seem reliable. Preliminary data for 2003 indicates a further decline of the utilization rate in several districts. During field visits to the health centers, the ICR team was able to confirm with field personnel these very low utilization rates. The utilization rate in the 14 project-supported districts is also low for family planning, immunization, pre-natal care, and nutrition consultation. Utilization of family planning services decreased in 13 districts, with a major decrease in 7 districts. In 11 project districts, the number of completely vaccinated children is in decline. The utilization rate of prenatal care decreased in 7 districts, and in the Zinder region where the project heavily invested in infrastructure and equipment (47% of all rehabilitated CIs, 54% of all newly constructed CIs, and 38% of all rehabilitated and constructed district hospitals are in this region) prenatal services utilization decreased on average by 13%. While at the national level maternal mortality is slightly declining, in 5 out of 8 project-supported districts with available data, maternal mortality seems to be increasing. Finally, the utilization rate for nutrition consultations decreased in project districts on average by about 20%. The impact of the project-financed new health centers is not fully considered yet in the utilization rate data, as most of the newly constructed health centers were only finalized in October and November Hence, it can be expected that the utilization rate might increase if improvements in the other criteria that determine the utilization rate can be achieved. The main reasons for the decreasing utilization rate in the project-supported districts seem to be: (i) poor service quality and reputation of the existing and rehabilitated CIs as perceived by the users and indicated in the 2003 Beneficiary urvey (see below); (ii) no permanent services as health personnel are frequently out for training or have left the CIs - fluctuation in project-supported districts is higher than in other districts (see below); and (iii) decreased financial accessibility due to the form of the introduced cost recovery scheme - while cost recovery has been introduced country-wide, data indicates that cost recovery might have been more forcefully implemented in project-supported districts.. (ii) In project-financed districts, increased potential geographic coverage did not transform into increased accessibility so far. Criteria to assess outcome and achievement of objective: PD and PAD did not exactly define the criteria to assess the increase in coverage of basic health services, hence the ICR used the following criteria: (i) potential geographic coverage defined as the population who lives within 5 km of a health center; (ii) real geographical coverage, assessing if these new health centers are functioning, measured by the following quantitative criteria: sufficient personnel in place, no out of stock drugs for the most essential drugs, infrastructure and equipment in place; and (iii) real coverage from clients' perspectives, i.e. considering financial and socio-cultural accessibility of basic health care services. Country-wide, potential geographic coverage increased from 32% to 48% between 1996 and 2002, but it appears that the increase in potential coverage did not fully transform into increased accessibility. Although the potential geographic coverage data does not appear to be fully reliable (in several districts the Case de anté, small health centers with a limited function, seem to be included) the overall trend of increased potential coverage is obvious. Yet this increase in potential coverage did not fully transform into - 7 -

13 real geographic coverage as several of the newly constructed facilities, including facilities financed by other donors, are not fully functioning. In addition, financial accessibility seems to be limited (negative impact of cost recovery as implemented, see below) and there appear to be socio-cultural access barriers (e.g. health care staff not able to establish long-lasting and trustful relationships with patients, see below). The project contributed to the increase of potential geographic coverage in seven districts through the construction of 50 health centers. In the project-supported districts, the coverage rate increased from 27% to 30%, yet, according to the 2004 National Project Evaluation report, this increase does not include the 50 project-financed health centers. Given that no data is available about the increase in potential geographic coverage, the ICR estimates, based on data collected during the field visits, that the project helped to increase potential geographic coverage for up to 300,000 people. However, at the time of the ICR mission, most of the project-financed health centers and district hospitals were either (i) closed, (e.g. all 10 new health centers in the Diffa region - lack of drugs); (ii) not functioning, (e.g. operating theatres in 9 of the 13 district hospitals not used due to lack of surgeons, anesthesists, radiologists); or (iii) only partially functioning. A limited number of financed infrastructures were fully functioning (1 out of 9 project-financed CIs visited during the ICR mission - Annex 14 provides a more detailed overview). Taking into account this ratio as well as the number of currently closed new CIs it can be estimated that the project increased real geographic coverage for about 27,000 people so far. Considering that most of the new CIs were only recently finalized it can be expected that the project's contribution to real geographic coverage will increase. Yet, transforming geographic coverage into real coverage from the clients' perspective will remain a major challenge, given the above mentioned problems concerning financial and socio-cultural accessibility. (iii) Limited access to essential generic drugs in project-supported districts Criteria to assess outcome and achievement of objective: The PAD defined the number and duration of out of stock drugs as an indicator to measure access to essential generic drugs. The ICR applied the following criteria for its assessment: (i) availability of essential generic drugs, reflecting the PAD indicator; (ii) financial and socio-cultural accessibility. Country-wide, though slightly improving, there is still restricted availability of the most essential generic drugs. While drug availability is a major problem, the bigger issue seems to be the limited access to essential generic drugs. This is linked to limited financial accessibility due to the current form of the cost recovery mechanism (see below). The 2002 Participatory Poverty Assessment revealed that for the poor, the affordability of drugs was among the three greatest weaknesses of Niger s health system, substantiating the country s PRP objective of ensuring that essential drugs are available at affordable prices. Project contribution: Access to the most essential generic drugs in the 14 project supported districts slightly improved but is still limited: Out of stock criteria Assessment Average number of days the 20 most essential drugs are out of stock % improvement Number of 20 most essential drugs that are not out of stocks % decline Number of most essential drugs that are out of stock for more than 30 days % decline Maximum number of days most essential drugs are out of stock % improvement - 8 -

14 In addition to these problems, one can conclude, given the significant decline in the utilization of health centers in the project-supported districts, that access to generic drugs in the 14 districts has also decreased. (iv) Poor quality of basic health care services and major problems concerning health care staff Criteria to assess outcome and achievement of objective: PD and PAD did not explicitly define how improved quality would be measured. The ICR used the following criteria for its assessment: Criteria Priority CI HD taff competencies: socio-cultural and technical 1 Able to establish trustful relations with patients and community Belongs to the community (e.g. personnel stays for long periods) Accountable to the population, no civil servant mentality Able to do correct diagnosis and treatments Permanently available Technically competent (e.g. surgery) Permanently available Access to drugs 2 Permanent availability of most essential drugs no out of stock drugs Financial and socio-cultural accessibility, i.e. affordable and accessible for all patients Organization of care 3 Waiting time Team work Outreach to risk groups Integration of care Buildings and equipment 4 Buildings and equipment provide basic functionality, but what is important is what is done inside ocially adapted in order to increase sociocultural accessibility Waiting time Team work ome parts of the hospital have to respond to high technical standards. Referral ystem 5 CI and HD function in a complementary way as an integrated system Well functioning hospitals and CIs are the pre-requisite Country-wide, patients perceive the quality of health care services as poor. While coverage, access to drugs, management effectiveness of health sector organizations and participation are important, they are not sufficient contributors for increased utilization of health services. ervice quality is the key. According to the 2002 Participatory Poverty Assessment, the poor rated the quality at reception as one of the three greatest weaknesses of the health system. "Producing quality health care services is a multi-faceted process. According to the 2003 Beneficiary urvey, availability of health care personnel able to establish trustful and continuous relationships with the community and to provide adequate quality service is the most important factor in developing quality health care services. Yet, health care personnel is the country's major bottleneck for improved quality. While PD and PAD identified this issue, it appears that problems surrounding health care staff have increased over the recent years: (i) significant shortage of health care staff, especially in more rural and remote districts, also linked to the 1999 law on retirement which led to the situation where many experienced health sector staff left and were not adequately replaced; (ii) dependency on expatriate staff, particularly medical doctors; (iii) delays in salary payments; (iv) too high specialization of health care staff; (v) nonexistence of an adequate incentive system to reward staff working in rural and remote areas; (vi) extremely high and increasing staff mobility; (vii) civil servant mentality; (viii) varying staff technical and socio-cultural competency; (ix) education and training approaches to a certain degree are not adapted to staff learning needs, over-emphasizing formal classroom training that provides an opportunity for health personnel to acquire per diems to make up for the low salary. Annex 15 provides a more detailed overview. Project contribution. Acknowledging its importance, the project tried to improve service quality by (i) first improving buildings and equipment of 43 existing health centers before financing new constructions; - 9 -

15 (ii) mainstreaming a quality assurance approach in 3 regions of the project area; (iii) consistently trying to ensure that an adequate number of health care staff would be placed in the respective districts; and (iv) heavily investing in the training of health care staff. Yet it appears that these inputs did not translate into significant improvement in service quality. In addition to the above mentioned points of limited functionality of CIs and HDs, limited access to drugs, and not explicitly defining what health care quality would constitute and how to best improve it, the project (i) was not successful, despite some efforts made, in helping the Government to effectively address the major problems surrounding health care staff, resulting in poor quality of health care services and ultimately in poor health outcomes; (ii) did not develop and implement fully appropriate learning approaches for health care personnel (see section 4.2). Most CI health care staff seem not to be able to establish trustful relationships with patients and the community, given the high fluctuation rate. In the project-supported districts, 48% of the staff left their post after having received training, with a significant increase in turnover since 2000, compared to 36% in districts outside the project area. An example is the project's attempt to introduce a quality assurance culture for CIs. The purpose of this approach was to train CI service providers to identify and solve problems. This training was not delivered in a work-based and hands-on manner, directly supporting specific CIs as a whole in improving service quality, but was mainly organized through a series of formal training. While health care staff might have improved their individual skills, it appears that these skills and the quality assurance approach have not been properly integrated into how CIs organize their care. Hence, the 2003 Beneficiary urvey indicates that service quality remains the major concern: 31.5% have a positive view concerning the quality of services offered, compared to 25% in the first Beneficiary urvey of This increase is acknowledged, but is not significant considering the important resources dedicated to improving quality. In addition, patients do not feel respected and they feel poorly received. According to 85.8% of the beneficiaries interviewed, the biggest need is to improve quality. (v) Current form of cost recovery might create negative impact and reflects limited efficiency of participation approach Criteria to assess outcome and achievement of objective: PD and PAD did not fully and explicitly define how participation of the private sector and the population in the delivery of health care services should be implemented. The ICR differentiated in its assessment between (i) different forms of participation, i.e. financial contribution, contributions in work, management contributions, and contracting out of services; and (ii) different stakeholder groups, i.e. formal and informal private sector, NGOs, community-based organizations, and individuals. Country-wide, the current form of cost recovery has led to a slight improvement of drug availability, but also contributes to low utilization of health services especially among the poor. Two studies support this assessment: (i) the 2002 Participatory Poverty Assessment confirms that most people did not derive benefits from the public health services offered, due to their low income. Hence, reducing consultation fees was one of the most expressed needs by the poor; (ii) according to the 2003 Beneficiary urvey, only 20.5% of the beneficiaries believe that the current cost recovery system is a way to improve the health sector. Even contraceptives are on cost recovery. This, in the country with the highest fertility rate worldwide, a country where more than 60% of the population lives in poverty, and a country where it appears that a large segment of the population wants to reduce the number of its children: data from the 2003 Beneficiary

16 urvey indicate that 40% want to have between two and three children against 23% who want to have six or more children. Project contribution: The negative impact of cost recovery, as practiced, appears to have been more severe in the project supported districts as cost recovery seems to have been more forcefully introduced in these districts. As mentioned above, data on cost recovery in selected regions indicate that the relative increase in cost recovery revenues between 1998 and 2000 was higher in project-supported regions. In order to limit the negative impact of cost recovery on the poor, the project supported the creation of CI management committees that can decide to waive consultation fees for the poor. However, it appears that this approach did not succeed in overcoming the exclusion of the poor. The heavy emphasis by the Government on cost recovery mirrors a too narrow focus on financial participation of the population. Non-financial participation through contribution in work was limited to the requirement of the population to build walls around the rehabilitated or newly constructed health infrastructures. Concerning participation in the management of health services, the 2003 Beneficiary urvey indicates that 37% of the established community management structures in the project-supported districts are functional, which is a good achievement, but that they are too much focused on the administration of the cost recovery scheme instead of bridging the gap between the CI and the community. Finally, attention to strengthening effective community participation decreased at a later stage of the project, as demonstrated in 2003 when the resources were spent on infrastructure at the expense of supporting and training of CI management committee members. Private sector participation in project-financed districts had a relatively narrow focus. At the outset, formal private sector participation was limited. In order to increase community and private sector participation, the project introduced the contracting out of services, but it appears that it was not able to adequately increase effective collaboration between the public health sector and formal private health care services (229 formal private health structures country-wide in 2003, 17% of them located in rural areas) as well as informal private sector services (traditional healers). Formal private health centers and clinics were not considered in coverage plans that, for example, led to the decision to fully equip the Konni district hospital located less than 50 km from a well functioning missionary hospital. The ICR concludes that the project achieved to a limited extent its objective to increase efficiency of private sector and population participation in the delivery of health care services. In particular, there seems to be an urgent need to re-think individual and community participation which goes beyond the project-financed districts. This is a national issue, linked to the PRP objective of creating financially affordable and equitable access to the health system for the poor in order to improve health outcomes. (vi) Limited efficiency and performance of sector organizations, low public expenditures on health and limited donor sector performance Criteria to assess outcome and achievement of objective: The PAD did not sufficiently define how management efficiency of health sector organizations would be assessed. The chosen indicators, namely the number of annual plans prepared, the number of planning meetings, annual recurrent expenditures of district health facilities, and the financial contributions of key financiers either do not seem to be appropriate or have not been adequately tracked or reported in PRs (see Annex 11). In an effort to assess this variable, the ICR applied the following additional criteria: (i) ability of national sector organizations to set priorities in a resource constraint environment; (ii) their ability to effectively address systemic health sector bottlenecks at the adequate level; (iii) efficiency of personnel management, (iv) public expenditures on health; (v) donor sector performance and contributions

17 Country-wide, efficiency and performance of sector organizations and management appear to be limited. Organizational performance at the most important level, the health centers and district hospitals, is relatively poor, and sector organization at the district and national levels do not appear to be able to adequately set priorities and to effectively address systemic bottlenecks on the most important determinants of health outcomes. Public expenditures on health have decreased over the last decade (1.8% of GDP in 2001 against 3.4% in 1990) and are well below the ub-aharan Africa average. Few donors are supporting the health sector in Niger and only 3.1% of all DAC aid in the years was dedicated to the health sector. Until recently, donors did not coordinate their support to the health sector very well. Instead of donor alignment, donor segmentation by district or by specific interest (e.g. vertical programs) seemed to be the mode of operation. This is recently changing for the better with the Belgian Cooperation being nominated by the Government as the chef de file to support the Government in leading sector donor coordination. Limited project contribution to increase efficiency and performance of sector organizations. The project contributed to the set up of district health teams in the 14 districts and supported the development of multi-year district development and annual action plans, aiming at consolidating donor support in one plan and at setting district level priorities. This is an important achievement (see section 4.2). The project also invested heavily in the introduction of performance-based management in the MOH with a focus on the districts since However, after six years of support, district health teams appear to be quite fragile. The ICR team randomly analyzed annual district plans prepared over the years, and it appears that they focus on inputs and do not set clear strategies and priorities, nor do they stress on improvement of service quality. At the time of the ICR mission in January 2004, several visited districts did not have an annual plan for 2004 prepared. Finally, the project contribution to increase efficiency and performance at the MOH central level was modest. It appears that as long as the systemic health sector bottlenecks are not adequately tackled, significant performance increase will not be possible. Conclusion: Unsatisfactory outcome as most of the relevant objectives were not met Relevance: From today s country and CA perspective, and given the above described overall situation of Niger s health system, most of the project objectives are still relevant (service quality, access, sector organization efficiency, partnership). Efficacy: Project objectives have only been achieved to a limited extent as (i) the utilization rate of basic health care facilities in the project area decreased significantly; (ii) potential geographic coverage in the seven targeted districts increased but did not transform into increased accessibility so far; (iii) real access to generic drugs in the 14 project-supported districts appears to have decreased; (iv) service quality in the project area has not considerably improved; (v) efficiency of private sector and population participation increased but only to a limited extent; and (vi) while performance of district health teams has been partially increased, the efficiency and performance at MOH central level were not adequately addressed. Hence, the project put in place many inputs necessary for achieving the objectives, but was not able to integrate these inputs in order to fully achieve project outcome and objectives. Efficiency: The use of scarce project resources to accomplish the objectives and achieve the outcome of increased utilization is questionable. Indeed, the project invested too heavily in infrastructure and equipment, and within the infrastructure sub-components, resources were not used efficiently as (i) the project invested too much in the over-expansion of existing health centers (on average expansion by 284% compared to the original size of the then existing health centers; average size of rural health centers is 216% larger than those supported by an IDA project in neighboring country Chad - see Annex 16) without

18 increasing coverage and without adequately contributing to improved service quality; (ii) questionable infrastructure priorities were set, for example concerning the Diffa region and district where 25.4% of the overall project resources were spent: the region had a significantly lower poverty rate than some other regions, the district's coverage rate in 1996 was already higher than the national objective for 2000, and construction costs in this region were higher compared to other regions; and (iii) average unit construction costs were much higher compared to other donor-financed infrastructure (214% higher compared to Belgian Cooperation; 24% higher compared to African Development Bank). Annex 16 provides a more detailed overview about the unsatisfactory infrastructure efficiency, and Annex 3 provides a short analysis of the economic costs and benefits of the project. 4.2 Outputs by components: A schematic overview about the achievement of project outputs per component has been prepared (see Annex 17). In addition to this overview and the points discussed in the previous section as well as the data in Annexes 1 and 13, a few key issues will be shortly analyzed and assessed applying the project s output indicators (see Annex 11). (i) Outputs component 1: Improving the quality and coverage of Basic Health ervices (Component is rated unsatisfactory) 107 health infrastructures rehabilitated/expanded or newly constructed and equipped (43 CIs expanded/rehabilitated, 50 new CIs constructed in seven districts, 13 district hospitals finalized/rehabilitated/constructed, one regional hospital rehabilitated/expanded), but few of them fully functioning, due to lack of quantity and/or quality of personnel, lack of initial drug inventory, or unsuitable locations. As described above, potential geographic coverage has been increased but has not been systematically transformed into real coverage from the clients' perspective yet (see also Annex 14). Investment in these 107 health infrastructures resulted in significant increase in recurrent costs in a country with major financial difficulties, considerably decreasing aid per capita and very low health sector expenditures. While increasing recurrent costs, the benefits have not been achieved so far (utilization rate, quality of care etc) - see Annex 3. 4,200 service delivery staff trained, but limited impact on increased service quality. On average, each health care staff in the districts received about three trainings financed by the project, but the success of this training is questionable: (i) primary focus on formal training and not sufficient work-based learning; (ii) formal training required staff to leave their work, and many staff left their jobs after having received the training, contributing to poor service quality; (iii) training did not fully match staff needs. According to an evaluation prepared for the MTR, only 22% of trained health care staff rated the training as being based on their needs. This result did not trigger further evaluation of training activities. Low client satisfaction in project-supported districts. The significant rehabilitation, expansion and new construction of health facilities and the major training efforts have not yet transformed into satisfactory quality services, as perceived by the poor and by clients and as assessed by the ICR in the section above. (ii) Outputs component 2: Improving the access of the population to essential generic drugs (Component is rated unsatisfactory) Most essential drugs are frequently out of stock. The project provided an inventory of initial drugs, but, as mentioned above, out of stock drugs remain a major bottleneck to improving Niger s health system, despite