IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-75450) ON A LOAN IN THE AMOUNT OF US$83.45 MILLION TO THE FEDERATIVE REPUBLIC OF BRAZIL FOR THE

Transcription

1 Public Disclosure Authorized Document of The World Bank Report No: ICR Public Disclosure Authorized IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-75450) ON A LOAN IN THE AMOUNT OF US$83.45 MILLION TO THE Public Disclosure Authorized FEDERATIVE REPUBLIC OF BRAZIL FOR THE SECOND PHASE OF THE FAMILY HEALTH EXTENSION PROGRAM June 22, 2015 Public Disclosure Authorized Health, Nutrition and Population Global Practice Brazil Country Management Unit Latin America and the Caribbean Region

2 CURRENCY EQUIVALENTS (Exchange Rate Effective December 31, 2014) Currency Unit = Real (R$) R$ 1.00 = US$ US$1.00 = R$ 2,6577 FISCAL YEAR January 1 December 31 ABBREVIATIONS AND ACRONYMS AMAQ AMQ APL ARI DAB DESID DPT-H ESF e-sus-ab FIOCRUZ GOB IBRD IPEA M&E MOH NCD MTR PDO PMAQ PNAB PROESF 1 PROESF 2 PSF QUALISUS SIS-AB SUS UBS Self-appraisal for Quality and Access Improvement in Primary Care (Autoavaliação para Melhoria do Acesso e da Qualidade de Atenção Básica) Appraisal for Quality Improvement of the Family Health Strategy (Avaliação para a Melhoria da Qualidade da Estratégia Saúde da Família) Adaptable Program Lending Acute Respiratory Infection Department of Primary Care (Departamento de Atenção Básica) Department of Health Economics, Investment and Development (Departamento de Economia da Saúde, Investimentos e Desenvolvimento) Diphtheria, Pertussis and Tetanus Family Health Strategy (Estratégia Saúde da Família) e-unified Health System - Primary Health Care (e-sistema Único de Saúde - Atenção Básica) Oswaldo Cruz Foundation (Fundação Oswaldo Cruz) Government of Brazil International Bank for Reconstruction and Development Institute of Applied Economic Research (Instituto de Pesquisa Econômica Aplicada) Monitoring and Evaluation Ministry of Health Noncommunicable Diseases Mid-term Review Project Development Objectives National Program for Access and Quality Improvement (Programa Nacional de Melhoria do Acesso e da Qualidade) National Primary Care Policy (Política Nacional de Atenção Básica) Family Health Extension Project, APL Phase 1 (Projeto de Expansão e Consolidação Saúde da Família, fase 1) Family Health Extension Project, APL Phase 2 (Projeto de Expansão e Consolidação Saúde da Família, fase 2) Family Health Program (Programa Saúde da Família) Brazil Health Network Formation and Quality Improvement Project (Projeto de Formação e Melhoria da Qualidade de Rede de Saúde) Health Information System for Primary Care (Sistema de Informação em Saúde da Atenção Básica) Unified Health System (Sistema Único de Saúde) Primary Care Unit (Unidade Básica de Saúde)

3 Vice President: Jorge Familiar Country Director: Deborah L. Wetzel Practice Manager: Daniel Dulitzky Project Team Leader: Tania Dmytraczenko ICR Team Leader: Daniela Pena de Lima ICR Primary Author: Suzana Abbott

4

5 BRAZIL Second Phase of the Family Health Extension Program CONTENTS A. Basic Information... i B. Key Dates... i C. Ratings Summary... i D. Sector and Theme Codes... ii E. Bank Staff... ii F. Results Framework Analysis... ii G. Ratings of Project Performance in ISRs... vii H. Restructuring (if any)... viii I. Disbursement Profile... ix 1. Project Context, Development Objectives and Design Key Factors Affecting Implementation and Outcomes Assessment of Outcomes Assessment of Risk to Development Outcome Assessment of Bank and Borrower Performance Lessons Learned Annex 1. Project Costs and Financing Annex 2. Results Framework and Outputs by Component Annex 3. Economic and Financial Analysis Annex 4. Bank Lending and Implementation Support/Supervision Processes Annex 5. Beneficiary Survey Results Annex 6. Stakeholder Workshop Report and Results Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Annex 9. List of Supporting Documents Annex 10. Detailed Project Description Annex 11. Timeline of Events during Project Life Annex 12. MAP... 67

6

7 A. Basic Information Country: Brazil Project Name: Second Family Health Extension Adaptable Lending Project ID: P L/C/TF Number(s): IBRD ICR Date: 06/18/2015 ICR Type: Core ICR Lending Instrument: APL Borrower: Original Total Commitment: Revised Amount: USD 83.45M Environmental Category: B FEDERATIVE REPUBLIC OF BRAZIL USD 83.45M Disbursed Amount: USD 34.75M Implementing Agencies: Ministry of Health Department of Primary Care (Departamento de Atenção Básica - DAB) Cofinanciers and Other External Partners: B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 03/01/2006 Effectiveness: 10/15/ /15/2009 Appraisal: 12/18/2006 Restructuring(s): 03/14/2013 Approval: 04/25/2008 Mid-term Review: 07/31/ /28/2012 Closing: 03/30/ /31/2014 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Borrower Performance: Unsatisfactory Substantial Moderately Satisfactory Unsatisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Moderately Quality at Entry: Moderately Satisfactory Government: Unsatisfactory Implementing Quality of Supervision: Satisfactory Unsatisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Unsatisfactory Performance: Performance: i

8 C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Performance (if any) Potential Problem Project at any time (Yes/No): Problem Project at any time (Yes/No): DO rating before Closing/Inactive status: Yes Yes Moderately Unsatisfactory Quality at Entry (QEA): Quality of Supervision (QSA): None None Rating D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 20 5 Health Sub-national government administration Theme Code (as % of total Bank financing) Child health Health system performance Participation and civic engagement 17 5 Population and reproductive health E. Bank Staff Positions At ICR At Approval Vice President: Jorge Familiar Pamela Cox Country Director: Deborah L. Wetzel John Briscoe Practice Manager: Daniel Dulitzky Keith E. Hansen Project Team Leader: Tania Dmytraczenko Gerard Martin La Forgia ICR Team Leader: Daniela Pena De Lima ICR Primary Author: Suzana Nagele de Campos Abbott F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The objectives of the second phase Adaptable Program Lending (APL) are: (i) increase access to Family Health-based primary care in large, urban municipalities; (ii) raise the technical quality of and patient satisfaction with primary care; and (iii) improve the ii

9 efficiency and effectiveness of Family Health service providers as well as the broader delivery system. Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Indicator 1 : Decrease of infant mortality per 1000 live births in participating municipalities. Value quantitative or Qualitative) Date achieved 12/31/ /09/ /31/2013 Target exceeded. The MOH set a target of a 5% reduction in infant mortality Comments (16.9) by Project completion. Data for 2014 not available. Source of data: (incl. % Information System for Mortality (SIM) and System for Registration of Live achievement) Births (SINASC). Indicator 2 : Increase of per capita contact with primary care providers (doctors and nurses) in participating municipalities. Value quantitative or Qualitative) Date achieved 12/31/ /09/ /31/ /31/2014 Comments (incl. % achievement) Target not met. Results reflect a partial reporting, based on data from the old information system. The indicator was revised to: increase of per capita contact with primary care providers (doctors and nurses) in participating municipalities ; the baseline was revised to 1.7 (2003) and the target to 1.8 in Source of data: Information System for Primary Care (SIAB). (b) Intermediate Outcome Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Indicator 1 : Percentage of PSF population coverage in participating municipalities. Value (quantitative or Qualitative) Date achieved 12/31/ /09/ /31/2014 Target exceeded for Phase 2 and for Phase 3 (40%). Coverage in Comments participating municipalities outpaced coverage in non-participating (incl. % municipalities (32%). Source of data: Department of Primary Care (DAB) and achievement) Brazilian Institute of Geography and Statistics (IBGE). iii

10 Indicator 2 : Proportion of patients with hypertension registered by PSF teams, among estimated population with hypertension (15 years of age and older). Value (quantitative or Qualitative) Date achieved 12/31/ /09/ /31/ /31/2014 Comments (incl. % achievement) Indicator 3 : Target not met. The indicator was originally calculated as a ratio of these two groups and in the Project restructuring was changed to be calculated as a proportion to better align with the MOH monitoring system. Source of data: SIAB and IGBE. Percentage of infants less than 1 year with full vaccination regimen (DPT-H, polio, measles, tuberculosis) in participating municipalities (as a group). Value (quantitative or Qualitative) Date achieved 12/31/ /09/ /31/ /31/2014 Comments (incl. % achievement) Indicator 4 : Target not met. The MOH vaccination program was upgraded to a penta- and hexavalent scheme. The indicator was modified to be calculated based on the sum of completed tetra-, penta- and hexavalent schemes in children less than one year of age. Baseline revised to 100.8% in 2008 and target to equal or more than 95% in Source of data: System for Vaccination Information (SI-PNI) and SINASC. Percentage of women with live births, attended by PSF teams, that had 7 or more pre-natal consultations. Value (quantitative or Qualitative) Date achieved 12/31/ /09/ /31/2014 Comments (incl. % achievement) Indicator 5 : Target not met. Indicator revised to Percentage of women with live births, attended by PSF teams that had 7 or more pre-natal consultations Baseline was revised to 60% based on actual data for Source of data: SINASC. Percentage of total medical consultations referred from PSF to hospital specialty services in participating municipalities (as a group). Value (quantitative or Qualitative) Maintain 10 Maintain Date achieved 12/31/ /09/ /31/2014 Comments (incl. % achievement) Target considered met because 0.1 marginally affects the result. Source of data: SIAB. Indicator 6 : Percentage of all PSF teams applied quality evaluation instrument and are ranked according to standard in participating municipalities (as a group). Value (quantitative or Qualitative) Date achieved 12/31/ /09/ /31/2014 Comments (incl. % Target exceeded. Source of data: Database of PMAQ-AB and AMQ, DAB/Ministry of Health (MOH). iv

11 achievement) Percentage of PSF teams in a sample of municipalities using evidence-based Indicator 7 : clinical guidelines for hypertension and diabetes (based on sample survey) in participating municipalities (as a group). Value (quantitative or Qualitative) N/A Date achieved 12/31/ /09/ /31/2014 Comments (incl. % achievement) Target exceeded. Source of data: Database of PMAQ-AB, and external evaluation, DAB/MOH. Indicator 8 : Proportion of municipalities that conduct self-assessment to improve access and quality of primary care. Value (quantitative or Qualitative) Date achieved 12/31/ /09/ /31/2014 Comments (incl. % achievement) Indicator 9 : Value (quantitative or Qualitative) Target exceeded. This indicator was updated to reflect the evolution and scale up of quality improvement instruments - AMAQ and PMAQ. Indicator revised to Proportion of municipalities that conduct self-assessment to improve access and quality of primary care. Source of data: Database of PMAQ-AB, DAB/MOH. Proportion of states that establish performance agreements with at least 25% of municipalities with less than 100,000 population. 0 40% states 25% municipalities 100% (26) states with 100% of the municipalities with defined goals, and 73.4% joined the Pacto pela Saúde 2010/11 Date achieved 12/31/ /09/ /31/2014 Comments (incl. % achievement) Target exceeded. Revised in the Project restructuring to reduce complexity. Target revised to 40% of states in Source of data: Project Information System SGP2. Proportion of states with monitoring and evaluation plans implemented and Indicator 10 : evaluated in participating states (as a group). Value (quantitative or Qualitative) Date achieved 12/31/ /09/ /31/2014 Comments (incl. % Target met. Source of data: SGP2. achievement) Proportion of states with 10% of municipalities with less than 100,000 Indicator 11 : inhabitants that implemented self-assessment. Value (quantitative 0 10% Municipalities 50% States 100% (26) states, 94.6% of v

12 or Qualitative) 50% States in 2013 municipalities use the AMAQ Date achieved 12/31/ /09/ /31/ /31/2014 Target exceeded. Revised in Project restructuring to reduce unnecessary Comments (incl. % achievement) complexity and to transform it into a numeric indicator, eliminating ambiguity. Also, AMQ was replaced by AMAQ and PMAQ that were implemented on a large scale. Target changed to 50 percent of states in Source of data: Database of PMAQ-AB, DAB/MOH. Establishment, at the federal level, of a results-based management system that Indicator 12 : links project financing to states and municipalities. Value (quantitative or Qualitative) 0 System developed and tested Implemented in 3,965 municipalities (in all states) Date achieved 12/31/ /09/ /31/2014 Comments Target exceeded. Indicator updated to reflect evolution of AMQ into PMAQ (incl. % system. Target revised to reflect continued progress. Source of data: DAB/MOH. achievement) Indicator 13 : Cost accounting system (at federal level). Value (quantitative or Qualitative) 0 Study concluded and system Developed Study on costs and tripartite spending on primary healthcare carried out in partnership with IPEA and DESID/MOH Date achieved 12/31/ /09/ /31/2014 Comments (incl. % Target met. Source of data: DAB/MOH. achievement) Inter-municipal cooperation plan implemented in each state to strengthen PSF, Indicator 14 : specifying activities in three areas: management, coordination, Service Provision. Value Communities of 1 per state (20 (quantitative 0 practice states) or Qualitative) established. Date achieved 12/31/ /09/ /31/2014 Comments (incl. % achievement) Indicator 15 : Value (quantitative or Qualitative) Target met. Details at: dab.saude.gov.br/portaldab comunidade_praticas.php. Source of data: DAB/MOH. Proportion of family health teams participating in the Program for Improving Access and Quality (PMAQ-AB). 58.3% in the 1st Model developed cycle of PMAQ; and tested 88.0% in the 2nd cycle. Date achieved 12/31/ /09/ /31/ /31/2014 vi

13 Comments (incl. % achievement) Indicator 16 : Target exceeded. Indicator was revised during restructuring. The baseline was revised to 50% in 2011, and target revised to 60% in Source of data: Database of PMAQ-AB, DAB/MOH. Proportion of participating municipalities that implemented the Health Information System for Primary Care (SIS-AB). Value (quantitative or Qualitative) Date achieved 12/31/ /31/ /28/2015 Comments (incl. % achievement) Indicator 17 : Value (quantitative or Qualitative) Target not met. This indicator was introduced in the Project restructuring. It monitored the upgrading of information systems and integrated data systems to support performance monitoring of primary health care and the PSF. Source of data: SISAB and DAB/MOH. Proportion of primary care units (UBS) in participating municipalities that have a Tele-health access point. 47% of municipalities have 0 20% Tele-health access point in support of PSF units Date achieved 12/31/ /31/ /31/2014 Comments (incl. % achievement) Target exceeded. This indicator was introduced in the Project restructuring. It monitored the upgrading of primary care units to units with tele-health access points. Source of data: Tele-health Monitoring System. G. Ratings of Project Performance in ISRs No. Date ISR Archived DO IP Actual Disbursements (USD millions) 1 06/25/2008 Satisfactory Satisfactory /16/2008 Satisfactory Satisfactory /19/2009 Satisfactory Moderately Satisfactory /09/2009 Satisfactory Satisfactory /30/2010 Satisfactory Satisfactory /26/2011 Satisfactory Satisfactory /23/2011 Satisfactory Moderately Satisfactory /12/2012 Satisfactory Moderately Satisfactory /11/2012 Satisfactory Moderately Satisfactory /05/2013 Satisfactory Moderately Satisfactory /28/2013 Satisfactory Moderately Satisfactory /13/2014 Satisfactory Unsatisfactory /23/2014 Moderately Unsatisfactory Unsatisfactory vii

14 H. Restructuring (if any) Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO Amount Disbursed at Restructuring in USD millions 03/14/2013 N S MS IP Reason for Restructuring & Key Changes Made a) The Results Framework - including indicators, baseline, and targets has been revised with updated available statistics to increase clarity and reduce repetition by improving the accuracy of indicator definitions and data. b) Two new activities were included for financing under Component 3: an International Seminar of Primary Care, and deployment of e-sus-ab. c) Proceeds have been reallocated to transfer US$30.2 million not utilized by the municipalities, states and the Federal District by March 30, 2013 to the federal component (Component 3). d) The closing date has been extended by 21 months from March 30, 2013 to December 31, 2014 to allow the Government sufficient time for implementation and goal achievement of three new priority activities under Component 3 of the Project. viii

15 I. Disbursement Profile ix

16

17 1. Project Context, Development Objectives and Design 1. In March 2002, the World Bank s Board approved the three-phase, seven-year US$550 million Brazil Family Health Extension Adaptable Program Lending (PROESF, the Program), 1 together with a US$68 million loan in support of the Program s Phase 1 (PROESF 1). The Program was meant to improve utilization and quality of publiclyfinanced health services by: (i) expanding and consolidating coverage of the Government of Brazil s (GOB) Family Health Program (Programa Saúde da Família - PSF) in about 187 municipalities, establishing well-articulated referral and counter-referral systems, and introducing performance-based financing and management arrangements; (ii) establishing family health as a core element of health professional and para-professional training; and (iii) strengthening Ministry of Health (MOH) capacity to monitor and evaluate PSF health services, policies and training activities on a systematic basis. 2. PROESF 1 closed on June 30, 2007 with satisfactory outcomes. 2 Phase 1 focused on extension of population coverage, establishment of a federal transfer mechanism in support of program-financed investments, pre-service and in-service training of family health professionals and para-professional, development and testing of a performancebased financing and management system, design of a quality assessment system, strengthening of a monitoring system, collection of baseline data for impact measurement, and development of protocols for referrals and care practices for PSF teams. 3. With originally established policy and implementation triggers substantially met, 3 a US$83.45 million loan in support of the Program s Phase 2 (PROESF 2) was approved by the Board on April 25, PROESF 2 would build upon the accomplishments of PROESF 1, while strengthening the quality of PSF care and states capacity to monitor and evaluate PSF services. Beyond supporting continued extension of coverage in the 187 Phase 1 municipalities, PROESF 2 would support quality improvements, strengthen the capacity of states to monitor and evaluate PSF, introduce the developed performance-based financing mechanism between the federal government and participating states and municipalities, and develop and test a results-based management system for PSF teams. 1.1 Context at Appraisal 4. Brazil had made significant progress in human development since PROESF was approved in While Brazil had achieved impressive gains in poverty and inequality 1 Adaptable Program Lending (APL) was a World Bank s lending instrument used to support long term programs. 2 World Bank, Implementation Completion and Results Report on a Loan in the amount of US$68.0 million to the Federative Republic of Brazil in Support of the First Phase of the Family Health Extension Adaptable Lending Program, Report No. ICR , dated December 21, The trigger related to an average of 35 percent of population in participating urban municipalities registered with Family Health teams was substantially met. At the outset of PROESF 1, the Government of Brazil decided to expand the number of participating municipalities from 40 to 187, nearly quadrupling the population base, but resulting in 34 percent coverage falling short of the 35 percent coverage target. 1

18 reduction, challenges remained. Poverty had decreased in the twelve years preceding PROESF 2 approval. Using reference lines for Brazil s Bolsa Família program and the Brasil Sem Miséria Plan to track the evolution of poverty, between 1995 and 2012, extreme (R$70) and moderate poverty (R$140) fell from 10 percent to 4.3 and 24.6 to 9.4 percent. 4 While the population s health status had improved significantly, the PROESF 2 Project Appraisal Document (PAD) highlighted the following shortcomings: (a) inequalities in health status, health financing and service utilization among regions, states and municipalities, income groups and between urban and rural areas; (b) inefficiencies in the health care delivery system that remained hospital- and specialty-centric; and (c) quality and effectiveness problems including the under-provision of health promotion and prevention interventions, weaknesses in the referral system, lack of dissemination and use of cost-effective treatments, and the absence of functional networks to facilitate the application of case management across all levels of care. 5. Brazil had implemented major health reforms, starting in 1988, which had transformed the organization, financing and provision of health services, and created the Unified Health System (Sistema Único da Saúde - SUS), while transferring responsibility to the sub-national level. However, although states and municipalities had supported the reforms with financial buy-in, low managerial capacity and a lack of performance orientation compromised their ability to make continued gains. Setting performance benchmarks regarding efficiency, effectiveness and quality, providing financial incentives to achieve them, and monitoring and evaluating results were identified in the PAD as important next steps in the reform process. 6. The PSF was introduced in 1994 as the MOH s flagship program and main strategy for extending primary care, defining health priorities, and ultimately, improving health outcomes. As a community-based, outreach-oriented primary health care model with multi-professional teams delivering primary health care interventions and prevention to households and communities, PSF is a departure from earlier, traditional primary care approach. PSF teams serve as gateways to higher-level services, and are responsible for enrolling a given number of families within a defined catchment area, and monitoring their health status, risks and needs. PSF seeks to implement three essential attributes of primary health care: first point of access, continuity of care, and coordination with the health system. The program focuses primarily on maternal and child health interventions and prevention and early treatment of a few noncommunicable diseases (NCDs), such as hypertension and diabetes. PSF teams (consisting of a family health physician or generalist, a nurse, nursing assistant, community health workers, and, in some cases, dental health professionals) act as the entry point to the health system, both through outreach activities 4 While Brazil does not have official poverty lines, the lines used for Bolsa Família and Brasil Sem Miséria are frequently referred to in government documents and often serve as a de facto official poverty rate. Due to slight differences in poverty lines and calculation methodologies, poverty rates may differ from official numbers. 2

19 (e.g., household visits and community-wide health promotion events) and directing patients to higher-level services as needed. 7. PSF s population coverage had increased to 45 percent by 2005, with 50 percent coverage in municipalities under 100,000 inhabitants, and for nearly all rural areas. Further, early research findings indicated that PSF had a positive impact on health outcomes, including in terms of decreases in infant mortality, higher immunization rates, and reduction in hospital admissions for preventable diseases, among others. Still, PSF coverage lagged in the 250 large, urban municipalities (over 100,000 inhabitants), where over 90 million people (60 percent of Brazil s population) resided. Further, the program faced several challenges, including: (i) low productivity of health teams contributing to long queues; (ii) difficulties in converting traditional primary care units to the family health approach; (iii) low prestige of family health practice among physicians, contributing to turnover; (iv) poor quality of some health professionals; (v) weak referral systems; (vi) lack of information on quality of care, efficiency and costs; and (vii) absence of performance orientation at the service delivery level. 8. Rationale for the World Bank Involvement. The rationale for involvement was strong as the World Bank had supported the reform of Brazil s health sector through a series of investment and policy loans since its early stages. The Adaptable Program Lending (APL), through PROESF 1, had already contributed to an expansion of the PSF from 32 to 45 percent of the population. And, although World Bank financing represented only about five percent of federal financing, it had provided significant leverage to further the GOB s reform agenda, as Project financing was additional to the regular federal transfers, and most municipalities complemented financing with their own investment resources. MOH studies had shown that PSF coverage increased in municipalities that had participated in the Program, while coverage decreased in those that had not participated. The expansion of primary care coverage helped to reduce inequalities and costly inefficiencies in Brazil s previous hospital-centered delivery system. Finally, PROESF had promoted improvements in health system governance by specifying performance targets, linking federal financing to performance, strengthening monitoring and promoting systemic evaluation. Based on the PROESF 1 experience, the MOH approved regulation that tied increases in federal grant financing for primary care to the achievement of performance targets negotiated between the MOH and municipalities. PROESF 2 would continue to build on this experience, introducing more advance mechanisms linking financing to performance, providing incentives for municipalities to continuously improve coverage and quality, and rolling out the innovative quality assessment and monitoring systems that had been developed and tested under PROESF Original Project Development Objectives (PDO) and Key Indicators (as approved) 9. The Project Development Objectives (PDO) of PROESF 2 were to: (i) increase access to Family Health-based primary care in large, urban municipalities; (ii) raise the technical quality of and patient satisfaction with primary care; and (iii) improve the 3

20 efficiency and effectiveness of Family Health service providers as well as the broader delivery system Progress towards the PDO was to be measured by the following APL program and Intermediate Results Indicators: APL Outcome Indicators ( ): 1. Infant Mortality 2. Per capita contact with primary care providers. PROESF 2 Project Indicators 6 For participating municipalities (as a group): Access/equity: 3. Percent PSF population coverage. 4. Ratio of patients with hypertension and followed by PSF teams to estimated number of patients with hypertension. Effectiveness: 5. Percentage of infants <1 with full vaccination regimen (DPT-H, polio, measles, tuberculosis). 6. Percentage of pregnant women attended by PSF teams that have 7 or more pre-natal consultations. 7. Percentage of total medical consultations referred from PSF to hospital specialty services. Quality: 8. Percentage of all PSF teams applied quality evaluation instrument and are ranked according to standard (AMQ system). 9. Percentage of all PSF teams in a sample of municipalities using evidence-based clinical guidelines for hypertension and diabetes (based on sample survey). Efficiency: 10. Rate of hospital admissions of children <5 for Acute Respiratory Infection (ARI). 11. Rate of hospital admissions for stroke. Institutional: 12. Percentage of municipalities apply quality evaluation instrument in the areas of PSF management and coordination (AMQ parts 1 and 2). 5 The wording of the PDO in the Loan Agreement was slightly different, although broadly consistent with that in the Project Appraisal Document. It read: The objectives of the Project are to: (a) increase access to family health-based primary care in Eligible Municipalities (defined as any of the municipalities with 100,000 or more inhabitants eligible for participating in Part 1 of the Project in accordance with the requirements of the Operational Manual ); (b) improve the technical quality of, and patient satisfaction with, primary care; and (c) improve the efficiency and effectiveness of family health service providers as well as the broader delivery system. 6 There are minor differences in the indicators, and in their wording, as presented in the Project Appraisal Document on page 11 and in Annex 3. The Section above lists the Intermediate Results Indicators as they appear in Annex 3. 4

21 For participating states (as a group): 13. Percentage of states establish performance agreements with at least X% municipalities with <100,000 population. 14. Percentage of states with monitoring and evaluation plans implemented and evaluated, including: (i) the establishment of an M&E unit within State Health Secretariats; (ii) definition of performance indicators; (iii) definition and collection of baseline data for monitoring system; and (iv) documentation of analysis of data derived from monitoring system. 15. Percentage of municipalities with <100,000 population in X% states participate in quality assessment program (AMQ), including completion of self-assessment and development of plan to address quality gaps. For the Federal Ministry of Health 16. Establishment of a results-based management system that links project financing to states and municipalities along the following dimensions: objectives, performance indicators, outputs, inputs, investment priorities and spending; this would include integrated information system, manual to support implementation, and monitoring of implementation of a results-based management system. 17. PSF costs collected and analyzed, cost accounting system developed and implemented. 18. Major research projects on PSF (at least one on chronic diseases, one on patient satisfaction, and one on PSF impact on hospital admissions) concluded. 19. Plans, methods and Terms of Reference for Impact Evaluation prepared. 20. Proficiency test of PSF professionals developed and applied to a sample of recent graduates of all PSF training centers. 21. Inter-municipal cooperation plan implemented in each state to strengthen PSF, specifying activities in three areas: management, coordination, service provision 22. Percentage of ESF implement results-based management system, including the signing of performance contracts. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 11. The Project s PDO was not revised. The Results Framework, including Intermediate Results Indicators, baseline and targets, was revised in a Level 2 Project Restructuring in March 2013 to increase clarity and reduce repetition by improving the accuracy of indicator definitions and data. 1.4 Main Beneficiaries 12. Like PROESF 1, the Project was not targeted specifically at low-income beneficiaries since the PSF model aimed at providing primary health care for Brazil s entire population. Nevertheless, while access is universal, the Project was expected to benefit mainly low-income populations that rely heavily on public providers, particularly for health care, and therefore represented the main users of the SUS. One of PROESF s goals was to reach populations with limited access to and underutilization of primary health care 5

22 services. There was already substantial PSF coverage among small municipalities with large rural areas, as these had been the focus of the program s initial scale up because of the absence of providers, low population density and political interest in those areas. PROESF 2 was to support further coverage extension exclusively in 187 municipalities with over 100,000 inhabitants that had participated in Phase 1. About 38 percent of those in the poorest income quintile lived in urban areas, and health indicators among the urban poor were as bad as or worse than indicators among the rural poor. PROESF 2 would also support states to provide technical assistance, training and managerial support to small municipalities in rural areas, and to the MOH in strengthening the Program s systems and its ability to provide support to states and municipalities. 1.5 Original Components (as approved) 13. The Project included three components: (1) expansion and consolidation of Family Health care in municipalities over 100,000 inhabitants that participated in Phase 1; (2) strengthening state capacity for technical support, supervision, and monitoring of family health services; and (3) strengthening federal policy and technical support to, and oversight of the family health program. Municipalities and states were to submit subprojects prepared by them on the basis of their needs for financing and implementation under Components 1 and 2. The concept involved linking levels of financing to levels of performance in terms of coverage extension, quality improvement, efficiency, and fiduciary actions. The accountability arrangement consisted of signed management contracts that specified performance indicators, as well as spending and implementation plans. Performance was to be assessed at subprojects mid-term and conclusion. Performance levels achieved at mid-term would determine financing levels for the subsequent 18 months of execution. Performance levels achieved at the conclusion of the APL Phase 2 would determine municipal and state participation as well financing levels in APL Phase 3 (Section 2.1). The full Project description is presented in Annex 10 and summarized below. 14. Component 1: Expansion and Consolidation of Family Health Care in Municipalities (US$ million Total Cost, US$55.0 million Loan Financing). Component 1 would support scaling up and strengthening PSF capacity at the municipal level, in 187 municipalities (out of 250 municipalities) with populations over 100,000. Coverage extension activities, aimed at raising PSF population coverage from about 34 to 37 percent would continue to focus on large urban areas that are characterized by poorly integrated, traditional delivery system, with large numbers of families living in poverty and confronting severe social risks. 15. Component 2: Strengthening State Capacity for Supervision, Monitoring and Technical Support of Family Health Services (US$17.45 million Total Cost, US$12.45 million Loan Financing). Component 2 would strengthen state capacity to: (i) support municipalities extend coverage, improve technical quality, and manage service delivery, with a special focus in municipalities with less than 100,000 inhabitants; (ii) improve the quality of family health human resources; and (iii) conduct monitoring and supervision as key functions of state health secretariats. The component would finance goods, materials, 6

23 training and technical assistance for activities implemented by 26 states and the Federal District. 16. Component 3: Strengthening Federal Oversight of the Family Health Program (US$15.8 million Total Cost, US$16.0 million Loan Financing). Component 3 aimed at strengthening PSF oversight capacity at the federal level, by strengthening the MOH s capacity to provide support to states and municipalities in implementing Components 1 and 2. This component was to finance goods, training, technical assistance and Project incremental operational costs for the development of PSF accountability arrangements, including results-based financing and management, and support the states and municipalities to develop training programs, research, monitoring and evaluation arrangements, and coordination and supervision capacity. It would also finance research and evaluative studies on specific aspect of PSF service organization and delivery as well as institutionalize primary care/psf impact evaluation in the MOH. Several activities were to support implementation of interventions under Components 1 and Revised Components 17. The Project s components were not revised, although new activities were included under Component 3. The Project restructuring in March 2013 (Section 1.3) introduced two new activities both consistent with the PDO for financing under Component 3: an international seminar of primary care, and deployment of e-sus-ab. Deployment of the e-sus-ab entailed implementation of electronic medical records, including patients medical and health service history, inventory control of pharmaceuticals and supplies, appointment scheduling, among other data of every primary care health unit. The two new activities were in support of the new National Primary Care Policy (Section 2.2), and represented an important attempt to adjust PROESF 2 to the Government s new priorities. The restructuring also reallocated about US$30 million from amounts allocated to municipalities under Component 1 (Category of Expenditure 2) and to states under Component 2 (Category of Expenditure 3) to Component 3 to finance these new activities. 1.7 Other significant changes 18. The closing date was extended by 21 months from March 30, 2013 to December 31, 2014 to allow the Government sufficient time for implementation and goal achievement of activities under Component 3 of the Project. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 19. As the second phase of an APL, the Project was prepared in a relatively short period, from Concept Review on March 1, 2006 to Appraisal in December PROESF 1 was still under implementation, and preparation of PROESF 2 was carried out together with final supervision of the first phase project. Negotiations took place only in March 2008, one year after the World Bank had sent the invitation to negotiate. The US$83.45 million 7

24 loan was approved by the Board on April 25, 2008, and signed almost 17 months after that. Delays in signing can be partially explained by a procurement issue not related to the Project the Brazilian Federal Court of Accounts (Tribunal de Contas da Uniao TCU) issued Acórdão No. 2690/2008, which prevented the Borrower to adhere to the World Bank procurement rules. 7 This fact made the Federal Government stop authorizing signing of any new external financed loan until May 2009, when the Acórdão was suspended. The Loan Agreement became effective on October 15, 2009, right before completing the 18- month deadline after Board approval. 20. World Bank support to the Government s Project preparation efforts was provided largely by the same team that had prepared and supervised PROESF 1. As a result, the Project s preparation and design benefited from the lessons learned of PSF s earlier implementation, which were described in detail in the PAD. PROESF 2 was an appropriate use of the APL instrument to support a program that needed a long-term horizon required to meet its objectives. The Project built upon the positive experience of PROESF 1, while further refining and adjusting the Program s structure and indicators to improve and better measure outcomes. For example, implementation under Phase 1 identified the need for a permanent cadre of federal- and state-based technical and fiduciary personnel to provide support to participating municipalities. This was important for municipalities with low capacity (as reflected in technical and fiduciary assessments of performance during Phase 1), for which capacity-building plans would be developed and implemented. PROESF 2 design gave states a greater role in monitoring program and project performance, and in providing technical and fiduciary support to participating municipalities under their jurisdiction. States were eligible for financing (under Component 2) to strengthen their capacity to carry out these and other functions. 21. PROESF 2 maintained, but further improved, the Program s results-based financing system to stimulate performance of participating states and municipalities. Financing continued to be linked to levels of performance in terms of coverage extension, quality improvement, efficiency and fiduciary actions. Signed management contracts between the MOH and the states and participating municipalities would specify a more manageable (than in Phase 1) number of easily verifiable performance indicators, as well as spending and implementation plans, that would be monitored at mid-term and conclusion, with subsequent levels of financing determined by performance. 8 Initial levels 7 With respect to the TCU decision on the need to provide detailed budgets to the bidders, the World Bank shared its concern through a letter to the Borrower dated February 4, Confidentiality of the bids is a policy requirement of the World Bank under paragraph 2.47 and 2.31 of the Procurement Guidelines for Goods and Works and Consultant s Services respectively. 8 The performance-based financing scheme involved one health (coverage) indicator and another composite indicator that was to measure efficiency, effectiveness and fiduciary improvement. Municipalities could increase their subproject financing by up to 2.5 times its original amount by complying with benchmarks, thereby providing a strong incentive to comply with performance indicators. Municipalities could also be awarded an additional 15 percent of financing by achieving targets linked to elective indicators. Conversely, poor performance could result in downgrading, and continued non-performance could result in exclusion from Project financing. A similar scheme based on different indicators was to be applied to the states under Component 2. 8

25 of financing to municipalities would be determined on the basis of performance under PROESF 1, thereby avoiding delays in initial implementation. In addition to linking payments to levels of performance as under PROESF 1, results-based financing under the Project would provide bonus payments for achievement of performance targets tied to elective indicators, eligibility criteria 9 and exclusion for non-execution. 22. Phase 2 was also to continue the successful pooled financing approach whereby loan financing was comingled with two large primary health care incentives--piso de Atenção Básica (PAB) fixo and PAB variável, which are the main sources of federal funding for primary care. 10 The pooling of World Bank and GOB funding would offer several advantages, including that it would: (i) allow for a single procurement, financial management and monitoring system, reducing the transaction costs of parallel systems; (ii) direct GOB and World Bank efforts to technical and fiduciary oversight, monitoring and strengthening of the larger PSF program rather than ring-fencing a much smaller World Bank-financed project; (iii) insert the Project into a high-priority programmatic and policy framework; and (iv) create an opportunity to link both World Bank and GOB financing to results. Approximately 65 percent of loan financing would be allocated through municipal subprojects (Component 1), and about 15 percent to state subprojects (Component 2), pooled with the MOH s regular transfers. 23. As support to an ongoing and well-established GOB program, PROESF 2 counted upon the Government s full ownership and financial commitment. Family health, particularly expansion of PSF coverage in urban municipalities, was undoubtedly the highest priority for the SUS. Further, the roles and responsibilities of the different levels of government were clearly defined. Government budgetary allocations to PSF had increased by 10 percent annually between 2001 and 2006, and the World Bank s loan represented a small fraction of overall PSF financing. In additions, the PSF had been discussed and strongly endorsed by stakeholders at the state, municipal and local levels, and the priorities and indicators to measure progress were defined based on consultations with these groups. 2.2 Implementation 24. Several issues impacted the implementation of PROESF 2 that resulted not only in implementation difficulties in terms of contracting by states and municipalities, but also and more importantly, the Project s design losing relevance. Despite initial delays, early implementation proceeded smoothly. A change in administration at the federal level, resulted in greatly increased priority to primary health through a new National Primary 9 Eligibility criteria for continued project participation for states and municipalities included: (i) presence of a full-time coordinator for family health and primary care; (ii) application of Project financing to investments and costs involved in primary care expansion and quality improvement; and (iii) timely provision of documentation of at least 75 percent of eligible expenditures in the previous year. 10 PAB fixo is a fixed amount, based on each municipality s population, transferred monthly from the federal health fund to municipal health funds. PAB variável is an additional amount, based on municipal adherence to specific MOH s health programs, such as the family health program (PSF), pharmaceutical program, health in school program, and indigenous program. 9

26 Care Policy (Política Nacional de Atenção Básica - PNAB) that introduced several new initiatives and provided greatly increased funding, but strained the capacity of the limited staff in the MOH. The gradual evolution of World Bank investment lending procedures, with more rigorous up-front fiduciary assessments, resulted in a slight, but important, modification in the application and acceptance of Brazil s national competitive bidding procedures. Difficulties in following the World Bank s procurement requirements (Section 2.4), coupled with increased funding under the PNAB led to decreased demand for PROESF 2 funding by municipalities and states that eventually resulted in only about one third of the original loan amount being disbursed. Other issues that affected implementation included: multiple staffing changes resulting from state and local elections and project staffing. 25. Delayed start-up. The MOH made progress in early implementation with its own resources, as signature of the loan only took place on September 9, 2009 and effectiveness on October 15, The lag between approval and signature is an issue that affects the entire Brazil lending portfolio because all projects require, among other steps, ratification by the Senate before signature. 11 Although this delay did not impact early implementation, it was clear that the Project s implementation schedule would be affected throughout. As a result, the MOH requested and the World Bank agreed, before signature, to postpone the loan closing date by 12 months to March 30, Despite early progress towards its Intermediate Results Indicators, several issues that surfaced early on affected PROESF 2 implementation through completion. These included: 26. Political Environment. Implemented at three levels of government, the Project was affected by government changes at the federal, state and municipal levels. Gubernatorial elections in the 26 states and the Federal District were held concomitantly with presidential elections in 2010, and municipal elections were held in 2008 and Changes resulting from state and municipal elections invariably affected staffing continuity at those levels (not only in the health secretariats, but in other agencies that had a role in approving procurement and making other decisions). Perhaps the most significant impact resulted from changes at the federal level. Dilma Rouseff of the Workers Party (Partido dos Trabalhadores) was elected President in November 2010 and assumed office in January Although from the same party as her predecessor, President Rouseff from the start assigned high priority to primary health and appointed a new Minister of Health to help design and implement a new primary health policy. 27. Primary health care in Brazil is financed by the federal government, states and municipalities, and delivered at the municipal level. States can establish their own delivery programs, to be implemented by municipalities under their jurisdiction. During implementation, for instance, the states of São Paulo and Rio de Janeiro were pursuing a strategy of establishing emergency and specialized units (Ambulatory Medical Care and 11 In addition to routine delays in obtaining Senate ratification, the loan for the Project was affected by an increase in the World Bank s fixed spread, for which the Brazilian Treasury had to recalculate the loan s debt analysis and resubmit the loan to the Senate for a subsequent approval. There were also discussions on how the external financing funds would flow to Treasury account. 10

27 Emergency Care Unit), and increased dramatically the funding for primary health care of these alternative models, which competed with further expansion of PSF coverage. 28. National Primary Care Policy. The new PNAB was approved by a ministerial Portaria in October 2011 providing directives and guidelines on primary health, including for the PSF (Section 3.1). The PSF remained at the center of the primary health care delivery model, but new initiatives were introduced, including changes in federal financing arrangements to include both equity and quality as criteria. Under PNAB, financing of primary health now includes resources: (i) per capita; (ii) for specific investments such as the Requalification Program for Primary Health Units (Programa de Requalificação das Unidades Básicas de Saúde); (iii) for adopting strategic and priority programs, such as the PSF (and others); and (iv) conditioned to results and ex-post evaluation of access and quality according to the National Program for Access and Quality Improvement (Programa Nacional de Melhoria do Acesso e da Qualidade - PMAQ). More importantly, resources allocated to primary health more than doubled in five years, from R$9.8 billion in 2010 to R$20 billion in Project Staffing. With a multitude of new programs and initiatives, dramatically increased funding and heightened priority, the PSF project unit in the MOH, established under PROESF 1 to coordinate the PSF and World Bank financial assistance, was integrated with the Ministry s staff in its Department of Primary Care (Departamento de Atenção Básica - DAB). Although there were some staff that worked exclusively on the Project, the PAD had contemplated that the PSF Project unit would be strengthened with consultants to assist in reviewing and approving municipal and state subprojects, providing assistance to municipalities and states for the implementation of their subprojects, carrying out site visits, developing training strategies and monitoring Project indicators. There were several attempts to hire additional staff, especially to provide hands-on assistance to states and municipalities, but these did not materialize until late in Project implementation. In 2013, the MOH entered into an agreement with Oswaldo Cruz Foundation (FIOCRUZ) to provide this assistance. 30. Issues at the Municipal Level. Municipalities also had issues that affected the contracting of specialists to their PSF units. Several municipalities and states were affected by the ceilings of the national Fiscal Responsibility Law, which precluded the contracting of staff, as specified in their convenios (service delivery and performance agreements) with the MOH. The Mid-term Review (MTR) also reported issues with availability of qualified medical doctors in some municipalities. Several municipalities found different options to overcome these issues and expand PSF (social organizations, short-term contracts for health personnel, etc.). 31. World Bank Investment Lending. PROESF 1 had introduced innovative fiduciary arrangements in which World Bank loan funds were pooled with those of the GOB in the grant-based financing system through which the federal government funds health services 12 Equivalent to approximately US$3.3 billion and US$6.7 billion, respectively. 11

28 managed by states and municipalities. These arrangements provided inter alia for: (i) financing a share of the government transfers instead of a share of each transaction; (ii) disbursing against Statements of Transfers by the MOH; (iii) applying national rules and systems for procurement and financial management; (iv) annually certifying expenditures by states and municipalities through internal and independent audits; and (v) carrying out fiduciary risk assessments and strengthening capacity. 13 An advantage of this approach was that implementation support could focus on sectoral, program issues as opposed to routine fiduciary management. PROESF 1 had been somewhat path breaking in adopting these procedures, at least for operations in Brazil. Since PROESF 1 was prepared in early 2001, World Bank lending instruments and procedures have evolved and sector-wide approaches and results-based lending, including the new Program for Results instrument, have been institutionalized. As a consequence, up-front fiduciary assessments have become more rigorous. The PROESF 1 and the PROESF 2 PADs had allowed for the application of national procurement procedures under subprojects; nevertheless, restrictions to fully accept national procurement legislation were introduced in PROESF 2, causing procurement to surface as an important implementation issue for states and municipalities (Section 2.4). 32. Impact of changes on PROESF. With increased funding and the new programs introduced by the PNAB, the PSF, which had already witnessed major expansion, remained a preferred delivery model, but became only one of several programs. The revised interpretation of the World Bank s procurement requirements (Section 2.4) and introduction of new MOH procedures, which made the reporting requirements for PROESF 2 different than those of the PROESF 1, resulted in the Project being managed as a ringfenced investment project, with separate procedures and reporting requirements. It became a complex Project implemented by 143 municipalities, 26 states and the Federal District and also the MOH, whose priority had changed from mere expansion of PSF to one of improving quality and efficiency in the primary health system. The complexity of operating in this new reality, coupled with massively increased funding from federal grant financing, reduced demand for PROESF 2 funding by states and municipalities who saw the PROESF 2 requirements as excessive vis-à-vis the amount of resources that could be made available. 33. Coordination with QUALISUS. Starting at the Project s MTR in May 2012, implementation support increased coordination with that of the QUALISUS Project. 14 There was substantial coordination among task teams supporting both projects, and an especially strong effort to build on the QUALISUS experiences, and apply them to PROESF 2. More importantly, when the GOB advised the World Bank that it no longer desired continued funding for municipal and state subprojects, provisions for funding the information technology equipment for the e-sus system were included with financing to be shared between both projects. Co-financing was an attempt to respond to the 13 Implementation Completion and Results Report, First Phase of the Family Health Extension Adaptable Lending Program, Report No. ICR dated December 21, BRAZIL QUALISUS-REDE Brazil Health Network Formation and Quality Improvement Project, Report No BR dated December 17,

29 Government s emerging priority and request for funding. The equipment was to be procured under one International Competitive Bidding process that would be co-financed through the two World Bank-financed projects: 60 percent from the PROESF 2 loan, and 40 percent from the QUALISUS loan. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 34. Design. Monitoring and evaluation (M&E) was central to the design of PROESF 2, building upon the progress under the PROESF 1. The Project aimed to strengthen the M&E system established under PROESF 1 while continuing to provide technical assistance to municipalities, states and the MOH. M&E was designed not only for implementation monitoring (and monitoring according to indicators in the Project s Results Framework), but, more importantly, to systematically collect performance information and data that would be used to link resources to performance targets. Five sets of indicators were to be monitored by the M&E system: 15 (i) PROESF 2 performance indicators included in federal-municipal agreements, including those on coverage, efficiency, quality and institutional capacity, that would determine levels of financing; (ii) PSF performance indicators not included in federal-municipal agreements that constituted a broader set of Program indicators used by the MOH to monitor primary care; (iii) Project-oriented institutional/managerial capacity performance indicators not included in federalmunicipal agreements, including existence of PSF coordinating units, enrollment of families by PSF teams, management contracts signed with PSF teams, etc.; (iv) fiduciary indicators, including compliance with implementation plans, existence of investment monitoring and reconciliation systems, existence of budget and bank account for project financing, etc.; (v) environmental indicators to monitor compliance with both building and environmental protection, and medical waste management and disposal legislation and guidelines; and (vi) ex-post survey data for impact evaluation. 35. There were some deficiencies in the design of the Project s Results Framework, however. First, the PDO was complex and there was no indicator to measure patient satisfaction, although this was part of the PDO. Furthermore, several indicators were either overly complex or were not adequately measuring the attainment of development goals as both positive and negative changes pointed at potential improvements in primary care (e.g., hospital admissions for acute respiratory infections). The PAD, in Annex 3, did not establish targets for PDO level indicators for PROESF More importantly, the Infant Mortality Rate was in retrospect probably inadequate as a development outcome indicator. It is a high level indicator influenced by many factors, including ones that are not specific to the health sector, such as real increases in the 15 The PROESF 2 M&E framework, indicators and arrangements are discussed in detail in Annex 3 of the PAD. 16 The PAD specified two outcome indicators for the three-phase APL, and included as a target an increase of at least ten percent over (2003) baselines in participating municipalities by the end of the APL program: (i) Infant Mortality (per 1000 births), baseline of 17.8 and target for Phase 3 of 16.0; and (ii) Per capita contact with primary care providers (increase) from baseline of 1.4 to 1.6. It did not include intermediary targets for PROESF 2. 13

30 minimum wage; a major expansion of the Government s conditional cash transfer program - Bolsa Familia, as well as of similar transfer programs at the state and local levels; improvements in infrastructure, notably, water supply and sewerage, as well as improvements in education. It would be difficult to attribute progress in infant mortality to World Bank-financed project that represented a very small subset of all investments federal, state and municipal in primary health. 37. Implementation. M&E implementation faced challenges, not the least of which was limited capacity of some municipalities to implement monitoring and a lack of technical support provided by the limited MOH project staff. Additional challenges included: (i) problems in the definition of indicators in the Project s Results Framework, including incompatibility with indicators required for reporting to the MOH; 17 (ii) the roll out of primary health care information systems upgrades, and the challenges in adopting new upgrades among municipalities, that led to lack of consistency and comprehensiveness of reporting among all states and municipalities; and (iii) the concomitant introduction of PMAQ, with its own set of indicators and evaluation processes. 38. Utilization. M&E was utilized not only to monitor progress towards expected outcomes, but also to determine state and municipal eligibility to participate in PROESF 2 and to establish the level of transfers, including a performance incentive. State and municipal eligibility for funding from PROESF 2 was to be established in two stages. Eligibility for funding in the first stage was determined based on performance under PROESF 1, and eligibility in the second stage would depend on continued good performance, and at least 30 percent execution of funding within 12 months of receipt. 2.4 Safeguard and Fiduciary Compliance 39. Safeguards. The Project was classified as Category B based on two potential environmental impacts, including those from: (i) new construction and rehabilitation of ambulatory health units, and (ii) medical waste management. An Environmental Framework was prepared and incorporated in the Operational Manual to guide selection, screening, construction and monitoring of new health unit construction. It also detailed the development and implementation of medical waste training program and plans to supervise the medical waste management system in family health units. No significant safeguards issues surfaced during implementation. A thorough assessment of safeguard policies compliance carried out during the MTR reported satisfactory compliance, with only minor recommendations with respect to medical waste management. 40. Fiduciary. Strengthening fiduciary capacity at state and municipal level was very much at the center of PROESF 2. Upon completion of PROESF 1, a financial management 17 For PROESF 1, the Government s and the Project s indicators were the same. When the Government introduced the PNAB, the indicators for PROESF 2 became different, so that the Project s indicators were superimposed on those that the Government used to monitor progress for all of its programs. 14

31 assessment found that all states and municipalities had maintained an accounting and financial management system for the specific purpose of recording PSF/PROESF information, and recommended that it would be possible and even preferable to rely on the MOH s existing financial management systems. Nevertheless, in practice, implementation of PROESF 2 represented a step backwards in terms of the fiduciary innovations. A new, strengthened Project information system (SGP2) had been put in place early on, which addressed planning, procurement, financial management, and monitoring. But several steps and procedures that were adopted resulted in fiduciary arrangements under PROESF 2 resembling those of a traditional investment operation, as described below. 41. On the financial management side, funds were transferred using the MOH s fundto-fund system, but separate special accounts were established in each state and municipality to channel loan funding. Funds were transferred to states and municipalities only after contracts were signed for the procurement of works, goods and services, and not against transfers as had been the case under PROESF 1. During implementation, the World Bank repeatedly requested strengthening of inventory control of goods acquired by states and municipalities, but the MOH was reluctant to address routine matters that in the federal environment are the responsibility of subnational governments. Financial planning was weak, as exemplified by the Project team s inability to secure enough resources in the MOH budget for loan funds to be spent in the Project s final year. Finally, delays in executing transferred funds by states and municipalities prevented the timely provision of statements of expenditures, and reimbursement of funds that were transferred but not executed. 42. With respect to procurement, states and municipalities had difficulties applying World Bank procurement requirements for goods and services, as well as for contracting of consultants (they were not necessarily familiar with the World Bank s consultant and procurement guidelines). The Loan Agreement for PROESF 1 had explicitly mentioned that goods and works for subprojects carried out by municipalities would follow the Procurement Law, meaning the Borrower s Laws 8666/93 and 10520/02 in their entirety. The Loan Agreement for PROESF 2 did not make reference to the Borrower s Laws; it only included the standard reference to National Competitive Bidding as an acceptable procurement method. As a result, in loco reverse auction (pregão presencial), provided for under Law No /02, was no longer recognized as an acceptable procedure for the loan. 18 Similarly, the e-reverse auction (pregão eletrônico), 19 which was still considered acceptable, was allowed only if carried out through certain financial institutions, not necessarily used throughout the country. In addition, states and municipalities were required, by the MOH, to prepare ex-ante procurement plans. Unfamiliar and uncomfortable to follow World Bank procurement requirements (including shopping procedures), despite limited training provided by the MOH and the states, officials in local governments (often in agencies outside the health sector) were presumably reluctant to not 18 Pregão presencial is an open bidding process for goods and services, in which the bidders presence in person is required. 19 Pregão eletrônico is an on-line bidding for goods and services, using information systems made available by several financial and other institutions. 15

32 follow national legislation. To compound the difficulties that these requirements created, the MOH was never able to contract the additional fiduciary specialists, either as staff or consultants, who were to provide hands-on guidance and technical assistance to states and municipalities on these matters. Although the MOH kept a team at DAB, this did not seem to be enough to mitigate procurement-related bottlenecks. 43. Application of the World Bank s procurement requirements also presented, according to the MOH, an obstacle to completion of several activities at the federal level. The studies planned under Component 3 were implemented with GOB resources, as the MOH found greater flexibility in contracting universities and think tanks by following national legislation. More importantly, the approval of the International Competitive Bidding (ICB) contract for information technology equipment for the e-sus system was delayed extensively, in view of the MOH s internal processes, especially with respect to defining technical specifications for the system. Once the required approvals were obtained in September 2014, there was no longer sufficient time to ensure that the goods and services would be provided before the loan s closing date. 44. Difficulties in following procurement and financial management requirements resulted in disbursements to municipalities and states of approximately US$1.4 million against ineligible expenditures. These were detected in the 2012 and 2013 audits by the Federal Controller General (Controladoria Geral da União - CGU). In addition, there were approximately US$5.8 million of amounts transferred to but not executed by states and municipalities. Thus, approximately US$7.2 million have been reimbursed to the World Bank in May Post-completion Operation/Next Phase 45. The Family Health Extension APL had originally contemplated three phases. However, the third phase will not move forward. The PSF is the Government s preferred model for delivery of primary health care under the PNAB. Financing to municipalities for PSF will continue to be provided through the Government s fund-to-fund transfer system. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 46. The PSF was, at the time that PROESF 2 was approved, and remains today a high priority for Brazil. Relevance of its objectives continues to be High. A recent World Bank document, Twenty Years of Health System Reform in Brazil: An Assessment of the Sistema Único de Saúde, reports that Brazil has seen impressive improvements in health outcomes, with dramatic reductions in child and infant mortality (and maternal mortality, to a lesser extent) and increases in life expectancy. Geographic inequalities in health outcomes were significantly reduced, with northeastern states benefiting the most, and disparities across socioeconomic groups also declined. However, significant inequalities in health status remain. The report highlights five key challenges facing Brazil s health care system: (i) 16

33 sustaining improvements in access to health care; (ii) improving efficiency and quality of health care services; (iii) clarifying roles and relationships across levels of government; (iv) raising the level and efficiency of government spending, and (v) conducting more and better health system monitoring and research. 20 The PDO of PROESF 2 responded to each of these priorities, and especially to the first. 47. The Government recognizes these challenges, which are a central feature of its recently approved PNAB. The PNAB prioritizes the PSF as a successful and preferred delivery model for the expansion and consolidation of primary health care, but it also incorporates other programs, with financing through its federal to municipal transfer system, to help address bottlenecks and constraints. These new programs build very much on the design, lessons and experience of PSF, supported by the PROESF APL. The new programs include: (i) a modification of the system of allocations of federal financing for primary care to include both equity (based on population and quality parameters); (ii) the PMAQ that evaluates and recognizes PSF units and municipalities that achieve previously agreed and contracted (among the federal, state and local units) results, and rewards them with supplemental resources; (iii) the establishment of primary care units for the homeless through mobile street units; (iv) an increase in the number of municipalities that can establish Núcleos de Apoio à Saúde da Família, interdisciplinary teams of medical specialists and other professionals that provide support to a pre-defined number of PSF teams; (v) a streamlining of procedures for establishing the PSF and other units in support of riverside populations; (vi) strengthening intersectoral links for health through, for example, early childhood initiatives; (vii) strengthening linkages among different levels of health care and among units, and providing access to distance health education through Telessaúde, that comprises integration of modern telecommunications and information systems and a new regulation policy; (viii) through the Unidades Básicas de Saúde program, modernizing and improving the services provided by primary health units through the construction of new and larger units, and refurbishing, expanding and providing information technology systems to existing units; and (ix) the Mais Médicos program aimed at augmenting the availability of medical doctors in remote areas and strengthening medical training through hands-on experience. 48. The Project s objectives are fully consistent with the second strategic objective of the World Bank s Brazil Country Partnership Strategy (Report #63731-BR) discussed by the Executive Directors on September 21, 2011: to improve the quality of public services for low income households, and expand their provision through public and private channels. 49. As a follow up to successful innovations with systems, monitoring and performance-based financing supported by PROESF 1, the Project s was relevant during preparation and early implementation. When the Government adopted its new PNAB in October 2011, introduced new programs and increased funding for them, PROESF 2 lost relevance quickly. Hence the Project s relevance for these dimensions is rated as Modest. 20 World Bank, Twenty Years of Health System Reform in Brazil: An Assessment of the Sistema Único de Saúde, Michele Gragnolati, Magnus Lindelow, and Bernard Couttolenc,

34 Federal funding for primary health care increased almost 40 percent from 2010, shortly after PROESF 2 began implementation, to A new program was established for the construction and rehabilitation of health centers and the PMAQ was launched. Reporting systems for these new programs were established, which differed from those required for monitoring PROESF 2 implementation. For instance, PMAQ became the standard for monitoring and evaluating performance by individual PSF units and municipalities. Implementation support made a strong effort to keep implementation on track. Nonetheless, demand for PROESF 2 funding by municipalities and states dropped considerably as financing from the federal government was readily available and did not require application of non-standard procurement procedures. 50. The World Bank s implementation support missions made a valiant effort to restructure the Project to support the GOB s priorities, to modify the results indicators to make them more compatible with the MOH s programs and systems and, thereby, to maintain its continued relevance (Sections 1.3, 1.6 and 1.7). The missions also tried repeatedly to foster the technical dialogue and contribute to activities and studies that were being implemented by the MOH. Unfortunately, continued difficulties in dealing with procurement bottlenecks (Section 2.4) as well as lack of interest from the Government to engage in a technical dialogue (e.g. the World Bank offer to support the impact evaluation of PMAQ or the analysis on costing of primary care), led to World Bank funds available under the federal component not being utilized for technical assistance. The changes to the Results Framework increased clarity and reduced repetition, but did not resolve the issue that PROESF 2 was rather a small Project within a much larger program. Its indicators still measured the Program s results, and not specifically those of the Project. Further, addressing weaknesses in the municipal and state components was no longer feasible as only the federal Component 3 was maintained after the restructuring, reprogrammed to finance information technology equipment. This was a very large contract that required additional approvals. The World Bank implementation team made strong efforts to provide support. Ultimately, delays in procurement precluded even this equipment from being financed under PROESF 2 in view of the Project s Closing Date Achievement of Project Development Objectives 51. As mentioned before, the Results Framework did not include targets for the outcome indicators in the second phase of the APL. However, the MOH established indicative targets for both of the APL PDO indicators: a 5 percent reduction in infant mortality in participating municipalities and a 5 percent increase in per capita contact with primary care providers in participating municipalities. By the time of the MTR, both of these targets had been met: infant mortality had been reduced from 17.8 per thousand live births in 2011 to 13.9 in 2003, and per capita contacts with primary care providers had increased from 1.4 in 2003 to 1.6 in At the time, PSF coverage in PROESF 2 municipalities had increased from 33.8 percent of the population in 2006 to 39.5 percent in 2011, exceeding the target of 37 percent. By Project completion, infant mortality had 21 The contract is expected to be fully financed with QUALISUS loan proceeds, which was extended until December 31,

35 been reduced further to 12.7 per thousand live births in The per capita contacts with primary care providers, however, had increased to 1.8 in 2012 but then began a downward trend, decreasing to 1.5 in 2014, falling short of target. 52. As described below, good progress was made towards PROESF 2 s objectives set out in Annex 3 of the PAD, as measured by the Intermediate Results Indicators: thirteen out of 17 indicators have been met. Yet, in considering PROESF 2 s accomplishments, a word of caution is in order with respect to attribution of results to the World Bank s financing. First, PROESF 2 funding made a very small contribution to the expansion and consolidation of PSF teams in participating municipalities and states. Of the 187 municipalities that were selected to participate in PROESF 2 based on their performance under PROESF 1, 170 signed subproject agreements with the MOH (171 were declared eligible and 13 were temporarily ineligible). Of these 170 municipalities that started, 55 were temporarily suspended for not having executed 30 percent of funding within 12 months (44 of these did not use any funding). Only 143 municipalities signed subproject agreements with the MOH for the second stage, and of these 67 never utilized the funding, seven utilized less than 50 percent, and only 32 utilized at least 90 percent of funding. All states and the Federal District signed subproject agreements, but their level of execution was low: only 22 percent of resources transferred were utilized, with the consequence that 15 of 26 states were temporarily suspended for not having implemented at least 30 percent of resources within 12 months. Equally important, although there are several studies that point to the relevance and effectiveness of the PSF, health outcomes are determined at least in part to developments outside the health system, all of which have shown improvements in recent years, as well: access to safe water and sanitation, quality food and education and the economic situation of households. Finally, the Government s increased funding outside the scope of PROESF 2, and the priority it assigned to new programs (some of which were incorporated in the Project, especially PMAQ), undoubtedly influenced Project outcomes and intermediate outcomes, as well. 53. Component 1 had three intermediate objectives, to: (i) increase access to family health-based primary care in about 187 large, urban municipalities; (ii) raise the technical quality of and patient satisfaction with primary care; and (iii) improve the efficiency and effectiveness of family health service providers as well as the broader delivery system. The achievement of these objectives, as measured by eight intermediate results indicators, was Negligible. As described above, the number of municipalities that participated in PROESF 2, and the utilized loan funding was significantly below what had been anticipated. Nevertheless, coverage of PSF in participating municipalities reached 48.2 percent, exceeding the 37 percent target for PROESF 2 and even the 40 percent target for a third phase of the APL. In addition, coverage in participating municipalities outpaced the 32 percent coverage in non-participating municipalities. The indicator on the proportion of patients with hypertension registered by PSF teams, among estimated population with hypertension (15 years of age and older) was 24.4 percent, falling short of the target of 35 percent. PROESF 2 met the intermediate objective of raising technical quality of and patient satisfaction, as measured by the indicators, but none of these actually measure patient satisfaction they measure technical quality. A study carried out by the Institute of Applied Economics Research (Instituto de Pesquisa Econômica Aplicada - IPEA) in 19

36 2010 found that the PSF was the highest rated strategy of the health care system: 80.7 percent of the persons interviewed responded that the PSF was good or very good. Finally, the third intermediate objective of this component was largely not met, as none of the targets for intermediate results indicators were achieved. Especially worrisome is the deterioration in the percentage of fully immunized infants (92.3 percent against a target of 95 percent), and percentage of women with live births attended by PSF that have 7 or more prenatal consultations (64.1 percent against a target of 70 percent). 54. Component 2 was to support the objective of improving the efficiency and effectiveness of family health service providers as well as the broader delivery system, at the state level, by improving their capacity for supervision, monitoring and technical support of PSF services. All of the states and the Federal District signed performance agreements with all of the municipalities, 52.8 percent of states have monitoring and evaluation plans implemented, and 94.6 percent of municipalities in 100 percent of states participate in quality assessment programs. Although targets for the three intermediate indicators have been met, the achievement of this objective is considered Modest as given the extremely low utilization of funds by states; it is more likely that improvements are linked to the establishment of the Government s PMAQ. In PMAQ s first cycle, 90 percent of PROESF 2 municipalities had entered the program. By the second cycle, 97 percent of PROESF 2 municipalities had joined, while countrywide only 91 percent had. 55. Lastly, Component 3 was to support the objective of improving the efficiency and effectiveness of family health service providers as well as the broader delivery system at the federal level by strengthening federal oversight of the PSF. As measured by six intermediate results indicators, progress was made towards this objective in the restructured Project, as evidenced by achievement of targets for three indicators. Yet, given the lack of updated data and the fact that it is difficult to attribute results to PROESF 2, the achievement of this objective is rated Negligible. A results-based management system linking financing to municipalities and states was established, a study on health care costs and tripartite spending on primary healthcare was carried out by the MOH in partnership with IPEA, communities of practice have been established, and 88 percent of municipalities have signed performance contracts with the MOH. However, the MOH implemented most of these activities with national resources as part of the new PMAQ program. The MOH has not provided information or recent data on either the number of municipalities that have implemented the new real-time, web-based health information system, e-sus. Most recent data shows, that 47 percent of PROESF 2 municipalities have implemented Telesaúde (supporting 3,979 PSF teams) which aims to expand and improve the delivery of primary health care by promoting better integration across the levels of the service delivery network, enhancing delivery at the first level of care and supporting realtime learning by health teams. The financing of information technology equipment for the e-sus was ultimately not financed by the Project. 3.3 Efficiency 56. A retrospective economic and financial analysis of the Project was prepared, and Efficiency is rated as Modest. This assessment was based on review and updating of the 20

37 analysis carried out during Project preparation, rather than a new analysis. It sought to assess whether the assumptions and projections made at the time held true, and assess the impact of possible changes in variables on the conclusions of the analysis. The initial evaluation performed for the PAD estimated the incremental costs and benefits over the Project lifetime (three years, mid-2008 to mid-2011) and over a longer period of 10 years, because of the expected lag in Project impact and their continuation over time. Costs were estimated at US$166.7 million, while the Net Present Value (NPV) of the net benefits were projected at US$745 million over 10 years. The Project was expected to yield a negative NPV during its implementation, and positive benefits thereafter. 57. Because PROESF 2 was intended to support an ongoing program (the PSF strategy) that expanded and evolved with or without the Project, it is nearly impossible to assign any observed improvement in health indicators to the Project alone. Therefore, for the purposes of this ICR, it was assumed that PROESF 2 has been a strong contributor to the impact of the PSF strategy as a whole, and the incremental benefits of PSF in general during Project implementation were measured. The results of this ex-post analysis, using more recent data, are less favorable than in the ex-ante analysis of 2008, but are still within the range of the sensitivity analysis performed during preparation. Actual 2014 results show Project costs at US$84 million, and a NPV of US$256 million. These results reflect recent trends in population growth and birth rate (lower than estimated in the PAD), lower initial coverage by PSF but faster growth, and recent evidence of somewhat lower impact on specific mortality rates; specifically, the number of births is much lower (20 percent) than estimated at the time, thus reducing the size of the potential impact on infant and maternal mortality. 58. Several factors are likely to affect the results of the analysis performed for the PAD, such as Project specific factors, changes in key variables, recent evidence of PSF impact, and Project costs. Discussion on these factors and their implications for the results of the analysis are detailed in Annex Justification of Overall Outcome Rating Rating: Unsatisfactory 59. PROESF 2 s Overall Outcome Rating is Unsatisfactory, based on (i) a Modest overall Relevance; (ii) Negligible achievement of intermediate objectives under Components 1 and 3, and Modest achievement of intermediate objectives under Component 2; and (iii) Modest Efficiency. While there is a strong body of evidence that the PSF has had a measurable positive impact, and PROESF 2 had the potential to continue contributing to this very important health service delivery model, its contribution to this success was marginal at best. Perhaps the APL program s largest impact was not the possible immediate results in terms of health care outcomes, although it undoubtedly did have some impact on these, but rather on piloting through the PSF a performance-based health financing model with an emphasis on monitoring and evaluation that has now been adopted in Brazil s federal health care financing through the PNAB generally, and the PMAQ specifically. 21

38 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development See Section 1.4. (b) Institutional Change/Strengthening 60. The most significant contribution of PROESF 2 was especially the performancebased financing modalities for linking federal transfers to results that have now been institutionalized in the Government s primary health care system, especially in the PMAQ. (c) Other Unintended Outcomes and Impacts (positive or negative) 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 4. Assessment of Risk to Development Outcome Rating: Substantial 61. It is difficult to assess the Risk to Development Outcome for an operation that did not produce expected outcomes, or better, for which contribution to the Project cannot be established. The Risk to Development Outcome for the PROESF APL is substantial, especially because the APL program lost relevance and the expected final phase will not materialize. Nevertheless, the PSF program that it supported is very much institutionalized in Brazil s health care delivery system, has produced promising results country-wide, and is now enhanced by several other complementary programs aimed at addressing the next steps in the primary health care agenda (i.e., quality), now that coverage of PSF has expanded considerably. Performance-based financing has been adopted as a complementary top-up to per capita and other transfers in the Government s fund-to-fund transfer system. 62. The Government remains fully committed to the PNAB, including the PSF, and new programs such as PMAQ, as well as to the continued roll out of its new e-sus primary health care information system (which has already begun). Although very low, the greatest risk to implementation of these programs is maintaining a sustained commitment to budgetary allocations. The macroeconomic scenario has deteriorated somewhat since the time PROESF 2 was prepared and since the PNAB was launched. Fiscal constraints, resulting from a more constrained budget, may come to affect if not the level of existing funding, at least the ability to maintain the past trend of increases in funding. Nevertheless, there is no reason to assume that funding for a priority program would be drastically reduced, and no budget cuts to primary health care programs have been announced. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 22

39 63. The World Bank s Performance in Ensuring Quality at Entry is rated as Moderately Satisfactory, as moderate shortcomings were observed. As the second phase of an APL, Project preparation was carried out in coordination with the supervision of the first phase, which was still under implementation. PROESF 2 built upon the successful outcomes of the earlier project and was fully responsive to the GOB s request for continued support for the PSF expansion. The preparation team extracted lessons, identified priorities for consolidating the highly successful PSF model, and incorporated them into the Project s design by making adjustments where justified. The same Task Team, based in the Country Office, that provided implementation support to the earlier project remained responsible for preparation of PROESF 2. The Team continued to provide constant support to the Project in its early phases, despite the extensive lag in effectiveness, to ensure that implementation would not be unduly affected by the delay. Yet, issues in the design of the Project s Results Framework (Section 2.3) resulted in difficulties in measuring the Project s outcomes. Also, the change in procurement procedures during preparation from those under PROESF 1, especially not accepting fully the terms of the Borrower s Procurement Laws No. 8666/93 and 10520/02 (Section 2.4), was an issue that came to affect the implementation of subprojects and the utilization of loan funds by both municipalities and states. (b) Quality of Supervision Rating: Satisfactory 64. The World Bank s Quality of Supervision is rated as Satisfactory, as minor shortcomings were observed. Supervision involved routine changes in Task Management who were based at headquarters (although the fiduciary team remained in the Country Office). Nevertheless, the transitions were seamless, which was all the more important in view of the environment in the sector that involved numerous changes implemented through the Government s PNAB, a dramatic increase in funding for primary health and the implications of these changes for the Project s implementation. Despite a statement to the contrary in the Borrower s Completion Report, the team made attempts to engage with the MOH in a technical dialogue regarding its new policy to no avail. During the MTR, the team specifically suggested that in light of reduced municipal and state demand for subproject financing, the MOH should consider requesting a partial cancellation of loan funds. The team also responded expeditiously to the Government s request to restructure the Project. When the MOH opted against a partial cancellation, the team worked with the Government to adjust the Results Framework to address issues with indicators and made revisions needed to adapt the operation to the Government s new priority programs, and coordinated closely with the Task Team supporting the QUALISUS project to synchronize the World Bank s response for assistance to the MOH, particularly regarding support for implementation and roll out of its e-sus system. The Development Outcome and Implementation Progress (IP) ratings in the Project s Implementation Status and Results Reports were only downgraded very close to loan s closing date to reflect the lack of progress in implementing actions agreed in the restructuring. However, given the implementation challenges described in Sections 2.2, 2.3, 2.4 and 5.2, the World Bank team could have been more realistic in performance reporting, especially for the IP rating that should, in retrospect, have been downgraded earlier. 23

40 (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 65. The Overall Bank Performance is rated Moderately Satisfactory in view of the ratings for both Ensuring Quality at Entry and Quality of Supervision, and the fact that Quality at Entry issues influenced the Project s outcome, despite strong efforts by the World Bank s Task Team to address them during implementation. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Unsatisfactory 66. The Government s performance is rated as Moderately Unsatisfactory, meaning that there were significant shortcomings. Despite continued priority to the PSF, and a continued desire for World Bank assistance in support of it, several factors impacted the Government s performance. First, Project preparation suffered greater than expected delays. Project preparation began in early 2006, but the corresponding loan was only approved more than two years later. Subsequently, effectiveness only took place almost eighteen months after approval. Although the effectiveness delay was not necessarily project-specific and initial Project activities were being implementing with Government funding, the delay impacted implementation of the municipal and state subprojects, slowing them down considerably. The new administration, which took office in 2011, reconfirmed the priority given to primary health care and significantly increased funding for it. The new programs adopted and implemented in accordance with the PNAB built on and expanded the performance-based financing model developed under PROESF 1, which was continued and improved under PROESF 2, while maintaining PSF as the preferred delivery mechanism. Nevertheless, throughout Project implementation, MOH s ownership and commitment to the Project implementation was questionable, and appointment of key staff to the Project Unit to support states and municipalities was delayed until There was no timely resolution of implementation issues as evidenced in the Aide Memoires and non-responsiveness, on occasion, by the MOH to management letters sent by the World Bank, which emphasized the Project implementation bottlenecks and called for MOH s actions. (b) Implementing Agency or Agencies Performance Rating: Unsatisfactory 67. The Implementing Agencies Performance is rated Unsatisfactory. Initially, the MOH/DAB began implementation of PROESF 2 without access to loan financing, and, as the PSF was an established program, already had in place the mechanisms to eventually fund PSF expansion in those municipalities that had qualified for the first round of funding under the Project. With the enactment of the new PNAB, the demands on the staff of the MOH s DAB increased dramatically; they were responsible to put in place the new programs, develop and implement new systems and manage the increased funding. The inability to contract additional specialists to support PROESF 2 resulted in the Project s coordination by MOH being absorbed by the existing staff of DAB, which was already 24

41 overstretched. In addition, administration of PROESF 2 became a low priority for the staff in DAB in view of the importance of new programs established under the PNAB. Difficulties encountered by municipalities and states in following requirements of PROESF 2, that had begun to diverge significantly from routine internal government procedures, compounded these difficulties, placing an even greater burden on the staff and heightened the need for additional supervision. The Project s restructuring required excessive time for internal approval, progress reporting was weak, the contracting of equipment for the e-sus suffered repeated delays (to the extent that it could not be financed by the loan) and there were large and consistent delays in addressing satisfactorily the agreements reflected in mission Aide Memoires and in responding to World Bank correspondences. In addition, DAB was not willing to engage in a technical dialogue with the World Bank on health sector policies, which could undoubtedly have served to maintain the relevance of PROESF 2 within the new PNAB. (c) Justification of Rating for Overall Borrower Performance Rating: Unsatisfactory 68. The Overall Borrower Performance is rated Unsatisfactory in view of the impact of the Government s and Implementing Agencies performance on the Project s outcomes. 6. Lessons Learned 69. A programmatic, medium-term approach to financing (previously provided as in the case of the Project through an Adaptable Program Loan) can be very valuable to help establish, and provide medium- to long-term continuity to a government program. However, even if the program established and supported continues to enjoy full government priority and funding, exogenous factors can come to affect the relevance, effectiveness and eventual results of a World Bank-financed program. Even a satisfactory or highly satisfactory program can rapidly lose relevance in view of changes in the overall sectoral policy environment or changes in the World Bank s own requirements (Section 2.2). These factors have to be deeply assessed during implementation, both by the World Bank team as by the Government, and clear decisions on how to proceed have to be taken at the management level of all institutions involved. 70. Even though a project or program supported by a World Bank-financed operation does not produce measurable, satisfactory outcomes, it can still have a major demonstrative impact. Specifically in the case of the PROESF APL, the innovative performance-based financing model that was piloted under the Project, along with the accompanying emphasis on monitoring and evaluation, have now been brought to scale and institutionalized in the government s primary health care program. Without a doubt, the PMAQ program, which incorporates an important quality aspect to health care financing and that includes formalized procedures for both internal evaluation and external evaluation (by universities), inspired upon the experience of the PROESF APL. In this sense, the World Bank s involvement was highly successful, and could have been used to foster the technical dialogue and other accomplishments. 25

42 71. Projects (and programs) that support increases in coverage produce notable results in their early phases. With coverage increased, and more basic needs met, further progress depends on improvements in quality of service delivery. But, quality improvements require the simultaneous adoption of inter-related measures, including professional qualifications, information systems, efficiency, etc., that require a longer implementation to converge. The focus and indicators need to incorporate a longer-term horizon. 72. Project implemented in federal frameworks, involving three levels of government, should consider more flexible procurement arrangements for the acquisition of relatively small goods and services. Requiring that all implementing agencies follow World Bank procurement guidelines especially when there are differences between those procedures and national legislation (Sections 2.2 and 2.4) can hamper implementation and increase costs of supervision while not necessarily producing better results. First, local governments are not familiar with World Bank procedures and dissemination of these requires a major, non-lasting investment. Second, even if dissemination and training is provided, local governments may be concerned that not following national legislation can have important implications with respect to compliance with national norms. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 73. In addition to the Executive Summary included in the Annex 7 of this ICR, The Project team in DAB provided marked-up comments (electronically) to the World Bank on a draft of this document that was submitted for the Government s consideration under cover of a letter dated June 8, These comments provided mostly clarifications to the writeups in paragraphs 29, 41, 42, 45, 50, 60, 62, 64, 66, 67, 70, and 72, and have been addressed in this ICR. (b) Cofinanciers (c) Other partners and stakeholders (e.g. NGOs/private sector/civil society) 26

43 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Components Appraisal Estimate (USD millions) Actual/Latest Estimate (USD millions) 22 Percentage of Appraisal 1. Expansion and Consolidation of Family Health Care in Municipalities 2. Strengthening State Management of the Family Health Program 3. Strengthening Federal Management of the Family Health Program Total Baseline Cost Physical Contingencies 1.3 Price Contingencies -5.2 Total Project Costs Front-end fee IBRD Total Financing Required (b) Financing Source of Funds Type of Cofinancing Appraisal Estimate (USD millions) Borrower International Bank for Reconstruction and Development Actual/Latest Estimate Percentage of (USD Appraisal millions) It was agreed with the Financial Management Specialist that only loan amount would be included in the IFRs. 27

44 Annex 2. Results Framework and Outputs by Component Baseline Target Revision in Restructuring Actual at Closing (Dec/2014) Comments Key Performance (Outcome) Indicators Project Development Objective: Contribute to Government efforts to reduce poverty and inequality by improving health outcomes, raising health system responsiveness, improving quality and reducing the financial burden of illness on the poor. Indicator One: Decrease in infant mortality (per 1,000 births) in 159 participating municipalities Indicator Two: Increase of per capita contact with primary care providers in 159 participating municipalities (Dec. 2013) The indicator was revised to: increase of per capita contact with primary care providers (doctors and nurses) in participating municipalities ; the baseline was revised to 1.7 (2003) and the target to 1.8. Target exceeded. The MOH set a target of a 5% reduction in infant mortality (16.9) 1.5 Target not met. This indicator had increased to 1.8 in 2012, and then declined to 1.5 in This however, reflects a partial reporting based on data from the old information system, which is gradually being replaced. Data for municipalities that have changed over to the new system have not been reported, and results may 23 The Phase 2 APL did not have a target for this indicator; the target for the Phase 3 APL is 16.0 according to the table in Annex 3 of the Project Appraisal Document. 24 The Phase 2 APL did not have a target for this indicator; the target for the Phase 3 APL is 1.6 according to the table in Annex 3 of the Project Appraisal Document. 28

45 Baseline Target Revision in Restructuring Actual at Closing (Dec/2014) Comments be biased downward as better performing municipalities may be more likely to migrate to the new system early. Intermediate Results Indicators Intermediate Result (Component 1): 1. Raise PSF population coverage; 2. Improve quality of and patient satisfaction with PSF service provision, and 3. Improve the efficiency and effectiveness of PSF services and the broader health system. IRI One: Percentage of PSF population coverage in participating municipalities IRI Two: Ratio of patients with hypertension followed by PSF teams to estimated number of patients with hypertension in participating municipalities (as a group) IRI Three: Percentage of infants < 1 with full vaccination regimen (DPT-H, polio, measles, tuberculosis) in participating municipalities (as a group) Target exceeded for Phase 2 and for Phase 3 (40%). Coverage in participating municipalities outpaced coverage in nonparticipating municipalities (32%) Indicator revised to proportion of patients with hypertension registered by PSF teams, among estimated population with hypertension (15 years of age and older) Baseline revised to percent in 2008 and target to > or = 95 percent in Target not met. The indicator was originally calculated as a ratio of these two groups and was changed to be calculated as a proportion to better align with the MOH monitoring system 92.3 Target not met. The MOH vaccination program was upgraded to a penta- and hexavalent scheme. The indicator was modified to be calculated based on the sum of 29

46 IRI Four: Percentage of pregnant women attended by PSF teams that have 7 or more pre-natal consultations in participating municipalities (as a group IRI Five: Percentage of total medical consultations referred from PSF to hospital specialty services in participating municipalities (as a group) IRI Six: Percentage of all PSF teams applies quality evaluation instrument and are ranked according to standard (AMQ system) in participating municipalities (as a group) IRI Seven: Percentage of PSF teams in a sample of municipalities using evidence-based clinical guidelines for hypertension and diabetes (based on sample survey) in participating municipalities (as a group) IRI Eight: Percentage of municipalities apply quality evaluation instrument in the areas of PSF management and coordination (AMQ parts 1&2) in participating municipalities (as a group) Baseline Target Revision in Restructuring Indicator revised to Percentage of women with live births, attended by PSF teams, that had 7 or more pre-natal consultations Baseline was revised to 60% based on actual data for 2006 Actual at Closing (Dec/2014) Comments completed tetra-, pentaand hexavalent schemes in children < Target not met. <10 < Target considered met Target exceeded. N/A Target exceeded Indicator revised to Proportion of municipalities that conduct self-assessment 95.1 Target exceeded. This indicator measured the proportion of municipalities applying quality evaluation 30

47 Rate of hospital admissions of children <5 for ARI in participating municipalities (as a group) Rate of hospital admissions for stroke in participating municipalities (as a group) Baseline Target Revision in Restructuring to improve access and quality of primary care Actual at Closing (Dec/2014) Comments instrument in the area of PSF management and coordination (AMQ parts 1&2). It need to be updated to reflect the evolution and scale up of quality improvement instruments, that now feature two follow-on systems: AMAQ and PMAQ. 24/ /1000 Indicator dropped This indicator was dropped, as it was not adequately measuring the attainment of development goals. Both positive and negative changes to the indicator point at potential improvements in primary care. 27/ /1000 Indicator dropped This indicator was dropped, as it was not adequately measuring the attainment of development goals. Both positive and negative changes to the indicator point at potential improvements in primary care. 31

48 Baseline Target Revision in Restructuring Actual at Closing (Dec/2014) Comments Intermediate Result (Component 2): Strengthen state capacity to: (i) support municipalities extend coverage, improve technical quality and manage service delivery; (ii) improve the supply, quality and stability of family health human resources; and (iii) conduct monitoring and impact evaluation as key functions of state health secretariats. IRI Nine: Percentage of states establish performance agreements with at least X% municipalities with <100,000 population in participating states (as a group) IRI Ten: Percentage of states with monitoring and evaluation plans implemented and evaluated in participating states (as a group) 25 IRI Eleven: Percentage of municipalities with <100,000 population in X% states participate in quality assessment program (AMQ), including completion of self-assessment and 0 40% states 25% municipali ties Indicator revised to Proportion of states that establish performance agreements with at least 25% of municipalities with <100,000 population and target revised to 40% of states in % (26) states with 100% of the municipalit ies with defined goals, and 73.4% joined the Pacto pela Saúde 2010/11 Target exceeded. Revised to reduce complexity Target met. 0 10% municipali ties 50% states Indicator revised to Proportion of states with 10% of municipalities <100, % (26) states, 94.6% of municipalit Target exceeded. Revised to reduce unnecessary complexity and to transform it into a 25 Percentage of states with monitoring and evaluation plans implemented and evaluated, including: (i) the establishment of an M&E unit within State Health Secretariats; (ii) definition of performance indicators; (iii) definition and collection of baseline data for monitoring system; and (iv) documentation of analysis of data derived from monitoring system. 32

49 development of plan to address quality gaps in participating states (as a group) Baseline Target Revision in Restructuring inhabitants that implemented selfassessment and target changed to 50 percent of states in 2014 Actual at Closing (Dec/2014) ies use the AMAQ Comments numeric indicator, eliminating ambiguity on how it was tracked over time. Also, AMQ was replaced by AMAQ and PMAQ that were implemented on a large scale. Target was attained at the time of restructuring. Intermediate Result (Component 3): Build MOH capacity to manage PSF, and support PSF implementation in states and municipalities through: development of PSF governance, policy and regulation, including results-based financing and management, human resources and incentives policies; and support states and municipalities developing training programs, research, M&E, and coordination and supervision capacity IRI Twelve: Establishment, at the federal level, of a results-based management system that links project financing to states and municipalities 26 0 System developed and tested Implement ed in 3,965 municipalit ies (in all states) Target exceeded. This indicator measured the proportion of PSF teams that implement a results-based management system. It was updated to reflect the current development in resultsbased management at the federal level, including the evolution of the AMQ 26 Establishment of a results-based management system that links project financing to states and municipalities along the following dimension: objectives, performance indicators, outputs, inputs, investment priorities and spending; this would include integrated information system, manual to support implementation, and monitoring of implementation of results-based management system. 33

50 IRI Thirteen: Cost accounting system (at federal level) IRI Fourteen: Inter-municipal cooperation plan implemented in each state to strengthen PSF, specifying activities in three areas: management, coordination, service provision Baseline Target Revision in Restructuring 0 Study concluded and system developed 0 1 per state (20 states) Actual at Closing (Dec/2014) Study on costs and tripartite spending on primary healthcare carried out in partnership with IPEA and DESID/M OH Communiti es of practice established. Comments system into the new PMAQ system. The target was revised to better reflect continued progress. Target met. Target met. dab.saude. gov.br/port aldab comunidad e_praticas. php 34

51 IRI Fifteen: Percentage of ESF implement results-based management system, including the signing of performance contracts IRI Sixteen: Baseline Target Revision in Restructuring 0 Model developed and tested Indicator revised to Proportion of family health teams participating in the Program of Improving Access and Quality (PMAQ-AB), baseline revised to 50 percent in 2011 and target revised to 60 percent in 2014 New indicator: Proportion of participating municipalities that implemented the Health information System for Primary Care (SIS- AB) ; baseline 0 percent in 2011; target 70 percent in 2014 New Indicator IRI Seventeen: Proportion of primary care units (UBS) in participating municipalities that have a Tele-health access point ; baseline 0 percent in 2011; target 20 percent in 2014 Actual at Closing (Dec/2014) 58.3% in the 1 st cycle of PMAQ; 88.0% in the 2 nd cycle (Feb. 2015) 47% of municipalit ies have Tele-health access point in support of PSF units Comments Target exceeded. This indicator monitors the upgrading of information systems and integrated data systems to support performance monitoring of primary health care and the PSF. This indicator monitors the upgrading of primary care units to units with telehealth access points. These access points play a key role in raising the technical quality of primary care and have a direct role in promoting continued professional learning for 35

52 Six major research projects on PSF including at least one on chronic diseases, one on patient satisfaction, and one on PSF impact on hospital admissions Impact evaluation Proficiency test of PSF professionals developed and applied to a sample of recent graduates of all PSF training centers Baseline Target Revision in Restructuring N/A Baseline collected 3 studies concluded Plans, methods and TORs completed 0 Test developed Indicator dropped Indicator dropped Indicator dropped Actual at Closing (Dec/2014) Comments family health professionals across the tele-health network. Government funded studies with own resources. The Government was funding the impact evaluation of its major quality improvement initiative, PMAQ, with its own resources. Major research studies focusing on the PSF were carried out and published by renowned research institutions. Hence, the need for studies and evaluations diminished. The need for a proficiency test of PSF professionals became obsolete due to the introduction of the PMAQ system, involving an external evaluation of PSF professionals. 36

53 Revisions to the Results Framework (in the March 2013 restructuring): Indicator 2 was revised to read Increase of per capita contacts with primary care providers (doctors and nurses) in participating municipalities. This definition is in accordance with the one adopted by the MOH. The baseline was corrected accordingly, and the target for Phase 2 was defined as an increase of 5 percent compared to the baseline; Indicator 4 was revised to read Proportion of patients with hypertension registered by PSF teams, among estimated population with hypertension (15 years of age and older) to better align the indicator with the GOB s monitoring system; The baseline and target of Indicator 5 were revised to be compatible with upgrade of Brazil s vaccination program to a penta-, and hexavalent scheme in children below one year of age; The baseline of Indicator 6 was revised to reflect actual data for 2006 from the SINASC information system; Indicators 10 and 11 were dropped as they were considered to not adequately measure the attainment of development goals as both positive and negative changes to the indicators pointed to potential improvements in primary health care; Indicator 12 was revised to read Proportion of municipalities that conduct selfassessment to improve access and quality of primary care to reflect the evolution and scale up of quality improvement instruments that besides AMQ 1&2, now feature two follow-on systems: AMAQ and PMAQ; Indicator 13 was revised to read Proportion of states that establish performance agreements with at least 25% of municipalities <100,000 population to reduce unnecessary complexity, and the target was revised to only 40 percent of states in 2014 (since the number of municipalities was in the revised indicator itself); Indicator 15 was revised to read Proportion of states with 10% of municipalities <100,000 inhabitants that implemented self-assessment and the target was revised to only 50 percent of states in 2014 (since the number of municipalities was in the revised indicator itself). Also, the reference to AMQ was eliminated since it was replaced by follow-on systems AMAQ and PMAQ that were implemented on a large scale; Indicators 18, 19 and 20 were dropped; the GOB funded research projects and an impact evaluation of its major quality improvement initiative, PMAQ, with its own resources; Also, major research studies focusing on the PSF were carried out and published by renowned research institutions, hence the demand and need for similar studies and evaluation funded with Project resources diminished; the PMAQ system, inter alia, involves an external evaluation of PSF professionals, hence Project funding for this activity was not required; and Indicator 22 was revised to read Proportion of family health teams participating in the Program of Improving Access and Quality (PMAQ-AB) to reflect evolution of results-based management at the federal level, including of the AMQ system into the new PMAQ system; The target was revised to better reflect continued progress on the indicator; and Two new Indicators were added, along with baselines and targets. The first, Proportion of participating municipalities that implemented the Health 37

54 Information System for Primary Care (SIS-AB), was to monitor the upgrading of information systems and integrated data systems to support performance monitoring of primary health care and the family health program. The second, Proportion of primary care units (UBS) in participating municipalities that have a Tele-health access point, was to monitor upgrading of primary care units with Tele-health access points that play a key role in raising the technical quality of primary care and have a direct role in promoting continued professional learning for family health professionals across the Tele-health network. 38

55 Outputs by Component: Of the 184 municipalities that were selected to participate in PROESF 2 based on their performance under PROESF 1,170 signed subproject agreements with MOH (171 were declared eligible and 13 were temporarily ineligible). Of these 170 municipalities that started, 55 were temporarily suspended for not having executed 30 percent of funding within 12 months (44 of these did not use any funding). Only 143 municipalities signed subproject agreements with the MOH for the second stage. All states and the federal district signed subproject agreements, but their level of execution was low: only 22 percent of resources transferred were utilized, with the consequence that 15 of 26 states were temporarily suspended for not having implemented at least 30 percent of resources within 12 months. The goods and services financed under the Municipal and State subproject agreements is presented below. Municipal Expenditures under PROESF2 Category of Expenditure R$ Vehicles 11,852,580 Medical/Dental Equipment 8,630,352 Construction/Expansion/Reconstruction 6,704,048 Information Technology Equipment 5,991,458 Furniture 5,121,390 Appliances 3,349,403 Medical Furniture 1,805,462 Training 1,735,737 Events/Seminars 1,572,529 Audiovisual Equipment 486,709 Graphic Services 271,068 Supervision 262,792 Publications, Books, etc. 246,226 Communications Equipment 35,489 Per Diem 6,150 TOTAL 48,071,390 State Expenditure under PROESF2 Category of Expenditure R$ Events/Seminars 2,827,450 Information Technology Equipment 2,503,003 Training 1,189,310 Vehicles 871,629 Per Diem 321,230 Graphic Services 305,752 Airfare, transportation 265,097 Audiovisual equipment 258,396 Communications equipment 9,239 TOTAL 8,551,

56 Annex 3. Economic and Financial Analysis Summary 1. This annex presents the results of the retrospective economic and financial analysis of the Brazil Family Health Extension Project APL Phase 2 (PROESF II) performed for the Implementation Completion and Results Report (ICR). This assessment was based on review and updating of the analysis performed for the PAD, rather than a new analysis. It seeks to assess whether the assumptions and projections made at the time held true, and assess the impact of possible changes in variables on the conclusions of the analysis. 2. The original Project (PROESF 1) was approved in 2002, for an amount of US$550 million over 7 years. Its intended goal was to improve utilization and quality of publiclyfinanced health services by: (i) expanding and consolidating coverage of the Government of Brazil s (GOB) Family Health Program (Programa Saúde da Família - PSF) in about 187 municipalities, establishing well-articulated referral and counter-referral systems, and introducing performance-based financing and management arrangements; (ii) establishing family health as a core element of health professional and para-professional training; and (iii) strengthening Ministry of Health (MOH) capacity to monitor and evaluate PSF health services, policies and training activities on a systematic basis. The project was closed in 2007 with satisfactory outcomes. 3. PROESF 2 was approved in 2008 for an amount of US$83.45 million over a period of three years, to build upon the achievements of Phase I, and strengthen the quality of care within PSF and the capacity for monitoring and evaluation at the state level. I covered the same 187 urban municipalities with populations over 100,000 already covered under PROESF The initial evaluation performed for the PAD estimated the incremental costs and benefits over the project lifetime (three years, mid-2008 to mid-2011) and over a longer period of 10 years, because of the expected lag in project impact and their continuation over time. Costs were estimated at US$166.7 million, while the Net Present Value (NPV) of the net benefits were projected at US$745 million over 10 years. The project was expected to yield a negative NPV during its implementation, and positive benefits thereafter (Table 1). 5. Because PROESF 2 was intended to support an ongoing program (the PSF strategy) that expanded and evolved with or without the Project, it is nearly impossible to assign any observed improvement in health indicators to the project alone. Therefore, it is assumed that PROESF 2 has been a strong contributor to the impact of the PSF strategy as a whole during Project implementation and measure the incremental benefits of PSF in general during Project implementation. 6. The results of this ex-post analysis, using more recent data, are less favorable than in the ex-ante analysis of 2008, but are still within the range of the sensitivity analysis performed for the PAD (Table 1). These results reflect recent trends in population growth and birth rate (lower than estimated in the PAD), lower initial coverage by PSF but faster growth, and recent evidence of somewhat lower impact on specific mortality rates; 40

57 specifically, the number of births is much lower (20%) than estimated at the time, thus reducing the size of the potential impact on infant and maternal mortality. Table 1: Results from Economic Analysis ex-ante (2008) and ex-post (2014) over 10 years PAD (2008) Sensitivity analysis Actual (2014) Project costs 2, , Project benefits 3, , NPV * Benefit/Cost ratio IRR 47% 20% * Lowest value, with 35% reduction in benefits; NPV for 20% reduction was -4.28, and for a 2 year delay Costs and Benefits in USD billion, discounted at 10%. 7. Several factors are likely to affect the results of the analysis performed for the PAD. Each of these factors and their implications for the results of the analysis are discussed below. When more recent information became available, variable values were updated to reflect them. When no new information was available the original values were kept. Project specific factors 8. Project-specific variables include change in Project startup and duration and the low disbursement of project funds. Project startup was delayed by over one year, as the loan signature took place only in September Further implementation delays resulted in the Project closing being postponed to December Overall, duration of the Project was of 5 years instead of the planned three years. These changes in Project startup and duration affect directly the projected costs and especially benefits, even more so because PROESF 2 aimed at supporting an ongoing program that continued to expand while the Project was delayed. Changes in key variables 9. Several key values in the analysis had revised or updated for more recent years (after the baseline for the PAD analysis), and some of them are quite different from initial values; when relevant calculations were adjusted to reflect these changes. 10. Population estimates and projections were revised by the National Statistics Institute (IBGE) in 2013, resulting in a slightly lower numbers for recent years (0.4% for 2009) and significantly lower birth rate and population growth (1.0% around 2010, decreasing to 0.84% by 2018, against initially projected 1.27%). This lower demographic growth naturally affected the projected size of the target population, reducing it by 1% in 2009 and 2.4% (or 2.5 millions) by 2017, while the number of births was 20% lower (from 1.8 to 1.4 million). The impact of this is to reduce both annual incremental recurrent costs (at maintained project costs) and total projected benefits. 41

58 11. The number of covered municipalities, and thus the target population, has also changed, for two reasons. Of the 224 municipalities with over 100,000 population covered under PROESF 1 (and considered in the PROESF 2 PAD), only 188 submitted proposals and were finally covered under PROESF 2; of these one did not participate in PROESF II. Even though the target population was already corrected in the PAD, the core indicators of mortality and morbidity still referred to the larger population (for consistency with PROESF 1). In the absence of a full list of these municipalities, we assumed no significant change in the profile of covered municipalities (e.g. PSF coverage). 12. PSF coverage data are available for more recent years, and are different from those used in the PAD. This is because in 2007 the MOH changed the basis for computing coverage, from the theoretical number of families covered by a PSF team (to actual registration of families. This change resulted in a drop of 4.5 percentage points in the estimated coverage for 2007 and following years. For that reason, the target of 37% coverage by 2009 and 41% in 2010 was not reached (32.6 was the recorded coverage), but was nearly reached according to the prior calculation method. In addition, coverage increased further than projected in the PAD analysis, but with a longer lag. During project actual implementation ( ) and based on the new data, coverage increased from 32.6 to 45.3% in December Maternal morbidity data has for long been known to be imprecise. The main available time series, published by MOH/Secretariat of Surveillance for reported maternal deaths unadjusted for under-reporting, indicates that MMR has been oscillating between 50 and 60 /100,000 live births, with no declining trend: the mean value for for Brazil as a whole was 57, while the mean since 2000 was 55. It is very likely that the actual rate has been declining but was compensated by improved reporting, but no recent precise estimate was available. Recent evidence of PSF impact 14. New evidence of the impact of PSF has been published after the preparation of the PAD. Generally, these new findings tend to indicate substantially lower impact than previous studies for mortality rates, and larger ones for hospitalizations; in the case of diabetes, a negative effect was found in one study. We summarize below some of these new findings in the literature. 15. Aquino et al (2009) 28 measured the impact of PSF coverage on infant mortality in 771 municipalities between 1996 and 2004, after controlling for fertility rate, income per capita, illiteracy, access to running water, availability of local hospital care, and income concentration (measured by the Gini index). They found that a significant impact that increased with coverage, from 13% (incipient PSF, coverage <30%), 16% (intermediate PSF, between 30 and 70%) and 22% (consolidated PSF, >70%). This suggests that the larger impact is obtained when PSF is incipient, and gradually declines as coverage increases. 27 These estimates are based on IBGE population data, and are therefore slightly different from MOH estimates, which use their own population estimates. 28 Aquino R, Oliveira NF and Barreto ML, 2009: Impact of the Family Health Program on Infant Mortality in Brazilian Municipalities. American Journal of Public Health, January 2009, Vol 99, No

59 16. Rocha and Soares (2010) 29 confirmed that PSF coverage had a large impact on infant mortality, but also significant - though smaller - impact on adult mortality, except for those over 59 (where impact was not consistently significant). 17. Macinko et al (2010) 30 and Tursi et al (2012) 31 showed that PSF also had a large impact on the proportion of hospital admissions for Ambulatory Care Sensitive Conditions (ACSC) between 1999 and 2007 in municipalities participating in PROESF: in almost every state admissions for ACSC dropped significantly (24% on average, from 18.4/1000 inhabitants to 14.3), and more so than admissions for non-acsc (9%, from 39.5 to 36.9). Hospitalizations for ACS chronic diseases (asthma, stroke, other CVD, hypertension, Diabetes, and COPD) fell twice as fast as for all other conditions. In municipalities with high PSF coverage, ACSC rates were 13% lower than in those with low coverage, after controlling for other factors. However, the opposite was true for diabetes: hospitalizations increased with PSF coverage, possibly due to either increased detection and access, weaknesses in disease management, or data classification errors. Project costs 18. Of the original project funding of US$166.7 million (50% each for the Bank and GOB), million were disbursed from the World Bank s commitment, or 51% of the total. We assume that GOB disbursements followed those from the Bank, thus spreading between 2010 and Recurrent costs arising from the project were calculated in the same way as for the PAD (on the basis of US$5.46 per capita per month), but starting in 2010 rather than 2008 due to Project delay, and computed on the new population and coverage data. 20. The resulting total costs are shown in Table 2. Table 2: Project and recurrent costs for PROESF (baseline) onwards Project costs Total recurrent costs ,971 Incremental recurrent costs ,60 Project benefits 21. Consistent with the initial Economic Analysis, this assessment concentrates on two major impacts that are easier to measure: the expected reduction in hospitalization rates 29 Rocha r. and Soares RR, 2009: Evaluating the Impact of Community-Based Health Interventions: Evidence from Brazil s Family Health Program. Discussion Paper No. 4119, IZA, Bonn. 30 Macinko J, Dourado I, Aquino R et al, (2010: Major Expansion of Primary Care in Brazil Linked to Decline in Unnecessary Hospitalization. Health Affairs, 29, no.12 (2010): Oliveira VB de, Tursi MA, Costa MFFL, and Bonolo PF, 2012: Avaliação do impacto das ações do programa de saúde da familia na redução das internações hospitalares por condições sensíveis à atenção básica em adultos e idosos. NESCON/FMUFMG, Projeto ICSAP Brasil. 43

60 (direct benefits) and mortality rates (indirect benefits) from selected conditions. Direct benefits are those accrued from savings in hospitalizations, and indirect benefits were measured in Potential Years of Life saved from educed mortality. The detailed indicators used for the assessment are shown in Table 3, with their original values (in the PAD) and revised/updated values. 22. The resulting benefits estimation applies these new findings without assessing their strength or reliability relative to previous studies; this means that the new impact estimations are not necessarily better than those used for the PAD, but tend to be on the lower range of impact. Table 3: Key indicators used in the analysis and their values Reduction per year % (PAD) New reduction data % Baseline: 2007 Original target: 2010 Actual at startup: 2009 Achieved/ Proj 2014 Target population (millions) Coverage rate (%) Covered population (millions) Mortality rates Infant /1,000 live births Maternal /1, Diabetes Mellitus Infectious & parasitic diseases ,24 Circulatory diseases Respiratory diseases Malnutrition Hospitalization rates /1,000 ARI in children Diarrhea in children Stroke, 40+ years old Cardiac insufficiency, , Intra-cranial hemorrhages, 0.8 0, cerebral infarction, stroke Hypertension & related diseases Respiratory diseases Diabetes Mellitus ACSC NA

61 Table 4: Benefits from PROESF II in the PAD and ICR PAD (2008) Actual (2014) Variati on Main reason for variation Project/loan costs % Low disbursement (51%) Incremental recurrent 5, , % Faster coverage growth costs Total costs, discounted 2, , % Result from above Direct benefits % Higher coverage, bigger impact Indirect benefits 6, , % Lower birth rate and mortality impact, higher coverage Total benefits, 3, , % Result from above discounted NPV, discounted % Result from above IRR 47% 20% Result from above Benefit/Cost ratio Result from above 45

62 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Responsibility/ Specialty Lending Gerard La Forgia Lead Health Specialist LCSHH Team Leader Joana Godinho Senior Health Specialist LCSHH Team Member Trajano Quinhões Health Specialist LCSHH Team Member Daniela Pena de Lima Operations Officer LCSHH Team Member Luciano Wuerzius Procurement Specialist LCSPT Procurement José Janeiro Financial Management Specialist LCSFM Financial Fabson Voegel Specialist LCSFM Financial Nicolas Drossos Specialist LCSFM Financial Regis Cunningham Specialist LCSFM Financial Patricia Hoyes Disbursement Specialist LOAG1 Disbursement Miguel Oliveira Disbursement Specialist LOAG1 Disbursement Valeria Pena Social Development Specialist LCSEO Social Develop. Mariana Montiel Senior Counsel LEGLA Lawyer Marta Molares Senior Counsel LEGLA Lawyer Benjamin Levinsohn Specialist SASHD Peer Reviewer April Harding Specialist LCSHH Peer Reviewer Carla Zardo Program Assistant LCC5C Team Member Lerick Kebeck Senior Program Assistant LCSHD Team Member Cassia Miranda Team Assistant LCSHD Team Member Marize Santos Team Assistant LCSHD Team Member Olga Pané Consultant Bernard Couttolenc Consultant Supervision/ICR Daniel Dulitzky Practice Manager GHNDR GP Manager Tania Dmytraczenko Senior Economist GHNDR Team Leader Joana Godinho Lead Health LCSHH Team Leader Daniela Pena De Lima Senior Operations Officer GHNDR ICR Team Leader Ezaú Pontes Senior Health Specialist GHNDR Team member Luciano Wuerzius Senior Procurement Specialist GGODR Procurement Susana Amaral Senior Financial Mgt. Specialist GGODR Financial Mgt. Monica Tambucho Senior Finance Officer WFALN Disbursement Tatiana de Abreu Finance Officer WFALN Disbursement Patricia Melo Finance Analyst WFALN Fin. & Accounting Marize Santos Program Assistant GHNDR Team Member Bernard Couttolenc Consultant Economic and Financial Analysis 46

63 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY FY FY Total: Supervision/ICR FY FY FY Total:

64 Annex 5. Beneficiary Survey Results Not applicable 48

65 Annex 6. Stakeholder Workshop Report and Results Not applicable 49

66 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR 1- Analysis of Objectives and Actions Carried Out 2- Indicators 3- Challenges and Lessons Learned 1- ANALYSIS OF OBJECTIVES AND ACTIONS CARRIED OUT Municipal Component (I) Objective a): Organization, oversight and delivery of primary care and extension services of the Family Health Program (FHP) Expected actions 1 - eligible municipalities to convert to the Family Health Program in cases where this model has not been adopted 2 - extension of the FHP in eligible municipalities where this model is at the early stage of implementation 3 - continued expansion of the FHP in eligible municipalities that have already made significant advances in family health, but have yet to reach their coverage targets Performance assessment Satisfactory Satisfactory Satisfactory Action carried out by the Primary Care Department (DAB) National Primary Care Policy, Program for Access and Quality Improvement in Primary Care, the Mais Médicos Program, and the Program for Placing Value on Primary Care Professionals Objective b): Increased quality and innovation in Eligible Municipalities that have shown significant progress in extending the coverage of the Family Health Program Performance Expected actions Action carried out by DAB assessment 1 - introduction of a quality certification system to apply to health care providers and the Satisfactory municipal administrators of the Family Health Program 2 - improvement of equipment and installations used by the FHP in support of its functions, to include inter alia the construction and renovation of the Family Health center, remodeling of health centers and polyclinics, and the purchase of medical and non-medical equipment, furniture and vehicles. Satisfactory Objective c): Capacity strengthening of Municipal FHP teams Performance Expected actions assessment 1 - development and deployment of training courses for members of the FHP Teams, e.g. Satisfactory training of internal municipal service providers. 2 - project on tools and methods for monitoring Satisfactory FHP Teams 3 - development and deployment of consultation systems to link FH Teams to higher Satisfactory level services Program for Access and Quality Improvement in Primary Care, and the Primary Health Units Rehabilitation Program Action carried out by DAB Brazil Telehealth Networks, Program for Access and Quality Improvement in Primary Care Specialized Institutional Support training course e_sus AB: Electronic Medical Records 50

67 Objective d): Strengthening of municipal health care management Performance Expected actions assessment 1 - Development and implementation of: (A) performance-based management systems; (B) instruments, policies and regulations for hiring and managing human resources; C) structuring Satisfactory FH coordination units; and (D) performancebased contracts signed between the administrators of eligible municipalities and the FH Teams 2 - development, deployment and dissemination of new health care practices, working procedures and protocols Satisfactory Objective e): Strengthening of monitoring and information systems Expected actions 1 - development and improvement of the information systems of Eligible Municipalities for managing data, tracking results and evaluating performance Performance assessment Satisfactory Action carried out by DAB Program for Access and Quality Improvement in Primary Care Community of Practice, Self- Appraisal Tool for Improving Access and Quality in Primary Care, Primary Care Booklets Action carried out by DAB e_sus AB 2 - development and deployment of monitoring and evaluation systems Satisfactory e_sus AB State Component (II) Objective a): Restructuring the State Health Secretariats with a view to them playing a major role in oversight, monitoring and quality improvement Performance Expected actions Action carried out by DAB assessment 1 - State FH coordination teams defined and tasked with providing technical support, and with monitoring the performance of the primary care and FH services deployed by the municipalities under the jurisdiction of the State FH teams 2 - introduction of a quality certification system to apply to FH providers in small and mediumsized municipalities Average Satisfactory 3 - to improve and equip office accommodation for the State FH coordination teams Satisfactory 4 - to strengthen State capacity to monitor municipal FH services Satisfactory Objective b): Support for the continuing education of professional staff Expected actions Performance assessment Induction actions developed based on the Institutional Support Strategy and various programs, especially the Program for Access and Quality Improvement in Primary Care External monitoring of the Program for Access and Quality Improvement in Primary Care for certificating the primary care teams Done with PROESF funding line Done with PROESF funding line Action carried out by DAB 51

68 1 - to develop and implement internal courses as demanded by the health professionals Satisfactory Specialization Course in Institutional Support, distance education courses focused on matrix support and care for street- dwellers Objective c): Formulation and implementation of a State plan for health services performance monitoring and impact assessment Performance Expected actions Action carried out by DAB assessment 1 - definition of the monitoring teams to carry out assessment 2 - improvement of information systems for handling data and tracking performance of municipal FH services 3 - improvement and equipping of offices for the monitoring and evaluation teams 4 - development of techniques required for monitoring and evaluating FH services 5 - undertaking an evaluative and operational survey to measure the performance of municipal primary care providers 6 - development and implementation of a performance-based municipal management system Average Unsatisfactory Satisfactory Average Average Average Done with PROESF funding line Federal Component (III) Objective a): Preparation and implementation of a Family Health management model Performance Expected actions Action carried out by DAB assessment 1 - development of management functions and protocols for the administration of primary health care units. 2 - development and deployment of tools and mechanisms for organizing and planning family health care 3 - development and implementation of a results-based management system, including the development and implementation of performance-based contracts between municipal managers and FH teams 4 - development and dissemination of tools, manuals and training to link funding to performance 5 - development and implementation of FH cost accounting systems, including the acquisition and development of software Satisfactory Satisfactory Satisfactory Average Average Primary Care Booklets (CAB), Self-Appraisal for Improving Access and Quality in Primary Care, and the Program for Access and Quality Improvement in Primary Care Self-Appraisal for Improving Access and Quality in Primary Care Program for Access and Quality Improvement in Primary Care Knowledge Portal Preparation jointly with DESID/MoH and IPEA of the Technical Note on Cost Estimates of primary care 52

69 6 - implementation of norms for regulating and providing incentives to hire human resources in the national health system 7 - organizing workshops and seminars aimed at developing, disseminating and implementing mechanisms for planning and reorganizing State and municipal health systems Objective b): Strengthening the quality of family health care Expected actions 1 - preparation of clinical guidelines for managing the primary care of specific diseases and conditions 2 - to supply technical support and training for extending the quality certification system related to primary health care and FH Objective c): Strengthening training of FH professionals Expected actions 1 - strengthening of residency programs and specializations linked to family health 2 - to promote research by FH nurses and doctors. 3 - to support the development and deployment of FH curricula in medical and nursing schools 4 - support the development of FH-related textbooks and training materials 5 - preparation and implementation of preservice and internal service courses for FH professionals 6 - support for the development and deployment of inter-municipal cooperation plans for sharing best practices, crossfertilization, consortiums for funding training and horizontal technical assistance between participating municipalities Satisfactory Performance assessment Satisfactory Satisfactory Performance assessment Satisfactory Satisfactory Satisfactory Satisfactory Unsatisfactory Satisfactory human resources: Family Health Teams (ESF) and Oral Health Teams (ESB) Mais Médicos Program and the Primary Health Care Provision Program - PROVAB 6th International Primary Care Seminar; 4th National Exhibition of Primary Care/Family Health Experiences; National Primary Care Management Forum Action carried out by DAB Primary Care Booklets (CAB) Program for Access and Quality Improvement in Primary Care Action carried out by DAB Program of Technological Development and Innovative Extension in Primary Health Care and Community Health Education; Incentive for establishing teams with resident physicians Program for Access and Quality Improvement in Primary Care; and publication of materials and scientific journals National Primary Care Policy and the set of primary care programs. Primary Care Booklets (CAB) Community of Practice; Self- Appraisal for Improving Access and Quality in Primary Care; Primary Care Booklets 53

70 7 - support for the establishment of a national family health performance bonus 8 - to implement Training and Capacity Building subprojects for the FH Teams Satisfactory Satisfactory Program for Access and Quality Improvement in Primary Care Brazil Telehealth Networks Program Objective d): Support for States and Eligible Municipalities to develop training programs for improving research, monitoring and evaluation, coordination and oversight capacities Expected actions 1 - research and evaluation studies to be conducted on equity, efficiency, and the quality of organization and delivery of Family Health Program services 2 - preparation of a methodology and plan for the FH services impact assessment Performance assessment Satisfactory Satisfactory Action carried out by DAB Program for Access and Quality Improvement in Primary Care Program for Access and Quality Improvement in Primary Care, and partnerships with research institutes Objective e): Consolidation and standardization of monitoring systems and tools for primary and family health care Performance Expected actions Action carried out by DAB assessment 1 - strengthening of monitoring systems to track the performance of the Family Health Program and to measure compliance with Municipal- State agreements. 2 - review of indicators used in the monitoring systems in order to enable a federal review of State and Municipal performance in primary and family health care 3 - improvement of information systems, to include developing integrated data systems to support the performance monitoring of primary care and the Family Health Program, including redesigning existing systems Objective f): Providing support for project coordination Expected actions 1 - provision of technical assistance, funding of the extension operation, operational and fiduciary staff, travel expenses and the acquisition of products and equipment required by the Project Satisfactory Satisfactory Satisfactory Performance assessment Satisfactory e_sus AB and the process of implementing and evaluating the Program for Access and Quality Improvement in Primary Care (PMAQ) e_sus AB and the process of implementing and evaluating the PMAQ e_sus AB Action carried out by DAB Team hired through FIOTEC 54

71 2- INDICATORS Municipal Indicators Access and quality No. Indicators Calculation Method Baselin e MS2.1 BM RI1 MS2.2 BM RI2 MS2.3 BM RI3 Estimated coverage of the population registered by the FH Teams(ESF) Proportion of persons with high blood pressure (HT) registered by the Family Health Teams DTaP-Hib vaccination coverage of children under one year of age (Number of Family Health Strategy Teams X 3,450 / Resident population) x 100 (Number of people with HT registered by Family Health Teams / Estimated number* of persons aged 15 or over with HT) X 100 * Estimated number of persons aged 15 or over with HT = (Total registered people aged 15 or over in the area covered by the team X estimated State prevalence parameter)/ 100 (Sum of the doses given (3rd dose) of vaccines with the same component (TETRA * + PENTA +HEXA) in <1 year-old child / population under one year old) X 100 Dec 2014 Goal 33% 48.2% 40% 23% 24.4% 35% NA 92.3% 95% Effectiveness MS2.4 BM RI4 Proportion of live births to mothers with 7 or more prenatal consultations * Scheme replacing the penta vaccine (Number of live births to mothers with 7 or more prenatal consultations/ Number of live births to resident mothers) X % 64.1%* 70% BM RI5 Proportion of medical referrals for specialized care Number of referrals for specialized care at specific place and time / Total number of basic medical consultations at the same place and time) x % 10.1% <10% Quality MS2.5 BM RI6 MS 2.6 BM RI7 Proportion of FH teams that applied Self- Assessment for Improving Access and Quality in Primary Care Proportion in FH teams with evidence-based protocols implemented for HT and diabetes care (Number of Family Health teams that applied self-assessment tools for improving access and quality (AMQ / AMAQ) / Total number of FH teams deployed) x 100 (Number of FH teams with evidence-based protocols implemented for HT and diabetes care / Total of FH Teams) X 100 7% 8.5% 15% NA 65.9% 15% Institutional MS 2.7 BM RI8 Proportion of municipalities that performed Self- Assessment for Improving Access and Quality in Primary Care (Number of municipalities that used selfassessment tools for improving access and quality / Total number of municipalities participating in PROESF and PMAQ) x (zero) 95.1% 10% 55

72 State Indicators No. Indicators Calculation Method MS 3.1 BM RI9 MS 3.2 BM RI10 MS 3.3 BM RI11 Proportion of states that established performance agreements with at least 25% of municipalities with <100,000 inhabitants Proportion in states with monitoring and evaluation plans (M&E) implemented and evaluated Proportion of states with 10% of municipalities of <100,000 inhabitants that have implemented self-assessment (Number of states with performance agreements with 25% of municipalities (minimum) / Total number of states (26 + DF)) X 100 (Number of states with M&E plans implemented and evaluated / Total number of states (26 + DF)) X 100 (Number of states with 10% of municipalities (minimum) that have implemented self-assessment / Total number of states) x 100 Baseline 0 (zero) Dec % (26) of the states have 100% municipalities with defined targets, of which 73.4% subscribed to the Pact for Health 10/11 (Pacto pela Saúde) Goal 40% 0 (zero) 52.8% 50% 0 (zero) In 100% (26) states 94.6% of the municipalities applied the self-assessment tool 50% Federal Indicators No. Indicators Calculation Method MS 4.1 BM RI12 MS 4.2 BM RI13 MS 4.3 BM RI14 Implementation of results-based management system in State Health Secretariats (SES) and Municipal Health Secretariats (SMS) Cost accounting study completed Inter-municipal cooperation strategy to strengthen Primary Care Baseline 0 (zero) 0 (zero) 0 (zero) Dec 2014 Implemented in 3,965 municipalities (in all the states) Tripartite study of costs and expenditure on primary care performed in partnership with IPEA and DESID/MoH Communities of Practice implemented No. System developed and tested Study completed and system developed Strategy implemented (in 20 states) 56

73 MS 4.4 BM RI15 Proportion of Family Health teams that participate in the Program for Access and Quality Improvement in Primary Care (PMAQ- AB) (Number of FH teams participating in the PMAQ-AB / Total number of FH teams deployed to PROESF municipalities) x % 58.3% during the first cycle of the PMAQ; 88.0% in the 2nd cycle 60% MS 5.1 BM RI16 Proportion of municipalities with Primary Care Information System (SISAB) implemented (Number of municipalities that regularly feed the national SISAB database / Total number of municipalities participating in PROESF) x (zero) 51.8%* 70% MS 5.2 BM RI17 Proportion of Primary Care Units (UBS) of the municipalities that participate in PROESF and possess a Telehealth point (Number of UBS of the municipalities in PROESF with installed Telehealth point / Total UBS of the PROESF municipalities) x (zero) 47% of the municipalities have installed Telehealth, serving 3,979 FH teams 20% * February 2015 data 3- CHALLENGES AND LESSONS LEARNED Project Design - PROESF Phase 1 of the Expansion and Consolidation Project for the Family Health Strategy (PROESF), launched in 2003, was based on a realistic diagnosis of the Primary Health Care situation in Brazil. In 2003, coverage of the Family Health Program, the main core of the primary health care model funded by the Federal Government, was at a low level in the urban centers. By the end of PROESF Phase 1 in 2007, however, the number of family health teams had increased, serving the 184 municipalities that participated in the program. The main goal of the project was thus achieved. Work on Phase 2 of the PROESF began in 2007, but this phase was not initiated until October 2009 for reasons related to negotiating and approving the loan agreement with the Brazilian Government. In 2011, under the new government, the National Primary Care Policy underwent major improvements, and new goals were set that led to some inconsistency between the original PROESF goals and the new Primary Care targets and incentives. In 2012 the Project restructuring proposal included making the necessary adjustments to bring the project more closely into line with the Ministry of Health s interests regarding Primary Care. The aim of the restructuring exercise was to implement the new Primary Care information system (the e-sus-ab) in the Primary Health Units in the areas covered by the QUALISUS Networks Project. However, the deadlines for implementing this proposal were too tight, since administrative formalities needed to be completed prior to publication of the official public announcement. In the event, these procedures began in 2012 and were not completed until the end of the third quarter of

74 Project Implementation At the beginning of Phase 2 the Project s financial execution structure was especially focused on the municipal and State components (55% and 12%, respectively). In other words, the Project was not to be implemented directly by the Ministry of Health, despite the fact that the Union was accountable for the loan agreement. We thus depended on other entities to deploy the Project resources. The Ministry provided support and incentives to the States and municipalities to enable them to use the resources for implementing the planned actions, but circumstances beyond MoH control made it difficult to achieve better results. Brazilian bidding rules, plus World Bank rules, also impeded the financial execution and implementation of the agreed Project actions. In this respect, we can point to certain rules that caused misgivings and doubts among the managers, such as the use of the shopping modality and Quality-and Cost-Based Selection (QCBS) for funding execution. A significant outcome of this was, for example, the use of funds to purchase vehicles (25%). Serviço de Formação/Capacitação 4% Mobiliário Médico 4% Eletrodoméstico 7% Equipamento de Informática 12% Mobiliário 11% Execução por grupo de insumo em percentual Serviços de Realização de Eventos 3% Veículo 25% Equipamento Médico/Odontológico 18% Ampliação/Reforma/C onstrução 14% Veículo Equipamento Médico/Odontológico Ampliação/Reforma/Cons trução Equipamento de Informática Mobiliário Eletrodoméstico Mobiliário Médico Serviço de Formação/Capacitação KEY: Vehicle Medical/dental equipment Extension/Renovations IT equipment Furniture Electrical appliances Training/capacity-building services Medical furniture Events organization AV equipment Printing services Construction Bibliographical material Project design and oversight Communications equipment Fares (travel) Subsistence allowances A further important question was the dissipation of resources on action plans that covered many different activities. This led to low-value amounts being transferred, causing considerable extra work being devoted to activities of limited effectiveness. Finally, the Federal management of the project was put in the hands of an exclusive team (the Project Management Unit) which, to a certain extent, led to the PROESF and other Primary Care Department projects being sidelined. Financing The amount financed by PROESF became increasingly irrelevant for the municipalities over the Project period in view of the amounts of Federal Government monies transferred to them. In 2003, Federal Government transfers to the municipalities to fund Primary Care activities amounted to over US$3 billion. This figure gradually increased over the years to over US$12 billion in 2012 and US$14 billion in Meanwhile, PROESF funds amounted to US$68 million in Phase 1 and US$83 million in Phase 2. 58

75 Results Achieved - agreed commitments This report has referred to the agreed actions to be carried out under the PROESF, and the detailed strategies adopted to achieve those (items 3.0 and 4.0). The achievement of most or virtually all the agreed actions resulted especially from the Federal Government s own-resource investment efforts and is not attributable to the use of PROESF resources. Weakness in the Brazilian Information System The information system used as a data source for calculating indicators in Phase 2 of the PROESF was the SIAB (Primary Care Information System), which has been in use in Brazil since This system has for long been the target of efforts aimed at improving or replacing it, given its many identified weaknesses. Since 2011, the Ministry of Health has invested in the SUS-AB strategy for Primary Care (SISAB, detailed in topic 4.4). When analyzing the agreed targets and indicators committed to by the PROESF, it is clear that there are weaknesses in the production of SIAB data such as: lack of a mechanism for avoiding wrong numerical values being entered; value-added capture leading to lack of confidence in the data; no institutional obligation to fill in the data; and little use-value for workers and managers. One example is the indicator Average number of visits by Family Health Team doctors and nurses. The historical series shows that the average number of visits peaked in 2007, alleging 4.4 consultations per inhabitant/year, while the average for was only 1.8 consultations per inhabitant / year. Relationship with the World Bank The oversight of PROESF implementation tended to focus more on monitoring physicalfinancial requirements than on providing technical support for actions that could have contributed to correcting the issues identified in this report, principally those concerning the improvement of management procedures. The meetings between the DAB/PROESF team and the World Bank were restricted to the routine missions, and there were no opportunities for sharing analyses on Brazil s Primary Care situation with a view to understanding how the new programs and actions designed to enhance the Family Health Strategy could be used to improve the Project actions. 59

76 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable. 60

77 Annex 9. List of Supporting Documents Ministério da Saúde, PNAB Política Nacional de Atenção Básica, World Bank, Project Appraisal Document, Brazil Family Health Extension Project in Support of the Second Phase of the Family Health Extension Program, Report No BR dated March 28, 2008 World Bank, Restructuring Paper on a Proposed Project Restructuring of Second Family Health Extension Adaptable Lending Project, Report No BR dated March 7, 2013 World Bank, Implementation Status and Results Reports, No. 1 to No. 13 World Bank, Aide Memoires, 2006 to World Bank, Twenty Year of Health System Reform in Brazil: An Assessment of the Sistema Único de Saúde, Michele Gragnolati, Magnus Lindelow, and Bernard Couttolenc, World Bank, Brazil Country Partnership Strategy , Report No BR dated September 21, World Bank, Implementation Completion and Results Report, First Phase of the Family Health Extension Adaptable Lending Program, Report No. ICR dated December 21,

78 Annex 10. Detailed Project Description Component 1: Expansion and Consolidation of Family Health Care in Municipalities (US$ million Total Project Cost, US$55.0 Loan Financing) 1. Component 1 would support scaling up and strengthening PSF capacity at the municipal level, in 187 municipalities (out of 250 municipalities) with populations over 100,000. Coverage extension activities, aimed at raising PSF population coverage from about 34 to 37 percent, would continue to focus on large urban areas that are characterized by poorly integrated, traditional delivery system, with large numbers of families living in poverty and confronting severe social risk. 2. The component would finance works (new construction and rehabilitation), goods, training and technical assistance for the following four sets of activities: (i) (ii) (iii) (iv) the extension of Family Health to municipalities where this model was in an initial phase of implementation, and continued expansion of the model in municipalities that had already made significant headway on family health, but had yet to attain coverage targets. This activity would also support the conversion of traditional health units to the PSF model; consolidation, quality improvement and innovation in municipalities that had shown significant progress in coverage extension during Phase 1, including strengthening supervision and managerial capacity, establishing links to higher level services for care management, and introducing the PSF quality assessment program; support for municipal-based in-service training, the financing of which would be decentralized to participating municipalities through the MOH s grant transfer subsystem (fundo à fundo); and improvement of monitoring and information systems. 3. This component would focus on improving management of primary health care in selected municipalities. Capacity building activities would include development and implementation of activities aimed at strengthening PSF teams (training courses; instruments and methods of supervision of teams; new health care practices, work processes and protocols), PSF management (performance-based management systems; instruments; capacity building of Family Health Coordination Units; and performancebased contracts between managers of eligible municipalities and Family Health teams), and PSF M&E (information systems to manage data, track results and assess performance, and M&E systems). 4. Funding would be made available to municipalities, under the pooled funding approach, on the basis of the following criteria: Eligibility. The 187 municipalities that participated in PROESF 1 would be eligible to participate, once they: (i) complied with fiduciary obligations; (ii) maintained or increase the number of PSF teams since 2003; (iii) presented a 62

79 investment plan by the announced date; (iv) presented revised investment plan addressing recommended modifications by the announced date; (v) had appointed a technician responsible for project coordination; and (vii) had sent representatives to project launch and training workshops. Subprojects and Results-Based Agreements. The 187 municipalities that participated in PROESF 1 were divided in groups according to technical and fiduciary risks. This grouping would determine the initial level of financing available to each municipality under the Project. Each municipality would sign PSF agreements with the MOH that would specify 12 results indicators. Achievement of these indicators would determine future levels of financing, including possible additional funding for elective bonus indicators. Municipality Exclusion. Municipalities that did not initiate financial execution of their subproject 12 months after receiving the transfers would lose eligibility for further funding until the issue was resolved. The state and federal levels would provide technical assistance to solve the financing or technical issues that impeded execution of municipal subprojects. Irregular use of project funds and/or low execution of project financing would exclude a municipality from subsequent financing. Component 2: Strengthening State Capacity for Supervision, Monitoring and Technical Support of Family Health Services (US$17.45 million Total Costs, US$12.45 million Loan Financing) 5. Component 2 would strengthen state capacity to: (i) support municipalities extend coverage, improve technical quality, and manage service delivery, with a special focus in municipalities with less than 100,000 inhabitants; (ii) improve the quality of family health human resources; and (iii) conduct monitoring and supervision as key functions of state health secretariats. The component would finance goods, materials, training and technical assistance for three sets of activities implemented by 26 states and the Federal District, as follows: (i) (ii) Organizational structuring of state health secretariats to play a central role in oversight, monitoring and quality improvement of municipal AB/PSF delivery systems. Activities would include: establishment of State Family Health Coordination teams to provide technical support and monitor the performance of primary care and family health services implemented by municipalities under their jurisdiction; managerial training of AB/PSF coordinators; development and training of teams to supervise small and medium-sized municipalities, implementation of the MOH s quality assessment of Family Health providers in small and medium-sized municipalities; and equipping offices for State PSF Coordination teams. Support for continuous education of family health professional and paraprofessional staff. This would involve, among other activities, the establishment of partnerships with universities and other training institutions to develop in-service courses for Family Health professionals and paraprofessionals, such as community health agents and dental hygiene assistants. 63

80 (iii) Institutionalization of monitoring and supervision functions in state secretariats. This would involve the development of a program for management and evaluation based on targets and performance, and the development of state plans to measure and evaluate municipal performance in Family Health. Specific activities include: (a) establishment of monitoring teams to assess, on a regular basis, the performance of municipalities in the operations of their primary care systems; (b) upgrading of information systems to manage data and track performance of municipal family health services; (c) equipping offices for monitoring and supervision teams; (d) development of skills required for monitoring and evaluation of family health services; (e) implementation of evaluative and operational research to measure the performance of municipal primary care providers; and (f) supporting implementation of information systems, including the development of a database with information to establish and track primary care indicators. 6. Funding would be made available to states, under the pooled funding approach, on the basis of the following criteria: Eligibility. The 26 states and the Federal District would be eligible to participate once they: (i) presented an investment plan by the announced date; (ii) presented a revised investment plan addressing recommended modifications by the announced date; (iii) had appointed a state AB/PSF coordinator; (iv) had appointed a technician responsible for project coordination; and (v) had sent representatives to the project launch and training workshops; Subprojects and Results-Based Agreements. States and the Federal District would be grouped according to technical and fiduciary risks. This grouping would determine the initial level of financing available to each under the Project. Each state would a sign PSF agreement with the MOH that would specify 6 results indicators. Achievement of these indicators would determine future levels of financing, including possible additional funding for elective bonus indicators. State Exclusion. States that did not initiate financial execution of their subproject 12 months after receiving the grant would lose eligibility for further funding until the issue was resolved. The federal level would provide technical assistance to solve the financing or technical issues that impeded execution of the state subprojects. Irregular use of project funds and/or low execution of project financing would exclude a municipality from subsequent financing. Component 3: Strengthening Federal Oversight of the Family Health Program (US$15.8 million Total Cost, US$16.0 million Loan Financing) 7. Component 3 aimed at strengthening PSF oversight capacity at the federal level, by strengthening the capacity of the MOH to provide support to states and municipalities in the implementation of Components 1 and 2. This component was to finance goods, training, technical assistance and project incremental operational costs for the development of PSF accountability arrangements, including results-based financing and management, and support the states and municipalities to develop training programs, research, monitoring 64

81 and evaluation arrangements, and coordination and supervision capacity. It would also finance research and evaluative studies on specific aspect of PSF service organization and delivery as well as institutionalize AB/PSF impact evaluation in the MOH. Several activities were to support implementation of interventions under Components 1 and 2, including: (i) development of inter-municipal cooperation agreements; (ii) development of a new monitoring system for federal performance assessment of states and municipalities; (iii) introduction of results-based management practices (e.g., performance contracts between municipal managers and PSF teams); (iv) development of results-based management system for federal financing to states and municipalities; (v) development and testing of cost accounting systems for AB/PSF; (vi) development of clinical guidelines for primary care case management of specific diseases and conditions; (vii) strengthening of formation of family health professionals through supporting specialization, and research for Family Health professionals; (viii) training and capacity building activities for Family Health Teams; and (ix) special initiative to share experiences and innovations of PSF teams. 65

82 Annex 11. Timeline of Events during Project Life 1988: Major health reforms result in the creation of the Unified Health System - SUS. 1994: Family Health Program (PSF) introduced as Ministry of Health s flagship program. March 2002: Approval of Family Health Extension Project, APL Phase 1 (PROESF 1), US$68 million, the first in threephase, sever year Program (US$550 million). 2005: PSF coverage is 45% (50% in municipalities < 100,000 population). 2006: Family Health Extension Project, APL Phase 2 (PROESF 2), Concept Note Review and Appraisal. March 2007: Invitation to Negotiate PROESF 2 sent. June 2007: PROESF 1 closed. March 2008: Negotiation of PROESF 2. April 2008: PROESF 2 (US$83.45 million) approved. September 2009: signing of Loan Agreement. October 2009: PROESF 2 Effectiveness. November 2010: Presidential elections and state elections. October 2011: National Primary Care Policy (PNAB) approved. October 2012: Municipal elections. May 2012: Mid-term Review. March 2013: Level 2 Restructuring. December 2014: PROESF 2 closes. 66

83 Annex 12. MAP 67