IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-7409) ON A LOAN IN THE AMOUNT OF US$ 300 MILLION TO THE REPUBLIC OF ARGENTINA FOR THE

Transcription

1 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Document of The World Bank IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-7409) ON A LOAN IN THE AMOUNT OF US$ 300 MILLION TO THE REPUBLIC OF ARGENTINA FOR THE Report No: ICR2618 PROVINCIAL MATERNAL-CHILD HEALTH INVESTMENT PROJECT IN SUPPORT OF THE SECOND PHASE OF THE PROVINCIAL MATERNAL-CHILD HEALTH PROGRAM June 25, 2013 Human Development Department Argentina, Paraguay and Uruguay Country Management Unit Latin America and Caribbean Region

2 CURRENCY EQUIVALENTS (Exchange Rate Effective December 31, 2012) Currency Unit = Argentine Peso 1.00 = US$ US$ 1.00 = AR$4.91 FISCAL YEAR January 1 - December 31 ABBREVIATIONS AND ACRONYMS APGAR AUH AE ASM CC CHD COFESA DEIS ECA FESP FM GAAP GDP HD HSRP IBRD IE IFR IMR IPR ISR MCHIP MTR M&E NEA NHSPT NOA NMH NPV PAD PDO PHIP PHSPT PMCH-SAL PMCH-SAL Appearance, Pulse, Grimace, Activity, Respiration Universal Child Allowance (Asignación Universal por Hijo) Universal Pregnancy Allowance (Asignación Universal por Embaraza) Supervision and Monitoring Group Congenital Heart Disease (Cardiopatías Congénitas) Congenital Heart Disease Federal Health Council (Consejo Federal de Salud) Directorate of Health Statistics External Concurrent Audit Essential Public Health Functions Project Financial Management Governance and Anticorruption Action Plan Gross Domestic Product Human Development Health Sector Reform Program International Bank for Reconstruction and Development Impact Evaluation Interim Financial Report Infant Mortality Rate Independent Procurement Review Implementation Status Report Maternal and Child Health Insurance Program Mid-Term Review Monitoring and Evaluation Northeastern Region National Health Service Purchasing Team Northwestern Region National Ministry of Health Net Present Value Project Appraisal Document Project Development Objective Provincial Health Insurance Project Provincial Health Service Purchasing Team Provincial Maternal-Child Health Sector Adjustment Loan Provincial Maternal-Child Health Adaptable Lending Program ii

3 PMH PPR QAG QALP QEA QSA RBF SEA UFI-S VDRL Provincial Ministry of Health Procurement Post Review Quality Assurance Group Quality Assessment of Lending Portfolio Quality at Entry Assessment Quality of Supervision Results-Based Financing Strategic Environmental Assessment International Financing Unit for Health National Ministry of Health Venereal Disease Research Laboratory Vice President: Country Director: Sector Manager: Project and ICR Team Leader: ICR Primary Author: Hasan A. Tuluy Penelope Brook Joana Godinho Andrew Sunil Rajkumar Katharina Ferl iii

4

5 ARGENTINA Provincial Maternal-Child Health Investment Project in Support of the Second Phase of the Provincial Maternal-Child Health Program CONTENTS Data Sheet... v 1. Project Context, Development Objectives and Design Key Factors Affecting Implementation and Outcomes Assessment of Outcomes Assessment of Risk to Development Outcome Assessment of Bank and Borrower Performance Lessons Learned Comments on Issues Raised by Borrower/Implementing Agencies/Partners Annex 1: Project Costs and Financing Annex 2. Outputs by Component Annex 3. Economic and Financial Analysis Annex 4. Bank Lending and Implementation Support/Supervision Processes Annex 5. Beneficiary Survey Results Annex 6. Stakeholder Workshop Report and Results Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Annex 9. List of Supporting Documents Annex 10. Background Information: The Health System in Argentina Annex 11: Quality Assessment of Lending Portfolio (QALP-2) Summary Annex 12. More Details on Preparation and Implementation of APL Annex 13. Impact Evaluation for Plan Nacer: Challenges and Accomplishments iv

6

7 A. Basic Information Data Sheet Country: Argentina Project Name: Project ID: P L/C/TF Number(s): IBRD ICR Date: 06/25/2013 ICR Type: Core ICR AR Provincial Maternal-Child Health Investment APL 2 Lending Instrument: APL Borrower: Republic of Argentina Original Total Commitment: Revised Amount: USD M Environmental Category: C USD M Disbursed Amount: USD M Implementing Agencies: National Ministry of Health, Argentina Cofinanciers and Other External Partners: N.A. B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 05/24/2006 Effectiveness: 05/31/ /31/2007 Appraisal: 08/28/2006 Restructuring(s): 04/26/ /26/2012 Approval: 11/02/2006 Mid-term Review: 08/08/ /07/2011 Closing: 12/31/ /31/2012 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Borrower Performance: Satisfactory Moderate Satisfactory Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Satisfactory Government: Satisfactory Quality of Supervision: Satisfactory Implementing Agency/Agencies: Satisfactory Overall Bank Performance: Satisfactory Overall Borrower Performance: Satisfactory v

8 C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments (if Indicators Performance any) Potential Problem Project at any time (Yes/No): Problem Project at any time (Yes/No): DO rating before Closing/Inactive status: Yes Yes Moderately Satisfactory Quality at Entry (QEA): Quality of Supervision (QSA): Rating Satisfactory Moderately Unsatisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 9 9 Compulsory health finance Health Sub-national government administration Theme Code (as % of total Bank financing) Child health Health system performance Population and reproductive health E. Bank Staff Positions At ICR At Approval Vice President: Hasan A. Tuluy Pamela Cox Country Director: Penelope J. Brook Axel van Trotsenburg Sector Manager: Joana Godinho Keith E. Hansen Project Team Leader: Andrew Sunil Rajkumar Cristian C. Baeza ICR Team Leader: Andrew Sunil Rajkumar ICR Primary Author: Katharina Ferl F. Results Framework Analysis Project Development Objectives The Project development objectives are: (a) to increase access by eligible uninsured mothers and children to basic health services; (b) to strengthen the incentive framework for efficiency and focus on results between the national level and the eligible provinces and among eligible provinces and service providers by linking financing to both services actually rendered to the vi

9 target population and to the achievement of the Maternal and Child Health Insurance Program (MCHIP) results as reflected by the selected ten tracers of the Trazadoras Matrix. Revised Project Development Objectives Not Applicable. vii

10 (a) PDO Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Indicator 1 : Proportion of eligible population voluntarily enrolled in the program. Value (quantitative 0% 80% 97% or Qualitative) Date achieved 11/02/ /31/ /31/2012 Comments (incl. % Target surpassed by large margin. achievement) Indicator 2 : Proportion of eligible pregnant women with first antenatal care visit before 20th week of pregnancy. Value (quantitative 23% 70% 67% or Qualitative) Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target almost achieved. Indicator 3 : Proportion of eligible pregnant women who get Venereal Disease Research Laboratory (VDRL) test during pregnancy and antitetanic vaccine previous to delivery. Value (quantitative 45% 90% 83% or Qualitative) Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target almost achieved. Indicator 4 : Proportion of eligible children less than 18 months old with coverage of measles vaccine or triple viral. Value (quantitative 45% 95% 77% or Qualitative) Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target not achieved. Indicator 5 : Proportion of eligible puerperal woman that received at least one Sexual and Reproductive Health Care consultation. Value (quantitative 27% 60% 90% or Qualitative) Date achieved 11/02/ /31/ /31/2012 viii

11 Comments (incl. % Target surpassed by large margin. achievement) Indicator 6 : Proportion of eligible children 1 year old or less, with all well child consultations up to date (percentile of weight and height). Value (quantitative 13% 50% 45% or Qualitative) Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target almost achieved. Indicator 7 : Proportion of newborns from eligible pregnant women weighing more than 2,500 g. Value (quantitative 47% 85% 90% or Qualitative) Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target surpassed. Indicator 8 : Proportion of newborns from eligible pregnant women, with Apgar score higher than 6 at minute 5. Value (quantitative 47% 92% 93% or Qualitative) Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target achieved. Indicator 9 : Percentage of National Ministry of Health-Provincial Ministry of Health (NMH-PMH) annual performance agreements successfully implemented. Value (quantitative 0% 60% 79% or Qualitative) Date achieved 11/02/ /31/ /31/2012 Comments (incl. % Target surpassed by large margin. achievement) Indicator 10 : Percentage of authorized providers under annual performance agreements and provider payment mechanism with its respective participant province. Value (quantitative 0% 60% 95% or Qualitative) Date achieved 11/02/ /31/ /31/2012 Comments (incl. % Target surpassed by large margin. achievement) Indicator 11 : Percentage of Tracer targets achieved by the participant provinces in last year billing ix

12 period. Value (quantitative 0% 70% 94% or Qualitative) Date achieved 11/02/ /31/ /31/2012 Comments (incl. % Target surpassed by large margin. achievement) (b) Intermediate Outcome Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) x Formally Revised Target Values Actual Value Achieved at Completion or Target Years Indicator 1 : Loan disbursements. Value quantitative or 0% 100% 100% Qualitative) Date achieved 11/02/ /31/ /30/2013 Comments (incl. % achievement) Achieved. Indicator 2 : Value quantitative or Qualitative) Capitation payment occurring according to approved enrollment lists and trazadora systems. In at least 9 eligible Capitation payments None provinces in central occurring in 24 out of and southern regions 24 provinces. Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target surpassed. Provincial Health Service Purchasing Team (PHSPT) and National Health Service Indicator 3 : Purchasing Team (NHSPT) function effectively according to concurrent and financial audits. Value quantitative or Qualitative) None In at least 50% of all participating provinces NHSPT functions effectively; PHSPTs function effectively in 24 out of 24 provinces. Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target surpassed. Indicator 4 : Number of authorized providers receiving the basic medical equipment / vehicles / communication equipment according to the annual performance agreement. Value None At least 50% of 85.3% of authorized

13 quantitative or Qualitative) those for whom an investment Project was approved. xi providers Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target surpassed. Indicator 5 : Number of PHSPTs established and functioning, capable of preparing and negotiating NMH-PMH and PMH-authorized providers annual performance agreements. Established and Value At least 60% of all functioning PHSPTs quantitative or None participant in 24 out of 24 Qualitative) provinces. participant provinces Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target surpassed. Indicator 6 : NHSPT is established and functioning, capable of preparing and negotiating NMH- PMH annual performance agreements. Value quantitative or Qualitative) Not Applicable One National Direction for purchase of medical services has been established. NHSPT has been established and is operational Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Achieved. Indicator 7 : Regular information process among stakeholders on maternal-child health care issues in general and entitlements in the Plan Nacer in particular. 23 (out of 24) provinces with Annual Plans of Communication Information and executed as planned. dissemination Various information Value campaign launched and dissemination quantitative or Not Applicable. at national level and activities undertaken Qualitative) in at least 80% of all at national level, e.g. participating via television and provinces. production/distributio n of relevant audiovisual and graphic material. Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target surpassed. Indicator 8 : Targeted groups increase, knowledge of their entitlements under the program and

14 Value quantitative or Qualitative) participation in Plan Nacer, and report satisfaction with process and results Not Applicable. At least 70% of eligible population reports (in surveys) knowledge of Plan Nacer. At least 50% of enrolled population reports (in surveys) that is satisfied with the Plan Nacer Program. According to the Estudio de Monitoreo de la Satisfacción del Usuario y de la Calidad de Atención del Plan Nacer (a household survey), conducted to date in 13 provinces (including both Phase 1 and Phase 2 provinces) between July 2012 and December 2012: (i) 84.1% of those interviewed knew about Plan Nacer, and (ii) 72.1% of interviewed beneficiaries indicated that they were either satisfied or very satisfied with the services provided by the program. Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target surpassed. Project implementation reports available as agreed, including financial reports Indicator 9 : supporting the capitation payments. Satisfactory reports from Concurrent and annual financial auditor. Value Satisfactory reports At least 80% of the quantitative or Not Applicable. available in 24 out of reports Qualitative) 24 eligible provinces Date achieved 11/02/ /31/ /31/2012 Comments (incl. % achievement) Target surpassed. Project evaluation implemented: (i) baseline at the end of Year 2; mid term impact Indicator 10 : evaluation at the end of Year 3; and final impact evaluation at the end of last year of APL-1. Value quantitative or Qualitative) Not Applicable. At least 80% of the reports Baseline done, midterm evaluation ongoing. Date achieved 11/02/ /31/ /31/2012 Comments Partially achieved. xii

15 (incl. % achievement) G. Ratings of Project Performance in ISRs No. Date ISR Actual Disbursements DO IP Archived (USD millions) 1 01/25/2007 Highly Satisfactory Satisfactory /20/2007 Highly Satisfactory Satisfactory /20/2007 Satisfactory Satisfactory /30/2008 Satisfactory Satisfactory /12/2008 Satisfactory Satisfactory /19/2009 Satisfactory Satisfactory /18/2009 Satisfactory Satisfactory /28/2010 Moderately Satisfactory Moderately Satisfactory /20/2011 Moderately Satisfactory Moderately Satisfactory /26/2011 Moderately Satisfactory Moderately Satisfactory /05/2012 Moderately Unsatisfactory Moderately Satisfactory /20/2012 Moderately Unsatisfactory Moderately Satisfactory /25/2012 Moderately Satisfactory Moderately Satisfactory H. Restructuring Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO Amount Disbursed at Restructuring in USD millions 04/26/2010 No S S IP Reason for Restructuring & Key Changes Made Loan proceeds reallocation, modification of financing percentage for Category (4) of disbursement table of Loan Agreement (capitation payments), inclusion of new provinces as Eligible Provinces, expansion of eligible list of services, closing date extended to December /26/2012 No MU MS Loan proceeds reallocation xiii

16 I. Disbursement Profile xiv

17 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Despite Argentina's recovery after the deep economic crisis of , many problems persisted at the time of appraisal, including low access to and quality of health services for the poor. This was the case despite sweeping health reforms, and despite the fact that in the early 2000s, Argentina was among the top 20 countries globally in per capita health spending. 2. Immediately after the crisis, the percentage of the population not covered by formal health insurance had risen to 44% in 2002 (from 38% in 1997). The rest of the population had formal health insurance coverage either by the Obras Sociales (national social health insurance programs) or by private health insurance policies. But those without formal health insurance had to use the public healthcare system, where there were significant shortfalls in quality and sometimes in service availability. This system was financed fully from public funds since health care must be provided for free at public health facilities in a traditional manner based on financing of inputs, without any link between health facility revenues and the quality of care provided. 3. In response to the crisis, the Government of Argentina developed the Health Sector Reform Program in 2003, with two main pillars the first one providing financing for essential public health actions. An innovative modality was used to provide provinces with financial incentives to produce these actions. This pillar was financed by the World Bank under the Essential Public Health Functions Project Phases 1 and 2 (P and P110599), starting in The other major pillar centered around the Maternal and Child Health Insurance Program (Plan Nacer), complemented by structural changes in the provincial health systems to improve the efficiency of public financing and the delivery of health services. The latter was supported by the World Bank Provincial Maternal-Child Health Sector Adjustment Loan (P072637). 5. Plan Nacer started in 2004 as a provincial public insurance program providing a free basic package of pre-defined cost-effective services, through participating healthcare providers, to women and children without formal health insurance in the country s nine poorest provinces those in the Northeast and Northwest. All uninsured pregnant and lactating women (up to 45 days after birth), as well as uninsured children aged under six, were eligible. No copayments or prepayments were (or are) required, in line with the principle of mandatory free health care in the public system in Argentina. 6. Plan Nacer is a pioneering Results-Based Financing (RBF) program in health. It is one of the first large-scale programs worldwide to use an RBF approach in the health sector, and various other RBF programs in health around the world have been modeled on it. 7. Participating health care providers under Plan Nacer all public providers in the provinces covered receive payments on a fee-for-service basis (conditional on quality) for the services covered, on top of the financing they receive from the public system on a traditional basis. These payments are only made for services provided to eligible (uninsured) women and 1

18 children enrolled in the program, and are conditional on pre-defined protocols being followed (to assure quality). Annex 10 lists the services covered all maternal and child health services. 8. At the same time, there is a second level of Results-Based Financing (RBF), whereby payments are provided to provinces on an RBF basis, as part of a two-stage incentive mechanism. The National Ministry of Health (NMH) makes performance payments to provinces called capitation payments based on the number of eligible persons that have been enrolled in the program, and on provincial achievement regarding a list of ten tracer indicators (e.g. prenatal care coverage in eligible women, following pre-determined quality protocols see Table 2 of Annex 10). 1 The financing received by the provinces via this mechanism is used to make the payments to the health facilities on a fee-for-service basis. Thus there are two stages of RBF payments: one from NMH to the provinces, and the other from the provinces to the health facilities. The achievement of performance indicators by both provinces and health facilities (including adherence to quality protocols) is verified by an external audit firm (see Annex 10). The goal is to move from a traditional health system based on inputs and fixed budgets to one geared toward outputs and results. 9. The initial phase of Plan Nacer, covering the nine poorer northern provinces of the country, was financed by the first loan of the Provincial Maternal-Child Health Adaptable Lending Program (PMCH-APL, P071025). This APL was approved by the Board for an amount of US$435 million in April The Project Appraisal Document for the first phase APL (APL-1) had anticipated that the program would be rolled out nationwide over three phases, and the first phase would consist of APL-1 (for US$135.8 million) covering the nine northern provinces. 10. However, it was decided to merge the planned second and third phases of this APL series into one single loan, APL-2 (P095515), financing a rollout of Plan Nacer across all 24 provinces of the country. This decision was taken due to the strong demand from the provinces that were not included in APL-1, as well as the positive assessment on the part of the Government and the Bank of APL-1 implementation in the nine northern provinces. APL-2 was approved by the Board (for US$300 million) in November Its anticipated closing date was December Original Project Development Objectives (PDO) and Key Indicators 11. The Project s PDOs were: (a) to increase access by eligible uninsured mothers and children to basic health services; (b) to strengthen the incentive framework for efficiency and focus on results between the national level and the eligible provinces and among eligible provinces and service providers by linking financing to both services actually rendered to the target population and to the achievement of the Maternal and Child Health Insurance Program (MCHIP) results as reflected by the selected ten tracers of the Trazadoras (Tracers) Matrix. The eligible provinces were defined in the original Loan Agreement as the 15 provinces not covered by APL-1, but this definition was later amended in April 2010 to also include the nine poorer provinces covered by APL-1, given that APL-1 was about to end in July A fixed maximum capitation amount is given to each province per eligible person enrolled in the program. Forty percent of this maximum amount is based on provincial performance regarding the ten tracer indicators 4% for each tracer indicator while the remaining 60% of this maximum amount is given regardless of tracer indicator performance. Thus, in effect the province receives somewhere between 60% and 100% of the maximum capitation amount depending on tracer performance per eligible beneficiary that is enrolled. 2

19 12. The Project had eleven PDO indicators (given in Table 3), of which eight were closely linked with the enrolment or tracer indicators used to determine provincial performance. These eight PDO indicators were measured at the national level, while the corresponding enrolment or tracer indicators were measured at the provincial level, for each province. 1.3 Revised PDO and Key Indicators, and reasons/justification 13. The PDO remained unchanged throughout the life of the Project. 1.4 Main Beneficiaries: 14. The main beneficiaries were pregnant and lactating women (up to 45 days after birth) and children aged under six years, without formal health insurance, in all provinces. The indigenous population as a whole also benefited from the Indigenous Peoples Plans (IPPs). Others in the population also benefitted indirectly from improved health services. 1.5 Original Components 15. Table 1 summarizes the activities covered, for each of the five components of the Project. Table 1: Project Components, Allocations and Activities Component Initial Allocation from Bank (with Final Expenditure in Parentheses), and Main Activities Covered 1 US$242.7 million (US$249.7 million): Capitation payments to provinces (in turn financing fee-for-service payments to participating health service providers); equipment (medical, transportation, and communications) for participating service providers; technical assistance and training for provinces and service providers; equipment and consultant services to upgrade information systems. 2 US$10.2 million (US$16.0 million): Reorganizing provincial Ministries of Health; improving epidemiological information, financial, and human resource management systems; studies for policy formulation. 3 US$17 million (US$667,000): Dissemination of detailed information about the program among major stakeholder groups; community outreach to increase participation of the eligible population. 4 US$14.6 million (16.8 million): Information technology design services, software and equipment, and related training; external concurrent audits; in-depth evaluations (including surveys and Impact Evaluation). 5 US$1.6 million (2.6 million): Operational expenses (e.g. travel, per diems). Note. Component 1 was co-financed by the Government in two ways. First, the Government financed salaries, infrastructure, etc. at the health facilities. Second, from January 2011 onwards for the Phase 2 provinces (and from August 2010 onwards for the Phase 1 provinces), 30% of each capitation payment (i.e. the monthly payment for each person covered) was paid by the province. The rest (70%) was covered by the Bank loan. 1.6 Revised Components 16. The Project s components were not revised during implementation. 1.7 Other significant changes 17. Two restructurings took place during implementation, in April 2010 and in June In both cases, there were changes in the allocation amounts for the different loan categories of disbursement (consultant services, goods, training, operating costs and capitation payments, which consisted of a separate category). The changes were not large, and the allocation for capitation 3

20 payments the heart of the Project ended up increasing from US$208.5 million to US$212.1 million. No other changes took place in the second restructuring. 18. In the first restructuring, further changes took place: (i) the closing date was extended to December 2012; (ii) the Bank co-financing percentage for the capitation payments category was changed; 2 (iii) coverage was expanded from just 15 provinces to all provinces; and (iv) the basic package of services covered by the program was expanded to also include congenital heart disease (CHD) treatment for children and additional complex maternal health services. The latter was due to analysis showing: (a) CHD was a major factor underlying child mortality in the country; and (b) the package needed additional services to better address higher-risk births. 2. Key Factors Affecting Implementation and Outcomes 19. This ICR s findings for the Project (APL-2) preparation and implementation phase, until June 2010, are based on the findings of a Quality Assessment of the Lending Portfolio (QALP-2) report, done in June 2010 by the World Bank Quality Assurance Group (QAG). Annex 11 provides details of this report, including ratings for different aspects of Project and Bank performance, and a comparison of these ratings with ISR ratings. QAG assessments are considered to be independent and often critical, and this ICR considers QALP-2 to be a largely accurate assessment of the Project at the time. See also Annex 12 for more information. 2.1 Project Preparation, Design and Quality at Entry 20. The ICR for APL-1 noted a deep understanding of Argentina s complex health sector, the lessons learned from the Bank s rich health sector portfolio, and the difficult circumstances the country was going through. This finding remains relevant for APL The lessons drawn upon at the preparation stage were appropriate, and were incorporated appropriately into the Project design. APL-2 took into account not just the lessons from the preparation of APL-1, but also those from implementation of APL-1, featuring in particular: (i) an additional element in the independent concurrent audit to verify the compliance of the provinces with key elements of Plan Nacer; and (ii) expanded technical assistance. 22. The rationale for Bank intervention for APL-2, as for APL-1, was solid. A major rationale was the Bank s long experience with Argentina s health sector (more than ten years), resulting in significant comparative advantages in supporting Government efforts that combine policy reform with investments to contribute to longer-term institutional improvements. 23. The QALP-2 report gave an overall rating of Satisfactory for the Quality of Design, highlighting some of the following as key strengths, which this ICR agrees with: 2 Instead of falling over time from 100% to 70%, and then to 40%, it would fall to just 70%. The portion of the capitation payments not financed by the Bank was supposed to be financed by the provinces. The original loan agreement was found to be abrupt and unrealistic in expecting the Bank co-financing percentage to fall to 40% (implying that the provincial contribution would rise to 60%) so rapidly. 4

21 Strategic relevance and appropriateness of Development Objectives: The Project designed a public insurance system that increased access to health care services for an underserved low income and vulnerable population group. Adequate attention to technical aspects and financial sustainability: The Project has a solid and internally consistent framework for technical aspects. There should be no undue risk to financial sustainability given the relatively small size of this operation (about 1% on average of total provincial public health expenditures). (See also Section 4 on sustainability.) Quality of institutional framework for the Project: The Project was framed to work within the institutional structure in existence at the time of appraisal. Extent of integration and quality of financial management and procurement aspects in project design: Adequate measures were in place to support early start-up of procurement activities, and team s procurement specialists were fully integrated with other Project activities during the design phase. The Project Financial Management (FM) risk assessment was thorough, and took into account the Bank's Country Financial Accountability Assessment of Quality of arrangements for governance and anti-corruption in Project design: The Project avoided altering the political economy ramifications while introducing an empirically driven performance system for allocating resources. The introduction of an external auditing system provided a measure of verification and third party monitoring. Extent of integration and quality of environmental aspects in Project design: This was rated as Highly Satisfactory by QALP-2. APL-1 financed a high-quality Strategic Environmental Assessment with detailed analysis of the environmental risks affecting maternal and child health. Government ownership of the design process: The QALP-2 report gives a Highly Satisfactory rating for this, saying: There has been clear political will on the part of the central and provincial authorities. The decision to widen the scope of the Project to cover a wider set of provinces was in response to demand from the provinces to be included. 24. Other strengths noted in the ICR for APL-1 regarding the quality of design are also relevant in this ICR for APL-2. These include (see Annex 12) high relevance; a pro-poor focus; innovative results-orientated systems of incentives; strong collaboration with various actors; and appropriate choice of lending instrument since the use of an APL allowed the approach to be tried out first in the northern provinces under APL-1 before including the other provinces under APL Some weaknesses in design were also identified by QALP-2, which ultimately did not turn out to be significant shortcomings given the information available after the Project close: 2.1(a) Factors seen by QALP-2 as the Main Reasons for Lagging Achievement in the PDO Indicators: 26. The evolution of several PDO indicators was lagging at the time of the QALP-2 report which rated achievement of these indicators at Project close as Moderately Likely and the following are the main reasons identified for this (see Annex 12 for full details): 5

22 i. Readiness for Implementation: QALP-2 stated: APL-2 was judged to be ready for implementation based on the successful implementation of the first two years of APL-1 in nine provinces.the addition of 15 provinces and thereby covering the whole country was ambitious and resulted in a slow start up. At the time of the QALP-2 in June 2010, many of the provinces that had been newly included under APL-2 (the Phase 2 provinces) still had low rates of coverage by the program, even though these were relatively richer than the Phase 1 provinces. ii. Realism of Project Design, and Varying Provincial Capacities: QALP-2 found that the overall Project design is realistic but could have taken into consideration varying provincial capacities referring particularly to the Phase 2 provinces where performance was lagging. iii. The role of municipalities in the larger Central Region provinces (Buenos Aires, Santa Fe and Cordoba): QALP-2 noted the institutional complexities here because in these provinces, oversight over the provision of health care services was handled not by the provinces but by the municipalities, which represented an additional layer of management in between the provinces and the providers. But the municipalities did not receive performance payments (nor any other payments) under the program. Providing them with such performance payments would have enhanced their incentive to facilitate good performance by providers. iv. Measuring and auditing performance requires a strong information system that was initially not present in some provinces. The depth of existing information systems in various provinces was initially insufficient in order to follow the fast pace of required reporting to obtain financial incentives. However, this improved over time, during implementation, as improvements were made to the existing information systems. 2.1(b) Other factors: v. Procurement aspects in Project design: QALP-2 rated this as Moderately Satisfactory (MS), explaining that this rating (rather than Satisfactory) was because of an INT investigation that had just been launched (see below), indicating that the initial procurement risk assessment now appeared naïve. The Bank Team explained that the Minister herself initiated the investigation, indicating Government pro-activeness to resolve any potential corruption issues. vi. Social aspects in Project design: The QALP-2 report gave an MS rating for integration of social issues in Project design, citing a sound basic orientation on essential concepts and approaches, coupled with an insufficient attention to the mechanisms of implementation. 27. Despite the weaknesses in design mentioned in QALP-2 (listed above), QALP-2 still rated the Project design as Satisfactory, and anyway these weaknesses were later addressed successfully and turned out to be at most minor shortcomings when assessed at Project close: The first four of the six weaknesses listed above were seen by QALP-2 as the main reasons for the then-lagging PDO indicators. These would have been significant shortcomings if the PDO indicators had continued to progress at a slow rate. However, as described below, an Action Plan implemented in the last year of the Project led to substantially improved performance (even in the large Central Region provinces). Hence, these four weaknesses ultimately did not significantly hamper performance when assessed at Project close. 6

23 The weakness noted by QALP-2 on procurement (fifth in the above list) was due to the ongoing INT investigation. But ultimately the investigation found no evidence of wrongdoing. Regarding the last (sixth) identified weakness in the above list, performance in social aspects and social safeguards improved substantially later during implementation (see Section 2.4). 2.2 Implementation 28. QALP-2 found implementation progress overall to be Moderately Satisfactory (MS) in June 2010 (see Annex 11). High ratings were given for several aspects of implementation (notably Government ownership and commitment), while some weaknesses were identified (see Table 2). 29. During the subsequent Midterm Review (MTR) of November 2011, the PDO rating was downgraded from MS to MU since at that time only three of the eleven PDO outcome indicators were on track to be fully achieved (based on end targets) but performance was good overall in the poorer Phase 1 provinces (i.e. the provinces that had entered earlier in the program under APL-1.) The lagging indicators were mostly those related to the tracers in the 15 Phase 2 provinces that had been included later in the program under APL-2 and was especially low in the provinces of the Central Region (see Table 1 of Annex 2). 30. The relatively good performance of the tracer indicators in the Phase 1 provinces was matched by good performance regarding various indicators of implementation, in particular the provincial resource transfer rate to the health service providers. The resource transfer rate in a province is the percentage of the total performance payments transferred under Plan Nacer to the province that has, in turn, been transferred onwards (on a fee-for-service basis) to providers. If the system is working well in a province, this percentage should be more than 80%. A low resource transfer rate implies a large unutilized balance in a provincial Bank account. At the time of the MTR, the resource transfer rate exceeded 80% in all Phase 1 provinces. Conversely, the poor overall performance of the tracer indicators in the Phase 2 provinces was matched by low provincial resource transfer rates. Of the 15 Phase 2 provinces, only four had a resource transfer rate exceeding 80%. In three provinces, the resource transfer rate was less than 40%. 31. The reasons for the poor tracer performance and poor implementation in many Phase 2 provinces were attributed mainly to a few major factors (which were later addressed), including weaknesses in information systems, overestimation of the capacity of the provinces to engage in a new results-oriented approach and issues with Component 3 (see Table 2). 32. After the MTR the Government conducted an in-depth analysis of the factors that hindered the performance and based on the results, initiated an Action Plan to address them (see also Annex 7). A special focus was given to the lagging Phase 2 provinces. A major focus of the Action Plan was on addressing the reporting, billing and information capture issues, including substantial improvements in existing information systems. The national Project team substantially increased the frequency of supervision visits to the provinces, and regional meetings were held regularly. Different selected individuals in the national team were each assigned to one province. As part of the Annual Performance Agreements signed between the national team and the provincial teams, a list of the larger health providers was identified in each province. The provincial teams committed to closely following these providers, and health coverage targets were included in the Annual Performance Agreements signed between the provincial teams and the providers. 7

24 Table 2: Implementation Weaknesses Identified in QALP-2 Report (June 2010) and MTR (Nov. 2011) Weakness/Challenge, and Source of Findings (QALP-2 and/or MTR) The provinces are Constitutionally autonomous. The Central Government was thus not able to easily control the actions of the provinces, and instead had to incentivize them to act appropriately regarding implementation (QALP-2, MTR). The QALP-2 provided a rating of MU for implementation by the Government on procurement issues, mainly due to an INT investigation that had just been launched at the request of the Minister of the Health (which was seen as positive by the QALP-2) (QALP-2). Execution of the amount allocated for Component 3 (Communications and Community Outreach) was very low 2% at the time of the MTR (QALP-2, MTR). Political changes at the highest levels slowing down the Government s response to implementation problems (QALP-2). Challenges in changing a medical culture to accept and follow through on working under an incentive system; underestimation of the capacity of the provinces to engage in a new results-oriented approach for the health care system (QALP-2, MTR). Incomplete or late reporting, incomplete capturing of information, incomplete or faulty health service billing and bottlenecks during the registration processes at the provider level, lack of appropriate information systems in many provinces and lack of harmonization of the systems across provinces (due to challenges posed by the Federal structure see above) (QALP-2 and MTR, but much more in-depth in MTR). Delays in Impact Evaluation (IE) activities and in the MTR (QALP-2). The MTR found that the PDO indicator targets were too ambitious, given the timeline for the Project especially considering the number of new provinces (15) and the size of the additional target population (70% of the total target population across all 24 provinces). The MTR also identified weaknesses in provincial target setting. There are additional complexities in the larger Central Region provinces, since oversight over the provision of health care services is handled not by the provinces but by the municipalities (see Section 2.1 for more details) (QALP-2, MTR). There was room for some potential efficiency gains from pricing of services (MTR). Towards the end of the Project, some provinces had problems in making their mandatory contribution of 30% of all capitation payments. Provinces would sometimes not make their payments, but they would typically repay in full the arrears of each year at the start of the following year. (See Annex 12 for details.) Note: MS = Moderately Satisfactory; MU = Moderately Unsatisfactory. Mitigation Measures, Negative Impacts (If Any), Steps Taken to Address Weakness As part of the project design and implementation, innovative ways were found to address this issue and to incentivize the provinces. Ultimately, the INT investigation did not result in any proven corruption. But a Governance and Anti-corruption (GAC) Plan was developed in early 2011, and agreed between the Government and the Bank. In June 2012 a Bank supervision mission found that most of the activities included in the GAC Plan had been completed, except for four. Good progress was noted on these four actions in the last ISR (see Annex 12). Execution for this component was 4% at Project close. But communication activities were carried out and financed by other funding sources (mainly Government own revenues), at the national and provincial levels. Annual Plans of Communication were executed for 23 (out of 24) provinces, as planned. Various information and dissemination activities were undertaken at national level, e.g. via television and production of audio-visual and graphic materials. Also, this component accounts for just 5.6% of the original Project allocation. Project units at the central and provincial levels were reinforced, and this was a key factor behind the substantially improved performance in the last year of the Project. These issues were successfully addressed as part of the Action Plan developed after the MTR, which required close engagement between the national Project unit and the provinces, and between the provinces and health providers. A major focus of the Action Plan was on addressing the reporting, billing and information capture issues. Substantial improvements were made in existing information systems, and this was key to the acceleration in Project indicators towards the Project close. Although IE activities were delayed, the Bank and Government teams showed a great degree of adaptability, and solid results have now been obtained despite the challenges (see Annex 13). The MTR took place later than expected, but as part of the MTR an indepth diagnosis was undertaken of the causes of the lagging tracer (and PDO) indicators, and this was instrumental for the development after that of a successful Action Plan. The high degree of achievement of the PDO indicators (see Section 3) suggests that most of the targets were in fact not too ambitious. The close engagement with provinces that took place as part of the post-mtr Action Plan addressed the weaknesses in provincial target setting. Ultimately the steps taken under the post-mtr Action Plan led to much-improved performance even in these larger Central Region provinces. This was addressed as part of the close engagement with the provincial level under the post-mtr Action Plan. The national Government required that under the new follow-on Plan Sumar program, capitation payments (in 2013) would not be made to any province that has not paid its debt (if any) under Plan Nacer. At this point, all provinces have paid their debt under Plan Nacer. 8

25 33. The Action Plan addressed the major weaknesses identified (see Table 2) and had a positive impact on the performance of the tracers, especially for the Phase 2 provinces and hence also on the PDO indicators. Between the January to April 2011 cuatrimestre (latest data at the time of the MTR) and the May-August 2012 cuatrimestre, the improvement in tracerrelated indicators for the Phase 2 provinces ranged from 22% to 43% of the target population within a period of 16 months (see Table 1 of Annex 2). Improvements were also made in several indicators for Phase 1 provinces, but less so since they had started at higher levels to begin with. Overall performance for the PDO indicators improved significantly, since seven of the 11 PDO outcome indicators were linked closely with provincial tracer indicators. 34. The disbursement rate and the provincial resource transfer rates also improved substantially. The disbursement rate rose sharply, from 59% at the time of the MTR to 100% just after Project close (see Annex 1). And by the Project close, the average resource transfer rate of the 15 lagging Phase 1 provinces was 83.4%, as compared to 62.5% at the time of the MTR. 35. A key feature of implementation was the constant interaction between the Bank and Government teams on the ground. Most of the Bank team consisted of country-based staff with strong skills who engaged constantly with the Government team. This close engagement, as well as the Government team s pro-active efforts, was key in addressing the many emerging challenges. 36. The following are some ratings changes that occurred towards the end of the Project: (a) The Project Management rating was raised from MS to S in the last two ISRs, given the steps taken by the Government to develop and implement an Action Plan successfully, after the MTR; and (b) in the last ISR, the social safeguards rating was raised from MS to S, and the PDO rating was raised from MU to MS (see Sections 2.4 and 3.2 for respective explanations). 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 37. Design of M&E: This included: i) internal oversight; ii) external concurrent audits; and iii) Impact Evaluations. The Project tracked results indicators for level of coverage and achievement of outcomes included in the tracer indicators, and monitored the quantity of services delivered. 38. Project Results Framework and Linkages at Different Levels: Eight of the Project s 11 PDO outcome indicators were closely linked to provincial enrolment or tracer indicators (which were used to determine the level of the performance payments to the provinces). Furthermore, the enrolment and tracer indicators in each province were closely linked to the services in the Nomenclador (the package of services whereby health service providers received a fixed fee per unit of service provided). 3 Thus, provincial performance which was linked to the PDO outcome indicators was in turn closely linked to the performance of service providers. This alignment of performance indicators at different levels represents good design, except for the larger Central Region provinces where municipalities were key but received no performance payments. 3 For example, one of the provincial tracer indicators was the proportion of eligible pregnant women with the first antenatal care visit before 20 th week of pregnancy. At the same time, antenatal care visits are services for which health facilities receive fee-for-service payments (subject to the required protocol being followed). So, stimulating production of these services at the health facility level in turn stimulates overall achievement of this particular tracer indicator. 9

26 39. Implementation and Utilization of M&E: The Project s outcomes and outputs were adequately monitored throughout implementation. The provincial tracer targets were not always closely linked to the PDO indicator targets, but this linkage was emphasized towards the end of the Project under the Action Plan developed after the MTR. The External Concurrent Audits (ECAs) generally worked well, despite procurement-related delays occurring twice (see Annex 12). 40. Evaluation Agenda Including Impact Evaluation (IE): The Project had a broad evaluation agenda, including in-depth IE, but this in turn led to initial challenges and delays. For example, the baseline IE survey for the Phase 2 provinces was conducted in 2008/09 instead of in 2007 as originally planned. There were other challenges with the surveys and survey data. But the Government and Bank teams have been innovative and adaptable in responding to unanticipated changes on the ground, and were ultimately able to use sources of information other than those originally anticipated, to adapt the evaluation strategy and generate solid results (see Annex 13). 41. Importantly, the Government shifted towards a more comprehensive evaluation agenda, going beyond IE. Quantitative and qualitative studies outside of the IE exercise were implemented to assess critical aspects of the program that could not be covered under the IE subprogram, including, among others: (i) a study examining how health workers motivation is affected by an RBF program; (ii) an assessment of users' satisfaction; (iii) an analysis of the synergy between the Universal Child Allowance program and Plan Nacer; and (iv) a review of the results linked to the implementation of a CHD Surgery Network. This evaluation agenda was financed mostly from the Government s own funds a sign of direct involvement and leadership on the part of the Government in institutionalizing a broad evaluation agenda for the program. 2.4 Safeguard and Fiduciary Compliance 42. Environmental and Social Safeguards: APL-1 financed a high-quality Strategic Environmental Assessment on environmental risks affecting maternal and child health. QALP-2 provided an MS rating for social safeguards in the quality of design, mentioning insufficient attention to the mechanisms of implementation (see Annex 11). It also accordingly provided an MU rating for implementation and Bank supervision on social aspects, based on information available as of March The rating for compliance with the Indigenous Peoples policy (OP 4.10) had been downgraded to MS in June 2009, not long before QALP-2. In November 2012, it was upgraded to Satisfactory again (in the last ISR), based on much-improved implementation at the later stages, including: (a) good overall progress on implementation of the Indigenous Peoples Plans (IPPs) of 2010/2011; (b) the activities in the 2010/2011 IPPs that were not completed were discussed in detail as part of consultations for a new round of IPPs, provided to the Bank for 10 provinces in October 2012; (c) for the 5 new provinces that triggered OP 4.10, 4 social assessments had been undertaken. 43. Financial Management (FM): The rating for FM was Satisfactory until December 2009 and was then downgraded to MS (due to moderate shortcomings at the provincial level identified by the concurrent Auditor). It remained that way until the Project closed. QALP-2 also noted delays in some of the audit reports and unaudited IFRs, but these delays were rapidly addressed by the client. 44. Procurement: (see also Annex 12): QALP-2 provided a rating of MU for implementation by the Government on procurement issues, due mainly to an INT investigation that ultimately did not result in any proven corruption (see Table 2). A Governance and Anti-Corruption (GAC) Plan 10

27 was developed in early 2011, and progress on this has been good (also in Table 2). The MTR of November 2011 found that overall performance for procurement implementation had improved. Just before the MTR, the procurement rating in the ISRs was upgraded from MU to MS. An action plan was anyway developed at the time of the MTR based on a recent review of procurement in the health sector program. Performance regarding this plan has been satisfactory. In May 2012, an external consultant conducted an Independent Procurement Review (IPR), and the Government then produced an Action Plan in late 2012 to address its findings (agreed to by the Bank). 2.5 Post-completion Operation/Next Phase 45. A new project, the Provincial Public Health Insurance Development Project, PHIP, Sumar (for US$400 million, P106735), was approved on April 28, 2011 to support similar goals as APL-2. But aside from uninsured under-six children and pregnant women, it also includes additional population groups: children and youth up to the age of 19, and women between the age of 20 and 64, that are uninsured (9.4 million instead of 2 million people covered). 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 46. The objectives, design and implementation of this flagship RBF Project were and still remain highly relevant for the Argentine health sector, for the country as a whole and for the many RBF initiatives in other countries. The national government and the provinces have demonstrated continuous support and strong interest throughout APL-1 and APL-2. The design of the operation included a highly innovative finance mechanism and linked the national level with the provincial and provider level. 47. This Project is considered one of the most important Projects in Argentina s portfolio in the Human Development sector in the past decade, and is mentioned as a key Project in the Country Partnership Strategy. It started an RBF approach in the health sector that has been followed now in many other countries, in Latin America and elsewhere. Design and implementation aspects are highly relevant for other countries with RBF initiatives, which look to the Argentina Plan Nacer experience to draw lessons. International RBF seminars often invariably feature the experience of Plan Nacer, discussing design as well as implementation aspects. 48. The Project is thus rated by this ICR as having High relevance of objectives, design and implementation. 3.2 Achievement of Project Development Objectives 49. After the implementation of the Action Plan formulated after the MTR (see above), the program s performance improved substantially, and ten of the eleven Project outcome indicators were found to be achieved or almost achieved with five targets being surpassed by a large margin at the Project close. This can be seen from Table 3. (See Annex 2 for details of data sources and related calculations.) Note that surpassed by large margin indicates a final value that exceeds the end target by more than 20%, while almost achieved indicates a final value that is within 90% of the end target. 11

28 Table 3: Achievement Level of PDO Outcome Indicators (Summary) PDO Indicator (A) Baseline Value (B) End Target (C) Final Value Ratio of (C) to (A) (where applicable) Description of Achievement With Respect to End Target PDO Sub-Objective (a): to increase access by eligible uninsured mothers and children to basic health services (1) % of eligible population voluntarily Surpassed By 0% 80% 97% enrolled in program. Large Margin (2) % of eligible pregnant women with first antenatal care visit before 20th week of pregnancy 23% 70% 67% 2.9 Almost Achieved (3) % of eligible pregnant women who get VRDL during pregnancy and antitetanic vaccine previous to delivery. (4) % of eligible children < 18 months old with coverage of measles vaccine or triple viral. (5) % of eligible puerperal women that received at least one Sexual and Reproductive Health Care consultation. (6) % of eligible children 1 year old or less, with all well child consultations up to date (percentile of weight and height). 45% 90% 83% 1.8 Almost Achieved 45% 95% 77% 1.7 Not Achieved 27% 60% 90% 3.3 Surpassed By Large Margin 12% 50% 45% 3.5 Almost Achieved (7) % of newborns from enrolled pregnant women weighing more than 2,500 g. 47% 85% 90% 1.9 Surpassed (8) % of newborns, from eligible pregnant women, with Apgar score > 6 at minute 5. 47% 92% 93% 2.0 Achieved Achievement Index for Sub-Objective (a) 105 Surpassed PDO Sub-Objective (b): to strengthen the incentive framework for efficiency and focus on results between the national level and the eligible provinces and among eligible provinces and service providers.. (9) % of NMH-PHM annual performance agreements successfully implemented. (10) % of authorized providers under annual performance agreements and provider payment mechanism with its respective participant province. (11) % of Tracer targets achieved by the participant provinces in last year billing period. 0% 60% 79% 0% 50% 95% 0% 70% 94% Achievement Index for Sub-Objective (b) 152 Surpassed By Large Margin Surpassed By Large Margin Surpassed By Large Margin Surpassed By Large Margin 50. An Index of Achievement a simple measure of the extent to which final values exceed or fall short of end targets, on average, for different groups of PDO Indicators shows that there was over-achievement for both PDO sub-objectives, but especially for Sub-Objective (b) where the final values far exceeded the end targets for all indicators. 4 An Achievement Index value more than 100 indicates that the extent to which final values have surpassed the end targets is larger than the extent to which final values have fallen short of the end targets. Table 3 shows that the Achievement Index of PDO Sub-Objective (a) slightly exceeded 100, while the Achievement Index of PDO Sub-Objective (b) exceeded 100 by a very large margin. 4 This Index is calculated by taking the simple average of the ratio of the achievement levels (final value divided by end target) of each Indicator included in the calculation (and scaling to 100). For example, the Index value for Sub- Objective (b) is the result of the following calculation: [(79/60) + (95/50) + (94/70)]/3 x

29 51. The final values far exceed the baseline values for all Project indicators in most cases there has been at least a doubling in indicator values indicating substantial progress during the Project period. For PDO Indicators 2 to 8 which are the most closely linked to final health outcomes, the ratio of the final value to the baseline value ranges between 1.7 and 3.5 (see Table 3). 52. Performance has also been good for the intermediate indicators. Out of ten intermediate outcome indicators, nine have been fully achieved, and the targets have been surpassed for seven of these (see Table 2 of Annex 2). The remaining indicator has been partially achieved. 53. An indication of the substantial impact of the program on targeted health outcomes can be seen from results of a recent rigorous Impact Evaluation (IE) of Plan Nacer. While this evaluation is still continuing, preliminary results indicate, for example, that controlling for other factors and for selection bias the number of infants with low birth weight (out of every 1,000 live births) are reduced by 28% for beneficiaries of Plan Nacer, as compared with non-beneficiaries who are eligible but not enrolled in the program. (See Annex 3 for more). 54. A further indication of the program s impact is the strong downward trends in child and neonatal mortality in Phase 1 and Phase 2 provinces, and also in maternal mortality (MM) in the areas where it was highest ( Phase 1 provinces). This can be seen from graphs in Annex 3. MM has not shown a clear rising or falling trend in the Phase 2 provinces. But it was much lower to begin with in these provinces, which also had a lower percentage of all pregnant women that were uninsured (30%) as compared to the Phase 1 provinces (41%). So the program lowered MM in the areas where the need was greatest. 55. According to survey data, beneficiaries and health workers are highly satisfied with the program. A survey of beneficiaries (see Annex 5) found their satisfaction level (based on an Index) to be almost 7 on a scale of 10, with higher satisfaction levels among beneficiaries with higher utilization levels under the program. Satisfaction was especially high (9.1) among mothers of children benefitting from CHD treatment paid for by the program. A separate health facilities survey found health workers to have a satisfaction level of 7 (on a scale of 10), when asked about the program s design regarding the use of funds at the facility level. 56. Taken together, the above information indicates that the Project rating for efficacy (achievement of objectives) should be Substantial. Although the final value was less than the end target in some cases, the shortfall was small for all but one PDO Indicator, and the final values far exceeded the baseline values for all PDO indicators. Even in the one indicator where the final value was substantially less than the end target, the final value was 1.7 times the baseline value. The extent to which the final values exceeded the end targets is larger than the extent to which the final values fell short of the end targets, for both PDO Sub-Objectives especially (b). Additional justification for the Substantial rating comes from the information available on the Intermediate Indicators, the IE results, and recent trends in child mortality, neonatal mortality and MM. 3.3 Efficiency 57. Economic Efficiency: The Project s internal (financial) rate of return was estimated at between 8.2% and 8.5% if one takes into account only the benefits from averted neonatal deaths 13

30 (see Annex 3 for details). The other benefits not factored into the calculations include benefits from averted deaths of older infants (after the neonatal period); higher productivity due to reduced incidence of low birth weight; reduced maternal mortality (at least in the areas with higher maternal mortality to begin with); and reduced morbidity among infants and mothers. If these additional benefits had been included in the calculations, the internal rate of return would have been far higher. 58. Institutional Efficiency: This ICR agrees with the following statement from QALP-2, which indicates a high degree of institutional efficiency: Within the general framework of a public sector with considerable discretionary power, and associated vested interests, the Project constitutes a serious attempt at introducing a system of performance indicators and associated resource allocation mechanisms, while leaving intact the institutional structure, thereby introducing an element of incentives for improved performance, which hitherto has been absent. 59. Minor shortcomings in design and implementation: As discussed in Section 2.1, QALP-2 rated the quality of the Project design as Satisfactory, highlighting both strengths and weaknesses but the weaknesses turned out to be only minor shortcomings when assessed at Project close (see explanations above). QALP-2 rated implementation in June 2010 as MS, and both QALP-2 as well as the subsequent MTR in November 2011 found a range of implementation weaknesses (see Table 2). But as Table 2 shows, the Action Plan developed just after the MTR addressed these weaknesses successfully, resulting in accelerated progress of the Project indicators. The result was an excellent level of achievement of the PDO indicators at Project close (see Table 3). Hence, the implementation weaknesses identified turned out to be only minor shortcomings. 60. The Project is thus rated by this ICR as having Substantial efficiency, since: (i) the internal rate of return is high even when taking into account just one category of benefits (those from averted neonatal deaths); (ii) institutional efficiency was high; and (iii) shortcomings in design and implementation were minor. 3.4 Justification of Overall Outcome Rating 61. This ICR assigns a Satisfactory rating for the Overall Outcome for the Project, based on relevance, achievement of PDOs, and efficiency. The ratings given by this ICR for relevance, achievement of PDOs (efficacy) and efficiency are High, Substantial and Substantial respectively (as explained in the previous three sub-sections). Hence, the Overall Outcome rating is Satisfactory. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 62. Poverty Impacts: Those covered by Plan Nacer were from the uninsured population, and consisted of vulnerable population subgroups, which if not enrolled in the program did not have access to adequate basic health care. The strategy of starting with the poorest part of the country under APL-1 the poorer northern provinces was appropriate. 63. Gender Aspects and Social Development: The Project had a strong gender emphasis, given its special focus on pregnant women, women post-partum (up to 45 days after birth or miscarriage) and their young children. The last ISR refers to good implementation on IPPs (see section 2.4 above), which is an important contributing factor to social development. 14

31 (b) Institutional Change/Strengthening 64. It is clear that the reforms and changes brought about by the program are having a lasting impact. It is hard to see how they can be easily reversed. (c) Other Unintended Outcomes and Impacts (positive or negative): None 4. Assessment of Risk to Development Outcome 65. The risk to the Development Outcome is rated as Moderate, more due to challenges in institutional sustainability than in financial sustainability. As mentioned in section 3.5 (b), the program has led to institutional changes and reforms that cannot be easily reversed. Furthermore, the program costs only about 1% of total provincial public health spending on average, and thus financial sustainability does not appear to be an issue, even at the provincial level. 66. The program (Plan Nacer and its successor Plan Sumar) is on its way to being fully mainstreamed at the provincial level, and provincial ownership is growing but will take time to be fully entrenched. There are clear indications of growing provincial ownership, such as a reduction over time in the scope of the problem (noted in Table 2) of delayed provincial Counterpart Funding contributions. At this time, all the provincial debt accumulated under Plan Nacer has been paid a big achievement in Argentina s Federal system. Under Plan Sumar, provinces have also begun to finance the salaries of some members of the provincial Project teams. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance 67. This ICR rates Bank performance on Quality at Entry as Satisfactory, in line with the Satisfactory rating on this issue in the QALP-2. As QALP-2 noted, the Project designed a social insurance system that increased access to health care services for an underserved low income and vulnerable population group of uninsured children and pregnant women. The Project introduced a system of incentives for improved performance in a difficult political context, with vested interests. It was based on an innovative APL approach, which showed to be a well-functioning model for the nine Phase 1 provinces. The Bank s preparation team consisted of experts with long experience in the field of health, and the Bank team worked in close collaboration with the Government in the preparation process. QALP-2 gave a Satisfactory rating for the following, among others: relevance and quality of the Project indicators; attention given to technical, financial and economic aspects; quality of institutional framework; effectiveness of the Bank s management of the preparation process; and overall quality of design (see Annex 11). The shortcomings in design noted by the QALP-2 turned out to be minor shortcomings when assessed at Project close (see Section 2.1). 68. This ICR rates Bank performance on Quality of Supervision as Satisfactory (S), for reasons which are now explained. The QALP-2 report of June 2010 rated overall Bank supervision as MU. This rating was based on the following sub-ratings (see Annex 11): (a) S for Focus on Development Effectiveness; (b) MU for Fiduciary, Safeguards and GAC aspects; (c) S for Supervision Inputs and Processes; (d) MS for Candor and Realism of ISRs. 15

32 69. Thus, the main problems identified by QALP-2 were with the sub-category of Fiduciary, Safeguards and GAC aspects, but the Bank team addressed these aspects well. Within this sub-category, Satisfactory ratings were given by QALP-2 for FM and procurement. But performance was found to be MS for GAC aspects and MU for social aspects. On social aspects, which relate in particular to poor implementation at the time of QALP-2 on issues relate to the IPPs, implementation improved markedly towards the latter phase of the Project (see section 2.4). This was facilitated by much closer supervision on the part of the Bank on social aspects, including on the part of a Bank social safeguards expert. On GAC aspects, the Bank supported the Government in the development in early 2011 of a solid GAC plan to address governance and corruption issues in the health sector, and implementation of the plan has been good (see section 2.4). 70. The candor of ISR ratings which had been rated as MS in the QALP-2 improved substantially after QALP-2, and indeed some of the subsequent ratings were lower than those in the QALP-2 (see Table 2 of Annex 11). In particular, at the time of the MTR of November 2011, the PDO rating was downgraded to MU as compared to the MS rating given by QALP-2. (The ISR PDO rating previously has been S). The end-of-project rating in the ISRs for the PDO was MS. 71. A competent task team is in place in the country and at headquarters with all relevant skills, as noted in the QALP-2, and this helped the Bank to provide technical assistance towards the production of an effective Action Plan to substantially improve Project performance after the MTR. The TTL in place at the time of the QALP-2 (June 2010) had been in place for two years, and remained until just before the Project close, indicating good continuity. 72. The successful interactions on the Action Plan were made possible by the constant and close engagement by the Bank team on the ground. Aside from the TTL, the core Bank team consisted of committed and capable county-based staff. Discussions focused on key implementation issues, and adaptations had to be made in response to often-changing circumstances on the ground. 73. The extension of the Project closing date by one year (to December 2012) was a sound move, allowing additional time to take the actions needed to improve the performance of the Phase 2 provinces. This was agreed to by the Bank and Government teams. 74. Since the ratings for Quality at Entry and Supervision are both Satisfactory, the overall rating for Bank performance is Satisfactory. 5.2 Borrower Performance 75. This ICR rates the performance of the Implementing Agencies (the National and Provincial Ministries of Health) as Satisfactory for the following reasons: The overall quality of the design process was rated as Satisfactory by QALP-2, and in this ICR. The reasons for this are explained in detail in Section 2. Furthermore, this is certainly attributable in very large part to the Government, since as mentioned in QALP-2: There has been clear political will on the part of the central and provincial authorities. The decision to widen the scope of the Project to cover a wider set of provinces was in response to demand from the provinces to be included in the program. In fact, Government ownership of the design process was rated as Highly Satisfactory by QALP-2. 16

33 76. Implementation progress was rated as MS by the QALP-2 report for the reasons mentioned above in Section 2.2. Implementation problems led to poor progress in many of the tracer indicators and the related PDO indicators, in particular in the Phase 2 provinces, and this led to a downgrading of the PDO rating to MU at the MTR of November However, during the MTR an in-depth diagnosis was undertaken of the reasons for the poor implementation and indicator performance, and the Government subsequently developed an in-depth Action Plan to address the issues raised. The Action Plan was implemented intensively, and the Government was pro-active in expanding the size of the provincial Project teams in several Phase 2 provinces, to help address local complexities that had not been anticipated in the design phase. Performance improved substantially since then, especially in the lagging Phase 2 provinces. As a result, tracer indicator performance improved substantially, and so did the performance of the seven PDO indicators linked to the provincial tracer indicators. 78. The Government team had strong skills and showed flexibility in making adaptations a number of times as needed, responding to new information or circumstances. For example, in 2010 the package of services covered was expanded to also include CHD treatment and additional complex maternal health services, due to an analysis showing the importance of these additional services. In November 2009 and May 2011, cash transfer programs were started first for disadvantaged children and then for disadvantaged mothers. In both cases, the transfers were made conditional on the child/mother being enrolled in Plan Nacer, as well as fulfillment of certain requirements regarding health checkups and immunizations. The Government team also adopted diverse strategies to deal with the delays in enrolment in the larger Central Region provinces, such as a contingency plan to improve performance in Buenos Aires province within 100 days. 79. There were sometimes delays in provincial counterpart funding contributions (due to issues such as inflexible provincial administrative procedures, cash flow constraints and financial planning difficulties), leading to several provinces owing debt under Plan Nacer at the time of Project closing. The national Government responded by making capitation payments under the follow-on Plan Sumar program (in 2013) conditional on full repayment of Plan Nacer debt on the part of each province. At this time, as a result, provincial debt under Plan Nacer has been paid in full, for all provinces. Despite the challenges with IE, the Government and Bank teams showed a great degree of adaptability, and solid results have now been obtained (see Annex 13). Finally, the agreement to extend the Project closing date was key, allowing additional time needed for the Phase 2 provinces. 80. All of this justifies an overall Satisfactory (S) rating for the Implementing Agencies for implementation. The overall rating for the Implementing Agencies performance is thus S, in line with an S rating for both the quality at entry (preparation) and implementation phases. 81. Performance of the Ministry of Economy and Public Finance (MEPF), with which the Loan Agreement was signed, is also rated as Satisfactory (S), and thus overall Borrower performance is also rated S. The Project was implemented by the Ministries of Health, who played the major role in its implementation. But MPEF played a suitable support role, allowing the NMH to take the lead in putting in place an innovative, transformative and effective program that was important for the country. MEPF s strong support for Bank-financed health projects is in line with its recognition of the health sector as key to the Bank s portfolio in Argentina. Since the performance by both the Government and Implementing Agencies is S, overall Borrower performance should also be rated S. 17

34 6. Lessons Learned 82. Differences in local institutional capabilities need to be analyzed and taken into account: For Argentina, and other countries with a federal institutional framework, it is critical to take diversity in terms of local institutional capacity into account and adjust the Project accordingly. An in-depth analysis is required to identify differences in local capabilities and develop a strategy on how to address the specific circumstances of different provinces or localities. 83. RBF can work well even in a Federal system, but all major actors at different levels need to receive performance payments, and the different sets of performance indicators at different levels need to be aligned. Alignment of province-level performance indicators and health facility-level performance indicators was key to overall performance of the program, and worked well in most cases. The exception is with the larger Central region provinces where the municipalities had direct oversight over the health facilities but did not receive performance payments. This clearly slowed down progress in these provinces. 84. Close engagement with actors at lower levels including health providers can make a large difference to overall performance of an RBF program. This type of close engagement was a key component of the successful Action Plan developed after the MTR. This Plan involved frequent supervision visits by national Project team members to the provinces, regular regional meetings and a commitment by provinces to work closely with larger health providers. 85. A strong focus on information systems (including for reporting and billing) is key for the success of a larger-scale RBF program. A major reason for the substantial improvement in performance resulting from the Action Plan after the MTR was the Plan s emphasis on strengthening information systems. 86. Institutional changes/strengthening takes time: Another reason for the better performance of Phase 1 provinces could have been that they had more time to implement the Project. The time frame for Phase 2 provinces was likely too short and better results could have been achieved if the Project period was longer. If the Project time frame could not be changed, then it would have been better to be less ambitious with the indicator targets. More broadly, provincial ownership and institutional sustainability have been growing but will take time to be fully entrenched. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 87. Through Plan Nacer, the Nation and the Provinces began to implement a policy of prioritization, public insurance and explicit health coverage for the most vulnerable members of the population. The program has achieved results along several different dimensions, many of which are not reflected in the PDO or Intermediate Indicators. See a summary of the Government s contribution to this ICR, in Annex 7, for more details. (b) Cofinanciers: Not Applicable. (c) Other partners and stakeholders: Not Applicable. 18

35 Annex 1: Project Costs and Financing Table 1: Project Cost by Component (in USD Million equivalent) Components 1. Implementation of the Maternal-Child Health Insurance Program Appraisal Estimate (USD millions) (of which from IBRD) Actual/Latest Estimate (USD millions) (of which from IBRD) Percentage of Appraisal 124% 2. Strengthening National and Provincial Ministries of Health % Stewardship Capacity 3. Communications and Community Outreach % 4. Program Monitoring, Evaluation and Concurrent Auditing % Systems 5. Project Management and Administration % Total Baseline Cost % Physical Contingencies Price Contingencies Total Project Costs Front-end fee PPF Front-end fee IBRD Total Financing Required % Source of Funds Table 2: Financing Type of Cofinancing Appraisal Estimate (USD millions) Actual/Latest Estimate (USD millions) Percentage of Appraisal Borrower % International Bank for Reconstruction and Development

36 Figure 1: Disbursement Over Time Category Table 3: Disbursement per Expenditure Category 1 Category Description Allocated USD Disbursed USD Undisbursed USD Level of Execution % 1 Consultants Services including Auditing Services 39,900, ,305, , % 2 Goods 45,150, ,828, ,321, % 3 Training 2,100, ,932, , % 5 Capitation Payments - Northern Region 39,000, ,128, ,871, % (Phase 1 Provinces) 6 Capitation Payments - Central, Patag, Cuyo Region (Phase 2 120,000, ,561, ,438, % Provinces) 7 Capitation Payments - Heart Surgery, 53,100, ,612, ,487, % Maternal Care Services 8 Operating Costs 750, , , % 9 Advances to Designated Account A 12,802, ,802, Advances to Designated Account B 388, , Total 300,000, ,000, As of the date of this ICR, the final Interim Financial Report (IFR) still has not been processed, hence the above figures are not the final figures. 20

37 Annex 2. Outputs by Component A. Evolution of Project PDO and Intermediate Outcome Indicators 1. Table 1 shows the values of the PDO Indicators of the Project, over time. The table reports the baseline values (column A), values at the time of the Midterm Review (column E), and latest known values/estimates (columns F, G and H). 2. For PDO Indicators 2 to 8 which are linked to the provincial tracer indicators two sets of final values are reported, since the results from the final external concurrent audit are not yet available for all provinces. For the tracer indicators, the procedure is that: (a) data are first obtained, and values calculated, providing pre-audit values; and then (b) the external concurrent audit is carried out, producing post-audit values for the tracer indicators. The audit always leads to a downward adjustment of the results, since some observations are rejected (i.e. the post-audit levels of achievement are lower than the pre-audit levels of achievement). The data are considered finalized for the tracer indicators only after the post-audit values are obtained. 3. At this time, the external concurrent audit for the Phase 2 provinces for the last cuatrimestre (September-December 2012) is still ongoing, and its results will not be known in time to be used for this ICR. The results from the external concurrent audit for the Phase 1 provinces for the same cuatrimestre are also not yet known, at present. The only period for which post-audit values are available for all provinces at this time is the second cuatrimestre of 2012 (May-August 2012). 4. Columns F and G report values for PDO Indicators 2 to 8 for the second cuatrimestre of 2012 (May-August 2012) the last cuatrimestre for which post-audit values for the tracer indicators are available for all provinces. Column F reports these values for the Phase 1 and Phase 2 provinces separately, while column G reports these values for all provinces as a whole. 5. Column H reports estimates for PDO Indicators 2 to 8 for the last cuatrimestre covered by the Project (September-December 2012), based on estimates of post-audit tracer values (calculated using known pre-audit values). Before the results of the external concurrent audit are known, post-audit values will not be available. But the pre-audit results are available for all provinces. Estimates of the post-audit results were calculated for all provinces, using the following procedure: (i) For each province, data on pre-audit and post-audit levels of achievement in the first and second cuatrimestre of 2012 (January-April 2012 and May-August 2012) were used, to obtain an estimate of the typical percentage downward adjustment applied as a result of the audits (for each tracer indicator); and then: (ii) this estimated percentage downward adjustment was applied to the known pre-audit levels of achievement for each tracer indicator for the last cuatrimestre (for each province), to obtain estimates for the post-audit levels of achievement. Finally: (iii) the estimated post-audit achievement levels for each tracer indicator for the last cuatrimestre were used to compute estimates for the final values of PDO Indicators, as of December 31, These are the values reported in column H. 6. The values reported in the Datasheet in front of the ICR are from column (H). However, the final values in column (H) are actually not so different from those of Column (G). It 21

38 was decided to use the values of column (H) because for PDO Indicators 2 to 8, these provide data for the last cuatrimestre of the Project (even though they are based on estimates of post-audit data, rather than actual post-audit data). 7. Key findings from the results shown in Table 1 are: Based on the latest known data (see columns F, G and H), seven out of eleven PDO indicators have been fully achieved or surpassed, while three more have been almost achieved (the latter is based on column H). Out of the seven indicators that were fully achieved or surpassed, the targets have been surpassed by large margins for all except two indicators. The Achievement Index for PDO Sub-Objectives (a) and (b) see Note 4 below the Table for an explanation on this shows that the extent to which the final values exceeded the end targets more than compensated for the extent to which final values fell short of the end targets. This was true for both PDO Sub-Objectives, but was especially true for Sub-Objective (b). Towards the end of the Project (after the Midterm Review), performance increased substantially for the Phase 2 provinces in particular, that had been lagging. 8. Note that the values reported in columns (E), (F), (G) and (H) are based on data coming in from the program, i.e. based on services invoiced and paid for as part of the program. Services rendered to the eligible populations that were not paid for under Plan Nacer are not included in the figures. So, the figures under-report the reality, e.g. the percentage of eligible pregnant women that had the first antenatal care visit before the 20 th week of pregnancy (Indicator 2) is more than what is reported in columns F, G and H since many eligible pregnant women did indeed have a first antenatal care visit before the 20 th week of pregnancy, but without this visit being recorded under Plan Nacer records at the time. 9. Thus, a large part of the improvements seen between the Column E values and the Column F (or G or H) values come from a large increase in the number of women and children registered in the program, and an increase in the number of services paid for and recorded under the program. They do not all reflect an increase in the number of services rendered to the eligible population per se (if one also counts the services that were not paid for and recorded under the program). 10. However, the baseline values (column B) taken from the Project Appraisal Document reflect all services given to the eligible population, not just services that were invoiced, paid for and recorded under the program. (The program was not even operational at the time of appraisal in the Phase 2 provinces). 22

39 (A) PDO Indicator (1) Proportion of eligible population voluntarily enrolled in the program. (2) Proportion of eligible pregnant women with first antenatal care visit before 20 th week of pregnancy (3) Proportion of eligible pregnant women who get VRDL test during pregnancy and antitetanic vaccine previous to delivery. (4) Proportion of eligible children less than 18 months old with coverage of measles vaccine or triple viral. (5) Proportion of eligible puerperal women that received at least one Sexual and Reproductive Health Care consultation. (6) Proportion of eligible children 1 year old or less, with all well child (B) Baseline Value (National) in PAD 1 Table 1: Performance of the Project PDO Outcome Indicators (C) End Target 0% 80% (D) Phase 1 or 2 Provinces? (E) Achievement at time of Midterm Review (by Apr 30, 2011 for Tracer indicators; by 31 Oct, 2011 for other indicators) (F) Achievement as of Aug 31, 2012 for Indicators 2 to 8 (using known post-audit values for tracer indicators), 2 and as of Dec 31, 2012 for other Indicators Phase % 99% Phase % 96% (G) Same as in column (F) but at National level (for all provinces) 2 (H) Achievement as of Dec 31, 2012 for Indicators 2 to 8 (based on estimated post-audit values for tracer indicators, calculated using known pre-audit values), 2 and as of Dec 31, 2012 for other Indicators 97% 97% PDO Sub-Objective (a): to increase access by eligible uninsured mothers and children to basic health services 23% 70% 45% 90% 45% 95% 27% 60% 13% 50% Phase 1 64% 69% Phase 2 31% 65% Phase 1 88% 89% Phase 2 49% 81% Phase 1 77% 85% Phase 2 24% 60% Phase 1 88% 94% Phase 2 49% 90% Phase 1 46% 53% Phase 2 15% 35% 67% 67% 84% 83% (I) Achievement at Project close description based on values in (H) 3 Surpassed by Large Margin Almost Achieved Almost Achieved 69% 77% Not Achieved 91% 90% 42% 45% Surpassed by Large Margin Almost Achieved 23

40 (B) Baseline Value (National) in PAD 1 (C) End Target (D) Phase 1 or 2 Provinces? (E) Achievement at time of Midterm Review (by Apr 30, 2011 for Tracer indicators; by 31 Oct, 2011 for other indicators) (F) Achievement as of Aug 31, 2012 for Indicators 2 to 8 (using known post-audit values for tracer indicators), 2 and as of Dec 31, 2012 for other Indicators (G) Same as in column (F) but at National level (for all provinces) 2 (H) Achievement as of Dec 31, 2012 for Indicators 2 to 8 (based on estimated post-audit values for tracer indicators, calculated using known pre-audit values), 2 and as of Dec 31, 2012 for other Indicators (I) Achievement at Project close description based on values in (H) 3 (A) PDO Indicator consultations up to date (percentile of weight and height). (7) Proportion of Phase 1 93% 91% newborns from eligible 47% 85% pregnant women weighing Phase 2 63% 90% more than 2,500 g. 90% 90% Surpassed (8) Proportion of newborns from eligible Phase 1 96% 93% pregnant women, with 47% 92% 93% 93% Achieved Apgar score higher than 6 at minute 5. Phase 2 66% 93% Achievement Index for PDO Sub-Objective (a) Surpassed PDO Sub-Objective (b): to strengthen the incentive framework for efficiency and focus on results between the national level and the eligible provinces and among eligible provinces and service providers by linking financing to both services actually rendered to the target population and to the achievement of the MCHIP results as reflected by the selected ten tracers of the Tracers (Tracers) Matrix (9) Percentage of NMH- PHM annual performance agreements successfully implemented. 5 0% 60% (10) Percentage of authorized providers under annual performance agreements and provider 0% 50% payment mechanism with its respective participant province. Phase 1 100% Phase 2 100% Phase 1 97% 97% Phase 2 87% 94% 100% 79% 95% 95% Surpassed by Large Margin Surpassed by Large Margin (11) Percentage of Tracer 0% 70% Phase 1 88% 98% 94% 94% Surpassed by 24

41 (E) Achievement at time of Midterm Review (by Apr 30, 2011 for Tracer indicators; by 31 Oct, 2011 for other indicators) (F) Achievement as of Aug 31, 2012 for Indicators 2 to 8 (using known post-audit values for tracer indicators), 2 and as of Dec 31, 2012 for other Indicators (H) Achievement as of Dec 31, 2012 for Indicators 2 to 8 (based on estimated post-audit values for tracer indicators, calculated using known pre-audit values), 2 and as of Dec 31, 2012 for other Indicators (A) PDO Indicator (B) Baseline Value (National) in PAD 1 (C) End Target (D) Phase 1 or 2 Provinces? (G) Same as in column (F) but at National level (for all provinces) 2 targets achieved by the participant provinces in Phase 2 67% 91% last year billing period. Achievement Index for PDO Sub-Objective (b) (I) Achievement at Project close description based on values in (H) 3 Large Margin Surpassed by Large Margin Notes: 1. The baseline values are all national values, based on national data (except for Indicators 1, 9, 10 and 11 where the baseline values are only for the Phase 2 provinces since the Phase 1 provinces were not included at the beginning). Since APL-2 was originally meant just for the 15 new Phase 2 provinces, it was planned that the baseline IE study would be used to obtain baseline values for the Phase 2 provinces only (as a group), and these would be used to replace the national baseline values reported in the PAD. But the baseline IE study was delayed substantially, and was ultimately conducted in 2008/09, well after the project started in late In addition, in mid-2010 the Phase 1 provinces were included in APL-2, in addition to the Phase 2 provinces. At that point, APL-2 became essentially a Project with national coverage. So, there was no longer a need for baseline values for the Phase 2 provinces only, and it is appropriate to use the original national-level baseline values as reported in the PAD. 2. See main text above for explanations regarding pre-audit values, post-audit values and estimations of post-audit values. 3. Not Achieved = final value lower than 90% of end target (e.g. lower than 45% if target is 50%). Almost Achieved = final value between 90% and 99% of end target (rounded to nearest % point). Achieved = final value between 100% and 104% of end target. Surpassed = final value between 105% and 119% of end target. Surpassed by Large Margin = final value 120% or more of end target. 4. The Overall Achievement Indices were calculated for PDO Sub-Objectives (a) and (b) by taking the simple average of the ratio of the achievement levels (final value / end target) for each PDO Indicator (and scaling to 100). For example, the Index value for Sub-Objective (b) is the result of the following calculation: [(100/60) + (95/50) + (94/70)]/3 x 100. The Overall Achievement Index would have a value of exactly 100 if the final values were exactly equal to the end targets for all indicators. A high Overall Achievement Index indicates that the extent to which final values have surpassed the end targets is larger than the extent to which final values have fallen short of the end targets. 5. The definition of PDO Indicator 9 (percentage of NMH-PHM annual performance agreements successfully implemented) was changed (made more stringent) towards the end of the Project. According to the previous definition, all annual performance agreements that had been signed and were being implemented were counted, regardless of the quality of the implementation (i.e. the word successfully had not been given sufficient weight). The level of achievement at the time of the Midterm Review was 100%, based on this previous definition. The final values reported for this indicator in the above table are based on the latest definition agreed to with the Bank which is more stringent. The latest definition is that one should only count performance agreements where: (i) the rate of enrolment of the eligible population is higher than the minimum target agreed to, and also: (ii) at least 70% of all projects in the Strategic Plan of Technical Assistance and Capacity Building (which is a part of the performance agreement) have been executed. 25

42 11. Table 2 shows achievement levels of the Project s ten Intermediate Outcome Indicators. Out of ten intermediate outcome indicators, nine have been fully achieved, and the targets have been substantially surpassed in seven out of these nine indicators. In the next section, achievement of individual indicators linked to each component of the Project is discussed. Intermediate Outcome Indicator Table 2: Achievement for Intermediate Outcome Indicators End Target Achievement as of December 31, 2012 (Project Close) Level of Achievement (1) Loan disbursements 100% 100% Fully Achieved (2) Capitation payment occurring according to approved enrollment lists and trazadora systems (3) PHSPT and NHSPT function effectively according to concurrent and financial audits. (4) Number of authorized providers receiving the basic medical equipment / vehicles / communication equipment according to the annual performance agreement (5) Number of PHSPTs established and functioning, capable of preparing and negotiating NMH-PMH and PMH-authorized providers annual performance agreements. (6) NHSPT is established and functioning, capable of preparing and negotiating NMH-PMH annual performance agreements (7) Regular information process among stakeholders on maternal-child health care issues in general and entitlements in the Plan Nacer in particular (8) Targeted groups increase, knowledge of their entitlements under the program and participation in Plan Nacer, and report satisfaction with process and results In at least 9 eligible provinces in central and southern regions In at least 50% of all participating provinces At least 50% of those for whom an investment project was approved At least 60% of all participant provinces One National Direction for purchase of medical services has been established Information and dissemination campaign launched at national level and in at least 80% of all participating provinces At least 70% of eligible population reports (in surveys) knowledge of Plan Nacer. At least 50% of enrolled population reports (in surveys) that is satisfied with the Capitation payment occurring in 24 out of 24 provinces. NHSPT functions effectively; PHSPTs function effectively in 24 out of 24 provinces. 85.3% of authorized providers Established and functioning PHSPTs in 24 out of 24 participant provinces NHSPT has been established and is operational 23 (out of 24) provinces with Annual Plans of Communication executed as planned. Various information and dissemination activities undertaken at national level, e.g. via television and production/distribution of relevant audio-visual and graphic material. According to the Estudio de Monitoreo de la Satisfacción del Usuario y de la Calidad de Atención del Plan Nacer (a household survey), conducted to date in 13 provinces (including both Phase 1 and Phase 2 Fully Achieved, and Surpassed Fully Achieved, and Surpassed Fully Achieved, and Surpassed Fully Achieved, and Surpassed Fully Achieved Fully Achieved, and Surpassed Fully Achieved, and Surpassed 26

43 Intermediate Outcome Indicator (9) Project implementation reports available as agreed, including financial reports supporting the capitation payments. Satisfactory reports from Concurrent and annual financial auditor (10) Project evaluation implemented: (i) baseline at the end of Year 2; mid term impact evaluation at the end of Year 3; and final impact evaluation at the end of last year of APL-1 End Target Plan Nacer Program. At least 80% of the reports Achievement as of December 31, 2012 (Project Close) provinces) between July 2012 and December 2012: (i) 84.1% of those interviewed knew about Plan Nacer, and (ii) 72.1% of interviewed beneficiaries indicated that they were either satisfied or very satisfied with the services provided by the program. Satisfactory reports available in 24 out of 24 eligible provinces Baseline done, midterm evaluation ongoing. Level of Achievement Fully Achieved, and Surpassed Partially Achieved Note: NHSPT = National Health Services Purchasing Team; PHSPT = Provincial Health Services Purchasing Team; NMH = National Ministry of Health; PMH = Provincial Ministry of Health. B. Discussion of Individual Components, Outputs and Achievements 12. Component 1: Implementation of the Maternal-Child Health Insurance Program (MCHIP) (Original Bank allocation of US$242.7 million; final expenditure of US$249.7 million from IBRD) 13. Description: This component aimed to ensure the sustainable implementation and functioning of the MCHIP. It provided support for: i) capitation payments for MCHIP services by the National Ministry of Health (NMH) to participating provinces covering a share of the costs of the basic service package, calculated on a per capita basis; ii) Equipment (medical, transportation and communications) for basic health care facilities that supply the MCHIP package; iii) Technical assistance and training programs for the provincial Ministries of Health to develop systems, instruments, and skills necessary to implement and run the MCHIP; iv) Health service delivery training for providers delivering basic services under the MCHIP; v) Information technology equipment and consultant services to upgrade and expand information systems for monitoring the implementation of the MCHIP; vi) Technical assistance and training for the management of participating health service providers to strengthen areas including their billing capacity, development and implementation of provider data systems. 14. Achievement of Intermediate Results for this Component (see also Table 2): 27

44 Capitation payment occurring according to approved enrollment lists and trazadora systems (Fully Achieved and Surpassed). PHSPT and NHSPT function effectively according to concurrent and financial audits (Fully Achieved and Surpassed). Number of authorized providers receiving the basic medical equipment/vehicles/communications equipment according to the annual performance agreement (Fully Achieved and Surpassed). 15. Component 2: Strengthening the National and Provincial Ministries of Health Stewardship Capacity (Original Bank allocation of US$10.2 million; final expenditure of US$16 million) 16. Description: The aim of this component was to adapt the provincial Ministries of Health to meet the implementation requirements (information, managerial) of the MCHIP. It includes the essential and major structural change of separating the purchasing and provision of services and setting up and training national and provincial health service purchasing team. This component provided financial support for: i) reorganizing participating Provincial Ministries of Health (PMHs) in both staffing and interrelationships, as necessary; ii) improving epidemiological information, financial, and human resource management systems; iii) completing studies essential for NMH policy formulation. 17. Achievement of Intermediate Results for this Component (see also Table 2): Number of PHSPTs established and functioning, capable of preparing and negotiating NMH-PMH and PMH-authorized providers annual performance agreements (Fully Achieved and Surpassed). NHSPT is established and functioning, capable of preparing and negotiating PMH-NMH annual performance agreements (Fully Achieved). 18. Component 3: Communications and Community Outreach (Original Bank allocation of US$17 million; final expenditure of US$667,000) 19. Description: The aim of this component was to ensure the effectiveness of the MCHIP through providing the eligible population, particularly those who have historically been marginalized, with enough knowledge about, and motivation to use, the services being offered. The component was supposed to provide financial support for: Consultant services, incremental ministry operating costs, event organization, and media communication services to support two main lines of communication: i) Dissemination of detailed information about the program among major stakeholder groups (provincial governments and their populations, the Federal Health Council (COFESA), the medical profession and insurance agency managers and staff). ii) Community outreach to increase participation of the eligible population. 20. Achievement of Intermediate Results for this Component (see also Table 2): Regular information process among stakeholders on maternal-child health care issues in general and entitlements in the Plan Nacer in particular (Fully Achieved and Surpassed). 28

45 Targeted groups increase, knowledge of their entitlements under the program and participation in Plan Nacer, and report satisfaction with process and results (Fully Achieved and Surpassed). 21. Component 4: Program Monitoring, Evaluation and Concurrent Auditing Systems (Original Bank allocation of US$14.6 million; final expenditure of US$16.8 million) 22. Description: This component provided financial support for: i) Information technology design services, software and equipment, and training for NMH and PHMs staff to upgrade the monitoring of health provider performance in collecting and reporting information; ii) External concurrent auditing of key elements underlying transfers of capitation payments; iii) In-depth Project evaluation, including the completion of the baseline for impact indicators and Project impact evaluations at mid-term and closing. 23. Achievement of Intermediate Results for this Component (see also Table 2): Project implementation reports available as agreed, including financial reports supporting the capitation payments. Satisfactory reports from concurrent and annual financial auditor. (Fully Achieved and Surpassed). Project evaluation implemented: baseline at the end of PY2; mid-term impact evaluation at the end of PY3; and final impact evaluation at the end of last year of APL-1. (Partially Achieved see Table 2 for more details). 24. Component 5: Project Management and Administration (Original Bank allocation of US$1.6 million; final expenditure of US$2.6 million) 25. Description: This component provided financial support for operational expenses such as travel costs for the provinces travel and per diem costs for coordination meetings, mainly for the National Health Services Purchasing Team (NHSPT) activities. Discussion of Achievement of Individual Components 26. All intermediate outcome indicators were achieved, for all Components, except for two indicators, for which achievement was partial. One of these indicators with only partial achievement was for Component 3 (Communications and Community Outreach), which also had a very low level of expenditure of just US$667,000 out of a total allocation of US$17 million. 27. This component had some difficulties, as raised in the ISRs. The Communication Strategy was ultimately only partially implemented, and disbursement of the component was just 4% at Project close. This component had a rating of MU at Project close. However, the government stated that first, communication campaigns were absorbed by the Government Communication Unit without the direct participation of the National Ministry of Health, and second, the government wanted to avoid an increase in the demand for health services in urban areas to an extend more than the health system could deliver. On the other hand, there was a lack of demand side interventions to bring people to health facilities particularly in rural areas were access was low. 29

46 28. However, the goals of the component were still partially achieved, as shown by the achievement levels of the two Intermediate Outcome Indicators for this component (one was fully achieved and the other partially achieved). This was because many of the activities were still carried out, financed from Government own sources. The Government also noted that communication campaigns were absorbed by the Government Communication Unit, outside of the National Ministry of Health. 29. The partial achievement of one of the indicators for Component 4 (Program Monitoring, Evaluation and Concurrent Auditing Systems) was due to challenges with the planned Impact Evaluation for the program. See Annex 13 for a full discussion of this. 30

47 Annex 3. Economic and Financial Analysis A. Trends in Enrolment 1. Enrolment in Plan Nacer has tended to rise rapidly from the time a province is newly enrolled, and at this time close to 100% of eligible children are enrolled. But the enrolment rate among women has been lagging to some extent (although it is still high). Figures 1 through 4 show basic trends in enrolment of eligible women and children in Plan Nacer, over time, separately for the Phase 1 and the Phase 2 provinces. 160, ,000 Figure 1. Number of women enrolled 120, ,000 80,000 60,000 Phase 2 provinces Phase 1 provinces 59,587 39,672 24, , ,113 81,699 40,000 20, ,863 43,977 43,773 42,655 44,245 48,262 43,202 22,450 3, ,000,000 1,800,000 Figure 2. Number of children enrolled 1,600,000 1,400,000 1,200,000 1,000, , , , ,000 0 Phase 2 provinces Phase 1 provinces 495, , ,183 1,240,090 1,078, , , , , , , , , ,512 30,

48 80 70 Figure 3. Percentage of women registered, out of eligible population Phase 1 provinces Phase 2 provinces % 0f eligible population Figure 4. Percentage of children registered, out of eligible population Phase 1 provinces Phase 2 provinces These figures show that enrolment has initially moved sharply upwards for both Phase 1 and Phase 2 provinces, but has then plateaued or even fallen to some extent once it has hit the 60% to 70% range for women. However, for children enrolment rates have continued to move up to 100% for both Phases 1 and Phase 2 provinces. 3. The reason for the lower enrolment rates for women is that it has proved challenging, at times, to attract early registration (enrolment) on the part of women. This would need to occur via health centers, and yet it has proved harder to get health centers to participate actively in the program, than hospitals. The reasons for this include higher capacity on the part of hospitals, as well as better record-keeping and registration systems, and better ability to invoice properly. 32

49 Thus, often, a woman is registered in Plan Nacer only when she goes to a hospital for her birth. Registration rates for children are high, on the other hand, because children get enrolled from the time the mother goes to the hospital to give birth, and they generally remain enrolled afterwards until they reach age six. B. Trends in Infant Mortality, Neonatal Mortality and Maternal Mortality 4. Plan Nacer aimed to address basic health care needs of uninsured pregnant/lactating women and young children. Thus it is illustrative to look at trends in infant, neonatal and maternal mortality, with the important caveat that there are many factors that affect these indicators aside from Plan Nacer and the characteristics of this program. Economic circumstances (at the national and local level) affect these indicators, for example, and so movements (or lack of movement) in these indicators cannot always be attributed to Plan Nacer. 5. With that caveat, the trends in infant, neonatal and maternal mortality are now analyzed. Figure 5. Infant Mortality Rate National, Phase 1 Provinces and Phase 2 Provinces Rate per 1,000 live births Phase 2 Phase 1 National 33

50 Figure 6. Infant Mortality Rate (IMR) - Ratio of IMR of Phase 1 Provinces to IMR of Phase 2 Provinces Figure 7. Neonatal Mortality Rate National, Phase 1 Provinces and Phase 2 Provinces Rate per 1,000 live births Phase 2 Phase 1 National 34

51 Figure 8. Neonatal Mortality Rate (NMR) Ratio of NMR of Phase 1 Provinces to NMR of Phase 2 Provinces Figure 9. Maternal Mortality Rate National, Phase 1 Provinces and Phase 2 Provinces 9 8 Rate per 1,000 live births Phase 1 Phase 2 National Linear (Phase 1) 35

52 6. The Figures (5 to 9) indicate the following: 7. The infant mortality rate (IMR) and neonatal mortality rate (NMR) have been falling in both the Phase 1 and Phase 2 provinces, but the fall has been faster in provinces included in Plan Nacer. Figure 5 (for the IMR) and Figure 7 (for NMR) show the falling trends in both groups of provinces. Figure 6 shows that the ratio of the IMR in the Phase 1 Provinces to the IMR in the Phase 2 Provinces fell substantially between 2004 and 2007 years in which the Phase 1 provinces were in Plan Nacer (but not the Phase 2 provinces). The Phase 2 provinces entered the program in 2007, and from then on the ratio of the two IMRs remained roughly constant. In short: The IMRs of the Phase 1 Provinces and Phase 2 Provinces converged to each other when Plan Nacer was operational in Phase 1 provinces (from 2004 to 2007). Once the Phase 2 provinces were also included (from 2007 onwards), the ratio of the two IMRs remained roughly constant. 8. Similar observations can be made for the NMR. As Figure 8 shows, the NMRs of the two provinces converged from 2004 to 2007, when only Phase 1 provinces were included in the program. Once the Phase 2 provinces were included from 2007 onwards, the divergence in the two NMRs ceased. In fact, the NMR of the Phase 2 provinces actually fell faster than the NMR of the Phase 1 provinces between 2009 and (This can be seen from the slight rise after 2009 in the line tracking the ratio of the two NMRs in Figure 8, after falling quite sharply for several years.) The improved relative performance in the Phase 2 provinces regarding the NMR may be due, in turn, to the faltering in the previously rising trend in the enrolment rate of eligible women from 2009 onwards in the Phase 1 provinces (see Figure 3). 9. Maternal mortality (MM) has shown a falling trend in the Phase 1 provinces, where it is highest to begin with. However, there is no obvious trend rising or falling in the Phase 2 provinces since they joined the program. This can be seen from Figure 9. While the lack of a falling trend in the Phase 2 provinces is not ideal, these provinces do start out with a relatively low maternal mortality rate already. 10. A recent analysis of MM in Argentina shows that there are many factors affecting it that need to be modified outside of a program like Plan Nacer, and by itself such a program cannot lower MM rates below a certain point. This was one of the findings underlying the recent Operational Plan for Reducing Maternal Mortality in Argentina, developed by the National Ministry of Health in Argentina. To reduce MM below levels already seen in the Phase 2 provinces, additional steps are needed, including, among others: (a) redefining the roles of hospitals, especially the ones that currently do not comply with all of the conditions for providing adequate maternal care; (b) strengthening the referral network, and assigning appropriate roles to health facilities in the network; and (c) providing additional technical assistance to hospitals and provinces to that they can maximize use of (and invoicing of) the new sub-package of additional complex maternal health services that were included in 2010 into the Nomenclador of Plan Nacer (see main text of this document). Utilization of and invoicing for these additional complex services, designed to address high-risk births, was not high in 2011 and much of 2012, but is now picking up. 36

53 11. The maternal mortality rate of the Phase 2 provinces should fall under the follow-on program Plan Sumar, as the new sub-package of additional complex maternal health services becomes entrenched, and as other steps are taken outside of Plan Sumar. Plan Sumar is the follow-on program, of Plan Nacer. It will support similar services as Plan Nacer, and will add additional population subgroups in addition to pregnant/lactating women and undersix children. As the steps mentioned in the previous paragraph and others are undertaken, in line with the Operational Plan for Reducing Maternal Mortality in Argentina, the chances of falls in MM even below the levels seen in the Phase 2 provinces are good. 12. Further analysis shows that there is a particularly close relationship between health care coverage rates of pregnant women either by formal health insurance or by Plan Nacer and the neonatal mortality rate (NMR). For this analysis, an indicator was created called Health Coverage Gap (HCG) Among Pregnant Women. Values of this indicator were calculated separately over time for the Phase 1 Provinces and for the Phase 2 Provinces. This indicator is defined as the percentage of all pregnant women in the entire population (either within the Phase 1 or the Phase 2 provinces) that are not covered by either formal health insurance or by Plan Nacer (which is conceptualized as a public health insurance program). Figure 10. Health Coverage Gap (HCG) Among Pregnant Women Versus Neonatal Mortality Rate (Phase 1 provinces) HCG among pregnant women, in preceding year Health Coverage Gap Among Pregnant Women, in Preceding Year Neonatal Mortality Rate Neonatal Mortality Rate 37

54 35 Figure 11. Health Coverage Gap (HCG) Among Pregnant Women Versus Neonatal Mortality Rate (Phase 2 provinces) 8.0 HCG among pregnant women, in preceding year Neonatal Mortaluity Rate Health Coverage Gap Among Pregnant Women, in Preceding Year Neonatal Mortality Rate 13. The HCG Among Pregnant Women was lagged by one year and then plotted on the same graph as the NMR, for the Phase 1 Provinces (see Figure 10). The same was done for the Phase 2 Provinces (see Figure 11). The figures show that the neonatal mortality rate (NMR) closely tracks the value of the HCG Among Pregnant Women in the preceding year. It makes sense that a fall (or rise) in health care coverage among pregnant women would lead to a rise (or fall) in the NMR in the following year, because: (i) in the case of Plan Nacer, as discussed above, a low enrolment rate among pregnant women is usually due to low registration of newly pregnant women during the earlier trimesters, rather than low registration of women at birth (when they are easily registered); and: (ii) low registration during the earlier trimesters may have an impact on NMR, but that would occur at the time of birth (or immediately afterwards), which would often. 14. Remarkably, the relationship between the two variables (HCG Among Pregnant Women in the previous year and the NMR) remains close even when the HCG Among Pregnant Women rose in 2010, for Phase 1 provinces. Furthermore, the proportionality of the relationship between the two variables i.e. the ratio of the change in one to the change in the other is roughly similar for the Phase 1 provinces as in the Phase 2 provinces. 15. The spike upwards in the HCG Among Pregnant Women in the Phase 1 provinces in 2009 (the year preceding 2010) is attributable to the fall in the enrolment rate of women in Plan Nacer in 2009 see Figure 3. The HCG Among Pregnant Women rose by 8.4% that is, by 84 for every 1,000 pregnant women in 2009 (the year before 2010), and the NMR rose by

55 (per 1,000 live births) in This implies that for every X additional women enrolled in Plan Nacer, the number of neonatal deaths saved will be X multiplied by the factor [0.23 / 84] = This value is used in the Benefit-Cost Analysis conducted below. 16. This analysis is not as rigorous as one that controls statistically for other confounding factors exogenous factors that lead to changes in the Plan Nacer enrolment rate and at the same time to changes in the NMR, resulting in spurious correlation between these two variables. It is hard to see what these exogenous confounding factors could be, in this case. The exogenous confounding factors would need to explain the fall in the Plan Nacer enrolment rate in 2009, and at the same time the rise in the NMR in Nevertheless, the magnitude of the impact on neonatal deaths, of being enrolled in Plan Nacer, as estimated using the above exercise, is remarkably similar to that found from the results of Impact Evaluation (IE) activities under Plan Nacer. Great care was taken to correct statistically for confounding factors in the IE analysis, which is discussed in the next section. The similarity of the findings from the two different sets of analyses can be seen from the similar estimates of the number of neonatal deaths averted, the Benefit-to-Cost ratio and the internal rate of return from the two different exercises in Section D. C. Results of Impact Evaluation for Plan Nacer 18. As detailed in Annex 13, while the Impact Evaluation (IE) activities under Plan Nacer had their challenges, reliable results have so far been obtained from two IE exercises involving analysis of administrative data: (A) an analysis of data from medical records at health centers and hospitals (as well as from Plan Nacer records) at Tucuman and Misiones provinces covering the years 2006 to 2009; and (B) an analysis of data from medical records at public maternity wards (as well as from Plan Nacer records) at 13 provinces the nine northern Phase 1 provinces as well as Cordoba, Entre Rios, La Rioja and Santa Fe. 19. The data and results from (A) have been finalized and are fully available, while only preliminary results are available at this time from (B). As discussed in Annex 13, results from survey data are not yet available, although data from the follow-up IE survey of the Phase 2 provinces are now being cleaned and analyzed and also be available in the next few months. 20. A summary of key results from (A) and some preliminary results from (B) are given below. The analyses only looked at the eligible population, i.e. only pregnant/lactating women and under-six children without formal health insurance. Among the eligible population, both Plan Nacer beneficiaries as well as those that chose not to be enrolled in the program were included in the analyses. In all cases, statistical techniques have been used to correct for possible bias, such as selection bias. 21. Table 1 presents the key results from these two IE exercises. All of the results shown are statistically significant, and in most cases highly statistically significant (significant at the 1% level). These results are very strong, and much stronger than what has been found in most other similar IE exercises. 39

56 Table 1. Implications of Results of Impact Evaluation (IE) Exercises for Plan Nacer Variable of Focus Implied change due to Plan Nacer 2 First IE Exercise for Tucuman and Misiones Provinces Number of Babies Born with Very Low Birth Weight 1 Fall of 26% (per 1,000 live births among eligible population) Number of prenatal consultations (per 1,000 pregnant Rise of 16% women among eligible population) Number of women receiving antitetanus vaccine (per Rise of 10.1% 1,000 pregnant women among eligible population) Number of consultations for infants aged 45 to 70 days Rise of 32.7% (per 1,000 infants among eligible population) Number of consultations for infants aged 70 to 120 Rise of 21.5% days (per 1,000 infants among eligible population) Number of consultations for infants aged 120 to 200 Rise of 18.2% days (per 1,000 infants among eligible population) Number of neonatal deaths (per 1,000 live births among Fall of 25% eligible population) Second IE Exercise for 13 Provinces (9 Phase 1 Provinces and 4 Other Control Provinces) Number of Babies Born with Very Low Birth Weight 1 Fall of 15.9% (per 1,000 live births among eligible population) Number of Babies Born with Low Birth Weight 1 (per Fall of 27.9% 1,000 live births among eligible population) Notes: 1. Very low birth weight means less than 1500 grams, while low birth weight means less than 2500 grams. 2. The implied change due to Plan Nacer is the change that is estimated conditional on enrolment, e.g. the first row indicates that out of every 1000 live births, the number of babies born with low birth weight would be reduced by 26% if all of these mothers were enrolled in Plan Nacer (as compared to the number of babies born with low birth weight if none of the mothers were enrolled in Plan Nacer). D. Benefit Cost Analysis and Calculation of Internal Rate of Return 22. As part of a Benefit-Cost Analysis (BCA), the benefits from reduced neonatal mortality as a result of Plan Nacer were estimated using two different methodologies: Methodology 1: Using the finding emerging from the graphical analysis of Section B above, that for every X additional women enrolled in Plan Nacer, the number of neonatal deaths saved was X multiplied by the factor [0.23 / 84]. Methodology 2: Using the finding of the first IE exercise described in Section C above, that the number of neonatal deaths was reduced by 25% (in the eligible population) as a result of being a beneficiary of Plan Nacer. 40

57 23. In the case of Methodology 2, it was necessary to estimate the number of neonatal deaths occurring in the eligible population in the first place, since the available data do not give this figure directly. This estimation was made by: (i) taking the number of neonatal deaths as a whole in each province (per year) which is a known figure from administrative data records; and (ii) multiplying this by the assumed ratio of the NMR of the eligible population to the NMR of the population as a whole. The latter ratio was assumed to be the same in all provinces (in all years) as the ratio found in the IE sample dataset drawn from Tucuman and Misiones provinces that was used for the first IE exercise mentioned in Section C. Table 2. Benefit-Cost Analysis: Number of Neonatal Deaths Averted Total Registered mothers (under APL-2) (thousands) No. neonatal deaths saved using Methodology No. neonatal deaths saved using Methodology The value of each neonatal life saved was estimated using the Human Capital Approach, i.e. by calculating the discounted stream of estimated lifetime earnings. The latter was estimated using the following assumptions and parameters: (a) per-capita GDP growth rate of 2% (which is conservative compared to recent trends); (b) discount rate of 3%; (c) income will be earned every year from age 18 to 62, at a level of 70% of the per-capita GDP level (given that these are disadvantaged members of the population). With these assumptions, the value of each life saved was calculated as US$246, This value was applied to the stream of benefits as calculated using each of the two methodologies. Benefit-to-cost ratios (BCRs) and internal rates of return were computed under the assumption that all benefits other than those from averted neonatal deaths are zero. This assumption is clearly not correct, but was nevertheless made for the purposes of obtaining a lower bound for the BCR and the internal rate of return. The cost figure used in the calculations was the cost of the IBRD contribution of US$300 million, together with the cost of the Provincial contributions to the capitation payments (US$39 million). 26. The results are given in Table 3 below. Both methods show similar results, that the Internal Rate of Return from the Project was either 8.2% or 8.5%. The BCR was calculated as around 1.4 in both cases. 41

58 Table 3: Estimated Benefit-to-Cost Ratios and Rates of Return (Taking into account only benefits from averted neonatal deaths) Benefit-to-Cost Ratio Internal Rate of Return Methodology % Methodology % 27. In summary, the internal rate of return from the Project was calculated as either 8.2% or 8.5%, from considering the benefits from averted neonatal deaths alone, without taking into account other benefits. The other benefits not factored into the calculations include the benefits from averted deaths of infants (after the neonatal period); higher productivity due to reduced incidence of low birth weight; reduced maternal mortality (at least in the areas with higher maternal mortality to begin with); and reduced morbidity among infants and mothers, among others. If these additional benefits had been included into the calculations, the internal rate of return would have been much higher. 42

59 Annex 4. Bank Lending and Implementation Support/Supervision Processes Task Team Members Names Title Unit Lending Cristian C. Baeza Task Team Leader LCSHH Luis Orlando Perez Sr. Health Specialist LCSHH Monique Francine Mrazek Sr. Public Health Specialist LCSHH Gaston Mariano Blanco Operations Officer LCSHH Jose Pablo Gomez Senior Economist LCSHH Marta Molares-Halberg Lead Counsel LEGLA Natalia Moncada Program Assistant LCSHD Martha P. Vargas Team Assistant LCSHD Paul Gertler Chief Economist HDNVP Jorge Uquilias Senior Sociologies LCSEO Emiliana Vargas Education Economist LCSHE Antonio Blasco Sr. Financial Management Specialist LCSFM Maria Lucy Giraldo Sr. Procurement Specialist LCOPR Xiomara Morel Sr. Finance Officer LOAG1 David Peters Sr. Public Health Specialist HDNHE Pablo Gottret Sr. Economist (Health) (Peer Reviewer) HDNHE April Harding Sr. Economist (Health) (Peer Reviewer) LCSHD Alejandro Solanot Consultant LCSFM Juan Sanguinetti Consultant Isabel Tomadin Consultant Supervision / ICR Rafael Cortez Task Team Leader, Senior Health Economist (TTL) LCSHH Andrew Sunil Rajkumar Senior Health Economist (TTL and LCSHH ICR TTL) Katharina Ferl ICR Author LCSHH Luis Orlando Perez Sr Public Health Spec. LCSHH Jose Pablo Gomez Senior Economist (Health) LCSHH Alexandre Arrobbio Lead Public Sector Specialist AFTP4 Gaston Mariano Blanco Sr. Social Protection Specialist LCSHS-DPT Vanina Camporeale Operations Officer LCSHH Daniela Romero Operations Analyst LCSHH Keisgner De Jesus Alfaro Senior Procurement Specialist LCSPT 43

60 Ana Maria Grofsmacht Procurement Specialist LCSPT Alvaro Larrea Senior Procurement Specialist LCSPT Alejandro Roger Solanot Sr Financial Management Specialist LCSFM Alejandro Alcala Gerez Senior Caunsel LEGES Efraim Jimenez Consultant LCSUW Rony A. Lenz Consultant MNSHD Luz Maria Meyer E T Consultant LCSFM Juan Luis Sanguinetti Consultant LCSHH Marcos Miranda Consultant LCSHH Paula Giovagnoli ET Consultant LCSHH Paul Gertler Consultant HDNCE Isabel Tomadin Consultant LCSSD Santiago Scialabba Program Assistant LCC7C Sarah Bailey Junior Professional Associate LCSHS Geraldine Beneitez Team Assistant LCSHH Natalia Moncada Senior Executive Assistant LCSHD Gabriela Moreno-Zevallos Program Assistant LCSHH Silvestre Rios Centeno Team Assistant LCC7C Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY FY Total: Supervision/ICR FY FY FY FY FY FY FY Total:

61 Annex 5. Beneficiary Survey Results A. Recent Survey of Satisfaction of Beneficiaries 1. Between July 2012 and May 2013, data were collected from 9633 users of Plan Nacer services as part of a household survey called the Estudio de Monitoreo de la Satisfacción del Usuario y de la Calidad de Atención del Plan Nacer (Study of Monitoring of Satisfaction of Users and of the Quality of Care Given Under Plan Nacer). The data were collected at the household level, from households in all 24 provinces, in 3 phases, as follows: Phase 1 (July to September 2012) Phase 2 (October to December 2012) Phase 3 (March to May 2013) Table 1. Number of Cases in Each Province Province Number of Cases Total in Phase Buenos Aires 600 San Juan 400 La Rioja San Luis 400 La Pampa 400 Chaco 400 Cordoba 400 Mendoza 400 Neuquen 371 Tucuman Santa Fe 400 Santiago Del Estero 400 Rio Negro 400 City of Buenos Aires 400 Catamarca 311 Corrientes 400 Entre Rios 400 Jujuy 400 Salta Chubut 400 Formosa 400 Misiones 400 Santa Cruz 400 Tierra Del Fuego 351 TOTAL Data were collected from households with children aged between 1 and 2 years, since these were considered to have the greatest potential in terms of using health services under Plan Nacer. A strategy of stratification was applied, with 3 different strata as follows: 45

62 Stratum 1: Children with high utilization of services (children aged between 1 and 2 years, with more than 3 consultations invoiced and paid for in the previous year). Stratum 2: Children with low utilization of services (children aged between 1 and 2 years, with between 1 and 3 consultations invoiced and paid for in the previous year). Stratum 3: Children with no utilization of services (children aged between 1 and 2 years, with no consultations invoiced and paid for in the previous year). 3. Interviews were conducted in users homes because, unlike in the case of exit surveys at health facilities, one would not have contamination due to possible temporary effects produced by an immediate feeling (positive or negative) generated by contact with the health system just before the interview. 4. At this time, results from the first and second phases are available. These are from both Phase 1 and Phase 2 provinces, as can be seen from Table A number of questions were asked at each interview some focusing on services received by mothers during the periods of pregnancy, childbirth and postpartum, and some focusing on services received by children up to two years of age. The answers allowed the formulation of an overall User Satisfaction Index, which is a conceptual and statistically consistent measure that accounts for the various factors that affect the satisfaction of beneficiaries with respect to public services provided under Plan Nacer. The methodology used for the construction of this Index is based on statistical tools such as factor analysis and binary logistic regression. 6. As is expressed in the Table 2, the User Satisfaction Index level on the whole, regarding use of Plan Nacer services, is 66 points (out of a possible 100) on average, reflecting a very high level of satisfaction. Table 2. Index of User Satisfaction Regarding Plan Nacer Services, Average Global Value Average Value on Scale of 0 to 100 Index of User Satisfaction Global Value (for all 66.0 users) 7. When the average value of the User Satisfaction Index is calculated separately for those in large urban centers, versus those in smaller urban centers, the value of this Index is found to be higher in larger urban areas (see Table 3). Table 3. Index of User Satisfaction, by Size of Urban Area Average Value on Scale of 0 to 100 Index of User Satisfaction Large Urban Centers 67.8 Index of User Satisfaction Smaller Urban Centers

63 8. The degree of entrenchment of Plan Nacer, for a particular health facility, can be measured in different ways. One simple way is to look at the proportion of services that are eligible under Plan Nacer, that are in fact invoiced and paid for by the program. If Plan Nacer is working as intended, then a higher degree of entrenchment implies more financing under the program, better motivated health workers, better provision of services, and so on. 9. Table 4 presents the values of the User Satisfaction Index for different groups of users, according to whether they use predominantly health facilities with a high, medium or low degree of entrenchment of Plan Nacer. Clearly, there is a positive relationship between the degree of entrenchment of Plan Nacer at a health facility and the degree of satisfaction of the users of health services at that facility. Table 4. Index of User Satisfaction, by Degree of Entrenchment of Plan Nacer Average Value on Scale of 0 to 100 Index of User Satisfaction Among Users of Health Facilities With High Degree of Entrenchment of Plan 67,3 Nacer Index of User Satisfaction Among Users of Health Facilities With Medium Degree of Entrenchment of 65,8 Plan Nacer Index of User Satisfaction Among Users of Health Facilities With Low Degree of Entrenchment of Plan 61,4 Nacer 10. Table 5 presents the values of the User Satisfaction Index for different groups of users, according to level of utilization. Clearly, there is a positive relationship between the User Satisfaction Index and the level of utilization of Plan Nacer services, which is a sign of the positive impact of Plan Nacer in terms of the satisfaction of users, particularly those that are very familiar with the program (i.e. among those with a high degree of utilization of the services under the program). Table 5. Index of User Satisfaction, by Level of Utilization of Plan Nacer Services Average Value on Scale of 0 to 100 Index of User Satisfaction, Among Those With High 68,9 Level of Utilization of Services Index of User Satisfaction, Among Those With Low Level 64,1 of Utilization of Services Index of User Satisfaction, Among Those With No 63,9 Utilization of Services 47

64 B. Survey of Satisfaction Regarding Congenital Heart Disease (CHD) Treatment 11. CHD treatment services were included in the benefits package during the period of implementation of Plan Nacer. Over the period February to April 2013, a telephone survey was conducted of households with beneficiaries of CHD treatment under Plan Nacer. Out of 1698 such households, it was ultimately possible to include 723 (with adequate information gathered) in the survey. 12. Similar to the case of the broader household survey mentioned above, in this case a Satisfaction Index was constructed using a statistical model. The average value of this Index among all households interviewed was 91.3 out of a scale of 10, which is an extremely high value for such studies (see Table below). Table 6. Index of Satisfaction Regarding CHD Services Under Plan Nacer, Average Global Value Average Value on Scale of 0 to 100 Index of Satisfaction Global Value

65 Annex 6. Stakeholder Workshop Report and Results Not Applicable. 49

66 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Evaluation and Dissemination of Experiences in Implementing Public Health Policies to Support Sustained Progress in Achieving Social Goals: the Final Evaluation of Plan Nacer 1. Plan Nacer is the Federal program of the National Ministry of Health through which, since late 2004, the Nation and the Provinces began to implement a policy of prioritization, public insurance and explicit health coverage for the most vulnerable members of the population. The program s immediate objective was to contribute to the reduction of maternal and child morbidity and mortality, in line with the Millennium Development Goals. In addition, the program aimed to strengthen the public health sector in terms of implementing Provincial Maternal-Child Health Insurance programs intended to provide health coverage to pregnant women, lactating women and children under 6 years without explicit (formal) health coverage as a mechanism to ensure, from an equity perspective, more and better access to health services of adequate quality. 2. Throughout its implementation, the strategy of generating Provincial Health Insurance schemes, initiated by Plan Nacer, was characterized by a continuous process of evaluation and dissemination of lessons learned. This enabled a critical analysis of the decisions implemented and the establishment of a strong foundation to face the challenges ahead. 3. This document is a part of the Final Evaluation of the implementation of Plan Nacer in the provinces of Argentina. This year marks the completion of the second phase of Plan Nacer (Phase 2). Phase 2 incorporated the provinces of the Cuyo, Centro and Patagonia regions, starting in mid The provinces of Northeast and Northwest Argentina were a part of Phase 1 starting in Phase 2 of Plan Nacer consists of a geographical expansion of a public health insurance program. This program aims to ensure a common set of prioritized health services, intended initially for mothers and children. Consequently, it will be evaluated as part of a process that seeks to promote universal coverage which is a key principle mentioned in the Constitution, through Provincial Health Insurance schemes. 4. This evaluation is intended to determine the degree of compliance with the objectives and targets in the program s original design and to identify key lessons facilitating the continuation and expansion of the strategy of public health insurance, featuring a set of prioritized services, to a broader population group and addressing more complex lines of care. In order to enable a comprehensive assessment of the implementation of Plan Nacer, in terms of both intended and unintended results at the time of the of design IBRD loan 7409-AR APL II, we propose the following principles that will enable good use of this document in the pursuit of learning, addressing gaps and building a strategic vision of the policy initiated through the program. A. Proposed Principles for the Final Evaluation of Plan Nacer 5. Prior to the Final Evaluation of Plan Nacer, it is necessary to establish a set of principles that will be useful for this final evaluation in particular, for better interpretation of the results and conclusions of this exercise. 50

67 Originally, Plan Nacer was expected to be rolled out in three phases, but then a decision was made to include in the program in 2007 all provinces that were not already included. This implied significant challenges on the part of the Nation (central government) in providing support to complex and diverse processes throughout the country. Phase 2 of Plan Nacer should not be evaluated only as a financing mechanism in isolation, but rather as a step in the process of rolling out the strategy of Provincial Health Insurance (SPS) schemes, in order to provide to the population full access to health care, in line with their rights as enshrined in the Constitution. It must be said that this is a priority today for the National Government. I am not referring here only to Plan Nacer which will continue with its original conception as planned, and which has proved to have very positive results but rather, I am talking about going beyond the original Plan Nacer and expanding its coverage. Dr. Juan Manzur National Minister Health Phase 2 of Plan Nacer was designed and began, in terms of objectives and implementation mechanisms, at a time when Phase 1 had not completed implementation of all mechanisms that were important for the generation of health results. (In June 2007, for example, health providers under Phase 1 were able to use only 54% of the resources received). Phase 1 was implemented in provinces with high rates of maternal and child morbidity and mortality, with few differences between provinces in institutional complexity. By contrast, the provinces of Phase 2 have a wide range of diversity in key aspects such as the size of the eligible population, the number of health facilities, health indicators (before implementation), and the size of the additional resources (from Plan Nacer) relative to the local budget, among others. The more advanced level of institutional development of the provinces of Phase 2 and better health outcomes at the outset led to the formulation of Project indicators with ambitious goals given the time period of implementation, without fully taking into account possible significant barriers to effective implementation. (But, most of the challenging targets were achieved or were close to being achieved at the Project close. This reflects the strong commitment to fulfill the Development Objectives on the part of the Government team and the provinces.) The implementation strategy of the Provincial Health Insurance schemes (SPSs) in Argentina, in particular Phase 2 of Plan Nacer, is characterized by the simultaneous search of progress in explicit universal coverage in various areas: population, health services and financial protection. This simultaneity was possible thanks to the program s Monitoring and Evaluation system, which provided relevant and timely information to support decisions made at each step. The ability to successfully implement Phase 2 of Plan Nacer was favored by the flexibility and adaptability of the design to local realities. No design can provide everything needed to ensure successful execution and be free of undesirable occurrences. Thus, in the first phase of implementation of the SPSs, more 51

68 emphasis was given to overall performance, while in the later phases of implementation of the SPSs, emphasis will also be given to enhancing equitable outcomes. The SPSs come with a set of tools and a framework to help translate the Nomenclador (services package) into effective access to health: Alliances with others responsible for good health outcomes (Maternal and Child Health Units of Ministries of Health, Municipalities, Social Security Schemes, Health Zones, etc.) Training and communication to health teams and to the public. Search of consensus with health establishments, definition of explicit targets and ongoing discussion with them of performance. Package of services that focuses on priority health services, whose prices are determined according to an assessment and prioritization of services within each Province. Adequate use of funds received via the program, on the part of the health establishments. Medical audits Application of financial penalties to health establishments for failure to follow the rules of the program. B. Implementation of the Different Axes of Universal Coverage: Results as Reflected in the Project Development Objectives (PDO) Indicators, And Other Results Not Reflected in the PDO Indicators Results Reflected in the PDO Indicators 6. In terms of insurance coverage under the program, 95% of the eligible population was nominalized (identified with ID i.e. Identification Number and names recorded and entered in a database, etc.) in the Phase 1 provinces, and 94% of the eligible population was nominalized in the Phase 2 provinces, in December This result was very much helped in the Phase 2 provinces by the Universal Child Allowance (Asignación Universal por Hijo) and Universal Allowance for Pregnant Women (Asignación por Embarazo) policies. 7. The program was able to identify a manageable number of performance indicators (Tracers) that allowed the institutionalization of the agreements between different levels of government, working towards improved health results via the use of a system of results-based transfers. 8. Among the tracers, one can distinguish between those measuring final health outcomes (e.g. birth weight and APGAR score) versus those measuring the processes for generating results. The former are measured at the time of birth, mainly in hospitals and maternity wards, and had high levels of achievement even before the start of Plan Nacer, according to baseline data for the Phase 1 and Phase 2 provinces. Other tracers like the percentage of pregnant women having prenatal checkups early in the pregnancy, and the percentage of children having well-child consultations, are measured (and administered) mainly at the level of the Centers of Primary Attention (health centers). These had lower levels of achievement before the start of Plan Nacer. 52

69 9. Measuring health outcomes using reported data, to track tracer achievement levels, is difficult because of deficiencies in reporting leading to an under-statement of the true levels of achievement. The reported levels of achievement in most provinces are well below the corresponding figures obtained from various statistically representative surveys (e.g. Surveys of Living Conditions, baseline surveys, the National Nutrition and Health Survey). 10. An analysis of the deficiencies in this capacity for reporting, done by the Plan Nacer national Government team (the Unidad Ejecutora Central or UEC) and the provinces at the time of the Mid-Term Review of the second phase of implementation of the program, identified the possibility of substantially improving this reporting for many of the tracers. 11. Plan Nacer, during its implementation, institutionalized formal agreements between the Federal government and the provinces, and between provinces and health facilities via Annual Performance Agreements (Compromisos Anuales de Gestión) signed between the Federal government and the provinces, and via Performance Agreements (Compromisos de Gestión) signed between the SPSs and the health facilities. As of December 2012 all provinces had signed Annual Performance Agreements and had succeeded in signing Performance Agreements with more than 90% of their health facilities. 12. The Mid-Term Review was a welcome opportunity to assess the determinants of the gaps in reporting and in results that led to the then-reported levels of achievement (vis-à-vis the targets). A set of actions was identified (Action Plan) relating to different aspects of the implementation of Plan Nacer, aiming to have a substantial impact on performance in the short term. A subset of these actions is presented in the following table. Table 1. Subset of Strategic Actions Implemented to Improve Performance in the PDO Indicators Actions in Original Action Plan Definition and Identification of Health Facilities of High Impact (i.e. contributing to a large extent to provincial performance because of size, etc.): Setting targets for these facilities Monitoring and close follow-up of performance of these health facilities Actions Implemented The national and provincial Plan Nacer teams closely and regularly monitored the performance of every one of these health facilities, and actions were agreed to in order to maximize the chances of attaining the targets agreed to for each facility. Deepening of analysis and Technical Assistance to the provinces with respect to the role of service fees (prices) in the implementation of the provincial fee-forservice insurance schemes: Training with respect to the strategic role of the service fees (prices) in the definition of the policy applied under the program. Training on production planning by health facilities, and its coordination with higher institutional structures. These trainings took place on October 10 and 11, 2011, with the participation of the technical units of the provincial Plan Nacer teams, the technical unit of the national Plan Nacer team (UEC), and leading academic experts who evaluated different aspects of the design and implementation of the program, and made valuable recommendations. 53

70 Actions in Original Action Plan Strengthened engagement with the provinces, vis-a-vis the application and exploitation of the Plan of Production and Application of Funds (PPAF): Training on the strategic role of this tool as a channel of communication between different institutional levels and as a planning and monitoring tool. Training for health facilities on the use of this tool. Agreement on targets by health facilities regarding the use of the PPAF by Visits to the provinces: joint search for strategies to improve performance, and joint agreement on plans of action. Regional Encounters: modalities of joint work with selected sub-groups from the provinces. Telephone conferences with provincial teams: modalities of joint work between a UEC team and provincial teams. Actions Implemented Took place in the second semester of 2011 via Video-Conference with the participation of all provinces. Tutors (persons to provide the training needed) in the use of funds were appointed for the provinces. Implemented through the Performance Agreements of Between January and August 2012, 22 provinces were visited. Between June and September 2012, 5 Regional Encounters took place, with the participation of the provincial Project teams from all the provinces. Between July and August telephone conferences took place with 9 provinces. 13. The identification of these issues that were critical for the generation of results, and for the planning and implementation of strategic actions, resulted in significant improvements in the reported levels of the provincial performance indicators related to the Project PDO Indicators. 14. However, it is necessary to make an observation about the periods used to evaluate the results reflected by the Project PDO Indicators. Periodically measuring (e.g. monthly and 4- monthly) progress in performance requires the measurement of a reference parameter that permits a normative judgment regarding the result, i.e. it requires measurement of a parameter that represents the potential outcome. The accurate measurement of this parameter requires the development and implementation of a consistent methodology in the context of the information available. In practice, whatever efforts one might make in terms of methodology, the available information may place limits on what can be done. 15. Under the framework of Plan Nacer, even with technically solid methodologies, the data do not permit estimations with intra-annual variation, due to a lack of reliable information on population and social phenomena leading to seasonality patterns. Moreover, the nature of the required information would make it necessary to make projections of variables taking into account a large number of socio-economic determinants where even in cases where these can be measured there are important data limitations (e.g. in the case of the percentage of the population without formal insurance coverage). 16. These limitations become especially relevant in the context of targets for intra-year periods (e.g. months or cuatrimestres ), making it impossible to adequately distinguish between variation in performance on the one hand and variation in seasonal phenomena (and measurement errors) on the other hand. To illustrate this with a concrete example, the Phase 1 provinces have reported, on average, a performance level exceeding the target for the proportion 54

71 of live births (among eligible mothers) weighing more than 2,500 grams (Tracer III). But these levels are significantly lower in the 3rd cuatrimestre (September-December) of each year, i.e. there is a seasonality pattern. 17. Below we describe the performance of the Project PDO Indicators as of December Note that for the last cuatrimestre (September-December 2012), the results of the External Concurrent Audit were not available for all provinces, and so the reported values incorporate estimates of the post-audit values (based on pre-audit and post-audit performance data from the first two cuatrimestres of 2012): Regarding the PDO indicators designed to monitor the sub-objective of establishing a new incentive scheme for provinces and provincial providers" (PDO Indicators 9, 10 and 11 of Table 3 of main text): Target of 60% substantially exceeded for percentage of Annual Performance Agreements successfully implemented. In 2012, 79% of the provinces achieved the agreed actions (relating to key dimensions of program management and implementation). The performance levels of the Phase 1 and Phase 2 provinces were 89% and 73%, respectively. Target of 50% substantially exceeded for percentage of authorized providers under Performance Agreements and provider payment mechanism with its respective participant province. Performance for this indicator reached 97% for the Phase 1 provinces and 94% for the Phase 2 provinces. It should be noted that all of the provinces succeeded in exceeding their provincial targets for this indicator, and the global achievement level was 95%. Target of 70% substantially exceeded for percentage of Tracer targets achieved by the participant provinces in last year billing period. Performance for this indicator reached 98% for the Phase 1 provinces and 91% for the Phase 2 provinces. The global achievement level was 94%. Regarding the PDO indicators designed to monitor the sub-objective of improving critical intermediary outcomes : Target of 85% exceeded for proportion of newborns from enrolled pregnant women weighing more than 2,500 grams (linked to Tracer Indicator III), with this indicator reaching 90% in the last cuatrimestre of The provinces incorporated in Phase 2 reached 94% for this indicator, while the provinces of Phase 1 achieved a performance equivalent to 98% of the target (84%). It should be mentioned that since the 2nd cuatrimestre of 2010, the Phase 1 provinces have been able to report, on average, performance levels exceeding the targets, but with dips in performance in the 3rd cuatrimestre. Furthermore, the last cuatrimestre of Plan Nacer coincided with a period of renegotiation of salaries and strikes in the health sector, adversely affecting reported levels of results. Target of 92% exceeded for proportion of newborns, from eligible pregnant women, with Apgar score > 6 at minute 5 (linked to Tracer Indicator II), with this indicator reaching 93% in the last cuatrimestre of The provinces incorporated in Phase 55

72 2 reached 96% for this indicator, while the provinces of Phase 1 achieved a performance equivalent to 95% of the target (87%). Similar to the situation for the previous indicator, the Phase 1 provinces, on average, succeeded in attaining much higher levels of 94% and 93% for the first and second cuatrimestre (January-April and May-August) of 2012, respectively, exceeding the target in each of those cases, before seeing a dip in the third cuatrimestre. Regarding the PDO indicators designed to monitor the sub-objective of increasing access to basic health services of the target population : Target of 80% exceeded substantially for proportion of eligible population voluntarily enrolled in program at the level of the country as a whole (97%), as well as for the Phase 1 and Phase 2 provinces individually (99% and 96% respectively). For the proportion of eligible pregnant women with first antenatal care visit before 20th week of pregnancy (linked to Tracer Indicator I), the achievement level at the national level in the last cuatrimestre of 2012 was 67%, or 96% of the target of 70%. On average, the Phase 1 provinces attained an achievement level of 95% of the target (67%), while the Phase 2 provinces attained an achievement level of 96% of the target (68%). For the proportion of eligible pregnant women who get VRDL during pregnancy and antitetanic vaccine previous to delivery (linked to Tracer Indicator IV), the achievement level at the national level in the last cuatrimestre of 2012 was 93% of the target of 90%, reaching 83% in the last cuatrimestre. On average, the Phase 1 provinces attained an achievement level of 92% of the target (82%), while the Phase 2 provinces attained an achievement level of 93% of the target (84%). It is important to emphasize the value of this result, given the complexity of coordination in real time and in a nominalized manner, for this indicator. The reporting for this indicator requires the reporting of two linked sets of results where the verification needs to take place at the level of hospitals as well as Centers of Primary Attention (health centers). For the proportion of eligible children less than 18 months old with coverage of measles vaccine or triple viral pregnant women with first antenatal care visit before 20th week of pregnancy (linked to Tracer Indicator VI), the achievement level at the national level in the last cuatrimestre of 2012 was 77%, or 80% of the target of 95%. On average, the Phase 1 provinces attained an achievement level of 98% of the target (93%), while the Phase 2 provinces attained an achievement level of 67% on average, with 5 provinces reaching 100% for this indicator. The target of 60% was substantially exceeded for the proportion of eligible puerperal women that received at least one Sexual and Reproductive Health Care consultation (linked to Tracer Indicator VII), with the achievement level at the national level in the last cuatrimestre of 2012 reaching 90%. This target was exceeded substantially for the Phase 1 provinces (86%) as well as the Phase 2 provinces (92%). For the proportion of eligible children 1 year old or less, with all well child consultations up to date (percentile of weight and height) (linked to Tracer Indicator VIII), the achievement level at the national level in the last cuatrimestre of

73 was 45%, or 91% of the target of 50%. Eight of nine Phase 1 provinces reached or exceeded the target for this indicator. The same was true for five Phase 2 provinces. 18. In summary, the first group of indicators (for new incentive scheme for provinces and provincial providers ) substantially exceeded the targets set, for all indicators. In the case of the second group of indicators (for improving critical intermediary outcomes ), the targets were exceeded for both indicators. Finally, for the third group (for increasing access to basic health services of the target population ), the targets were exceeded for three of the eight indicators, while for the other indicators, the level achieved varied between 80% and 96% of the targets originally set. Results not Reflected in the PDO Indicators Institutional The incorporation of the provinces of the Northeast and Northwest into IBRD Loan 7409-AR (APL-2) provided timely financing to accompany and support the progress in the implementation of the Provincial Health Insurances in those provinces. The key elements of Plan Nacer identification of the target population, prioritization of services for these populations, design of targets and consensual search for results among key actors, financing based on results and clear mechanisms for supervision as well as internal and external auditing constituted critical tools for the provincial health systems seeking to move towards better and more equitable health outcomes. Plan Nacer has gained strong recognition that is based on the support provided by the health teams and provincial health ministries, which led to Plan Nacer being the only national program where participation is a requirement for the most relevant Social Policies of the National Government: the Universal Child Allowance (Asignación Universal por Hijo) and the Universal Allowance for Pregnant Women (Asignación por Embarazo). Plan Nacer participated jointly with ANSES (the National Social Security Administration) in the design of the Universal Allowance for Pregnant Women (Asignación por Embarazo) policy. This coordination, in planning as well as implementation of policy actions, represents an unprecedented step forward in the path of institutionalization of the lessons of the program. With the Universal Child Allowance policy which will necessitate enrolling in Plan Nacer and having all the needed checkups on the part of mothers we are making a very strong commitment to life and also to the objectives of lowering infant mortality and lowering maternal mortality, and I am sure we will be able to succeed. Cristina Fernández de Kirchner President of the Nation Opening of the 129th Period of Ordinary Sessions of the National Congress of Argentina. 1 March

74 Thanks to the presence of Plan Nacer at the national level, the State has ceased to be seen as a threat and has become a facilitator, a creator of opportunities, enhancing creativity in the management of resources. This translates into a relationship between the Nation and the Provinces where each party contributes and both gain. Cristian Baeza Director of Health, Nutrition and Population, Human Development Network, World Bank. National Meeting featuring results of implementation of Plan Nacer as an innovative strategy in health financing. March 2011 I was impressed by how Plan Nacer helps to improve the health system. UNICEF will continue to monitor this policy which puts children in the first place. Nils Kastberg Regional Director for Latin America and the Carribbean, UNICEF.. August 2008, Capital Online Journal of Misiones Province. The ability of the national government and provincial governments to implement, gradually but steadily, a program of financing for results in a middle-income Federal country like Argentina, as well as the design of the program and the initial results from its impact evaluation, have gained strong international recognition. The characteristics of the design, implementation and execution of the program, as well as the results noted, were considered sufficient for the program to be recognized by IBRD over other projects, and to be awarded a "Good Practice Award". Advances in the process of establishing health insurance for the population were based on information obtained from unpublished studies, health service provider capacity, the costs of services, and utilization levels observed and desired for priority services all making possible the formulation of agreements between the National and Provincial ministry teams, based on this evidence gathered. The program generated transfer mechanisms between the Federal government and the provinces with very low administrative burden, allowing a flow of resources almost in real time. ISO Certification of the Processes of Nation-to-Province Transfers Plan Nacer implemented, from July 2012 onwards, a System of Quality Management following international standards. Thus allowed the process (termed "Transfer for Health Results Under the Sumar Program Involving the Nation and the Provinces") to be certified under ISO 9001: 2008 by IRAM (Instituto Argentino de Normalización y Certificación), with registration Number 4959 in January IRAM is an institution recognized for its role in standardization and quality certification, locally and internationally. Resource transfers from the Provincial Health Insurances to health facilities initially did not have a predetermined circuit, generating large administrative burdens and delays in payments, which was detrimental to the notion of an incentive scheme as envisaged under the program. This led to the need for the program to ensure the existence of specific, formal and transparent payment circuits, involving extra-budgetary transfers from the accounts of the Provincial Health Insurances to health facilities, which for the first time started to receive and administer funds in an autonomous manner. 58

75 The national government and the provinces co-financed the capitation payment paid to the Provincial Health Insurances to cover all services in the benefits package of the program. The Nation contributed 70% of the capitation value and the Provincial Governments the remaining 30%. This integrated funding mechanism is unprecedented in the public health sector of the country. The ability by the health facilities to decide on the use of the funds received led to changes in the levels of satisfaction on the part of the health workers, independently of whether or not the funds could be used for monetary incentives to individuals. Study on the Financing and Use of Resources by Health Facilities in the Provinces of Northern Argentina Given the diversity of institutional settings and the different relationships established by different province with their health facilities, a study was carried out to: Assess the model implemented for the process of decisionmaking regarding the use of funds, applied in different provinces. Investigate the extent to which the policy regarding the use of funds has had an impact on the motivation of health workers. Identify deficiencies in monitoring of the use of funds in each province and allow for more effective supervision. Main Results The level of satisfaction with the model implemented for the process of decisionmaking regarding the use of funds under Plan Nacer was given a score of nearly 7 out of 10 according to the managers of the health facilities. Among the health workers, the satisfaction level was 7.2 out of 10. Identification of Principal Determinants Among the reasons mentioned for the above, 8 out of 10 health workers highlighted the direct benefits to the population that are generated by the program. Since the implementation of Plan Nacer started, health personnel at the facilities covered mention the enhanced value placed on teamwork and the increased possibilities of professional and personal fulfillment, as a result of the program. The application of the model for the use of funds designed under Plan Nacer was found to improve the organization of work: Plan Nacer encourages one to be more organized since there is more supervision, and it is a task that we all do together. (Testimony from a health team at a Primary Attention Center i.e. health center in Chacos Province.) Optimizes professional practice due to the availability of more supplies and equipment, an increase in the coverage of the target population and an improvement of the physical working environment. Approximately 65% of Directors/Heads of the health establishments noted that funds received under Plan Nacer were distributed across the institution contributing to supplies, instruments and medical equipment to enhance care for the entire population. The flexibility of the program s design to be able to allow for provincial realities provided a lot of latitude in the determination of coordination mechanisms between the 59

76 responsible parties at the ministerial levels and the health facilities, in the search for enhanced results. UEC provided technical assistance to the provinces, and put at their disposition the Plan of Use of Funds tool. In its design, the program gives the provinces the power to set the prices (fees) for the services in the package of services (the Nomenclador ). This characterizes the design of the program as one that strengthens the Federal nature of the Argentine government, and provides flexibility in dealing with the heterogeneity of health conditions among provinces. However, the use of prices for services in the public sector is a tool never before used. Thus, the technical unit of UEC conceptualized and proposed to the provinces an assessment tool to value the services in the Nomenclador. The tool assisted the provinces to assess the contribution of each service to the national and provincial health objectives. On their part, the valuation of the services allowed the provinces to incorporate in their reasoning aspects related to resources needed for service provision, and the related budgetary constraints, allowing the formulation of a consistent set of prices (fees) for the services in the Nomenclador. The mechanisms for contracting health facilities in other provinces were refined. Framework Agreements between provinces were promoted, and later a contracted Provincial Health Insurance was allowed to pay a different set of prices for health facilities outside of its own province. An example of this is an Agreement signed between Salta and Jujuy Provinces, promoted by UEC. Monitoring and Evaluation Plan Nacer is characterized by implementation of a system of records, audits and evaluation, which ensure accountability and the establishment of clearly defined responsibilities. It also promotes and guarantees the use of funds in line with agreements made under a scheme where results achieved are documentated. Unlike other plans, it aims at improving the management of health system resources and promotes transparency in the provision of health care." Philip Musgrove Adjunct Editor, Health Affairs. Provincial Maternal and Child Health Insurance Using Results-Based Financing (RBF). March 2011 Plan Nacer is a dream come true for any child. To enable all children to have coverage and, furthermore, that such coverage is evaluated. It is no use having coverage without monitoring. Fina Rodriguez Executive Medical Director of Garrahan Hospital The program, from the information generated as well as the analysis of that information and the dissemination of the results of this analysis, has allowed the implementation of mechanisms to implement public policies based on evidence (evidence-based decisions). The monitoring and evaluation (M&E) component of the program included the completion of an impact evaluation (IE), both for Phase 1 and Phase 2 of the Project. The IE strategy for Phase I included the use of quasi-experimental methods with the provinces of La Rioja, Córdoba, Santa Fe and Entre Rios as control provinces. This initial strategy 60

77 had to be abandoned as Plan Nacer was expanded throughout the country ahead of schedule. The joint work of the Plan Nacer (Government) and World Bank technical teams allowed the exploration and identification of possible new IE strategies: experimental methods focusing on exogenous interventions at the health facility level, exploitation of the administrative databases of the Provincial Health Insurance schemes, exploitation of the data from clinical birth records from 2004 to 2008 for Phase 1 provinces, and other quasiexperimental methods using instrumental variables, among others. Impact Evaluation Using Perinatal Data: Provinces of Northeast and Northwest The exercise conducted to put together the data for this IE was done using perinatal data from public maternity wards, through an operation that involved teams visiting health facilities, and digitalizing the perinatal data, at thirteen provinces (Catamarca, Chaco, Córdoba, Corrientes, Entre Rios, Formosa, Jujuy, La Rioja, Misiones, Salta, Santa Fe, Santiago del Estero and Tucumán). Between November 2010 and November 2011, on-site digitalization took place for 467,281 birth records at 287 health facilities. More than 170 consultants participated in this operation. A team of specialists led by Professor Paul Gertler (of UC Berkeley) is working on the databases to identify program impacts in terms of the variable available in the database. While research is ongoing, preliminary results show significant impacts of the program on relevant health variables (see Annex 4, especially Table 1). IE should not be seen as the most important source of information for the design of new operations, but rather as a means of providing ex-post empirical validation of some dimensions of the existing operation. The program already has established IE results in terms of the impact on service utilization levels and the health status among the eligible population in Misiones and Tucuman provinces. This IE exercise used data from provincial information systems in order to identify the impact on (among others) utilization levels of health services prioritized by the program, the quality of prenatal care and of well child care, children's health status at birth and decreases in neonatal mortality (see Annex 3 for some results). Recognition of the limitations of quantitative assessment tools in monitoring results along all dimensions of interest promoted the search for other assessment strategies such as qualitative ones. This included, for example, a study of the use of funds (received under the program) by health facilities finance in the northern provinces. A distinctive aspect of the program is its measurement of the levels of satisfaction of users of prioritized health services. This is a necessary first step in the incorporation of civil society in the process of prioritizing public policy actions. Among the major challenges facing Plan Nacer is enhancing transparency vis-à-vis beneficiaries making them take on a central protagonist-type role in the program. Because, after all, Plan Nacer is not just a payment mechanism for provincial health facilities, but also a mechanism to empower people. So that they know their rights, and have (and act on) expectations regarding service delivery. I think this challenge is very important and the Plan has all the elements needed to address it." Amanda Glassman Center for Global Development National Meeting featuring results of implementation of Plan Nacer as an innovative strategy in health financing. March

78 Study of Satisfaction of Users of Plan Nacer (2012) In 2012, a study was conducted to measure user satisfaction, with a methodology that determined the level of overall user satisfaction with the basic services package under Plan Nacer to be 66 out of a maximum score of 100. Additionally, there was evidence that the level of satisfaction is higher when the level of utilization of services is higher (63.9, 64.1 and 68.9 for users with very low use, low use and high use, respectively). Also, the higher the level of integration (insertion) of Plan Nacer in a health facility, the higher was the level of user satisfaction (measured at 61.4, 65.8 and 67.3 respectively for health facilities with a low, medium and high level of integration, respectively). In turn, there is also evidence that levels of user satisfaction are sensitive to the level of complexity of services. Among users of the congenital heart disease services that were a part of the Plan Nacer benefits package, the level of user satisfaction was 91.3 out of 100. The methodology applied provided evidence that Plan Nacer achieved higher levels of satisfaction among users, as compared to the situation without the program. This can be seen from the positive relationship between satisfaction levels and the level of integration of Plan Nacer in health facilities, and also from the positive relationship between satisfaction levels and service utilization levels (higher levels of the latter indicating greater knowledge of the services and their quality). The External Concurrent Audit was an important management tool that facilitated, among other things: Better monitoring of the processes and results on the ground Warnings regarding key errors and limitations Possibility of guiding behavior towards what was desired via effective monetary penalties Training in processes for key personnel responsible for implementation of these processes Greater transparency in the process of using the financial resources of the program Expansion of Program Coverage Among the virtues and the challenges of the Plan is its continuity. Plan Nacer has had more continuity than many public policies in Argentina. The challenge in order to achieve positive results lies in the continuity of its implementation." Ariel Fiszbein Chief Economist of Human Development Network, World Bank National Meeting featuring results of implementation of Plan Nacer as an innovative strategy in health financing. March 2011 Plan Nacer is a centralized and serious attempt to promote greater equity. Mirta Roses Director of the Pan-American Health Organization April 2007, La Razón Journal As occurred in Phase 1, and also for the provinces within Phase 2, Plan Nacer was a program that allowed people without health insurance coverage by the Obras Sociales (Social Security schemes) in these provinces to be for the first time nominalized 62

79 within the health system. Furthermore, the program provides an explicit list of services for those without formal health insurance coverage. A mechanism was developed that allowed the incorporation into the benefits package of new health services with a substantial impact on hard-to-reduce factors affecting infant and maternal mortality, such as treatment for children with congenital heart disease and maternal/neonatal services of high complexity, among others. The incorporation into the Health Services Plan (benefits package) of cost-effective services allowed one to make explicit the rights to health contained in the newest regulations. This advance in the area of health coverage was facilitated by resource availability which also facilitated an acceleration in its implementation. The introduction of congenital heart diseases services for children in the benefits package constitutes a successful and model experience for the Argentinian health sector. It marks the first time that catastrophic care interventions have been included in a public health insurance scheme in the country, and also marks the first Federal network of health facilities providing specialized complex care in the country, and consisting of facilities whose quality has been rigorously pre-certified. The incorporation of more complex services into the benefits package required the definition and implementation of strategies for strengthening and improving existing health networks at the provincial and national level. The need to strengthen and improve existing health networks required the formulation of appropriate coordination structures both provincial and interprovincial. These structures were institutionalized by creating administrative circuits that combine the necessity of providing timely responses with the ability to document the whole process. The inclusion of the more complex services led to a significant increase in financial protection offered by the program, prompting some provinces to show greater interest in participating more effectively in the implementation of the strategy. The exercise of prioritization of the services under Plan Nacer took place in a context of limited resources and, for this reason, aimed to address fewer lines of care but in a comprehensive manner in order to increase the chances of organizational changes and improvements in performance. The program has made available to the Ministry and to the Provinces information on costs and coverage gaps for services that form part of a line of care under Plan Nacer. Training Plan Nacer, through efforts on the ground, has identified provincial counterparts for indigenous health issues and for work from an intercultural perspective, advancing significantly from 2009 in terms of participation by indigenous people. Indigenous Peoples Plans have been agreed on and validated locally. In cases where participation was achieved not only in planning but also in the implementation of these activities, empowerment was enhanced among this population. 63

80 The Strategic Plan for Technical Assistance and Training has been recognized by the provinces as a management tool suitable for generating proposals for training and technical assistance tailored to their particularities. It is also valued because it adapts to the dynamics and heterogeneity of each province, allowing the contracting of additional personnel. There has been a deepening in the training of personnel, shifting to a greater focus on processes, with the generation and dissemination of tools such as the Distance Learning Program (e-learning). This was aimed at promoting the growth of skills that enable better implementation of the program and the achievement of better health outcomes. From 2011 onwards, Plan Nacer incorporated a new modality of training, e-learning, through the development of the National Distance Learning Program. The objective of this Program is to allow more people to be incorporated in the health system, and to gain knowledge regarding the management of Plan Nacer and of public health. The National Distance Learning Program is linked with other programs of the Ministry of Health: the National Programmes for Control of Vaccine-Preventable Diseases, Comprehensive Adolescent Health, Oral Health, Tobacco Control, Hearing, Maternal and Child Health, and Sexual and Reproductive Health. A special platform, with assistance in its utilization, was put in place to facilitate with these linkages. This program represents one of the options to address the need for continuous education on a large scale, promoting equity and inclusion through access to education. The program is free, and exponentially expands the access of people to training and retraining regarding the health system, allowing users to choose their own time schedules vis-à-vis their learning, and to access the platform from any location with an Internet connection. The program supports learning and allows new concepts to be added, while strengthening existing knowledge and skills. 64

81 Not Applicable. Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders 65

82 Annex 9. List of Supporting Documents National Ministry of Health. Various Documents. Buenos Aires, Argentina. National Ministry of Health. October Informe Final: Estudio de Financiamiento y Uso de Recursos de Efectores del Plan nacer en las Provincias de la Fase 1. Buenos Aires, Argentina. National Ministry of Health Evaluación de la Implementación de la Segunda Fase del Proyecto de Inversión en Salud Materno-Infantil Provincial en Argentina: Desafíos, Retos y Aprendizajes. Documento de Base para la Evaluación de Medio Termino Préstamo BIRF AR APL II. Buenos Aires, Argentina. National Ministry of Health. December Informe de Desempeño de los indicadores de los Objetivos de Desarrollo del Proyecto. Buenos Aires, Argentina. National Ministry of Health Informe Ejecutivo: Monitoreo de la Satisfacción del Usuario y de la Calidad de Atención del Plan Nacer. Buenos Aires, Argentina. National Ministry of Health Informe Ejecutivo: Monitoreo de la Satisfacción de Beneficiarios de la Cobertura para la Corrección Quirúrgica de Cardiopatías Congénitas. Buenos Aires, Argentina. National Ministry of Health. June Evaluation Final de la Implementation del Proyecto de Inversión Materno-Infantil Provincial en Argentina: Documento de Base para la Evaluación Final del Préstamo BIRF Buenos Aires, Argentina. World Bank. Various Aide Memoires, ISRs, Independent Procurement Reviews and Post- Procurement Reviews, Loan Agreements and Amendments, and Other Documents. Washington DC. World Bank. September Project Appraisal Document for the Provincial Maternal-Child Health Sector Adjustment Loan. Washington DC. World Bank. March Project Appraisal Document for the Provincial Maternal-Child Health Investment Project. Washington DC. World Bank. May Country Assistance Strategy Washington DC. World Bank. October Project Appraisal Document for the Provincial Maternal-Child Health Investment Project in Support of the Second Phase of the Provincial Maternal-Child Health Adaptable Program Loan. Washington DC. World Bank. October Project Appraisal Document for the Essential Public Health Functions and Programs Project. Washington DC. 66

83 World Bank. June Country Partnership Strategy Washington DC. World Bank. June Quality Assessment of Lending Portfolio (QALP-2) for Provincial Maternal-Child Health Project. Washington DC. World Bank. November Project Appraisal Document for the Essential Public Health Functions and Programs II Project. Washington DC. World Bank. January Implementation Completion and Results Report for the Provincial Maternal-Child Health Investment Project in Support of the First Phase of the Provincial Maternal-Child Health Adaptable Program Loan. Washington DC. World Bank. March Project Appraisal Document for the Provincial Public Health Insurance Development Project. Washington DC. World Bank. March Impact of Plan Nacer on the Use of Services and Health Outcomes: Intermediate Results Using Administrative Data from Misiones and Tucuman Provinces. Washington DC. 67

84 Annex 10. Background Information: The Health System in Argentina 1. Argentina is a federal country with 23 provinces and the city of Buenos Aires. While the National Ministry of Health is mainly responsible for general health policies and regulation, the 23 provinces and the city of Buenos Aires are responsible for the provision of health care. The Argentine health sector can be divided into three sub-sectors: public, social security and private. The combination of Federalism and sectoral fragmentation make it a complex system. 2. Formal insurance typically covers only public employees, not the large population that does not belong to an Obra Social 5 or have private insurance or the financial ability to pay the full cost of care. The publicly-funded services in each province operated in the traditional Latin American manner, with investment and recurrent costs paid out of fixed budgets. Public providers often did not bill for services and had little or no autonomy in the use of the resources transferred to them from the provincial Ministries of Health. These budgets were often insufficient to provide all the care needed by the uninsured population, and the rigid budgeting process meant that the necessary inputs were not necessarily available in the right proportions and substitution among them was difficult or impossible. These features continue to characterize most of the Argentine health system and required some flexibility in the use of resources from the introduction of a new insurance scheme. Since the provinces differ greatly in per capita income, disease burden, insurance coverage and public funding capacity, care for the uninsured was not only inadequate on average but varied inequitably from province to province. In 2001, for example, the highest provincial infant mortality rate was more than twice as high as the lowest, and the variation in maternal mortality was much greater, more than 10 to The new program Plan Nacer intended to increase access to health care, reduce inequalities and speed the recovery in health following the recession especially for the most vulnerable part of the uninsured population, pregnant women, newborns and young children. 4. For women, the coverage is from the start of pregnancy (or at the time the pregnancy is confirmed) until 45 days after birth or the loss of the fetus if that occurs a maximum, that is, of 10-1/2 months. Each new pregnancy starts a new interval of coverage; even health problems that are caused by having been pregnant, such as complications from cesarean surgery, are not covered if they occur after 45 days. Women needing the corresponding services for such problems are referred to the province's reproductive health program. For children, in contrast, coverage lasts up to age six, with a concentration on interventions during the first year of life. 5 Obras Sociales are national or provincial social health insurance organizations which administer funds contributed by employers and employees and provide health care to their members and their direct families. The obras sociales are organized by occupation or trade Penchaszadeh V. et al, Italian Journal of Public Health, Vol. 7. Nr 4, 2010) 68

85 Basic Design of Plan Nacer 5. Plan Nacer is an innovate approach and includes several features: an investment in improved capacity to deliver services; a formal insurance scheme for people who have no access to employment-based coverage and cannot afford private insurance; a pair of legally binding agreements, one between levels of government and one between a provincial government and providers; a scheme of voluntary affiliation by providers; and a system of record- keeping, auditing and evaluation to ensure accountability and that funds are used as intended and results documented. 6. The program includes two results-based incentive mechanisms for the inclusion of the target population and the improvement in the quality of services. The first incentive is in terms of eligible population and rewards provinces when eligible beneficiaries get enrolled since they receive a capitation payment for each pregnant woman and child who joins the program. The second incentive is linked to the providers who deliver services, since they bill and are paid for these services (fee for service) that are covered by Plan Nacer. The services that are included, the benefit package, is known as the Nomenclador, and is the same in every province. The Nomenclador contains over eighty services. Table 1 summarizes the services in the Nomenclador in the first few years of the Project. (In 2010, as mentioned in the main text of this report, two new sub-packages of services were included: Congenital Health Disease treatment services, and additional complex maternal health services to better address high-risk births). Table 1: Services in Nomenclador of 2008 Group or Function Services Pregnant (low risk) Consultations for healthy pregnancy Pap test for cervical dysplasia or cancer Tetanus immunization Women Pregnant (high risk) Consultations for control of pregnancy risks Women Ambulatory treatment for HIV infection Childbirth Childbirth Care in childbirth and for the newborn Following childbirth or Rubella immunization fetal loss Consultation on postnatal health Newborn Immunization (including BCG and hepatitis B) Care in incubator for up to 48 hours Children Immediate treatment for vertical transmission of HIV Up to age six Eye examination Consultations on mouth and dental health Laboratory tests and procedures Pregnancy test Colposcopy following positive Pap test Taking blood samples Blood tests Imaging in pregnancy Sonogram Chest x-ray Community-level services Active recruitment of pregnant women in first trimester Complete rounds in rural areas by health agent Socio-epidemiological diagnosis of population at risk Group meetings to promote healthy nutrition Group meetings to promote child development Transportation Emergency transport of newborn as needed 69

86 7. Ten indicators (tracers, trazadoras) are defined to measure the program output and health outcomes, and are used by the National Ministry of Health to determine the financing to the provinces (see Table 2). Targets for each tracer are negotiated annually with each province. This is an essential feature for the program because it allows a province with a weaker health system to aim for a lower target than better-off provinces. In terms of achieving each tracer the program used initially an all-or-nothing approach. However, this approach was then changed to a threshold approach with minimum, medium and maximum level. Depending on which target was achieved, a certain percentage was paid. The full 4 percent are only being paid for reaching a higher target while 1 to 3 percent is being paid for lower targets. Therefore, the provinces have a strong incentive to achieve each target for a given tracer (but no marginal incentive to exceed it). This scheme aims to motivate providers to increase their efforts to deliver better care even after a target was met. Table 2: The Ten Tracers (Trazadoras) in Plan Nacer No. Health objective Corresponding tracer: number of as a share of all the eligible women, newborns, or children 1 Early enrollment of pregnant women Pregnant women with first prenatal consultation before the 20th week 2 Effectiveness of childbirth care and care for the newborn Newborns with Apgar score of 6 or better, five minutes after Delivery 3 Effectiveness of prenatal care Women with newborn weighing at least 2,500 grams (2.5 kg) and prevention of prematurity 4 Effectiveness of prenatal care and care in childbirth Women with VDRL (tests for STDs) and tetanus immunization during pregnancy 5 Attention in cases of maternal or infant death Cases of maternal death or death of infant under one year of age, fully evaluated as to cause 6 Immunization coverage Children under the age of 18 months with measles or measlesmumps-rubella (MMR) vaccination 7 Sexual and reproductive health Women receiving sexual and reproductive health consultation within 45 days after giving birth 8 Care of healthy child to age one Children under age one with complete record of consultations and height, weight, and head diameter 9 Care of healthy child from age Children between one and six in age with complete record of one to age six 10 Inclusion (coverage) of the indigenous population consultations and height and weight Providers who deliver services to indigenous populations, with personnel trained in that group's specific culture and health needs 70

87 Figure 1: Comparison of Input Based and Results-Based Financing Model Source: Ministry of Health of Argentina, 2009 Finance Mechanism 8. The Ministry of Health transfers the funds on a per-capita basis to the provincial ministries in two steps: 60 percent of the financing is provided only upon verification of enrollment and 40 percent after accomplishment of the tracers. Since provinces differ in their capacity and resources to deliver the services listed in the plan, the target levels for the ten tracers are negotiated between the national and provincial government and differ between the provinces of Phase I and Phase II. 9. The funds flow from the provinces to the contracted healthcare providers (public or private), who can use up to half of the funds to pay incentives to staff to improve productivity and quality of services. The program s incentive scheme is based on two major relationships: the relationship between the national and provincial governments, and the relationship between the provincial government and healthcare service providers. The national government periodically provides capitation transfers to the provincial governments. The capitation-transfer is based on the established per capita payment multiplied by the number of beneficiaries registered in the databases that the province maintains. Sixty percent of the total amount is transferred monthly based on enrollment statistics, and forty percent is added to the capitation-transfer once output of outcome results (accomplishment of the ten health indicator targets or tracers) is presented. The second relevant relationship is between the province and each of the health facilities. Provinces 71

88 pay health care providers for services rendered to the target population. Prices of services in the Nomenclador can be reviewed twice a year. The combination of a fixed capitation payment for enrollment that goes to the provinces, and the Fee For Service (FFS) payments to providers, which vary according to the type and the amount of care delivered, presents a financing scheme that aims to avoid the negative effects that the capitation or FFS payment alone might cause. Table 1 demonstrates financing flow as described above. Figure 2: Relationship between National Government, Provinces and Providers Supervision and Audits 10. In addition to the financial incentives, management and auditing mechanisms have been instituted to monitor the program s progress. Legally binding management agreements signed between the National Ministry of Health and the provincial government, and between the provincial government and healthcare providers outline their respective roles and responsibilities and hold parties accountable. Annual Performance Agreements between the National Ministry of Health and provincial governments are especially important to ensure compliance to specific targets on enrollment and tracers. Internal audits verify the work performed, while independent auditors complete the process through regular (every four months) detailed reports to the National Ministry of Health, monitoring the progress of the program. The feedback from the audits and management reports is used to correct any mismanagement, break bottlenecks, and improve the program s functioning. The result is an innovative management model for a resultsbased financing program with an effective control and monitoring system that aims to deliver substantially better results in the health sector. 72