The World Bank IMPLEMENTATION COMPLETION AND RESULTS REPORT

Transcription

1 Public Disclosure Authorized Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR Public Disclosure Authorized IMPLEMENTATION COMPLETION AND RESULTS REPORT ON A LOAN IN THE AMOUNT OF US$ MILLION TO THE Public Disclosure Authorized Public Disclosure Authorized THE STATE OF RIO DE JANEIRO WITH THE GUARANTEE OF THE FEDERATIVE REPUBLIC OF BRAZIL FOR THE BRAZIL - RIO DE JANEIRO RENOVATING AND STRENGTHENING PUBLIC MANAGEMENT ( P ) Governance Global Practice Latin America And Caribbean Region November 30, 2017

2 CURRENCY EQUIVALENTS (Exchange Rate Effective November 30, 2017) Currency Unit = BRL 3.01= US$1 US$ = SDR 1 FISCAL YEAR January 1- December 31 Brazilian Real Regional Vice President: Jorge Familiar Calderon Country Director: Martin Raiser Senior Global Practice Director: Deborah L. Wetzel Practice Manager: Arturo Herrera Gutierrez Task Team Leader(s): Lorena Viñuela, Roland N. Clarke ICR Main Contributor: William R. Dillinger, Lorena Viñuela

3 ABBREVIATIONS AND ACRONYMS CPS DPL GORJ IBRD IDEB INEA ITERJ LRF PAHI PDE PISA Pró-Gestão RioPrevidência RPF RRI SAERJ SEPLAG SEEDUC SES SIAFEM-RJ SIGAP SIGMIS SISPAT TAL Country Partnership Strategy Development Policy Loan Government of the State of Rio de Janeiro International Bank for Reconstruction and Development Basic Education Development Index (Índice de Desenvolvimento da Educação Básica) State Institute for the Environment (Instituto Estadual do Ambiente) Institute of Land and Cartography of the State of Rio de Janeiro (Instituto de Terras e Cartografia do Estado do Rio de Janeiro) Fiscal Responsibility Law (Lei de Responsabilidade Fiscal) Program to Support Municipal Hospitals (Programa de Apoio aos Hospitais do Interior) School Improvement Plan (Plano de Desenvolvimento Escolar) Program for International Student Assessment Program for Renovating and Strengthening Public Management (Programa de Renovação e Fortalecimento da Gestão Pública) Pension Fund of the State of Rio de Janeiro (Fundo Único de Previdência Social do Estado do Rio de Janeiro) Report on Physical Progress (Relatório de Progresso Físico) Report on Indicators (Relatório de Revisão de Indicadores) Rio de Janeiro State System of Evaluation for Education (Sistema de Avaliação da Educação do Estado do Rio de Janeiro) State Secretariat of Planning and Management (Secretaria de Planejamento e Gestão) State Secretariat of Education (Secretaria de Estado de Educação) State Secretariat of Health (Secretaria de Estado de Saúde) Integrated System of Financial Management for States and Municipalities (Sistema Integrado de Administração Financeira para Estados e Municípios) Integrated Social Security Management System (Sistema Integrado de Gestão Administrativa e Previdenciária) Information System for Management of Health Targets and Indicators (Sistema de Gestão de Metas e Indicadores de Saúde) State Property Management System (Sistema de Gestão do Patrimônio Imóvel) Technical Assistance Loan

4 TABLE OF CONTENTS DATA SHEET... 1 SUMMARY... 6 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES... 7 A. CONTEXT AT APPRAISAL...7 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) II. OUTCOME A. RELEVANCE OF PDOs B. ACHIEVEMENT OF PDOs (EFFICACY) Substantial C. EFFICIENCY D. JUSTIFICATION OF OVERALL OUTCOME RATING E. OTHER OUTCOMES AND IMPACTS (IF ANY) III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION B. KEY FACTORS DURING IMPLEMENTATION IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME.. 30 A. QUALITY OF MONITORING AND EVALUATION (M&E) B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE C. BANK PERFORMANCE D. RISK TO DEVELOPMENT OUTCOME V. LESSONS AND RECOMMENDATIONS ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ANNEX 3. PROJECT COST BY COMPONENT ANNEX 4. EFFICIENCY ANALYSIS ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ANNEX 7. CONTEMPORANEOUS PROJECTS ANNEX 8. FISCAL CONTEXT... 74

5 ANNEX 8. STATE OF RIO DE JANEIRO MAP... 77

6 DATA SHEET BASIC INFORMATION Product Information Project ID P Country Brazil Original EA Category Not Required (C) Project Name BRAZIL - RIO DE JANEIRO RENOVATING AND STRENGTHENING PUBLIC MANAGEMENT ( P ) Financing Instrument Technical Assistance Loan Revised EA Category Not Required (C) Organizations Borrower State of Rio de Janeiro Implementing Agency Secreatriat of Finance (merged with the former Secretariat of Planning and Management, the original Project Development Objective (PDO) Original PDO The objective of the project is to support the State of Rio de Janeiro to introduce performance-based management and information technologies to drive service delivery improvements in key public services, including secondary education and hospital care. Page 1 of 77

7 FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing IBRD ,673,000 18,673,000 12,237,479 Total 18,673,000 18,673,000 12,237,479 Non-World Bank Financing Borrower 46,683 46,683 46,683 Total 46,683 46,683 46,683 Total Project Cost 18,719,683 18,719,683 12,284,162 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 26-Aug Dec May Jun Jun-2017 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) 17-Feb Dec-2014 Amount Disbursed (US$M) Key Revisions 0 Change in Results Framework Change in Components and Cost Other Change(s) 7.03 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Change in Financing Plan Reallocation between Disbursement Categories Change in Safeguard Policies Triggered Change of EA category Change in Implementation Schedule Other Change(s) KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial Page 2 of 77

8 RATINGS OF PROJECT PERFORMANCE IN ISRs No. Date ISR Archived DO Rating IP Rating Actual Disbursements (US$M) Nov-2010 Satisfactory Satisfactory Jun-2011 Moderately Satisfactory Moderately Satisfactory Jan-2012 Satisfactory Satisfactory Jul-2012 Satisfactory Satisfactory Jan-2013 Moderately Satisfactory Moderately Satisfactory Mar-2013 Moderately Satisfactory Moderately Satisfactory Oct-2013 Moderately Satisfactory Moderately Satisfactory Apr-2014 Satisfactory Satisfactory Oct-2014 Satisfactory Satisfactory Apr-2015 Satisfactory Satisfactory Sep-2015 Satisfactory Satisfactory Mar-2016 Satisfactory Satisfactory Sep-2016 Satisfactory Satisfactory Mar-2017 Moderately Satisfactory Satisfactory SECTORS AND THEMES Sectors Major Sector/Sector (%) Public Administration 100 Other Public Administration 36 Page 3 of 77

9 Information and Communications Technologies 100 Public Administration - Information and Communications Technologies 2 ICT Services 5 Other Information and Communications Technologies 2 Education 100 Public Administration - Education 27 Health 100 Public Administration - Health 18 Health 10 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 29 Enterprise Development 29 MSME Development 29 Social Development and Protection 14 Social Protection 14 Social Safety Nets 14 Human Development and Gender 58 Health Systems and Policies 30 Health System Strengthening 30 Education 28 Access to Education 14 Education Financing 14 ADM STAFF Role At Approval At ICR Regional Vice President: Page 4 of 77

10 Country Director: Senior Global Practice Director: Practice Manager: Task Team Leader(s): ICR Contributing Author: Roland N. Clarke Lorena Vinuela, Roland N. Clarke Page 5 of 77

11 SUMMARY The objective of the project was to support the State of Rio de Janeiro to introduce performance-based management and information technologies to drive service delivery improvements in key public services, including secondary education and hospital care. While Rio de Janeiro remains one of the wealthiest states in Brazil, it compares unfavorably to other states in Brazil on several social indicators. The project was the first direct engagement with the State on public management issues in over a decade. The GORJ had not implemented a multi-sectoral project prior to the re-engagement with the Bank. The State launched in 2008 a program to strengthen public management called the Program for Renovating and Strengthening Public Management (Programa de Renovação e Fortalecimento da Gestão Pública or Pró-Gestão.) The program was part of a broader strategy to address the fiscal imbalance and structural problems in service delivery. Shortly after the project had been approved by the Board of Directors and became effective, the Bank and the GORJ agreed to add a new component to support disaster risk management as part of the response to an unprecedented environmental catastrophe. The preparation of an Additional Financing, which would have added resources to cover the new component, was initiated in August After a prolonged preparation, it would be approved as a stand-alone project in 2014, making necessary to restructure the original project again. A second restructuring was completed to better align the structure and resourcing of components with the activities and sectoral objectives and eliminate those that could no longer be financed. The restructuring also triggered additional safeguard policies and extended the closing date to match the one of the Additional Financing. The large majority of the activities were completed, and 92 percent of the original amount in the local currency was disbursed the loan amount of USD 18.7 million was equivalent to BRL 32.9 million at approval, while the total disbursements reached BRL 30.1 million in June The activities were budgeted and contracted in Brazilian reais. If the disbursements had not been suspended in August 2016, because of the Judiciary s seizing of the advanced resources, the project would have disbursed close to 94 percent of the dollar loan amount and would have delivered several additional complementary outputs. Substantial savings were generated from good procurement and administration practices along with foreign exchange gains. The project substantially achieved its development objectives. The project contributed to performancebased management in several areas (through better monitoring of health and education results, as well as the introduction of methodological framework and performance agreements in a small number of secretariats). The project succeeded in creating or improving the State s information systems for property management, monitoring of education outcomes, and performance evaluation of grants to municipal hospitals. A new integrated management information system for pension administration was designed, developed, and is being implemented. The State s ability to prevent and respond to natural disasters was strengthened through the addition of two state-of-the-art weather radars to its alert system and more effective institutional coordination. The enhancements in monitoring and management, supported by institutional reforms and information systems, were associated with improvements in service delivery, particularly in secondary education, the quality and coverage of health services offered by municipal hospitals, and disaster risk management. 6

12 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context Rio de Janeiro accounts for 8 percent of the total population of Brazil with 16 million inhabitants and has the second largest gross domestic product (GDP), representing approximately 12 percent of the national GDP. The state economy is heavily dependent on the oil sector and services. The State has suffered a protracted economic decline for several decades and has been plagued by problems of violence, poor service delivery, and corruption. While Rio de Janeiro remains one of the wealthiest states in Brazil, it compares unfavorably to other states in Brazil on several social indicators. At the time of appraisal, nearly a quarter (22 percent) of Rio s population was categorized as poor, and almost 10 percent lived in inadequate housing (the largest share of any state). The education system had deteriorated precipitously in previous years, and outcomes had fallen to the lowest levels in the Southeast Region. Less than half of students enrolled in secondary school graduated (44.5 percent, compared with 60 percent for the rest of the Southeast region.) Health outcomes also trailed behind other states in the Southeast Region, with significantly higher rates of maternal mortality (77 per 100,000 compared to the regional average of 51) and mortality by heart attacks (56 per 100,000 compared with a regional average of 43.) In 2007, in response to a severe economic crisis, the Government of Rio de Janeiro (GORJ) initiated a series of reforms aimed at regaining control of the State s fiscal position, improving public sector management, and increasing efficiency in the provision of public services. The fiscal situation was difficult and administrative capacity was deficient in most areas of the Government. The revenue base had been eroded, and planning and financial management were very weak. The State launched in 2008 a program to strengthen public management called the Program for Renovating and Strengthening Public Management (Programa de Renovação e Fortalecimento da Gestão Pública or Pró-Gestão.) Its main objective was improving public management to increase the quality public spending and service delivery. The program was part of a broader strategy to address the fiscal imbalance and structural problems in service delivery, with a strong focus on education, health, and security. The State also sought to regain the capacity to invest in maintaining and expanding its infrastructure. The Technical Assistance Loan (TAL) was initially designed to support improvements in public sector management under the purview of the then newly created Secretariat for Planning and Management (SEPLAG.) Following the example of other states that had implemented comprehensive management overhauls, Rio de Janeiro created a dedicated structure and competitively recruited a new cadre of public management specialists who were deployed in line secretariats to support the implementation of administrative reforms. By the late 2000s, the Bank had accumulated considerable experience with public sector reforms at the sub-national level in Brazil. It has worked and continues to work with numerous states to help improve their financial management capacity and implement results-based public sector management reforms. The Bank s engagement in the definition of the project was the result of collected knowledge in the areas of public administration, health, education and financial management. The Bank s involvement with the GORJ was also strengthened by its close working relationship with the Municipality of Rio de Janeiro and other states in Brazil. 7

13 The project was the first direct engagement with the State on public management issues in over a decade since the Privatization Loan approved in 1998, although a Development Policy Loan (DPL) would be approved first. The GORJ had not implemented a multi-sectoral project prior to the re-engagement with the Bank. Between 2007 and 2013, Rio de Janeiro would go on to receive four development operations and three investment projects, in addition to the project being discussed. The State also benefitted from substantial financial support from the Federal Government and other development partners for the preparation of the 2014 World Cup and the 2016 Olympics. As part of its program for fiscal consolidation and management modernization, the State requested a loan from the Bank to support (as reflected in the original request for a federal guarantee or Carta Consulta, dated September 2008) in the following areas: (i) implementation of results agreements (setting performance targets for secretariats, autarchies and foundations); (ii) training of staff in budgeting and planning; (iii) modernization of document processing; (iv) development and implementation of a data bank of socio-economic data; (v) introduction of a system for managing properties owned by the State; and the (vi) development of an integrated financial and administrative system for the agency responsible for managing the State Pension Fund. Over the course of the preparation, some activities were excluded, while additional components were introduced to provide technical assistance to the education and health sectors. The new activities sought to complement the policy reforms supported by the first policy development operation. Shortly after the project had been approved by the Board of Directors and a month after it became effective, the Bank and the GORJ agreed to add a new component to support disaster risk management in response to an unprecedented environmental catastrophe that killed over a thousand individuals in January The TAL was entirely consistent with the objectives of the World Bank Group Country Partnership Strategy (Report No BR) which called for: (i) strengthening macroeconomic fundamentals and public sector management, and; (ii) reducing poverty, vulnerability and social exclusion by increasing efficiency, quality of spending and accountability of the health and education sectors, especially at the subnational level. Theory of Change (Results Chain) The interventions have as objective to contribute to better service delivery by improving the results-orientation of service delivery and introducing transparent and efficient management practices within the public sector. The project built on and complements the reform efforts undertaken by the GORJ. The Government has emphasized the modernization of financial management and planning practices through several programs. Early efforts focused on regaining control of fiscal accounts, increasing own source revenue, paying overdue salaries, creating specialized careers for financial management and planning personnel, and improving public servant selection. Results-based management is a process that involves defining and monitoring indicators to measure and improve the performance of organizations and programs on a regular basis (Poister 2008). The approach relies on the adoption of systems that provide managers and organizations with greater discretion, techniques to measure and improve performance, and market-like mechanisms to replace input-focused and hierarchical steering with resultsoriented management. By focusing on outcome measures, results-based models give managers more space to adapt programs to different or changing conditions. Measuring performance is expected to impact the behavior of employees and managers positively and motivate people and organizations to improve productivity by shifting the focus from processes and activities to results (Behn 8

14 2003). Critically, these systems can contribute to better aligning organizational activities and processes with the organization s goals and connect activities and outputs to outcomes. The information allows principals to compare achievements in programs and services against targets, support processes of internal change and adjust programs as needed. However, performance systems and interventions are embedded within an institutional and political context and should not be considered in isolation (Radin 2006). While the interventions may contribute to improving performance by affecting distinct subsets of the incentives in place, the institutional context influences the extent to which these interventions may, in fact, be able to drive performance. The combination of institutional context and interventions has come to be referred to in the performance literature as a performance regime (Moynihan 2008). As explained by Talbot (2010: 81), considering performance management systems together with their context is particularly useful because what evidence does exist (...) suggests that it is the totality of a performance regime which potentially shapes or steers performance for specific services rather than the narrow purchase-provider or principal-agent assumptions often made about performance drivers. Translating results-based management into sustained gains requires adjustments and perfecting over time. Table 1: Project s Results Value Chain Activities Outputs PDOs Long-term Outcomes - Improve controls in asset management. - Streamline and automatize processes for pension administration. - Introduce performance agreements and monitoring. - Increased revenue collected from properties. - More efficient administration processes. - Strengthen monitoring and evaluation. - Pilot impact evaluations. - Improve data management. - Expand the offer and customization of training for teachers. - Implementation of a new monitoring system for supervising the indicators and targets included in results agreements signed with municipal hospitals receiving state transfers. - Reengineering of supply management processes and development of an information system to improve and integrate controls. - Up-to-date registries and regularized titles of state properties. - Integrated Information System for the Pension Fund. - Performance agreements and regular monitoring of data. - Improved monitoring and evaluation of programs and policies. - Improved teacher training. - Better targeting of interventions to aid lagging schools. - Improved monitoring and evaluation systems. - Implementation of School Development Plans. - Sistema Integrado de Gestão de Indicadores e Metas de Saúde SIGMIS) - Increased number of hospitals being monitored under the PAHI. - Improved supply management practices and controls. - Improved learning outcomes. - Reduction of performance gaps between schools. - Improved compliance with the Program PAHI s targets and requirements. - Increased efficiency of spending in the health sector. - Increased efficiency in the supply management. - Improved efficiency and results-orientation in the use of public resources. - Strengthen the center of government. - Improved productivity and livelihoods of lower income segments. - Increased access and quality to municipal health services in rural communities. - Reduced chronic disease in the lower income population. - Increase the efficiency of spending in health. 9

15 - Redesign of the institutional and operational arrangement for the Disaster Risk Management. - Acquisition and installation of two Weather Radar Systems. - Training of staff and contract personnel in the operation and maintenance of radars. - Training of staff in the development of weather prediction models. - Restructuring of the Civil Defense. - Revision of protocols for emergency response and inter-agency coordination. - Establishment of information sharing and coordination mechanisms between the Civil Defense, the Department of Mineral Resources, and the Institute of the Environment. - Weather monitoring center working 24 hours 7 days a week. - Greater coverage and accuracy of weather alerts. - Improved effectiveness of the response to natural disaster event. - Prevention and reduction of the impact of extreme weather events. Project Development Objectives (PDOs) The Project Development Objectives were to introduce performance-based management and information technologies to drive service delivery improvements in key public services, including secondary education and hospital care. Key Expected Outcomes and Outcome Indicators Originally, there were seven PDO indicators. One focused on reducing the time required to appraise the state-owned property. Two related to the state employees pension system (Fundo Único de Previdência Social do Estado do Rio de Janeiro or Rio Previdência). The first focused on reducing the time to collect social insurance taxes; the second sought to reduce staff. Two concentrated on education, the first aimed at reducing the number of schools with critically low scores on standardized tests; the second focused on improving the secondary school completion rate. The remaining two focused on the health sector and specifically on intermediate health centers in poor neighborhoods (UPAs). The first monitored the adoption of health protocols by UPA staff; the second monitored the waiting time for low-risk patients. This list was supplemented by eleven intermediate outcome indicators, as shown in Table 2. Table 2: Original Outcome and Intermediate Indicators Type Indicator Name 1. Core Government Systems and Management Tools for Improved Performance and Efficiency Outcome Average time (days) to produce real property appraisals Outcome Average time (days) to collect Social Security employment tax debts Outcome Reduction in the number of permanent personnel of RioPrevidência Intermediate Number of Results Agreements between the Governor and line departments or other bodies Intermediate Percentage of requests from public bodies concerning availability of state property that are answered 10

16 Intermediate Intermediate Average number of working days for RioPrevidência to prepare and deliver information demanded for judicial proceedings Average elapsed time (working days) for RioPrevidência to transfer financial data to the SIAFEM-RJ (Sistema Integrado de Administração Financeira para Estados e Municípios) 2. Strengthening Education Management Capacity, Teacher Instructional Practice and Incentives for School Effectiveness Outcome Outcome Intermediate Intermediate Intermediate Intermediate Intermediate Number of schools with critical (2.9 or lower) IDEB scores Secondary school completion rate Secondary school completion rate Schools that use SAERJ results in school-level improvement plans Schools with annual targets for learning improvement established with SEEDUC Website downloads from the Casa do Educador site Number of teachers who have taken an online training course Number of teachers who have taken a face-to-face course 3. Health Management and Information Systems Outcome Outcome Intermediate Intermediate Number of UPAs with at least ten treatment protocols adopted by health care staff Waiting time (minutes) for UPA patients with green risk classification (TERV) UPAs with completed quality accreditation by Joint Commission Methodology UPAs reporting the full cost of medical services by Cost Center in their Annual Reports Components The TAL was originally comprised of three components, each including several subcomponents: Component 1: Core Government Systems and Management Tools for Improved Performance and Efficiency (US$7.65 million) Subcomponent 1.1: Results-based Management. This subcomponent was intended to support the introduction of a results-based management tool into the state planning and monitoring system. Rio would establish a portfolio of strategic projects, each tied to a set of performance indicators. Results agreements, setting out targets for each indicator, would then be formalized through quasi-contracts between state administrative bodies and the Governor. The performance targets would be incorporated into the state s INTRAGOV.RJ 1 monitoring system to allow real-time data access by the governor and key state managers. Performance against these targets would be used as an input to budget formulation and approval. The TAL to finance a consultancy to support the identification of suitable indicators and design the GORJ version of a results agreement. It would also support improvements in the INTRAGOV.RJ data portal by financing the acquisition of INTRAGOV.RJ hardware, the preparation of physical space for the data warehouse servers, personnel training, and testing and installation. 1 INTRAGOV.RJ is a website (currently accessible only to authorized individuals) which combines primary databases of the state government, with information on education, health, and public security, among others. It builds on the Public Sector Management by Indicators (Gespi) system. 11

17 Subcomponent 1.2: Real Estate Property Management. This component was to support an inventory of properties owned by the State government. At the time the project was appraised, the State owned more than 6,900 properties. The exact number was not known. Nor was the State fully aware of the uses and economic value of its real property, as records were outdated and incomplete. Queries directed to the State government as to the availability of real property for use by a public enterprise or other government bodies averaged 60 per month in However, fewer than ten percent of these requests received a response each month, and replies inevitably were based on imprecise data. The remaining requests were denied for lack of information. At the time of appraisal, the State had developed and tested a new information system for managing its real estate: SISPAT-RJ (Sistema de Patrimônio de Rio de Janeiro). Field data, however, remained to be collected. The subcomponent would finance the survey and valuation of the estimated 5,000 properties that were not properly registered. The survey would gather additional data (blueprints, photos, descriptions of current use and occupants, if any, etc.) and prepare the materials needed for regularization of property titles. Subcomponent 1.3: Management Information System for Social Security. RioPrevidência is the agency responsible for the certification and payment of social security (pension) benefits for all the State s public employees and their dependents. With approximately 500,000 participants and beneficiaries, the pension fund held approximately BRL49 billion in assets and issued payments to beneficiaries totaling roughly BRL500 million per month at the time of appraisal in At the time, the state intended to introduce a new Integrated System for Social Security Management (SIGAP) in order to improve internal control over contributions to the pension fund, investments, cash flow management, contracts and billing, and bank account management. This new system would also allow services to be decentralized, improving services to beneficiaries (most of whom were elderly and infirm). It would also permit a substantial reduction in staff (from 350 to 280). The TAL would finance consulting services to define the functional and technical requirements of the proposed database modules, which would then provide the basis for a second contract for the production and installation of the priority modules for the SIGAP itself. The necessary hardware for SIGAP would be financed from RioPrevidência s own resources. Component 2: Strengthening Education Management Capacity, Teacher Instructional Practice and Incentives for School Effectiveness (US$4.83 million) The State Government bears primary responsibility for secondary, vocational, and adult education, accounting for about 80% of enrollments at those levels in the state. At the time of appraisal, the State was managing nearly 1,500 schools, with 1.34 million students. The performance of Rio s education system was poor. Graduation rates (for secondary students) and performance on standardized tests lagged regional averages. At the time of appraisal, the State Secretariat of Education (SEEDUC) identified three critical areas where the TAL could complement the State DPL investments to improve graduation rates and learning outcomes. Subcomponent 2.1: Strengthening Evidence-based Policy. The first would strengthen SEEDUC s policy-setting ability, helping it identify programs and policies that achieve positive results and modify or discard policies that underperform. To this end, the TAL would finance: (a) a rigorous evaluation of recent program innovations such as the programs Projeto Autonomia and Conexão Educação and (b) a study of the determinants of Rio s poor secondary school graduation rates and learning results, focusing on: the level and sources of funding, the uses of funding across different expenditure and program categories, and teaching practices. Subcomponent 2.2: School-level Performance Targets/School Development Plans. Following the models of Minas Gerais, Sao Paulo, Pernambuco, and the Federal District, the state intended to introduce performance targets for 12

18 annual improvements in math and Portuguese performance (measured through annual state and federal learning assessments) and student flows (repetition and graduation rates). These would be measured at the level of individual schools and introduced over two years. To support this effort, the project would finance the creation of an integrated education database and the design of specific performance indicators. At the same time, the State would encourage individual schools to prepare school development plans (Planos do Desenvolvimento Escolar or PDEs) setting out the schools goals for improvement and the specific actions and projects the school would undertake to achieve those goals. Subcomponent 2.3: In-Service Teacher Training. The third subcomponent in the education sector was intended to support Rio s teacher training institute, the Casa do Educador. At appraisal, the institute, which had both a physical and virtual presence, was still in its infancy. With the support of the project, SEEDUC was to contract specialized technical experts to further develop the program concept, define specific training courses, both in subject areas as well as pedagogy, classroom management, and the use of information technology in the classroom. Technical assistance would also be provided to construct a Casa do Educador Web portal as a forum to share information and to provide general support to the SEEDUC core team in developing evidence-based policies to address system challenges, carefully monitoring their implementation, and rigorously evaluating their costs and effectiveness. Component 3: Health Management and Information Systems (US$5.37 million) Subcomponent 3.1: Accreditation for Urgent Care Facilities (UPAs). Urgent Care Facilities (Unidades de Pronto Atendimento or UPAs) are 24-hour ambulatory care facilities equipped to perform low- and medium-complexity surgeries as well as to treat some severe cases. Each UPA has general medicine, pediatric, orthopedic and dental clinics, in addition to an X-Ray laboratory. They have observation beds for adults and children and a Red Room to stabilize patients with grave medical conditions before transfer to a hospital. In an effort to improve health outcomes and increase access to health services by the poor, GORJ has established 21 UPAs in low-income areas since To encourage quality improvements in UPAs, the project would help the State develop an accreditation system for UPAs, which are operated by non-for profit organizations, and carry out the first round of accreditations. Subcomponent 3.2: Cost-Accounting Methodology for UPAs. This subcomponent would develop a cost accounting system for UPAs, in order to help them control costs. The TAL would finance two phases of the system: (i) a detailed specification of the functional and technical requirements of the system; and (ii) the production and installation of the core modules in at least 10 UPAs. Subcomponent 3.3: Information System for Health Supply Chain Management. This subcomponent would focus on improving the procurement and distribution of medical supplies in all state health institutions. All stages in the supply chain process were to be addressed, from requisition, budget authorization, purchase, and distribution to the final service delivery unit. The TAL would help the state health secretariat design and develop an improved supply-chain system, including information systems and warehouse management. Subcomponent 3.4: Monitoring of the Program to Support Municipal Hospitals. Under Brazil s publicly-funded health care system (SUS), the State Government is primarily responsible for providing tertiary (hospital-level) health care. (Basic primary health care is the responsibility of municipal governments.) Nevertheless, there are hospitals owned by municipalities and philanthropic organizations. In order to improve the quality of health care they provide, the State gives grants to them. Only hospitals located in municipalities with fewer than 110,000 residents are eligible. Rural municipalities without hospitals are also eligible to receive grants, provided they refer patients to hospitals 13

19 located in other municipalities that belong to the Program of Support to Hospitals in the Interior (PAHI). 2 The PAHI grant has two parts: (i) a fixed component that is scaled according to the size and characteristics of the hospital; and (ii) a variable component based on performance benchmarks. In the years prior to appraisal, the state government had substantially increased its funding for PAHI. However, it did not have clear evidence of the impact that was being achieved by those additional resources. This component would help the State Health Secretariat develop a monitoring system to track the use and impact of these PAHI grants and to develop proposals to improve the effectiveness of its performance incentives. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets Although the project was restructured twice, its development objectives were not changed. Revised PDO Indicators The Result Framework was revised in the second restructuring to reflect the changes in the original subcomponents, some of which were transferred to the second Technical Assistance Loan to the State of Rio de Janeiro (see Table 3.). Furthermore, several indicators were changed to improve the capacity of the Government and the World Bank to assess the impact of the project activities, refining the methodologies used to calculate them and clarifying responsibilities for data collection and reporting. The indicator average time to produce property appraisals was replaced by number of (state-owned) properties with 'situção ocupacional' regularized because the original indicator could not be calculated accurately: the time required to produce appraisals was too difficult to measure. The indicator average time (for RioPrevidência) to collect Social Security employment tax debts was replaced by average time to record debt owed by renters of state property as the former indicator was not under state control, whereas the replacement indicator was. The indicator RioPrevidência: reduction in permanent personnel was dropped altogether. The indicator number of schools with critical (low) IDEB scores was replaced by points improved in the index of educational development (IDERJ) in the 30 percent worst-performing schools, on the grounds that the latter (which, until recently, was conducted annually) was a better monitor of performance than the former (which is only conducted every two years). The indicator monitoring the secondary school completion rate was dropped altogether, because it was redundant. Both indicators monitoring UPAs were dropped. 3 A new indicator percentage of false alerts issued by flood warning system was added to monitor the impact of the disaster management component. All but one of the original eleven intermediate indicators were dropped. Five new intermediate indicators were added, bringing the total to six (see Annex 2). 2 This rule removes a potentially perverse incentive for small municipalities to create their own hospitals. 3 The indicator number of UPAs with at least ten treatment protocols adopted by health care staff was dropped as the target was achieved without project financing. 14

20 Table 3: Revised PDO Indicators Status Indicator Name 1. Core Government Systems and Management Tools for Improved Performance and Efficiency Added Revised Number of the State's properties regularized Average number of days to record debt owned by renters of State Government's properties 2. Strengthening Education Management Capacity, Teacher Instructional Practice and Incentives for School Effectiveness Revised Points improved in the Index of Educational Development of Rio de Janeiro (IDERJ) by the 30 percent worst performing schools 4. Development of Institutional and Technological Capacity for Preventing and Performing during High-intensity Weather Events [NEW] Added Percentage of false alerts issued by the Flooding Warning System Revised Components The original project was restructured for the first time in February 2011 in response to an emergency request made by the GORJ in the wake of unprecedented extreme weather events that exposed the weaknesses in the State s disaster risk management a month before. The redesign of institutional and operational arrangements for disaster risk management and the acquisition and installation of two weather radar systems were added under the first component of the project. Intermediate outcome indicators to measure the results of these activities were incorporated accordingly. More than a third of the project resources had to be reallocated to finance the new activities. The preparation of Additional Financing (AF) to the loan was initiated in August 2011 to supplement funding for the initial activities that could no longer be financed and incorporate complementary activities. As part of the preparatory work, an Environmental and Social Assessment of the activities of the first project was completed by the Borrower with World Bank support. However, after a long preparation process, the AF had to be approved as a stand-alone Technical Assistance Loan (TAL II), named Rio State Strengthening Public Management and Integrated Territorial Development Technical Assistance Loan (P126735), on February 12, An administrative error had been made earlier in the processing of the Federal Guarantee but was only identified during the negotiation stage in late The unanticipated development created the need for a second restructuring to update the safeguards triggered and eliminate the indicators related to the activities to be financed by the AF/TAL II. The new restructuring, initiated on February 2014 and approved by the Board on December of that year, sought to align better the structure and resourcing of components with the activities and sectoral objectives remaining in the project. It added a separate component entitled Development of Institutional and Technological Capacity for Preventing and Performing during High-intensity Weather Events for the activities related to disaster risk management previously included under the first component and a Project Management component. The restructuring also eliminated subcomponents 2.2 ( School-level Performance Targets for State Schools ), 3.1 ( Accreditation System for UPAs ), and 3.2 ( Cost-Accounting Methodology for UPAs and Hospitals), which had been transferred to the AF/TAL II. 15

21 Other Changes The second restructuring led to the reduction of the counterpart funding from US$ 9.16 million to US$ million; and a transfer of US$ 0.55 million from the expenditure category Operating Costs to category Goods, Nonconsultant services, consultant services and training. The counterpart financing corresponded to resources allocated to the activities transferred to the TAL II. In addition, the closing date of the project was extended from June 30, 2015 to June 30, 2017, which was the closing date agreed for the AF. The extension was needed to compensate for the initial implementation delays encountered because of the technical and legal complexity of the various tendering processes, and the additional administrative burden created by the addition of the acquisition of the two weather radars. The second restructuring also required a change in the project s Environmental Assessment rating (from category "C" to "B") and triggered safeguard policies for Environmental Assessment (OP/BP 4.01), Natural Habitats (OP/BP 4.01) and Involuntary Resettlement (OP/BP 4.12) in a precautionary mode. Table 4 below presents the reallocation of funds under the two restructurings. Table 4: Reallocation of TAL Financial Resources to Project Components Original 1st Restructuring 2nd Restructuring Components US$ millions 1. Core Government Systems and Management Tools for Improved Performance and Efficiency Results-based Management Real Estate Property Management Management Information System for Social Security Strengthening Education Management Capacity, Teacher Instructional Practice and Incentives for School Effectiveness Building SEEDUC s Technical Capacity for Conducting Studies and Evidence-based Policy 2.2. School-level Performance Targets for State Schools [ELIMINATED] 2.3. Developing the Escola SEEDUC as an Effective In-Service 1.5 Teacher Training Network Health Management and Information Systems Accreditation for Urgent Care Facilities (UPAs) [ELIMINATED] Cost-Accounting Methodology for UPAs and Hospitals [ELIMINATED] 3.3. Information System for Health Supply Chain Management Monitoring of the Program to Support Municipal Hospitals Development of Institutional and Technological Capacity for 7.10 Preventing and Performing during High-intensity Weather Events [ADDED] 4.1. Design of the Institutional and Operational Arrangement for 0.5 the Disaster Risk Management 4.2. Acquisition and Installation of Two Weather Radar Systems Project Management [ADDED] 0.20 Price Contingencies Front-end Fee

22 Rationale for Changes and Their Implication on the Original Theory of Change The results chain was updated to include the activities, outputs, and PDOs associated with the new disaster risk management component (see last row of Table 1,) and eliminate the activities that were transferred to the TAL II. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating Substantial The objectives of the project remain relevant to the State and the current Country Partnership Strategy (CPS). Education and health (along with public security) continue to be the primary functional responsibilities of the State Government. Efforts to improve the efficiency of those services thus continue to be important. Improvements in pension administration are also highly relevant, as it continues to be an important administrative function of the State that has significant fiscal implications. Disaster management is pertinent, as well, considering the vulnerabilities exposed by the 2011 catastrophe. The project was fully consistent with and closely linked to the objectives of the CPS Improving governance and service delivery and increasing public sector efficiency and the quality of spending were the central themes of the CPS. This project was one of several sub-national DPLs and SWAps designed to strengthen public sector management and service delivery, as an integral part of the Bank s Country Partnership Strategy in Brazil. The reforms supported by the project were aligned with the CPS objectives of (i) increasing the efficiency of public and private investments; and (ii) improving the quality of public services for low income households. Its emphasis on multisector approaches reflected the multi-dimensional nature of addressing key development problems. The new CPS for also gives priority to supporting Brazil in strengthening of fiscal management at all levels of Government. These policies are to be accompanied by reform of the budgetary process and efforts to increase the efficiency of public spending in health, education, social assistance, labor market, and industrial-support programs, as well as in public procurement. The objectives of the loan were and continue to be salient for improving the public administration in the State Government. The objectives were and are consistent with the Government s reform program, but also with the identified needs of the State in important areas of the public sector. The relevance is enhanced by the continuity of the initiatives. Consistency in pursuing different phases of the reform over an extended period adds to the significance of the project. Furthermore, many of the objectives entailed fundamental changes in the operation of the sectors, in the way that resources are allocated, incentives are structured, performance is evaluated, and services are rewarded. One of the most important aspects in terms of both objectives and design was the expansion of the use of indicators and improved management methods. 17

23 The design of the program was also very relevant for achieving the expected results, focusing mostly on the policies and measures that were critical to the success of the Government s reform program. It ensured continuity and furthered the transformation of key areas of public administration. It is possible to argue that the project should have focused more on the State s fiscal situation, in light of subsequent developments. But these issues were addressed (with various degrees of success) in the series of DPLs. They were also a target of the TAL II. The State s fiscal crisis is largely a result of exogenous factors. One was the global decline in oil prices. Another was the constraints on wage bill reductions imposed by Brazil s Federal Constitution. (Inter alia, confirmed public employees cannot be dismissed except for cause, cannot have their salaries reduced, and--with some exemptions--are entitled to indexed pensions equal to 100 percent of their exit salaries.) The resulting combination of falling oil prices and rising personnel costs are the principal causes of the State s fiscal crisis. Because the solutions to these problems lay outside State s control, they could not have been addressed through state-level technical assistance. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome Substantial Justification of Overall Efficacy Rating The project substantially achieved its development objectives, which are divided into two parts: Introduction of performance-based management and information technologies to drive improvements at the center of government and in key public services The project contributed to performance-based management in several areas (through better monitoring of health and education results, as well as the introduction of methodological framework and performance agreements in a small number of secretariats) and the streamlining of central processes. The project succeeded in creating the State s information systems for property management, monitoring of education outcomes, and performance evaluation of grants to municipal hospitals (PAHI). A new integrated management information system for pension administration (SIGAP) was designed, developed, and is being rolled out. In addition, one module of this system will be adopted by all state agencies in Component 1. Core Government Systems and Management Tools for Improved Performance and Efficiency. The number of properties with records registered and updated in the state property management system increased from 728 to 4,578. As a result, annual revenues collected from renting state properties increased from BRL 1.6 million to BRL 11.5 million per annum. In addition, the efforts to regularize the titling of state-owned properties were increased substantially. Only 363 such actions occurred over the period In 2016, the number properties reached 3,024 (including the Maracanã Stadium and other landmark properties and far exceeding the target of 2,468). The larger aim of the subcomponent to establish a management information system for state-owned properties was also successful. SISPAT is fully functional. The enhanced efficiency of the management of real estate assets has allowed the Government to identify and sell idle buildings and land to capitalize the Pension Fund (for a total of US$ 200 million). 18

24 Similarly, the re-engineering of administrative processes in the State s Pension Fund administration is already yielding positive results and time savings that are expected to be expanded when the new integrated management information system is rolled out in The architecture and design of the SIGAP were completed in Five modules financial, judicial, actuarial, asset, and investment management were developed in 2017 and are being rolled out. RioPrevidência is financing the development of the other modules using its own resources. The remaining modules are expected to be completed in Component 2. Strengthening Education Management Capacity, Teacher Instructional Practice and Incentives for School Effectiveness. Many of the proposed tasks of the first subcomponent were implemented with the State s own resources and through the direct technical assistance of the Bank education specialists, who trained the monitoring and evaluation team of the Secretariat and provided ongoing advice on the design and implementation of evaluations. A comprehensive database of school-level education statistics was designed and implemented, yielding results for These were then incorporated in individual school improvement plans. Regrettably, the annual evaluation required to update the database has since been suspended due to the opposition from students. However, several other key indicators are still being tracked, and the school development plans and monitoring process remain in place. The Federal Government continues to monitor school performance through its own biannual Portuguese and Math proficiency test, Prova Brasil, and the Index of Educational Development (IDEB). Although the Prova Brasil is currently conducted only every other year, the Federal Government (with the Bank s assistance) intends to make it an annual exercise, which will fill the gap left by the interruption of the SAERJ. A consultancy proposed teaching guidelines, organizational strategies, and an action plan for the teacher training institute. The teacher training institute was reformed in line with its recommendations. The new institute, officially opened in 2012, offers training courses for teachers in classroom settings as well as online. In addition, Bank staff provided significant and impactful assistance to the Secretariat of Education in a more direct form, helping design the monitoring and evaluation system and providing support in producing reports, drafting legislation, and calculating performance targets. Component 3. Health Management and Information Systems. The third component, focusing on increasing the efficiency of the distribution of medical supplies in all state health institutions resulted in the delivery of a diagnostic and recommendations on how to improve the supply management arrangements despite the challenges encountered. Notably, during the preparation of the diagnostic, the consultancy provided support to the Secretariat in responding to a particularly glaring example of mismanagement. Over 1,000 tons of expired drugs and medical supplies that had to be discarded in early 2016 because they were not properly managed even when the State was encountering increasing difficulties to finance health services. As part of the response to this crisis, the State canceled the contract with the company responsible for managing the main warehouse and reassumed the responsibilities for supply management. The project supported the development and implantation of an information management system, the Sistema de Gestão de Metas e Indicadores de Saude or SIGMIS, to monitor the results agreements signed with small hospitals in rural areas. Inter alia, the project supported the design and the implementation of a system for gathering the relevant data, integrating that data into a common data base, defining performance indicators and agreeing upon targets with the relevant municipalities and hospitals. The new system was successfully implemented. All seventyseven hospitals participating in the program are using the system (since 2015), exceeding the target of 70. In addition 19

25 to allowing the performance of rural hospitals to be monitored, the system can be used for tracking the performance of state-run hospitals and UPAs. The Secretariat is planning to use it as part of the efforts to improve monitoring within the state network. Component 4. Development of Institutional and Technological Capacity for Preventing and Performing during Highintensity Weather Events. The review of the organizations involved in disaster risk management in Rio and recommendations for ways to improve collaboration among them led to the establishment of an inter-agency mechanism for the three key state agencies INEA, Civil Defense, and Mineral Resources to better coordinate their activities. The three institutions developed a matrix setting out their respective responsibilities in the implementation of the recommendations and continue to hold regular meetings to follow up on their advancement. Remarkably, the diagnostic and recommendations also informed the restructuring of the Civil Defense approved in 2012, the revision of emergency response protocols, and the coordination to develop a system for sharing geological, hydrological, and weather event information in real time. Two weather radars were successfully installed, allowing the State to expand the coverage of the weather alert system from 16 to 92 municipalities. The information enables the more efficient mobilization of resources and improves the results of prevention actions before and during extreme weather events. Improvements in service delivery, including secondary education, hospital care, and disaster risk management The enhancement in monitoring and management, supported by institutional reforms and information systems, are associated with improvements in service delivery, particularly in secondary education and the quality and coverage of health services offered by municipal hospitals, and disaster risk management. In the education sector, there is evidence that the technical assistance financed through the project constituted a critical link in the reform chain that led to improved school performance and teacher training. Performance-based management and targeted interventions that were informed by enhanced and frequent monitoring of indicators and supported by customized training for teachers translated into substantial improvements in education outcomes. The State achieved one of Brazil s most remarkable turnarounds in the education sector, mostly because of the targeted and decisive interventions in the worst-performing schools. After showing no change in three rounds of IDEB, the State recorded a sizable increase of 14 percent in 2011 and 12.5 percent in 2013 and As a result, Rio de Janeiro rose from the 26th position in the national ranking of IDEB 2009 to 4th place in The State adopted a new plan at the beginning of 2011 with the aim of improving learning performance and establishing a meritocratic selection process for critical functions such as the regional directors that have traditionally been captured by the State s Assembly. The program provides bonuses to school units and Regional Boards that meet or exceed annual improvement goals. In order to assess goals, Rio de Janeiro, like Minas Gerais and Pernambuco had done before, adopted an annual version of the IDEB evaluation and three other evaluations. As well, a career plan for regional directors and principals was established, and a competitive recruitment process was made mandatory to fill those positions. By the end of 2012, all 28 regional directors and 172 school principals had been hired through a competitive selection process, despite the strong opposition of legislators. Permeating the reform agenda was a strong focus on data and school management. To allow school directors to focus on education quality, young, information technology-savvy management specialists were assigned to each school, which improved rapidly the timeliness of school data reporting and budget management. The Secretary 20

26 established a routine meeting that brought the entire state management team together every six months to review progress and collectively address problems. The new model put a priority on assessing the performance of, and a developing a career plan for, every superintendent and principal. Isolating the overall impact of the specific measures supported by the TAL is difficult, but Federal survey data (IDEB) indicate broad improvements in educational performance that are contemporaneous with the project implementation. The average IDEB score of schools operated by the state government increased significantly between 2009 and 2015 (from 2.8 to 3.7). The 30 percent worst-performing schools improved 0.90 points in the Index of Educational Development of Rio de Janeiro (IDERJ) in The total number of schools with a low IDEB decreased from 429 in 2009 to 64 in The 30 percent worst-performing middle schools observed an increase in their IDEB score twice as large as rest of the schools (of 1.05 points versus 0.42). The number of secondary schools with a high dropout rate (higher than 22.6 percent) declined from 228 in 2009 to 21 in The approval rate improved from 64 percent in 2009 to 82 percent in 2015 in the worst performing secondary schools. The gap between the worst-performing schools and the rest was reduced from 17 percentage points to 3 in the same period. Improvements in the hygiene standards and disease management in municipal hospitals, as well as an increase in the number of patients that had access to hospitals in nearby municipalities and the number of intensive care beds available, were achieved through the PAHI program. With respect to disaster risk, the flooding alert system service coverage increased from 0 million in 2011 to 16 million in Since 2014, there have been more than 1,000 landslide and flooding events, but no related human losses were registered in the State. This is in large part due to the improved risk management and the timely preventive evacuation of affected areas, made possible by the information provided by the weather alert system and the more effective response protocols and inter-agency coordination. C. EFFICIENCY Assessment of Efficiency and Rating Substantial Economic Efficiency The benefits of investments in institutional developments that help public organizations improve their human, financial, and asset management capacity are difficult to evaluate using standard cost-benefit analysis. Such investments generate indirect effects on the efficiency of service delivery rather than necessarily translating into direct changes in the price, cost, quantity or quality of the services delivered. Cost-benefit analysis requires that the services analyzed have measurable, defined benefits. While it is possible to estimate part of the costs of management reforms, their impacts often affect the overall entity or government and valuing the benefits poses a major challenge as it is not always possible to define the scope of the project within the organization. Management reforms often involve a wide range of functions and indirect, widespread and diverse effects on public sector and the economy. For these reasons, at the time of appraisal, it was not possible with the information available to conduct a standard economic analysis that provided an estimate of the Present Value of the stream of economic benefits from the 21

27 project net of project costs. It was not possible to separate the specific benefits from improving institutional capacity and performance and value them in dollars or Brazilian reais, nor was possible to ascribe efficiency and effectively gains directly and exclusively to the specific activities included in the technical assistance project. However, some counterfactuals could be considered. The Brazilian procurement legislation makes it very difficult to contract (high quality) consulting services. Using Bank rules, the Government of Rio was able to mobilize external technical expertise, which otherwise it would have not been able to access. Without the Bank s financing and procurement framework, it would have taken much longer and undoubtedly would have been more expensive to complete certain activities. These include the inventorying of close to 5,000 real estate properties (at the time of the appraisal, the State was only able to complete a small number of inspections and title regularizations a year), and the development of the management system of the Pension Fund administration. The Bank rules permitted internationally contracting the acquisition of the radars under a turn-key agreement that not only resulted in cost savings and provided the State with the most advanced technology available at the time but also considerably reduced the installation time. Without the Bank s engagement and convening role, it would have been very difficult to complete a diagnostic on the institutional arrangements for disaster risk management or revise the supply chain processes in the health sector, given the prevailing bureaucratic politics and interests. Other activities that were crucial in the results chain, such as developing a unified database to monitor education outcomes to target better interventions and a comprehensive strategy to train teachers and improve the skills of the staff of the Secretariat of Education, would likely have been of a lesser quality and taken more time if implemented without the technical assistance of the Bank. Some may not have taken place, such as the development of the monitoring system for the PAHI grants. Administrative Efficiency and Quality of the Activity Design The competitive contracting yielded considerable savings, which allowed the project to add complementary activities using those resources. The large majority of the original activities were completed and 92 percent of the original amount in local currency was disbursed at the time of approval, the loan amount of USD 18.7 million was equivalent to BRL 32.9 while the total disbursement reached BRL 30.1 million. The original activities were budgeted and contracted in Brazilian reais. If the disbursements had not been suspended in August 2016, because of the Judiciary s seizing of the advanced resources, the project would have disbursed close to 94 percent of the loan amount in dollars and would have delivered several additional complementary outputs thanks to the savings generated by good procurement and administration practices along with foreign exchange gains. Nearly all the activities were completed with fewer resources than initially anticipated. In the case of the system for the management of the Pension Fund administration, the savings attained in the tendering process allowed the State to finance five modules instead of four with project resources. The strategy of contracting the process of streamlining and the design of the system architecture separately from the development of the various modules proved to be more economical, whereas at the same time guaranteed that the system met and was well-customized to the specific requirements of the Pension Fund administration. The Government team was proactive and worked closely with both firms to ensure that the lengthy and challenging process of designing, developing, and rolling out such complex system went smoothly. 22

28 Similarly, in the case of the inventory and valuation of real estate properties, the manner in which the Terms of Reference were structured and procured was very innovative. At the time of the contracting, no firm in Brazil offered the services required as they included a combination of engineering, legal, and property valuation expertise. The firm contracted was an international consortium that was created to address State s demands. The design of the services was feasible because the unit in charge of supervising the State s properties had previously mainstreamed its processes and carried many normative and organizational changes. The model for structuring the services designed by the State of Rio de Janeiro has served as a reference for several states and municipalities in their efforts to update the valuation and regularize the titling of their real estate assets, which is a requirement of the new public accounting rules established by the Federal Government following the International Public Sector Accounting Standards. The State regularly receives counterparts from other parts of Brazil who are interested in learning about the State s experience and approach. The market of services also grew as a result and the contracted firm, together with several new firms, expanded their capacity and are now supporting multiple subnational clients in similar efforts. The contracting of the inventory services was very competitive as well. Using similar Terms of Reference, the State of Rio Grande do Norte contracted the same firm for a price per property of BRL 3,308 (or USD 1,034) for an estimated number of 4,000 properties versus the price that the State of Rio de Janeiro paid of BRL 1,269 (or USD 635 at the exchange rate of 2012) for 4,635 properties. No comparative information is available for other consulting services but, in general terms, six or more bidders participated allowing the State to have a stronger negotiating position. The State of Rio de Janeiro paid a substantially lower price for the acquisition of two weather radars than other states, their installation, and a year of assisted operations, which amounted to USD 6.5 million. In 2012, the Federal Government provided grants to eight states (mostly in the Northeastern region) to buy radars with the similar specifications to the ones purchased by Rio de Janeiro (wavelength and frequency S) and the average price was approximately BRL 12 million per radar (in 2012 equivalent to USD 6.5 million). The tenders used reversed auction and did not include the cost of building the towers or the auxiliary infrastructure, which in some cases as in the State of Ceará reached an additional cost of BRL 2 million. At the time the State of Rio de Janeiro procured the radars, no subnational government had acquired such technology. Except for those used for aviation and by the military, the radars that were operating used frequencies of a much lower range and precision. The careful oversight of the Project Implementing Unit and the Bank during the execution of the contracts led to additional savings. For example, the contract for conducting the inventory of real estate assets was completed with a 6 percent savings. Similarly, the savings from the execution consultancy on the institutional arrangements for disaster risk management were used to finance an additional six months of assisted operations and training by the company that provided the radars. The close relationship between the Bank s and the State s technical teams and the proactive approach of the Project Implementation Unit positively affected the quality of the activity design. As a result, the technical quality of the Terms of Reference was generally very high, which, in turn, translated into better procurement results and products. Investment Returns In addition, the return of the US$ 2.5 million contract for the inventory of properties and the support to regularization of titling and the improvement of the management of these assets allowed the State to realize a sizable growth in the annual revenue it collects from renting some properties. It also led to the sale of assets that were not being utilized of approximately US$ 200 million, which were used to capitalize the State s Pension Fund. 23

29 It is not possible to accurately quantify the returns derived from the improvements observed in education attainment, coverage and quality of municipal health services in rural areas, the time and economic savings generated by the new administration system, or the human and economic losses prevented by better disaster alert systems and better inter-agency coordination and response during disaster events. However, these results, which are partially attributable to the project that financed crucial activities within the respective results chain, are likely to have substantially surpassed the cost of the consultancies and services that supported them, as well as the cost of the technical assistance directly provided by the Bank s team. D. JUSTIFICATION OF OVERALL OUTCOME RATING Satisfactory The project s major objectives were achieved efficiently. Several PDO indicators were achieved or partially achieved (with the expectation that the remaining will be attained in the medium term). There were no discernible shortcomings in the efficiency of the project (see previous discussion). The relevance of the project components remains substantial. Implementation did not always match the ambition of the design, but can still be considered good, particularly when evaluated from the perspective of the achievements under the overall Pro-Gestão program. There were considerable efforts to focus actions on the most important components, and the result was, on balance, positive. Important reforms were undertaken in the education area and disaster risk management, significant improvements were made on asset management, there were gains in the efficacy of pension administration, and key monitoring and evaluation tools were adopted in both the health and education sectors. The major shortcoming remains in the improvement of the health supply management. Most of the project activities were successfully implemented, and their results and subsequent reforms are likely to be extended. This is most evident in the areas of asset and pension management. The subcomponents contained important reforms (of systems and institutions) that are now well established within the administrative structure of the State. The institutional coordination mechanisms created for the management of disaster risks are likely to be preserved and improved in the future. The same can be said for improvements to monitoring in the health and education sectors given the commitment and stability of the technical teams in both secretariats. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender The lower quintiles of the income distribution and the most vulnerable groups in society, and women specifically, are the primary beneficiaries of the reforms and the improvement in the service delivery attained in education and health provision. Women benefit particularly in the case of the program supporting municipal hospitals, where most of the patients are from low-income families, and rural places. 24

30 Institutional Strengthening The project has improved information management in critical functions of the state government, e.g., property management and the administration of grants to hospitals. If work on SEAERJ and SIGAP resumes, the management of information in these sectors will improve as well. One of the most important outcomes of the project has been the construction of a new and stronger institutional structure in the State Government. Institutional strengthening has been observed in all the components. In all cases, there have been substantive reforms introducing or reinforcing the elements that are critical for institutional capacity. In public sector management, the measures to build institutions included the adoption of new systems and processes, the modernization of organizational arrangements, the issuance of norms and procedures, the application of knowledge and information, the establishment of monitoring and evaluative capacity, and the creation of incentives and rewards, among others. In the health and education sectors, several of the components also instituted new processes that reinforce institutional capacity, particularly through learning and results orientation. They support a culture of evaluation and analysis, along with a focus on targets. The institutional changes in the education sector could become even more far-reaching once the annual testing of learning outcomes is resumed. As argued by Bruns (2013) and Vinuela and Zoratto (2015), the system of evaluation and results-based rewards has had positive effects in Rio and other Brazilian states. Mobilizing Private Sector Financing Not applicable. Poverty Reduction and Shared Prosperity While Rio de Janeiro remains one of the wealthiest states in Brazil, it compares unfavorably to other states in Brazil on several social indicators. At the time of appraisal, nearly a quarter (22 percent) of Rio s population was categorized as poor, and almost 10 percent lived in inadequate housing (the largest share of any state). The education system had deteriorated precipitously in recent years, and outcomes had fallen to the lowest levels in the Southeast Region. Less than half of students enrolled in secondary school graduated (44.5 percent), compared with 60 percent for the rest of the Southeast region. Health outcomes also trailed behind other states in the Southeast Region, with significantly higher rates of maternal mortality (77 per 100,000 compared to the regional average of 51) and mortality by heart attacks (56 per 100,000 compared with a regional average of 43). 25

31 As initially designed, the project was expected to have very positive impacts on poverty. With its focus on lowperforming schools and municipal hospitals and UPAs, the project would directly benefit low-income households. Now, with SAERJ in abeyance and the UPA subcomponent transferred to TAL II (and subsequently canceled), the direct poverty impacts of the project are less certain. Nevertheless, there is evidence, described in Annex I, of substantial improvements in the education indicators of the worst-performing schools in the state, advancements in education outcomes across the board, and enhancements in the coverage and services provided by municipal hospitals in rural areas. Other Unintended Outcomes and Impacts The program design and institutional arrangements used in the asset management, education and disaster risk management areas have served as an example for other subnational governments implementing similar reforms. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION The original design was simple and adequate to the needs and institutional capacity that the State had at the time of the appraisal. The project had a strong focus on improving management across the Government and a flexible design. During the preparation of the project, and modeled on the engagement in other states, more sectors were added. While including technical assistance activities in the project created important incentives for line secretariats to commit to policy actions that were supported by Development Operations, it made project design more complex. Several of the original activities included in the project had to be dropped to free resources for the new sectors added. Soundness of background analysis. The design of the components drew on the Bank s knowledge and experience gathered through similar projects in other states. Detailed diagnostic work was conducted in each of the areas proposed. Assessment of Project Design. In general, the original design of the project made sense. The original design was simple and adequate to the needs and institutional capacity that the State had at the time of the appraisal. The project had a strong focus on improving management across the Government and a flexible design. When preparation started, the objective was to assist the State Government in establishing an agenda for public sector management reform. That focus was diluted with the subsequent addition of the health and education components. Nevertheless, the principal targets of intervention pension administration, education, and health are critical functions of the State and were clearly in need of improvement. (The fourth original target- -property management was important to the State s fiscal condition.) The specific mode of intervention consulting services focused on systems improvements was appropriate to the instrument at hand a Technical Assistance loan. It also dovetailed with other operations the Bank was conducting in Rio at the same time specifically the DPLs. 26

32 It could be argued that the project should have addressed other constraints on the targeted functions (pension administration, education, health functions, and property management) particularly the systems used to recruit, discipline, and motivate staff. This possibility was discussed with the State Government during project preparation and found no traction. In any event (as discussed earlier) the State has little control over the terms under which its employees work once they are confirmed. During the preparation of the project, and modeled on the engagement in other states, more sectors were added. While including technical assistance activities in the project created important incentives for line secretariats to commit to policy actions that were supported by development policy operations, it made project design more complex. Several of the original activities included in the project had to be dropped to free resources for the new sectors added. However, there was an appropriate plan for monitoring activities and results, support to create the capacity for SEPLAG to manage a multi-sectoral project, and risk mitigation through strong supervision and training of the Project Implementation Unit. However, there was an appropriate plan for monitoring activities and results, support to create the capacity for SEPLAG to manage a multi-sectoral project, and risk mitigation through strong supervision and training of the Project Implementation Unit. B. KEY FACTORS DURING IMPLEMENTATION As detailed below, several major factors affected the implementation of the project; some positively and some negatively. On the positive side, the project benefitted from frequent supervision by the Bank and a stable and committed group of staff in the PIU (although the number has dropped since the State s fiscal crisis deepened.) In addition, the existence of the series of DPLs contributed to the success of certain subcomponents of the project. As described in Annex VII, the first DPL required the state to take the initial steps in the design and implementation of performance monitoring systems in the education and health sectors (SAERJ and SIGMIS, respectively). The third DPL required both systems to be up and running. As such, the DPLs offered substantial financial incentives for the state to take advantage of the technical assistance provided under the project. On the negative side, the project suffered because of the high turnover among key senior-level officials in the State Government. The changes were particularly frequent after For example, the Secretariat of Health had four different secretaries appointed in one year. According to technical staff in the PIU, each new Secretary had to be persuaded to continue to support the various subcomponents of the project, as they existed at the time. Many had other ideas. Political support for the project was dampened as the administration dealt with scandals involving the former and current Governors and the State Legislature. At the outset, progress was also impeded by the PIU s unfamiliarity with Bank procurement rules, the complexity and variety of contracts to be administered many of which involved international bids and capacity constraints in sectoral secretariats that were responsible for the design of Terms of Reference. Over time, however, the PIU became extremely adept at procurement and developed a support structure to assist line secretariats in the design of activities. 27

33 There were also external factors that negatively affected the project performance. The commitment of the PIU and the close supervision helped address some of these challenges, but others were beyond the control of the project team. High Volatility of the Real and Competing Priorities/Resources. The high volatility in the exchange rate between the US dollar and the Brazilian real (from 1.70 to 4.20 and back to 3.20) observed during the implementation made the planning of the activities very difficult and required a constant revision of the Procurement Plan. In parallel, the prioritization of activities that were linked to the hosting of the World Cup and the 2016 Olympics and the unprecedented amount of external resources that the State received led to a loss of focus on the more structural reforms, such as the ones supported in the project. The political cycle and the early departure of Governor Cabral triggered a period of political and administrative instability, even putting the PIU at risk of being disbanded twice as the leadership of SEPLAG changed. Delays in the Preparation of Additional Financing and Separation of the Projects. The project performance was significantly and negatively affected by the delays in the preparation of the Additional Financing, which would end up taking close to two years. The signing of several tendered consulting contracts was considerably deferred because the original project did not have sufficient resources to finance them (a problem that was aggravated because of the substantial appreciation of the real until 2013.) The administrative error (filing the request to the Federal Government as a new project instead of an AF), which eventually led to the approval of the TAL II as a stand-alone project, created further delays as a new restructuring was required to address outstanding issues in the project and generated a needless lack of predictability that complicated the dialogue between SEPLAG and the line secretariats. Fiscal and Political Crisis. Rio de Janeiro s fiscal situation rapidly deteriorated in 2015 as the Brazilian economy entered a recession, the price of petroleum fell sharply, and the drastic reduction of Petrobras investments and operations in the state that affected the service industry. Aside from transfers from the Federal Government, Rio relies heavily on royalties from petroleum production in the fields off its coast to finance pensions, while the value-added tax (ICMS) mobilized from the services in the petroleum value chain is another important source of revenue for the State. In 2016, the State Government started paying salaries in installments, and accumulated arrears as the deficit of the pension fund grew because of the decrease in royalties received. The crisis revealed the inadequacy of the State s fiscal policy and that the efforts to diversify the economy and increase tax revenue have not been successful. The fiscal crisis was aggravated by the absence of Governor Pezão, on sick leave for over six months, at the beginning of 2016 and the fact that several senior officials are being investigated for corruption charges in relation with the Lava Jato scandal. The Government struggled to implement adjustment measures as it faced strong opposition from the State Assembly and protests. The commitments linked to hosting of the 2016 Olympics also made curbing spending difficult. The lack of action, in turn, deepened the crisis. After hosting the 2016 Olympics, the State declared a state of emergency ( estado de calamidade ) and called for a Federal Government intervention to avoid the collapse of social and security services. The State started a lengthy process of negotiation with the Federal Government on an adjustment program. Suspension of Disbursements. Disbursements were suspended in 2016 because of the actions of the State Judiciary. The World Bank and the Government of Rio de Janeiro were not responsible for these actions. On August 9, 2016, the resources deposited in the project's designated account were impounded by an injunction of the state's Judiciary, which ordered the State to use them to pay the salaries of the Judiciary instead. This 28

34 violated the terms of the Loan Agreement. The Bank then sent an official letter to the state and Federal governments informing them of the amounts seized and requesting their immediate refund. The letter also stated that until the situation was resolved no further IBRD disbursement advances could be made to those designated accounts. After multiple high-level meetings between the World Bank and the Federal and state governments, and at the request of the State, the Supreme Court of Justice issued a ruling forbidding lower courts from seizing and diverting IBRD (and other externally financed) investment accounts in the state. In response, in November 2016, the Bank management authorized direct payments for large local contracts to resume. The State Secretariat of Finance, however, issued by that time a decree prohibiting new disbursements from active loans because the State had gone over the allowable 200 percent debt: revenue ratio of the Fiscal Responsibility Law. The suspension took a toll, but not a severe one. By the time the disbursements of the project were suspended, most of the initial activities supported by the project had been completed. The only exception was the work on pension administration which continued using the State s own resources and would be used to justify part of the resources that were seized. Several new activities, which were in various stages of contracting, were suspended or canceled, however. The largest of these the computerization of the state s administrative processes (BRL 6.9 million) was canceled entirely. Had the suspension of disbursements not taken place, several other activities supported by the project would have been completed. Political Cycle, Policy Reversal, and Interruption of Programs. Since mid-2014, when Governor Cabral stepped down, and Vice-Governor Pezão assumed office, there has been a high turn-over in the leadership in the secretariats involved in the project including Planning and Management, Education, Health, and the State Institute of the Environment. The changes were particularly frequent in the Secretariat of Health that in one year had four different secretaries appointed. The governing coalition in Rio became increasingly unstable, and in 2016 the Worker s Party withdrew its support because of the impeachment process of former President Rousseff, leading to further changes in the cabinet. The situation remains fragile, and the Governor has formally been charged because of his handling of the fiscal situation. The fiscal crisis and the political instability led to the interruption of several programs and created opportunities for opposing groups to successfully pressure the Government to reverse long-standing and successful policies. The advancement in monitoring and evaluation, associated with significant gains in learning outcomes, were interrupted as students occupied schools and the Secretariat of Education. The students, with the support of one of three teachers unions, demanded that the annual evaluation that is the basis for determining if schools met their target was changed. The Government is to propose a new methodology to be applied in Organizational Restructuring. As part of the package of adjustment, in January 2017 the Government merged several secretariats, reducing the number from 25 to 12, and eliminated seven entities and foundations. The measure affected two of the secretariats that are part of the project. The Secretariat of Planning and Management was merged with the Secretariat of Finance, and the Project Implementation Units were consolidated into one. The Secretariat of Health became responsible also for the area of social protection. The organizational restructuring introduced has also affected the indicator related to performance-based management. While the State was close to accomplishing the target for this indicator in 2015, the reorganization merged some secretariats that have implemented results agreements and thus reduced the total number of entities that adhere to it. The Secretariat of Planning and Management had successfully rolled 29

35 out a protocol for the adoption of results agreements and assisted line secretariats and units to implement performance-based accountability. The results in some areas were significant. In addition, the Secretariat of Education, which had one of the longest-running results-based management programs, effectively discontinue part of the application of results agreements because of the interruption of the evaluation of learning outcomes. Responding to Emergency Situations. The addition of the disaster management component was a response to an unforeseen disaster. While it did not fit with the overall focus of the project, it reflects a desirable degree of flexibility on the part of the Bank. It prompted the second major change in the project design the decision to shift two health care subcomponents and one major education activity to TAL II to free resources. The project design did not, of course, anticipate the State s financial collapse. The state was running operational surpluses in the years just prior to appraisal although those surpluses disappeared in Although it might be argued that the State s fiscal problems were foreseeable, the project itself cannot be faulted for failing to do so. The project s objective was to improve certain state management systems, not to address the overall condition of the State s finances. Procurement Challenges. Bank procurement rules made it difficult to contract competent local individual consultants, according to the PIU. The problem arose in two respects. First, Bank rules do not permit government officials to be hired as consultants. Many of the most competent local consultants in Brazil have some link to the public sector, either as employees of government-sponsored think tanks or professors in government-funded universities. Second, Bank procedures for evaluating consultants differ from those required by Brazilian law. Neither of these problems is, of course, attributable to the specific design of this project. Adequacy of Government Commitment. The technical staff assigned to the project were highly committed. Many of the staff of the PIU have remained in their positions despite delays in salaries. Commitment by the governor and the relevant state secretaries was uneven. Fortunately, most of the activities supported by the project (with the notable exception of SAERJ) were politically uncontroversial. As noted earlier, incoming state secretaries did have to be convinced of the merits of the subcomponents relevant to them. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design The project s development objective was to introduce performance-based management and information technologies to drive service delivery improvements in key public services. The original eighteen indicators (seven PDO and eleven intermediate) that were intended to monitor progress toward this objective ranged from narrowly defined measures closely tied to specific subcomponents of the project (e.g., UPAs reporting the full cost of medical services by cost center, to somewhat broader indicators with less direct linkages. Improvement in the PDO indicator secondary school completion rate, for instance, might be attributable to the project s support for the creation of the performance monitoring system and its support for teacher training, but could also be attributable to separate actions supported by the DPLs (such as the introduction of merit-based appointment of school directors) or the State s GIDE 30

36 (Gestão Integrada da Escola) program, which diagnosed specific causes of performance problems in individual schools and devised solutions. It might also have been attributable to the wide range of other factors entirely outside the state government s control. In the event, only one of the original eighteen was still in use when the project closed. Some were dropped because they were difficult to measure, e.g. average time to produce property appraisals. Others were dropped because better indicators were found (e.g., the replacement of biennial IDEB scores with annual IDERJ scores to measure school performance). Others, such as the UPA-related indicators, were dropped because the subcomponents they tracked were dropped from the project. Overall, the near-total abandonment of the project s initial indicators suggests that more time should have been spent at the outset testing their practical feasibility and relevance. The final indicators, nevertheless, are measures of outcomes rather than activities or outputs. M&E Implementation The PIU staff went to great lengths to monitor the indicators. Every six months, the PIU produced a report (Relatorio de Acompanhamento de Indicadores) tracking progress against each indicator, with explanatory comments. In 2014, the PIU completed a Relatório de Revisão de Indicadores, a report explaining the changes to the indicators and documenting the new baselines. The PIU also issued regular semi-annual reports on the progress of individual subcomponents (Relatório de Progresso Físico), tracking the status of individual contracts and deliverables. M&E Utilization The monitoring indicators provided a basis for discussion during project supervision and informed decision-making on the project such as the need to revisit the results framework to ensure the indicators relevance and close link to the activities. It also informed both the Bank and GORJ in deploying what and where critical support is warranted to mitigate for risks and ensure achievement of desired outcomes. Justification of Overall Rating of Quality of M&E Substantial Despite the limitations of the original framework, the considerable efforts made by the Government team and the strong supervision together with the strengthening of monitoring and evaluation in the sectors that were the focus of the project contributed to widening the range of indicators that were being tracked. The improvement in monitoring provided additional information to monitor and evaluate the effectiveness of the interventions and capture other impacts. In the original framework, two types of issues should have been avoided. The inclusion of the original indicators that were not measures of outcomes and those that were difficult to measure and attribute to the interventions. The temporary interruption of measurement of several the revised indicators due to the political and financial created additional difficulties. However, the measurement is expected to resume in 2018, in some cases with the support of 31

37 the Federal Government. In the education sector and health sectors, despite the severe disruption that the crisis caused, monitoring continues albeit of a more limited set of indicators. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Environmental and Social Safeguards. The compliance with environmental and social safeguards was consistently satisfactory. Although no direct negative social or environmental impacts, good precautionary measures were designed, such as an environmental management and involuntary resettlement policy frameworks. Financial and procurement aspects were well covered at entry and were largely satisfactory. The restructured loan was expected to have positive indirect environmental impacts, given the focus on capacity building of key institutions of the GORJ, including the State Institute for the Environment. The fourth component "Development of Institutional and Technological Capacity for Preventing and Performing during High-intensity Weather Events," which entailed the strengthening of disaster risk management systems and the acquisition of two weather radar systems, could have indirect environmental implications, including on natural habitats. The project supported technical assistance activities for disaster risk management and the preparation of an Action Plan that may have included actions that entailed temporary or permanent preventive evacuation. The Involuntary Resettlement Policy was consequently triggered as part of this restructuring to ensure that actions for handling adverse events through preventive evacuation did: (i) take into consideration the impacts on the preventively relocated population, the population that may continue living at the sites, and the host population in the neighboring communities of the new settlements; and, (ii) include proper principles and guidelines to be applied by the State for land acquisition for the preventive relocation of people living in riskprone areas and/or for the post-disaster relocation of affected people in places other than their original residences and places of business; and, abandoned lands that reverted to the State for relocation were encumbered by pre-existing use claims. For these reasons, the project s rating was revised from category "C" to "B" and safeguard policies were triggered for Environmental Assessment (OP/BP 4.01), Natural Habitats (OP/BP 4.01) and Involuntary Resettlement (OP/BP 4.12) in a precautionary mode. An Environmental and Social Assessment, mandatory to all projects that may potentially have an environmental and social impact, was prepared by the Borrower for the Additional Financing, later approved as a separate project (Strengthening Public Management and Integrated Territorial Development Technical Assistance Loan P126735). This assessment covered the activities corresponding to the strengthening of disaster risk management in Component 4. The Assessment was submitted to the Infoshop on December 19, 2011, and disseminated in Brazil on December 20 of the same year. A Resettlement Policy Framework (RFP) was prepared by the State to set the principles and guidelines to be followed by the technical assistance project and will guide the preparation of the Action Plan for implementing the new institutional arrangements for disaster risk management. The RPF had been publicly disseminated on the project's website prior to approval of the second Technical Assistance Loan. The updated Integrated Safeguards Data Sheet discussed the specific risks involved triggering the safeguard policies mentioned above. During implementation, there were no adverse impacts related to involuntary resettlement due to land acquisition for project activities. The two weather radars systems have been installed in public lands, which 32

38 were free of occupants and had been voluntarily conceded by federal agencies (by force of the Brazilian legislation, the power of eminent domain cannot be exercised by the State of Rio de Janeiro upon entities of the federal government). The RPF principles and guidelines were fully incorporated by the supported technical assistance activities. Financial management. The performance of the State Government was rated as Satisfactory until the second part of 2016, when the rating became Moderately Satisfactory, due to changes in the financial management staffing complement and difficulties in the flow of funds arrangements (see below). The financial management risk rating was considered Moderate throughout the project s life. The PIU effectively carried out the technical, administrative and financial management of the Project. The audit report covering 2011 to 2012 was received with a delay; the audit reports after that, were received on time, and in all the reports, the auditors expressed an unmodified opinion therein. The combined 2016 and 2017 audit report will be due by February 28, All IFRs received during the life of the project were considered acceptable and were received in one to 30 days past the due date. There was a delay in the launch of SIAFI-Rio the new budgeting and accounting state system that would replace SIAFEM, which had an impact on the generation of the IFRs and had to be manually prepared using both SIAFEM and Rio State Integrated Financial Management System (SIAFE-Rio) information. Funds in the Designated Account were seized in August 2016, to comply with a judicial order which had an impact on Project execution (approximately USD 1 million) that the Bank had advanced to move Project implementation. This led to a halt of any further advances to the Designated Account, while the Federal Government reacquired the commitment to continue with the Project and committed to reimburse the sequestered amount. This event created a critical disruption, delaying payment for services already under implementation, forcing some contracts to be canceled or withdrawn, and preventing others to be signed, in view of the perceived uncertainty. The undocumented balance in the Designated Account per Client Connection is BRL ,92 (as at 10/31/2017); any balance that remains undocumented by the end of the Grace Period (11/30/2017), will need to be refunded to the Bank. There were no instances of ineligible expenditures identified to date. C. BANK PERFORMANCE Quality at Entry Moderately Satisfactory As noted earlier, the original design of the project was realistic and straightforward. Technical assistance was focused on tasks that state personnel could not perform on their own and would not be financed from the State s own budget. The project also dovetailed with parallel DPLs, with several subcomponents of the project providing the technical supported required for the achievement of the prior actions specified in those larger lending operations. The Bank s support was justified for two reasons. First, it is unlikely that the state would have financed these activities on its own, despite their low (relative) cost. Technical staff involved in the project argue that, in the absence of a strong backing from an important political constituency, such activities would have fallen by the 33

39 wayside in the course of annual budget negotiations. Second, the technical assistance provided a vehicle for the Bank to provide a wide range of on-site support. Technical staff noted that the sequence of supervision mission over the life of the project provided a venue for wide-ranging discussions on management issues. Nevertheless, the limitations of the original M&E framework at entry discussed earlier explain the downgrading of the rating. Quality of Supervision Highly Satisfactory The Bank conducted over two dozens formal supervision missions over the life of the project, along with frequent ad hoc visits, VCs, and calls. As noted above, these interventions were successful in identifying problems in implementation and helping the PIU to resolve them. The achievements of the project can be partly attributed to the intensive and continuous presence of the Bank throughout implementation. The number of close supervision missions ensured more consistent progress towards achieving the result targets. There was an ongoing dialogue, as evidenced by mission aide memoires, on many facets of the project. There was a focus on ensuring results and on the measures needed to move the program forward. The staff followed up on indications of difficulties and the technical needs of the State. There is widespread recognition that the technical assistance was a source of considerable help to the Government in overcoming implementation obstacles and finding solutions to several problems. Strong supervision was present in all sectors of the project. The nature of the support provided under each component varied according to needs. Within the public management component, for example, the assistance covered broad policy issues, the norms and regulations governing the operation of the various systems, and detailed technical issues of data collection, monitoring, and analysis. There was a heavy concerted effort in education to find solutions during implementation and more importantly to set up the evaluation system. These actions were significant and made a major difference in the results of the loan. Project supervision went far beyond the monitoring of indicators, as the Government team required significant support in the preparation of TORs and the procurement of consultancy services. Over the course of project supervision, the Bank also provided advice to SEPLAG on procurement systems and human resource management reforms. The education team provided ongoing policy advice to SEEDUC, including hands-on support on the completion of several evaluations. These inputs were particularly valuable as the team had access to high level decision makers. Justification of Overall Rating of Bank Performance Satisfactory The Bank s overall performance was commendable for its close involvement during the preparation and implementation of the reform program supported. The Bank s engagement and support to the State under this project were widely recognized by the authorities. The authorities also pointed to the critical role of the Bank s 34

40 oversight and scrutiny. The constant inquiries by the Bank and the on-going exchange of views led the authorities to examine in greater detail the work being done and to be more self-critical. In many instances, in response to Bank s concerns, the authorities adapted their approach to take into consideration the recommendations. D. RISK TO DEVELOPMENT OUTCOME Substantial The management and administrative reforms supported by the project in asset and pension management specifically those that are now in place are expected to remain in place. However, the disruptions in the reforms in health and education may delay progress despite the professionalization and stability of the technical teams in charge of monitoring and evaluation within these secretariats. The dislocation of the trained staff in INEA is also a source of concern. Yet, they are career civil servants that remain in the Institute but who have been tasked with new programs as cuts have forced the Institute to reduce its workforce. V. LESSONS AND RECOMMENDATIONS There were several lessons learned from the experience of the program supported by the project. Those lessons can be grouped into three categories: i) the design and selection of the components; ii) the value of close supervision and assistance; and iii) the importance of building institutional arrangements. Flexible Design and Focus on Few Sectors. One of the primary reasons for the achievements of the project was its focus on management and flexible design. The loan covered few sectors and concentrated on those areas where the Bank had accumulated experience in the State. By including only four areas, both the authorities and the Bank could devote greater attention to the details of the policies and measures agreed under the project. There was also the ability to follow up more closely on measures needed to achieve stated goals and to monitor outcomes. Primarily, the scope of the project was manageable. This overlap with the sequence of Development Policy Operations was crucial, not simply because it contributed with more in-depth knowledge about the sectors, but also because it helped build a closer working relationship between the Bank and the Government. Staff belonged to both supervision teams and had close and continuous contact with the authorities. The project s links with the series of DPLs was very useful, providing a strong financial incentive for the State to follow through on activities supported by the project as well as an additional platform for high-level dialogue with policymakers. As described in Annex 7, DPL-1 included several prior actions related to public sector management, including the capitalization of the social security system for state public servants and changes in the budget process, as well as conditionality in the education and health sectors. DPL-3 supported key sectoral reforms in education and health. The prior actions in both DPLs benefitted from technical assistance financed by the project. Strong Supervision and Direct Technical Assistance. A complementary lesson is the importance of close supervision and sharing of technical expertise. In all instances, the continuous presence of the Bank, with its 35

41 regular supervision missions and technical assistance via monthly video conferences and regular calls, contributed much to the successful implementation of the Pro-Gestão program. The support was possible because of the Bank staff, with relevant expertise, had the financial resources to provide the assistance. The internal budget for supervision, along with available funds from the DPLs and other existing investment projects, provided them with the ability to have a greater presence than usual. The Bank was able to provide timely technical support by tapping into the expertise and resources developed under the investment project. The implementing staff recognized that the on-going exchange with the World Bank-led them to evaluate problems and search for new solutions in a manner they had not considered before. This occurred mainly as they attempted to respond to Bank queries on the reach of the systems. In the process of analyzing the data and the characteristics of systems, the staff gained insights, discovered ways of correcting deficiencies, and adapted solutions that had not been considered before. The close supervision did not simply entail direct transfers of knowledge, but it also generated a process of learning by doing.. Focus on strengthening technical teams and institutionalization. Finally, the experience of the implementation revealed the importance of fostering new institutional arrangements and investing in career civil servants. The most significant institutional change was the introduction of monitoring and a results-oriented culture within the Government, even if the agenda has lost salience because of the crisis. The response to the crisis will require deepening some of the reforms, for example in procurement and efficiency of service delivery. The establishment of performance indicators helped to create a new structure of incentives. Finally, monitoring systems were introduced, which raised the level of scrutiny of actions. 36

42 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Introduction of performance-based management and information technologies to drive improvements at the center of government and in key public services Objective/Outcome: Improvements in service delivery, including secondary education, hospital care, and disaster risk management A.2 Intermediate Results Indicators Component: Introduction of performance-based management and information technologies to drive improvements at the center of government and in key public services Component: Improvements in service delivery, including secondary education, hospital care, and disaster risk management 37

43 A.1 PDO Indicators Objective/Outcome: Introduction of performance-based management and information technologies to drive improvements at the center of government and in key public services Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Number of the State's properties regularized Number Jun Jun Dec-2016 Comments (achievements against targets): Target Achieved and Surpassed. The achieved value is 22 percent higher than the target and represents an 833 percent increase compared to the baseline. Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Average number of days to record debt owned by renters of State Government's properties Days Jun Jun Dec-2016 Comments (achievements against targets): Target Partially Achieved. The results represent a reduction of 61 percent in the time it takes to record debt. The objective is expected to be achieved in 2018 when the remaining modules of the Integrated Management System for the Pension Fund are rolled out. Objective/Outcome: Improvements in service delivery, including secondary education, hospital care, and disaster risk management Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion 38

44 Points improved in the Index of Educational Development of Rio de Janeiro (IDERJ) by the 30 percent worst performing schools Number Jun Jun Dec-2015 Comments (achievements against targets): Target Partially Achieved. Measurement discontinued in The performance of the bottom thirty percent of schools has improved across the board as discussed in the main document. The results represent an increase of 125 percent and 50 percent of the desired improvement. It is not possible to determine what was the result for 2016, but the trend was positive. The administration of the test used to calculate the Index of Educational Development of Rio de Janeiro was discontinued in 2016 as a result of a student protest and the extended occupation by opposition groups of the premises of the Secretariat of Education. The measurement of learning outcomes is scheduled to resume in The national test of learning outcomes was administered in November 2017, which will help fill the gap for monitoring of results. Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Percentage of false alerts issued by the Flooding Warning System Percentage Dec Jun Dec-2015 Comments (achievements against targets): Target Partially Achieved. Measurement discontinued in The Institute of the Environment is developing a new methodology for measuring accuracy. A.2 Intermediate Results Indicators Component: Component I - Core Government Systems and Management Tools for Improved Performance and Efficiency Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion 39

45 Number of State secretariats and agencies with signed results agreements Number Jun Jun Jun Dec-2016 Comments (achievements against targets): Target Partially Achieved. The fiscal crisis and the organizational restructuring introduced as part of the State's fiscal consolidation efforts have affected this indicator. The Secretariat of Planning and Management had successfully rolled out a protocol for the adoption of results agreements and assisted line secretariats and units to implement performance-based management. The results in some areas were considerable. While the State was close to accomplishing the target for this indicator in 2015, the reorganization merged some secretariats that were implementing results agreements and thus reduced the total number of entities that adhere performance management. Other secretariats suspended the implementation of the results agreements as no resources were available to pay performance bonuses. Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Average number of working days to issue a reminder letter Days Jun Jun Jun Dec-2016 Comments (achievements against targets): Target Mostly Achieved. The results represent a reduction of 84 percent in the time it takes to issue a reminder letter and is equivalent to 89 percent of the desired time savings. The objective is expected to be achieved in 2018 when the remaining modules of the Integrated Management System for the Pension Fund are rolled out. Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Number of public real estate properties with records registered and updated in System of Real Estate Management Number Jun Jun Jun Dec

46 Comments (achievements against targets): Target Achieved and Surpassed. The results represent 136 of the target and an increase in the number of properties with updated records of 629 percent. Component: Component II - Strengthening Education Management Capacity, Teacher Instructional Practice and Incentives for School Effectiveness Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Percentage of public servants in strategic positions of the Secretariat of Education receiving performance evaluation Percentage Jun Jun Jun Dec-2016 Comments (achievements against targets): Target Not Yet Achieved. The activities related to establishing guidelines and process to determine competencies and evaluate staff in strategic positions have been completed. However, the program was not fully implemented according to the original timeline. The fiscal and political situation has impacted this indicator in several ways. The interruption of disbursements caused delays in the implementation of the consultancy. More importantly, the fiscal situation and the occupation by students of state schools and the premises of the Secretariat of Education led to the interruption of several monitoring and evaluation activities that impeded performance evaluations during The performance evaluation program may take at least another year to be fully implemented, but the Secretariat of Education remains committed to the objective and is taking the necessary provision to advance the program. Despite all the challenges, the Secretariat Component: Component III - Health Management and Information Systems Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Number of hospitals participating of the Program PAH I with targets monitored through the System of Number Jun Jun Jun Dec

47 Management of Indicators and Targets Comments (achievements against targets): Target Achieved and Surpassed. The program achieved the adherence of 100 percent of hospitals participating of the Program for Hospitals in the Interior, which surpassed the target by 10 percent. Indicator Name Unit of Measure Baseline Original Target Formally Revised Target Actual Achieved at Completion Percentage of unmet requests of the State Hospital "Rocha Faria" for hospital supplies Percentage Jun Jun Jun Dec-2015 Comments (achievements against targets): Target Not Measured. Measurement discontinued in The hospital and other of comparable size were municipalized in 2016 as a result of the fiscal crisis. Thus, it was not possible to measure the performance of the supply management system of the Secretariat of Health. 42

48 B. KEY OUTPUTS BY COMPONENT 1. Core Government Systems and Management Tools for Improved Performance and Efficiency Objective/Outcome 1 Management Tools for Improved Performance and Efficiency of the Management of Properties/Real Estate Assets Outcome Indicators Outcome Indicator (Additional) Intermediate Results Indicators Key Outputs by Component (linked to the achievement of the Objective/Outcome 1) 1. 3,021 properties with regularized titles in 2016 (from a baseline of 363 in 2009; the target was 2,468). 1. Increase in revenue from rented state properties from BRL 1 million in 2009 to BRL 11 million in 2016 (equivalent to USD 3.3 million). 2. Total rent collected since 2010 of BRL 80.9 million (US$ 25.2 million). 2. Total of 90 properties sold for a total of BRL million (equivalent to US$ million) since Total of 28 properties transferred to the State s Pension Fund (RioPrevidência) representing a value of BRL million (equivalent to US$ 40.5 million) since ,578 real estate properties with records registered and up-to-date in the System of Real Estate Management in 2016 (from a baseline of 728 in 2009; the target was 3,376). 1. Revision of norms and streamlining of procedures for management of properties. 2. Reorganization and strengthening of the unit in charge of managing properties. 3. Inventory of 4,578 properties (including physical inspection, update of blueprints, archival research for the identification of titling documentation, and geo-referencing). 4. Updating e of the information of 4,578 properties in the State s 43

49 System of Real Estate Management. Objective/Outcome 1.2 Core Government Systems and Management for Improved Performance and Efficiency of the State s Pension Fund (RioPrevidência) Outcome Indicators Intermediate Results Indicators Key Outputs by Component (linked to the achievement of the Objective/Outcome 2) 1. Reduction of the average number of days to record debt owned by renters of State Government's properties from 120 in 2009 to 47 by 2016 (the target was 30). [It is expected that the recording of debt will be automatic once the module for property management becomes operational.] 1. Average number of working days to issue a reminder letter from 90 days to 14 (the target was 5) [It is expected that letter will be issued automatically once the module for the property management is implemented.] 1. Reengineering of administrative processes in nine areas (pension management, administration, financial management, property management, inventory and asset management, monitoring of legal cases, audit and compliance, corporate governance, and actuarial management) completed and adopted. 2. Design of the architecture of an integrated management system (SIGAP) for the State s Pension Fund and identification of the technical specifications of nine modules (pension management, administration, financial management, property management, inventory and asset management, monitoring of legal cases, audit and compliance, corporate governance, and actuarial management) following the reengineered processes. 3. Development of four modules of the integrated administrative system (SIGAP), namely actuarial management, monitoring of legal processes, financial administration, investment, and inventory and asset management. 44

50 [The five modules have been completed and staff is being trained for their implementation starting in The inventory and asset management module will be adopted by the entire executive. The State s Pension Fund is using its own resource to develop the remaining modules.] 4. Development of the State System of Asset Management (Sistema Estadual de Bens Móveis or SBM/RJ), by adapting the SIGAP module. Objective/Outcome 1.3 Streamlining of Administrative and Human Resource Management Processes and Adoption of Electronic Processes Key Outputs by Component (linked to the achievement of the Objective/Outcome 1.3) 1. Design of the architecture of an e-procurement system. [The system was developed and implemented using the State s own resources. A revision of the system is being supported by the Bank with resources of a Trust Fund.] 2. Reengineering of 22 administrative processes for resource management, asset administration, monitoring of services contracted out to civil society organization, payments and procurement. 3. Design of the architecture of an electronic processing system. [The development of the system was not funded by the project, but the Federal Government has agreed to donate its electronic processing system (SES) to the State that will be customized to implement the streamlined processes.] Objective/Outcome 1.4 Improved Monitoring and Performance-based Management Intermediate Results Indicators Key Outputs by Component (linked to the achievement of the Objective/Outcome 1.4) agreements signed with State secretariats and agencies by 2015 (only 2 remain in effect; the target was 12). 1. Legal and regulatory framework for the implementation of results agreements between the Governor and line secretariat and autonomous agencies. 2. Methodology for the design and monitoring of results agreements. 45

51 3. Civil servants trained in results-based management. 4. Technical assistance provided to 10 secretariats and agencies to implement results-based management. 5. Establishment of a State Management Network for the training of career development of public management specialists deployed in line secretariats and agencies. 2. Strengthening Education Management Capacity, Teacher Instructional Practice and Incentives for School Effectiveness Objective/Outcome 2 Building SEEDUC s Technical Capacity for Conducting Studies and Evidence-based Policy and Providing Effective In-Service Teacher Training Outcome Indicators Outcome Indicators (Additional) 1. Points improved in a year in the Index of Educational Development of Rio de Janeiro (IDERJ) by the 30 percent worst performing schools from 0.40 in 2009 to 0.90 in 2015 (the target was 1.40) [The measurment of this indicator was discontinued in 2016.] 1. Rio de Janeiro improved its position in the national ranking for the secondary education Index of Educational Development (IDEB) from the 26 th place out of 27 in 2009 to the 4 th place in The total number of schools with an IDEB lower than 2.9 decreased from 429 in 2009 to 64 in The 30 percent worst-performing middle schools observed an increase in their IDEB score twice as large as rest of the schools (of 1.05 points versus 0.42). 3. The approval rate in middle school improved from 54.5 percent in 2009 to 78.7 percent in 2015 in the worst-performing schools. The gap between the worst-performing schools was reduced from 17 percentage points to 7 in the same period. 4. Between 2015 and 2009, dropout rates decreased 22.6 percent in the schools that had dropout rates higher than 20.6 percent in 2009, while the rest of the schools observed a decrease of 6.9 percent. 5. The number of secondary schools with a high dropout rate (higher than 22.6 percent) declined from 228 in 2009 to 21 in

52 6. The approval rate in secondary school improved from 64 percent in 2009 to 82 percent in 2015 in the worst performing schools. The gap between the worst-performing schools was reduced from 17 percentage points to 3 in the same period. Intermediate Results Indicators Key Outputs by Component (linked to the achievement of the Objective/Outcome 2) 1. Percentage of public servants in strategic positions of the Secretariat of Education receiving performance evaluation from 0 in 2009 to 26 in 2016 (the target was 100 percent). 1. Design of the structure a dedicated school for professional training of teachers and its curricula. 2. Establishment of school for professional training of teachers working in the state network Escola SEEDUC. 3. Strengthening and training of the statistics and monitoring team in the Secretariat of Education. 4. Technical assistance for the design of a consolidated database for school indicators to inform the schools improvement plans and other targeted interventions. 5. Technical assistance for the processing and evaluation of school census data. 6. Design, quality control, and publication of 4 annual reports consolidating all results and policy indicators of the Secretariat of Education in English and Portuguese. 7. Design of a training and impact evaluation on class observation to improve teacher performance using the Stalling Method. 8. Design of a study to identify the skills that each position in the Secretariat requires and development of a customized training curricula. [The study and a system to support the implementation of the skill monitoring and development program was completed under the TAL II.] 9. Design of an impact evaluation for two programs of professional training of teachers (Programa de Formação Continuada de Docentes and Programa de Formação de Docentes no Programa Nova EJA). 47

53 3. Health Management and Information Systems Objective/Outcome 3.1 Improve Monitoring of Results of the Program to Support Municipal Hospitals 10. Design of study to determine the standards and levels of performance targets in state secondary schools. 11. Training of staff on impact evaluation methods. Outcome Indicators Intermediate Results Indicators Key Outputs by Component (linked to the achievement of the Objective/Outcome 3.1) 1. Improved compliance with the Program PAH I s targets (i.e., increase in the number of hospitals providing services to patients from other municipalities) (100 percent) of hospitals participating of the Program PAH I with targets monitored through the System of Management of Indicators and Targets (target was 70) 1. Development and implementation of a new monitoring system for supervising the indicators and targets included in results agreements signed with municipal hospitals receiving state transfers (Sistema Integrado de Gestão de Indicadores e Metas de Saúde SIGMIS). [The system could be used for other programs as well as for monitoring the UPAs.] 2. Design of an evaluation of costing of UPAs and Hospitals. 3. Design of a study to inform the preparation of the new state policy and guidelines on basic health attention. Objective/Outcome 3.2 Improve the Efficiency of Supply Management in the Health Sector Outcome Indicators 1. Percentage of unmet requests of the State Hospital "Rocha Faria" for hospital supplies (target was 5 and the baseline 35) [This indicator could not be measured as the hospital that was the pilot and others of similar size were municipalized in 2016 and were no longer under the purview of the State.] 48

54 Intermediate Results Indicators (Additional) Key Outputs by Component (linked to the achievement of the Objective/Outcome 3.2) 1. Revision of the warehouse management arrangements and suspension of the firm contracted to manage drug and medical supplies storage and distribution. 1. Diagnostic of weaknesses and gaps in the supply chain management in the health sector. 2. Redesign of supply management processes in the health sector, including warehouse management arrangements. 3. Support to the Secretariat in the response to the crisis created by the discovery that 10tn of medicine had expired and had to be discarded in Design of system of a new information system for health supply management in the health sector. 4. Improve Institutional and Technological Capacity for Preventing and Performing during High-intensity Weather Events Objective/Outcome 4 Institutional and Operational Arrangement for the Disaster Risk Management and Weather Monitoring for Preventing Outcome Indicators Outcome Indicators (Additional) Key Outputs by Component (linked to the achievement of the Objective/Outcome 4) 1. Percentage of false alerts issued by the Flooding Warning System reduced from 56 in 2013 to 52 in 2015 (the target was 44) [The measurement of this indicator was suspended in 2016 because of technical problems.] 1. Increase in the number of people covered by the State flooding alert system from 0 million in 2011 to 16 million in Zero human fatalities registered in over a 1,000 landslide events from 2014 to Design of the institutional and operational arrangement for the Disaster Risk Management. 2. Restructuring of the Civil Defense. 3. Revision of protocols for emergency response and inter-agency coordination. 4. Establishment of information sharing and coordination mechanisms 49

55 between the Civil Defense, the Department of Mineral Resources, and the Institute of the Environment. 5. Acquisition and installation of two Weather Radar Systems. 6. Training of staff and contract personnel in the operation and maintenance of radars. 7. Training of staff in the development of weather prediction models. 8. Creation of a weather monitoring center working 24 hours 7 days a week. 9. Design of an integrated system to consolidate information on geological risk, disaster events, and weather information. 50

56 Exchange Rate BRL/USD Millions The World Bank Several risks materialized during the implementation of the project. The project execution was affected by external and internal factors. However, most activities were concluded and the project disbursed approximately 92 percent of the original budget in reais (see Figure 1.) The considerable devaluation of the real observed in 2015 freed resources that were allocated to complementary activities. These were subsequently cancelled when the disbursements halted in August Yet, by then, most of the original activities were completed or in the final stages. Figure 1: Planned and Actual Disbursements and Main Events Affecting the Project Actual Disbursements USD Original Disbursement USD Formally Revised Disbursement Plan USD Exchange Rate Actual Disbursements BRL Original Disbursement Plan BRL Formally Revised Disbursement Plan BRL Notes: The following are the milestones and events that affected the project: Approval TAL I August 26, 2010 Effectiveness TAL I December 27, 2010 Natural disaster (Landslides) January 11, 2011 First restructuring TAL I February 17, 2011 Pre-identification Additional Financing April 11-20, 2011 Project Concept Note Review of the Additional Financing June 27, 2011 Invitation to Negotiate of the Additional Financing Oct 10, 2012 Approval of Additional Financing as a stand-alone Project Pro-Gestão II February 12, 2014 Request to the Federal Government for authorization to restructure TAL I February 25, 2014 Authorization from the Federal Government to restructure the TAL I approved August 24,

57 Second restructuring of TAL I approved (authorized by the Federal December 3, 2014 Government in August 2014) Seizing of resources/suspension of disbursements August 9, 2016 Component 1. Core Government Systems and Management Subcomponent 1a. Results Agreements. This component was originally intended to support the introduction of a results-based management tool into the state planning and monitoring system. Following in the footsteps of some other Brazilian states, Rio would establish a portfolio of strategic projects, each tied to a set of performance indicators. Results agreements, setting out targets for each indicator, would then be formalized through quasi-contracts between state administrative bodies and the Governor. The performance targets would be incorporated into the state s INTRAGOV.RJ 4 monitoring system to allow real-time data access by the governor and key state managers. Performance against these targets would be used as an input to budget formulation and approval. As proposed in the PAD, the TAL would finance a consultancy to support the identification of suitable indicators and design the GORJ version of a results agreement. It would also support improvements in the INTRAGOV.RJ data portal by financing the acquisition of INTRAGOV.RJ hardware, the preparation of physical space for the data warehouse servers, personnel training, and testing and installation. The technical assistance ultimately took the form of two contracts. Under the first, the consultants were to analyze other states experience with results contracts, propose a model for Rio, and familiarize staff of the state government with the concept (6 months and BRL 674 thousand) The second was to operationalize the system itself, implanting the technical unit, training it, and advising it (13 months and BRL 106 thousand.) Both contracts were successfully completed. A total of ten agreements were in effect in 2014, covering inter alia, the secretariats responsible for security, education, planning, and the environment, along with specific departments and institutes responsible for transport (DETRAN) and environmental protection (INEA). Two agreements were made available to the ICR mission as examples. The first, with INEA, called for a total of 148 targets to be pursued over the period May, 2012-April, These included 14 strategic and institutional indicators (such as the amount of revenues derived from environmental mitigation agreements) and 134 performance targets. The latter included, for example, the number of licenses issued by the each of INEA s regional offices and the number of conservation units (Unidades de Conservação de Proteção Integral) that were fully staffed. The second agreement, with SEPLAG, called a total of 17 targets to be pursued in the first semester of 2014, including the implantation of an integrated system for human resource management (Sistema de Gestão de Recursos Humanos - SIGRH), the execution of a strategic plan for supply management (Gestão Estratégica de Suprimentos) and the training of staff in the units responsible for human resource management, logistics, planning and budgeting. An additional target (improved accuracy in forecasting own-source revenues) was added to the targets for the second half of INTRAGOV.RJ is a website (currently accessible only to authorized individuals) which combines primary databases of the state government, with information on education, health, and public security, among others. It builds on the Public Sector Management by Indicators (Gespi) system. 52

58 Both agreements set out the institutional arrangements for evaluating compliance with the targets. In the case of INEA, it consisted of an evaluation committee comprised of one representative of the secretariat of environment, two representatives of INEA, and one representative of SEPLAG. The agreements also established a system of bonuses to be paid to the staff in each organization, based on achievement of the targets. The impact of these agreements could not be assessed with the data at hand. It is instructive, however, that only two of the organizations that participated in the initial exercise continued to do so after A new agreement, with the revenue administration department of the Secretariat of Finance, came in effect in Given that many results agreement were discontinued, the performance this subcomponent is considered moderately unsatisfactory. Subcomponent 1b. Management of data on state-owned property. This component dealt with property owned by the state government. At the time the project was appraised, the state owned more than 6,900 properties. The exact number was not known. Nor was the state fully aware of the uses and economic value of its real property, as current records were outdated and incomplete. Queries directed to the state government as to the availability of real property for use by government agencies averaged 60 per month in However, fewer than ten percent of these requests received a response each month, and replies inevitably were based on imprecise data. The remaining requests were denied for lack of information. At the time of appraisal, the State had developed and tested a new information system for real property: the Sistema de Patrimônio de Rio de Janeiro (SISPAT). Field data, however, remained to be collected. An engineering firm was needed to survey and value the estimated 5,000 properties that were not properly registered. Additional data (detailed property descriptions, photos, descriptions of current use and occupants, if any, among others) also needed to be gathered. Subcomponent 1b was intended to finance these activities. This ultimately took the form of a single contract, calling upon the consultant to conduct a tratamento técnico patrimonial and tratamento da documentação legal on approximately 4,880 state-owned properties and incorporate the results into SISPAT. The contract was successfully completed. In the original version of the results framework, progress on this subcomponent was to be tracked with two indicators: (1) the average time (days) to produce property appraisals and (2) the percentage of requests from public bodies concerning the availability of state property that were answered. After the second restructuring both indicators were dropped. Two new ones were introduced. The first directly measured the results of the activity itself: the number of state properties with records registered and updated in SISPAT. The second tracked a benefit that might emerge from it: the number of (state-owned) properties with 'situação ocupacional' regularized or more precisely, the number of actions taken by the state to this end. 5 5 When a property owned by the state is occupied by a private individual (a particular) the occupier is required to pay a taxa de ocupação (rent) and sign a contract formalizing the occupation. A state agency (COORE) is responsible for notifying occupiers of their payment obligations. Another agency (ASJEC) is responsible for managing the contracts formalizing the occupation. In the case of properties occupied by low income persons, information is to be sent to the Institute for Land and Cartography (ITERJ) which is responsible for regularizing the title. According to the Relatório de Revisão de Indicadores (RRI) for 2014, the base value for this indicator is the number of these actions (the notification to pay the taxa, the signature of the contract, and the sending of data to ITERJ) undertaken during the period Progress is measured by adding the number of such actions in the current year to the base value. 53

59 The subcomponent achieved the desired results according to both indicators. From a base of only 728, the number of properties updated and registered in SISPAT reached 4,578 at the end of The number of regularization actions increased as well. Only 363 such actions occurred over the period In 2016, the number was 3,024, far exceeding the target. Because of these measures, annual revenues from state owned properties increased from BRL 1.6 million to BRL 11.5 million. Based on these results, this subcomponent can be rated highly satisfactory. Subcomponent 1c. Pension Fund Management Integrated Information System. RioPrevidência (Fundo Único de Previdência Social do Estado do Rio de Janeiro) is responsible for managing social security (pension) benefits for all the State s retired public employees and their eligible survivors. At the time of appraisal, the State intended to introduce a new integrated information management system (SIGAP) in order to improve internal control over contributions to the pension fund and other internal administrative processes. The new system would also allow services to be decentralized, improving services to beneficiaries (most of whom were old and infirm). It was expected that the introduction of SIGAP would permit a substantial reduction in staff (from 350 to 280). This subcomponent was linked to other reforms in the pension system supported by the first DPL: the capitalization of the system for state public servants with oil-based revenues and the protection of transfers from executive discretion. The project would finance consulting services to define the functional and technical requirements of the proposed database modules, which would then provide the basis for a second contract for the production and installation of the priority modules for the SIGAP itself. The necessary hardware for SIGAP would be financed from RioPrevidência s own resources. The technical assistance took the form of two contracts. The first one was to determine the steps necessary for the development of the administrative and pension processes and the draft terms of reference for the development of SIGAP. That contract was completed in The second contract was to develop and implant the four modules of the new system (including financial, legal, investments and actuarial management modules.) That contract was completed in June 2017 and RioPrevidência has contracted with its own resources the development of the remaining modules. The completed four modules are currently being tested and are expected to be up and running by January, At the outset, this activity was linked to four indicators. The first tracked the average time required by RioPrev to collect Social Security employment tax debts. The second tracked reductions in RioPrev staffing; the third, the average number of days required prepare and deliver information demanded for judicial proceedings; and the fourth, the average time to transfer financial data to the state s financial management system (Sistema Integrado de Administracao Financeira para Estados e Municipios SIAFEM). The Indicators Report for 2014 reported improvement in only one of the indicators: the number of staff (which declined from 350 to 262). After 2014, all four indicators were dropped and replaced by two new ones. The first monitors the average number of days required to register unpaid occupation charges in the state s debt registry. 6 The second monitors the average number of working days it takes for RioPrevidência to issue a reminder letter (carta amígavel) the first step in the debt collection process. 6 Note that the definition of this indicator (as tracked in the IRRs) differs from the indicator tracked in the ISRs. The ISRs track the number of days required to record debt owed by renters of state property. The figures reported in the two documents are nevertheless identical. 54

60 The State has shown improvement in both indicators. The average number of days required to register an unpaid occupation charge declined from 120 in the base year to 47 at the end of The time required to send a reminder letter reportedly shrank from 90 days in the base year (mid-2014) to ten days at the end of Presumably, both indicators will further improve once SIGAP is fully operational. Based on the data at hand, this subcomponent is rated satisfactory. New Subcomponents. To take advantage of the resources made available by the appreciation of the US dollar, SEPLAG proposed additional activities under this component that would come to complement the original interventions. These included the training of public sector management specialists (Gestão da Rede da SEPLAG), the computerization of administrative processes (including personnel management), and the development and implantation of a 'sectoral' model of personnel management, which would address the allocation of staff, personnel management procedures, and the use of performance indicators. SEPLAG also proposed the acquisition of data storage for SEPLAG, and the formatting and printing of various planning documents. Due to the suspension of disbursements in August 2016, these activities (some that have been contracted or for which procurement processes were in their final stages) were ultimately not undertaken. Component 2. Strengthening Education Management Capacity The State Government bears primary responsibility for secondary, vocational and adult education throughout the state, accounting for about 80 percent of enrollments at those levels. Municipalities bear primary responsibility for primary schools. At the time of appraisal, the State was managing nearly 1,500 schools, with 1.34 million students. The performance of Rio s education system was among the poorest in Brazil. Graduation rates (for secondary students) and performance on standardized tests lagged far behind regional averages. Subcomponent 2a. Strengthening education policy. As specified in the PAD, the TAL would include three subcomponents in the education sector. The first would strengthen SEEDUC s policy setting ability, helping it identify programs and policies that achieve positive results and modify or discard policies that underperform. To this end, the TAL would finance: (a) a rigorous evaluation of recent program innovations such as Projeto Autonomia and Conexão Educação and (b) a study of the determinants of Rio s poor secondary school graduation rates and learning results, focusing on: the level and sources of funding, the uses of funding across different expenditure and program categories; and teaching practices (using the Stallings Classroom Observation System). Subcomponent 2b: School-level Performance Targets/School Development Plans. Following the models of Minas Gerais, Sao Paulo, Pernambuco, and the Federal District, the state intended to introduce performance targets for annual improvements in math and Portuguese (measured through annual state and federal learning assessments) and student flows (repetition and graduation rates). These would be measured at the level of individual schools and introduced over two years. To support these efforts, the TAL would finance the creation of a feedback system to communicate state education performance data 7 to individual schools on a timely basis. To help schools improve their results, the TAL would also finance the preparation of School Development Plans (Planos do Desenvolvimento 7 Rio de Janeiro State System of Evaluation for Education (SAERJ) 55

61 Escolar PDEs.) These involve each school conducting a self-diagnosis, followed by preparation of a brief document setting out the school s goals for improvement and specific actions and projects that the school will undertake to achieve those goals. 8 Using TAL resources, SEEDUC was to contract high-level technical specialists to assist the Secretariat and school personnel in translating test results into PDEs with monitorable targets. Subsequently, these specialists would assist in monitoring school performance against those targets though an annual monitoring system. Subcomponent 2c. Teacher Training. The third subcomponent in the education sector was intended to support Rio s teacher training institute, the Casa do Educador. At appraisal, the institute was still in its infancy. Under the project, SEEDUC was to contract specialized technical experts to further develop the program concept, define specific training courses, both in subject areas as well as pedagogy, classroom management, and the use of information technology in the classroom. Many of the proposed tasks foreseen in subcomponent 2a were implemented with the State s own resources and through the direct technical assistance of the Bank education specialists, who trained the monitoring and evaluation team in the Secretariat and provided ongoing advice on the design and implementation of evaluations. The project funds under this subcomponent were focused on classroom performance. That work was divided into two contracts. Under the first, a consultant would devise a classroom performance survey based on the Stallings classroom observation system and train staff in its use. Under the second, the consultant would manage the implementation of the survey itself throughout the state school system. Although contracts were awarded for both activities, could not be executed because the learning outcome evaluation that would be used to assess the impact of the intervention was discontinued. Other forms of evaluation were, however, undertaken using the State s own resources. In the second subcomponent, a consulting firm was hired to integrate education data from existing sources into single data bank and devise performance indicators. This work was successfully completed. A second contract, to construct an education performance monitoring system comparing the state s performance with other states and the performance among municipalities and individual schools within the state was also successfully completed. The results are reported in an annual publication, as well as on the SEEDUC website. Regrettably, the annual testing of learning outcomes was suspended in 2016, which was central to the monitoring, due to political opposition specifically a series of student invasions, ostensibly instigated by teachers who opposed the monitoring system. The federal government continues to monitor school performance though the federal test of learning outcomes, Prova Brasil, and the Index of Education Development. These, however, are presently conducted only every other year and for secondary schools only provides information for a sample of schools. As a result, the performance monitoring system is not currently fully in use. However, the next test will be administered in December 2017 and the Ministry of Education is in the process of rolling out an annual evaluation that would cover all schools. Other source of data that the Secretary of Education is still using to monitoring school performance come from the annual education census, which provides information on approval, drop-out, and age-distortion rates. The third component has been successful. A consultant was contracted to proposed teaching guidelines, organizational strategies, and an action plan for the creation of a teacher training institute (now termed 8 At appraisal, all schools with critical IDEB scores were eligible for direct funding from MEC in support of their plans, through a competitive review process. SEEDUC also intended to focus priority attention on the schools with critical IDEB scores, but was encouraging all schools to engage in the PDE planning process. 56

62 Escola SEEDUC. This contract was successfully completed. Its recommendations have been adopted and are reflected in structure and course offerings of the Escola SEEDUC. Over the course of project implementation, several other activities in the education sector were proposed. Some were ultimately dropped. These included: (1) an analysis of the determinants of student performance in core subjects (Portuguese, math, natural and human sciences), to be followed by the development of relevant indicators and tests; and (2) an analysis of the impact of two existing teacher training programs (Programa de Formação Continuada de Docentes.) TORs for both activities were prepared and the studies designed. Due to the suspension of the project in August 2016, however, the work has not yet been undertaken. A third exercise-an inventory of skills in the education secretariat, the regional offices, and individual schools with the aim of better matching skills to needs was also initiated. It was later transferred to the TALII project, where it was completed. The original PAD for this operation proposed three indicators to monitor progress in the education component: (1) the number of schools using SAERJ results to prepare their school- improvement plans, (2) the number of schools with annual learning improvement targets agreed with SEEDUC, and (3) three sub-indicators of the use of teacher training instruments offered by the teacher training institute: the number of downloads from the institute s website, the number of teachers who have taken online training courses and number who have taken face-to-face training courses at the institute. These were all dropped at the time of the second restructuring and replaced by a new indicator: the percentage of public servants in strategic positions of the Secretariat of Education receiving performance evaluations. Performance against this indicator has been impressive reaching 34 percent of the relevant staff in (The percentage has since declined slightly. As at end-2016, it stood at 26 percent. The efforts to extend performance evaluation have not been abandoned, but were negatively affected by the student invasion of the Secretariat. The State adopted a new plan at the beginning of 2011 with the aim of improving learning performance and establishing a meritocratic selection process for key functions such as the regional directors that have traditionally been captured by the State s Assembly. The program provides bonuses to school units and Regional Boards that meet or exceed annual improvement goals. In order to assess goals, Rio de Janeiro, like Minas Gerais and Pernambuco had done before, adopted an annual version of the IDEB evaluation and three other evaluations. As well, a career plan for regional directors and principals was established and a competitive recruitment process was made mandatory to fill those positions. By the end of 2012, all 28 regional directors and 172 school principals had been hired through a competitive selection process, despite strong opposition of legislators. Permeating the reform agenda was a strong focus on data and school management. To allow school directors to focus on education quality, young, information technology-savvy management specialists were assigned to each school, which improved rapidly the timeliness of school data reporting and budget management. The Secretary established a routine meeting that brought the entire state management team together every six months to review progress and collectively address problems. The new model put priority on assessing the performance of, and a developing a career plan for, every superintendent and principal. Isolating the overall impact of the specific measures supported by the TAL is difficult. Yet, Federal survey data (IDEB) indicate broad improvements in educational performance over the period of project 57

63 implementation. The average IDEB score of schools operated by the state government increased significantly between 2009 and 2015 (from 2.8 to 3.7). 9 The total number of schools with an IDEB lower than 2.9 decreased from 429 in 2009 to 64 in The 30 percent worst performing middle schools observed an increase in their IDEB score twice as large as rest of the schools (of 1.05 points versus 0.42). The approval rate in middle school improved from 54.5 percent in 2009 to 78.7 percent in 2015 in the worst performing schools. The gap between the worst performing schools was reduced from 17 percentage points to 7 in the same period. Between 2015 and 2009, dropout rates decreased 22.6 percent in the schools that had dropout rates higher than 20.6 percent in 2009, while the rest of the schools observed a decrease of 6.9 percent. The number of secondary schools with a high dropout rate (higher than 22.6 percent) declined from 228 in 2009 to 21 in The approval rate in secondary school improved from 64 percent in 2009 to 82 percent in 2015 (dropping 75 in 2016, largely due to the extended student strike) in the worst performing schools. The gap between the worst performing schools was reduced from 17 percentage points to 3 in the same period. The State achieved one of Brazil s most remarkable turn-arounds in the education sector, largely because of the targeted and decisive interventions in the worst performing schools. After showing no change in three rounds of IDEB, the State recorded a sizable increase of 14 percent in 2011 and 12.5 percent in As a result, Rio de Janeiro rose from the 26 th position in the national ranking of IDEB 2009 to 6 th place in There is a clear discontinuity that is concomitant to the introduction of various reforms after The national average IDEB score for public secondary schools has improved at an average of 4.3 percent every two years since the index was adopted in However, between 2007 and 2013, Goiás, Pernambuco, and Rio de Janeiro had much larger improvements in their scores than other states that had similar scores at the beginning of the period. Many observed no change or declining scores. While results can be partially attributed to the TAL, other influences were at work. Some were the result of other Bank operations, including the series of DPLs. As described in Annex VII, the first DPL (2010) included two prior actions in the education sector: (1) the implementation of a special accelerated learning program to reduce age-grade distortions in basic and secondary education and (2) the implementation of new school management information system. (It should be noted that the second of these the school management information system required the technical assistance which was provided through the TAL). The third DPL (2013) required the establishment of annual school level targets for improving student learning/graduation rates, based on the data from the new management information system (again drawing of TAL-financed technical assistance). Perhaps most importantly, the third DPL required the adoption of a merit-based selection process for regional and school directors a critical departure from the previous practice. The remarkable improvement in education performance from 2009 to 2015 is also due to a factor outside the World Bank s purview: the presence of a highly competent and motivated Secretary of Education throughout most of the implementation period. This is considered, anecdotally, to have been the sine qua non of the improvements in the state s education system. Overall, this component is rated moderately satisfactory. The monitoring system has been set up and has been used in the preparation of school improvement plans. Although the annual learning outcome test SAERJ which was one of the cornerstones of the monitoring system has been interrupted, it can be readily 9 The IDEB score is a composite index that reflects student performance on standardized tests (in Portuguese and math) and graduation rates. Similar statistics are reported in the state s Management and Public Policy Report: Education (2014). 58

64 revived once political conditions improve and/or with the implementation of the annual IDEB. Reforms in the teacher training institute have also succeeded. Component 3. Health Management and Information Systems This component was intended to support state efforts to improve health care services provided by state institutions. In its initial form, it had four subcomponents. Subcomponent 3a. Accreditation for UPAs. The first two subcomponents focused on the State s Urgent Care Facilities (Unidades de Pronto Atendimento or UPAs, see Box 1.) To encourage quality improvements in UPAs, Subcomponent 3a would help the GORJ develop an accreditation system for UPAs and carry out the first-round accreditations. Subcomponent 3b. Cost Accounting Data for UPAs. This subcomponent would develop a cost accounting system for UPAs, in order to help them control costs. The TAL would finance two phases of the system: (a) a detailed specification of the functional and technical requirements of the system; and (ii) the production and installation of the core modules in at least 10 UPAs. Box 1: UPAs and PAHI: Definitions UPAs: In an effort to improve health outcomes and improve access to health services by the poor, the state government has GORJ established 24-hour health care facilities in certain low income areas. These are known as UPAs (Unidades de Pronto Atendimento). UPAs are ambulatory care facilities equipped to perform low- and medium-complexity surgeries as well as to treat some severe cases. Each UPA has general medicine, pediatric, orthopedic and dental clinics, in addition to an X-Ray laboratory. They have observation beds for adults and children and a Red Room to stabilize patients with grave medical conditions before transfer to a hospital. Most traditional non-hospital health units provided services only during working hours, making it difficult for poor families that faced urgent health needs at other times of the day. Rio s first UPA was established in May 2007 in Favela da Maré, one of the largest slums in the municipality of Rio de Janeiro. Since that time, 21 additional UPAs had been established at the time of appraisal. PAHI: Under Brazil s publicly-funded health care system (SUS), the state government is primarily responsible for providing tertiary (hospital-level) health care. (Basic primary health care is the responsibility of municipal governments.) Nevertheless, there are hospitals owned by municipalities and philanthropic organizations within the state. In order to improve the quality of health care they provide; the State provides grants under a program called PAHI (Programa de Apoio aos Hospitais do Interior). Only hospitals located in municipalities with fewer than 110,000 residents are eligible. Rural municipalities without hospitals are also eligible to receive the grants, provided they refer patients to hospitals located in other municipalities that belong to the Program. The PAHI grant has two parts: (i) a fixed component that is scaled according to the size and characteristics of the hospital; and (ii) a variable component based on performance benchmarks. Subcomponent 3c. Medical Supply-Chain Management. This subcomponent was to focus on reducing waste in the distribution of medical supplies in all state health institutions. All stages in the supply chain 59

65 process were to be addressed, from requisition, budget authorization, purchase, to distribution to the final service delivery unit. The TAL would help the state health secretariat design and develop an improved supply-chain system, including information systems and warehouse management. Subcomponent 3d. Reforming the Program of Support for Hospitals in the Interior (PAHI). As described in Box, the state government provides grants to improve the quality of health care in small hospitals owned by municipalities. In the years prior to appraisal, the state government had substantially increased its funding for this program. However, it did not have clear evidence of the impact that was being achieved. This component of the TAL would help the State health secretariat develop a monitoring system to track the use and impact of these PAHI grants and to develop proposals to improve the effectiveness of its performance incentives. The health component, like the education component, was revised during the second restructuring of the loan. To accommodate the costs of the new disaster management component, subcomponents 3a and 3b were dropped from this project and shifted to TAL II. The first activity was eventually dropped as all UPAs had been accredited. The second was dropped as well. Subcomponent 3c (gestão da cadeia de suprimentos) was retained, in the form of a consulting contract calling for the documentation of the existing system, the design of a new model, and the monitoring (acompanhamento) and implementation of proposed improvements to supply management. The resulting report had to be revised, however, in part because the relevant agencies did not initially provide the required information. After a subsequent scandal in which ten tons of expired drugs had to be destroyed the consultancy firm provided significant assistance in finding an alternative arrangement for stock management. In any event, the volume of medical supplies to be managed under the new system has been sharply reduced. Many of the UPAs have now been municipalized and the management of state hospitals has been turned over to social organizations (OSs.) Both handle their own medical supplies. As a result, the value of supplies managed by the state shrank from RS million in 2013 to only BRL 308 million in This activity was to have been monitored on the basis of the percentage of unmet requests for hospital supplies by the state hospital "Rocha Faria". 10 The indicator was later dropped because that particular hospital and those of a similar size were municipalized. Component 3d was also retained. It was to be executed through a contract calling for the development and implantation of a management information system to support the PAHI program (SIGMIS). Inter alia, the contract called for designing and implementing a system for gathering the relevant data, integrating that data into a common data base, defining performance indicators and agreeing upon targets with the relevant municipalities and hospitals. The contract was successfully completed and the new system is in use. Seventy-seven hospitals were using the system (as of 2015), exceeding the target for that year (50) and the final target: 70. The success of component 3d may be due to its links to the DPLs. The first DPL required the establishment of a performance-based transfer system between the State and municipalities based on improvements in mother-child health and hospital care to create incentives for municipalities to deliver more efficient and 10 This activity was not to be monitored in the original version of the PAD. The indicator first appears in RRI 2015-II. 60

66 higher quality health care services. The third DPL required the evaluation of mechanisms for transferring bonuses/incentives to municipalities/municipal hospitals and regional state hospitals. Overall, the performance of this component, in its revised form, may be considered moderately satisfactory. On the positive side, the system for monitoring the performance of PAHIs has been successfully implemented. On the negative side, efforts to improve medical supply management have not entirely succeeded an outcome mitigated by the new government-wide initiatives that the State Secretariat of Finance is implementing to increase the efficiency of procurement. Component 4. Disaster Management This component had two subcomponents. The first was a review of the technical/scientific organizations involved in risk management in Rio and propose improvements, including an action plan for their implementing. Although the contract for this work was completed, the report was not fully adapted to the Rio context. It did, however, spur the three key state agencies INEA (environmental protection); Civil Defense, and the Mineral Resources Department (whose geological survey is responsible for monitoring landslide risks) to coordinate their activities and take action. With its own resources and federal grants, the State has now installed 180 sirens in the 16 most vulnerable municipalities. The second subcomponent called for the installation of two meteorological radars. The radars were successfully installed. The State s fiscal difficulties are jeopardizing their use however. The supplier s guarantee on the radars is contingent upon preventive maintenance and the state only has one staff left who is capable of performing it. There are also concerns over the technical capacity of state personnel to operate system in future. Data from the radars must be interpreted and analyzed in conjunction with data from other sources (e.g., stream gauges) to assess the risk of landslides and flooding. The contract for the purchase of the radars included a turnkey-provision and assisted operation for the initial six months. The State has difficulty hiring permanent staff to assume these functions because 24/7 monitoring is required. The State is currently employing contractors to carry out the work in the night and weekend shifts. They are being paid from the revenues that INEA collects from environmental fines. Progress on this component was monitored on the basis of the percentage of alarms that turned out to be false (measured, according to IRR 2016-I, as the proportion of alerta notifications that did not progress to the occurrence of a flood). The figure was 52 percent in 2015, down from 56 percent in State staff believe that this indicator is not very meaningful, however. What matters is the amount of time the warning system gives to vulnerable populations. But what is technically feasible varies by watershed: in mountainous areas, even the best monitoring system cannot give much warning. A better indicator would have been the length of time required to give advance warning in each watershed. This component is nevertheless considered successful. Despite the weakness of the consultant s report, coordination among the agencies involved in disaster management has much improved. And despite the uncertainties associated with its funding, the radar-based warning system is up and running. Component 5. Project Management Additional funds for overall project management were included in the second restructuring of this project. This covered goods, consulting and non-consulting services and training. As of mid-2016, six activities had 61

67 been successfully completed, of which the largest (BRL 159,000) was with a travel agency. Another three contracts were in various stages of execution. 62

68 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Lorena Vinuela, Roland N. Clarke Sinue Aliram De Souza Miguel-Santiago da Silva Oliveira Gunars H. Platais Alberto Coelho Gomes Costa Ezau Pontes Leandro Oliveira Costa Andre Loureiro Frederico Ferreira Fonseca Pedroso Courtney Price Ivins Task Team Leader(s) Procurement Specialist(s) Financial Management Specialist Environmental Safeguards Specialist Social Safeguards Specialist Team Member Team Member Team Member Team Member Team Member B. STAFF TIME AND COST Stage of Project Cycle No. of staff weeks Staff Time and Cost US$ (including travel and consultant costs) Preparation FY , FY , FY , Total , Supervision/ICR 63

69 FY , FY , FY , FY , FY , FY , FY , FY , Total ,

70 ANNEX 3. PROJECT COST BY COMPONENT Components 1: Core Government Systems and Management Tools for Improved Performance and Efficiency 2: Strengthening Education Management Capacity, Teacher Instructional Practice and Incentives for School Effectiveness 3: Health Management and Information Systems 4: Development of Institutional and Technological Capacity for Preventing and Performing during High-intensity Weather Events Amount at Approval (US$M) Actual at Project Closing (US$M) Percentage of Approval (US$M) : Project Management Total

71 ANNEX 4. EFFICIENCY ANALYSIS In evaluating the project s efficiency, a key question is whether the roughly US$4 million spent on consulting services was the most cost effective means of achieving the desired end: the implantation of the various management tools supported by the project. This question cannot be answered definitively. What can be said is that: (1) the consultants provided technical expertise that was not available in house; thus outside expertise was definitely required; and (2) the targets of the TA were relevant to the desired outcomes--most of the consulting assignments were tied to production of specific management tools. It can also be inferred that the costs of the consulting services were reasonable, as most of the contracts were awarded on a competitive basis in a procedure closely overseen by the Bank. All was not perfect. There were cases of consulting work did not produce all the expected results. This is true of the results-based-management contracts (total cost: US$350,000 at the 2014 exchange rate). As noted in the main text, this contract was successfully completed but the concept of results-based management was limited to a small number of secretariats. The largest consulting contract, on the other hand, did produce concrete results. This was the property inventory and regularization component (final cost US$ 2.5 million). As noted in the main text, this produced an increase in annual revenues from state property of BRL 12 million (approximately US$ 4 million.) This, of course, represents the financial rate of return to the State. A calculation of the economic rate of return would have to take into account the loss in benefits accruing to the residents of affected property (due to increased rents) over an extended period of time, suitably discounted. Whether the expenses on implementing an integrated management information system (US$ 485,000, not counting the State s subsequent contribution) turn out to be cost-effective will depend on how successful the component is in improving the various targets of intervention: internal controls, cash flow management, contracts and billing, and bank account management. As those modules are in various stages of testing and implantation, it is early to assess the impact. The direct costs to the Bank of project preparation and totaled US$ 702,000. Of this, US$ 208,000 was spent on preparation and US$495,000 on supervision. This appears to have been well spent. Extensive preparation was required to identify the most effective targets of intervention in an environment where Bank had not worked since Similarly, supervision costs reflect the extensive hands-on assistance provided by the team, not only for the implementation of the project components (assistance with preparation of TORs, etc.) but also in the form of advice to senior policy makers by Bank experts. 66

72 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS The Borrower reviewed the document and did not have major comments. The edits suggested by the Secreatriat of Finance have been incorporated in the final version. 67

73 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) 68

74 ANNEX 7. CONTEMPORANEOUS PROJECTS The Bank s involvement in public sector management in the state of Rio de Janeiro dates back to the 1998 Rio State Privatization Loan. That loan (of US$250 million) was to be disbursed in three tranches upon the privatization of several major state enterprises. Ten years later, preparations for the operation under review (TAL I) began. TAL I was originally designed as a technical assistance loan focused solely on management reform in staff (as opposed to line) functions. The request for federal guarantees or Carta Consulta for the project, dated September 2008 (Programa de Renovação e Fortalecimento da Gestão Pública do Estado do Rio de Janeiro) called for a US$24 million Bank loan, with US$ million counterpart contribution by the state government. It comprised twelve components, of which nine were to be financed by World Bank, and three by the state. The nine to be financed by World Bank were: (1) consultant services to redesign the system of results agreement with 19 state secretariats (along with Procuradoria and Defensoria do Estado); (2) the same, for 40 autarchias and fundações; (3) training of staff in budgeting and planning, and acquisition of related hardware; (4) modernization of document processing (i.e., a shift to electronic document handling); (5) development and implementation of a data bank of socioeconomic data in SEPLAG, along with related hardware; (6) implantation of a system for managing the estimated 8000 properties owned by the state, (7) analysis and redesign of system for managing the processing of certain legal disputes (contenciosos fiscais); (8) development and implantation of a system of pension management consistent with federal legislation governing the organization of (subnational) pension funds; and (9) project management. The Carta Consulta also contained a list of actions that the state promised to undertake, including reductions in the number of secretariats, reductions in staffing levels through voluntary separations, and reductions in the number of political appointees (cargos comissionados). Between the submission of the Carta Consulta and Board presentation, the project was significantly altered. The added activities were largely intended to support prior actions in the education and health sectors required by the first DPL (in this series), the Fiscal Sustainability, Human Development and Competitiveness Development Policy Loan (US$ 485 million, December 2009). DPL I included several prior actions related to public sector management, including the capitalization of the social security system for state public servants and changes in the budget process. But it also included conditionality in the education and health sectors. In education sector, DPL I included two prior actions: (1) the implementation of a special accelerated learning program to reduce age-grade distortions in basic and secondary education 11 and (2) the implementation of new school management information system. The second of these the school 11 The specific wording of the prior actions was: (1) implementation of accelerated learning program to reduce agegrade distortion in a significant subset of state schools and development and (2) implementation of a new school management information system to increase accountability among teachers and students and efficiency of resource use.. Note that although municipalities are responsible for most basic (primary) schools the state government also operates some of its own. The prior action was intended to cover 7,000 primary and 6,000 secondary students in the state system. 69

75 management information system required technical assistance. According to the PAD for DPLI, the Secretariat of Education planned to introduce performance targets for individual schools over the next two school years. With support from TAL I, the State was to take the first step in developing a system to communicate the State s annual learning assessment (SAERJ) results to individual schools on a timely basis and to provide technical support in analyzing results and planning improvement programs. In the health sector, DPL I required the expansion of a more efficient model of provision of urgent and emergency care in poor areas of the Rio de Janeiro Metropolitan Areas and the establishment of a performance-based transfer system between the State and municipalities based on improvements in mother-child health and hospital care to create incentives for municipalities to deliver more efficient and higher quality health care services. DPL I was followed by DPL II, the Metropolitan Urban and Housing project (US$ 485 million, 2011). This largely focused on reforms in the state s low-income housing and transport policies. It was followed by DPL III, the Fiscal Efficiency for Quality of Public Service Delivery project (USD 300 million, 2012). This focused on tax administration and investment planning but also support key sectoral reforms. In the education sector, it required the adoption of merit based selection process for school directors and the establishment of annual school level targets for improving student learning/graduation rates and a bonus pay system. In the health sector, it required the state to (1) establish a framework to assess and contract social organizations to manage public hospitals and health units, and (2) evaluate mechanisms for transferring bonuses/incentives to municipalities/municipal hospitals and regional state hospitals. In 2013, the Bank approved DPL IV, Enhancing Public Management for Service Delivery (500 million) which focused on financial management and transport reforms. Finally, in 2014, the Bank approved a second technical assistance loan, Strengthening Public Management and Integrated Territorial Development-TAL-II (US$ 48 million). As described in the PAD, TAL II consisted of three major components (in addition to a project management component; one aimed at strengthening metropolitan management one aimed at improving living conditions and the third aimed at strengthening core finance and taxation functions.) 70

76 Table A.5: Rio de Janeiro Operations, Approval Date CPS Priority Areas Fiscal Consolidation and Public Financial Management Tax Administration Public Investment Private Sector Education DPL 1 Fiscal Sustainability, Human Development and Competitiveness (US$ 485 million) - Capitalization of Social Security System for State public servants - Reorganization of budget processes - Strengthening of the Revenue Secretariat. - Adoption of electronic invoices. - Streamlining of business registration process - Implementation of learning program to reduce of age-grade distortion in basic education. - Implementation of new school management information program TAL (P106768) Public Sector Modernization (US$ 18.6 million) - Adoption of Public Sector Performance Indicators - Real Estate Property Management System - Social Security Management System - Building of SEEDUC Technical Capacity for Studies and Evidencebased Policy. - School-level Performance Targets for State Schools. DPL 2 Metropolitan Urban and Housing (US$ 485 million) DPL 3 (P106465) Fiscal Efficiency for Quality of Public Service Delivery (US$ 300 million) - Piloting of a methodology for costing public policy programs - Restructuring of the internal audit agency. - Strengthening of synchronized taxpayer registration system. - Adoption of standards to improve collection of State taxes. - Implementation of a results-based management system for tax administration. Adoption of a methodology and procedures for evaluating, selecting and approving proposals for large public investment projects. - Adoption of a meritbased selection process for regional and school directors. - Establishment of annual school-level targets for improving student learning and graduation DPL 4 (P147695) Enhancing Public Management For Service Delivery (US$ 500 million) - Treasury s operational risk policy - Adoption of budget preparation practices to improve its accountability and fiscal discipline - Adoption of financial and debt management tools. TAL II (P126735) Strengthening Public Management and Integrated Territorial Development (US$ 48 million) - Strengthening core taxation functions - Developing In-Service Teacher Training Network. - Mapping of demographic and infrastructure needs. 71

77 Health Housing and Land Management Urban Management - Adoption of national standard practices in 24- hour operating Urgent and Emergency Care Units (UPAs). - Establishment of a system of performancebased transfers from the State to municipalities - Information System for Health Supply Chain Management Evaluation of Financial Incentive Programs for Municipalities - Enhancement of lowincome housing policy and financing mechanisms. - Strengthening land management and titling. - Enhancement of the governance of the State Housing Fund (FEHIS). - Strengthening of ITERJ land tenure regularization capacity - Restructuring of the Secretariat of Social Assistance and Human Rights. - Strengthening metropolitan management through improved integration and coordination in urban development and transport. - Establishment of a Directive Committee for Metropolitan Strategy and an Integrated Program. - Targeted and integrated social development programs to the urban poor. rates and a bonus pay system for schools. - Framework to assess and contract social organizations to manage public hospitals and health units. - Evaluation mechanisms to transfer bonuses and monetary incentives (i) to municipalities and municipal hospitals and (ii) to regional State hospitals. - Cost-Accounting Methodology for UPAs and Hospitals. - Improving living conditions in the most vulnerable social settings - Strengthening metropolitan management through integration and coordination in urban development, housing, transport, environment and disaster risk management 72

78 Transport - Establishment of the UPP Social Program, on a pilot basis, in the State slums. - Creation of the Single Ticket (Bilhete Único) for public transportation in the RJMR. - Adoption of the Urban Transport Master Plan (PDTU) and a stakeholder mechanism to accompany implementation. - Implementation of a policy to physically integrate municipal and Statemanaged mass transit systems. Adoption of a new policy to promote nonmotorized transport. Adoption of a policy for competitively awarding licenses for bus routes, instituting a resultsbased management system for concessions, and increase transparency on bus service performance. DRM - Radar system. - Establishment of a State - DRM. disaster risk policy. Natural Resource Management - Strengthening of the Institute of Environment to capacity to protect environmental assets. Improvement of effective watershed management and decentralization. PDO Rating Satisfactory Satisfactory Moderately Satisfactory Moderately Satisfactory Unsatisfactory Unsatisfactory 73

79 percentage The World Bank ANNEX 8. FISCAL CONTEXT Rio de Janeiro is Brazil s third largest state in terms of population (16 million people; roughly 8 percent of Brazil s population), and is second only to São Paulo in terms of state gross domestic product (GDP). In 2010, the year the project was appraised, the state s GDP was US$250 billion, or about 12 percent of the national GDP. Over the following six years, it grew by only 2.5 percent in real terms. Rio s economy is heavily dependent on the services sector. Services accounted for about 70 percent of state GDP in Government services (administration, education, public health and public security) account for about one-quarter of value added in the services sector, followed (remotely) by professional and technical services. The industrial sectors account for the remaining 30 percent of GDP. Half of industrial value added is contributed by the oil sector. As shown in Figure 2, Rio s economy grew robustly during the period just prior to appraisal, averaging 3.5 percent annually from 2005 to Figure 2: Trends in RJ GDP (Percentage) Source: Secretariat of Finance of Rio de Janeiro. At the time of appraisal, the finances of the state government appeared to be in reasonable shape. In 2007 and 2008, the state ran operational surpluses 13 equal to four percent and six percent of gross current revenues, respectively. In 2009, however, the State ran an operational deficit equal to three percent of gross revenues. Nevertheless, the appraisal team was optimistic about the State s finances. The Program Appraisal Document (PAD) noted that the fiscal situation of the state has been strong in recent years and that the State had managed to meet all the federal Fiscal Responsibility Law indicators since According to the PAD, the Government had already taken 12 Figures are for Source: CEPERJ 13 In this calculation, the operational balance is defined as gross current revenues less gross expenditures other than amortization. Gross revenues include tax revenues prior to deductions for FUNDEB and constitutional transfers to municipios. Gross expenditures include expenditures on these two items. 74

80 some important steps to cut costs, focusing particularly on the wage bill. These included reductions in staffing and removal of ghost workers from the payroll. The State was also in the process of installing a biometric registry for all public employees and pensioners. The fiscal analysis recognized important vulnerabilities but assumed they could be contained. The State s dependency on oil royalties was acknowledged (mostly as a risk coming from royalties reform, not from the oil price as materialized). The PAD s projections of State and national economic growth were also optimistic, but were broadly in line with what others (markets, International Monetary Fund) were thinking about Brazil at that time. The fiscal analysis undertaken for a complementary DPL Fiscal Sustainability, Human Development and Competitiveness Development Policy Loan (approved February, 2010) was somewhat less sanguine. It noted that during the second quarter of 2009, revenue collection had deteriorated and was limiting the Government s ability to keep pace with planned investments and other expenditures. The DPL was intended to partially compensate for this fall in revenues and help sustain the implementation of the Government s development program. The latest fiscal sustainability analysis that the Bank conducted was in 2013 in the context of the preparation of the fourth DPL. That analysis flagged the doubling of personal expenditures in five years ( ) a warning sign. The negative primary balance in a context of already large debt service obligations (155 percent debt of NCR, projected to reach 194 percent of NCR) was also a reason for concern. Nevertheless, the fiscal team assumed that in the case of adverse shocks the State would contain expenditures, even though it should have been clear that containing expenditures (other than investment) would be difficult given the high rigidity of spending. Expenditures were projected to grow at very moderate rates, well below the historical trends in personnel and other current expenditures. They also ignored well known spending pressures (World Cup, Olympics, protests, and so on). Nevertheless, the fiscal situation deteriorated dramatically after the operation was approved. The state s gross operating balance fell to nearly six percent of net current revenues in Some of the factors driving the deterioration could not have been anticipated (oil price collapse, fallout from Petrobras corruption investigations, and prolonged nationwide recession and political crisis). Comparing the DSA projections with the fiscal outcomes in recent years (Table 6) shows that current expenditures were underestimated in 2013 and capital spending was underestimated over the entire forecasting period. More importantly, revenue was overestimated; due to the dramatic fall in revenues beginning in

81 Table 6: Comparing the DSA Projections with Fiscal Outcomes DSA Projections Outcomes I. Revenue Taxes ICMS IPVA Others Social Contributions Transfers Current Transfers Capital Transfers Other Current Revenues Interest Revenues Net Current Revenues II. Expenses Compensation of Employees Interest Payments Other Current Expenditures Legal Constitutional Transfers Goods and Services III. Gross Operating Balance (I-II) % of NCR IV. Transactions in Financial and Non-Financial Assets Investment in Non-Financial Assets Investment in Financial Assets Total Expenditure V. Net Lending/ Borrowing (III-IV) VI. Primary balance % of NCR VII. Transactions in Financial Assets Asset Sales New Loans Amortizations, net VIII. Gross Financing Needs % of NCR VIII. Overall Balance (VI + VII) *Figures in DSA and outcomes are not strictly comparable, since outcomes are based on public data (RREO and DCA) while DSA was built based on data submitted directly by the government. Additionally, accounting changes to tax revenue transferred to municipalities means that these transfers had to be estimated for the updated table. 76

82 ANNEX 8. STATE OF RIO DE JANEIRO MAP 77