IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-76220) ON A LOAN IN THE AMOUNT OF US$4.4 MILLION TO THE UNITED MEXICAN STATES FOR A

Transcription

1 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Document of The World Bank Report No: ICR IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-76220) ON A LOAN IN THE AMOUNT OF US$4.4 MILLION TO THE UNITED MEXICAN STATES FOR A RESULTS-BASED MANAGEMENT AND BUDGETING PROJECT June 30, 2014 Poverty Reduction and Economic Management Colombia and Mexico Country Management Unit Latin America and the Caribbean Region

2 CURRENCY EQUIVALENTS (Exchange Rate Effective April 10, 2014) Currency Unit=Mexican Pesos US$ 1.00 = MXP $13.05 MXP $1 = US$ FISCAL YEAR January 1 December 31 ABBREVIATIONS AND ACRONYMS CONEVAL GoM IADB IEG IFAI IT M&E NAFIN NDP/PND OECD PBR PFM PGCM PI PMG PSBR RBB SE SED SFP SHCP SIDAFF SIGAIF SISED National Council for the Evaluation of Social Development Policy (Consejo Nacional de Evaluación de la Política de Desarrollo Social) Government of Mexico Inter-American Development Bank Independent Evaluation Group Federal Institute for Access to Public Information and Data Protection (Instituto Federal de Acceso a la Información y Protección de Datos) Information Technology Monitoring and Evaluation Development Banking Institution (Financial Agent: Nacional Financiera, S.N.C.) National Development Plan (Plan Nacional de Desarollo) Organization for Economic Co-operation and Development Results Based-Budgeting (Presupuesto Basado en Resultados) Public Financial Management Close and Modern Government Program (Programa de Gobierno Cercano y Moderno) Performance Information Management Improvement Program (Programa de Mejora de la Gestión) Public Sector Borrowing Requirements Results-based budgeting Undersecretary of Expenditures (Subsecretaría de Egresos) Performance Evaluation System (Sistema de Evaluacion del Desempeno) Ministry of Public Administration (Secretaría de la Función Pública) Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito Público) Integrated Federal Financial Management System (Sistema Integral de Administración Financiera Federal) Harmonized Governmental Financial Information System (Sistema Gubernamental Armonizado de Información Financiera) Performance Evaluation Program Information System (Sistema de Información para el Sistema de Evaluación del Desempeño) Vice President: Jorge Familiar Country Director: Gloria M. Grandolini Sector Manager: Arturo Herrera Gutierrez Project Team Leader: Pedro Arizti ICR Team Leader: Pedro Arizti

3 MEXICO Results-based Management and Budgeting Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design Key Factors Affecting Implementation and Outcomes Assessment of Outcomes Assessment of Risk to Development Outcome Assessment of Bank and Borrower Performance Lessons Learned Comments on Issues Raised by Borrower/Implementing Agencies/Partners Annex 1. Project Costs and Financing Annex 2. Outputs by Component Annex 3. Economic and Financial Analysis Annex 4. Bank Lending and Implementation Support/Supervision Processes Annex 5. Beneficiary Survey Results Annex 6. Stakeholder Workshop Report and Results Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Annex 9. List of Supporting Documents MAP

4 A. Basic Information Country: Mexico Project Name: MX Results-based Management and Budgeting Project ID: P L/C/TF Number(s): IBRD ICR Date: 06/30/2014 ICR Type: Core ICR Lending Instrument: TAL Borrower: Original Total Commitment: Revised Amount: US$ 4.4M Environmental Category: C GOVERNMENT OF MEXICO US$ 17.2M Disbursed Amount: US$ 0.37M Implementing Agencies: Secretaría de Hacienda y Crédito Publico (SHCP) Cofinanciers and Other External Partners: B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 09/20/2007 Effectiveness: 03/19/ /19/2010 Appraisal: 10/02/2008 Restructuring(s): 11/15/2012 Approval: 12/11/2008 Mid-term Review: 02/06/2012 Closing: 12/31/ /31/2013 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Borrower Performance: Moderately Unsatisfactory Moderate Moderately Unsatisfactory Moderately Unsatisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Moderately Government: Unsatisfactory Unsatisfactory Quality of Supervision: Moderately Implementing Moderately Unsatisfactory Agency/Agencies: Unsatisfactory Overall Bank Performance: Moderately Unsatisfactory Overall Borrower Performance: Moderately Unsatisfactory

5 C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Performance (if any) Potential Problem Project at any time (Yes/No): Problem Project at any time (Yes/No): DO rating before Closing/Inactive status: No Yes Moderately Unsatisfactory Quality at Entry (QEA): Quality of Supervision (QSA): None None Rating D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration Theme Code (as % of total Bank financing) Managing for development results Public expenditure, financial management and procurement E. Bank Staff Positions At ICR At Approval Vice President: Jorge Familiar Pamela Cox Country Director: Gloria M. Grandolini Axel van Trotsenburg Sector Manager: Arturo Herrera Gutierrez Nicholas Paul Manning Project Team Leader: Pedro Arizti Roberto Adrian Senderowitsch ICR Team Leader: ICR Primary Author: Pedro Arizti Azul Del Villar F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The development objective of the project is to establish that federal departments and agencies in Mexico provide decision makers and the public with rigorous, timely, user-friendly information on the efficiency and effectiveness of government organizations and program expenditures. Revised Project Development Objectives (as approved by original approving authority) The objective of the project is to strengthen the Ministry of Finance s (Secretaría de Hacienda y Crédito Público, SHCP) capacity to (i) allow the use of standardized performance information of priority public programs during budget preparation; (ii) generate harmonized subnational fiscal

6 information; and (iii) disseminate budget performance information to the Borrower's citizens on a regular basis. (a) PDO Indicator(s) Indicator Indicator 1 : Value (quantitative or qualitative) Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Performance information using a performance framework is generated and used during budget request discussions between line secretariats and SHCP. Performance information (PI) is used by SHCP during Performance budget information (PI) Performance is not widely preparation in presented in a used by line secretariats conversation timely fashion to in their budget requests. with line Congress in a Budget preparation ministries. format that allows discussions are mainly SHCP interpretation and focused on inputs and are uses meets their needs. not based on standardized standardized Content includes performance information performance outcome and output that allow to measure information measures. Line relevance, efficiency of that allow to ministries send programs. measure budget requests to relevance, SHCP with PI. efficiency of effectiveness of programs. Date achieved 10/03/ /31/ /31/2013 Comments (incl. % Achieved achievement) Indicator 2 : An active network and a portal to inform civil society at large on a regular basis about budget performance increasing transparency and accountability to citizens. There is a portal containing Portal containing PI financial and Budget performance on budget non-financial information is readily performance is Value performance available to civil society available to citizens quantitative or information via the new portal but has + Citizen s Budget Qualitative) about the not been expanded to all + Citizen s Public budget and its levels of Government. Accounts (Cuenta process Pública) available for citizens, and the

7 information has been expanded to all the levels of Government. Date achieved 10/03/ /31/ /31/2013 Comments (incl. % Achieved, the portal site address is: achievement) Indicator 3 : Percent of public programs with improved strategic design and reporting Value quantitative or Qualitative) In 2008 the percentage of public programs using performance information was 28%. For the 2013 budget preparation the percentage of programs using performance information is set to be over 75%. The majority of the programs have a more robust design across key dimensions (objectives, indicators, baselines, links to strategies) enabling better reporting and evaluation. (832 programs out of 984) Date achieved 10/03/ /31/ /31/2013 Comments (incl. % Achieved at 84.5% achievement) Indicator 4 : Percent of municipalities that have a simplified financial management system installed and working. Around 80% of The Accounting approximately Unit changed Value 1,200 small priorities to focus quantitative or 0 municipalities on developing the Qualitative) (of total Government approximately Accounting System 2,500 (SCG) first. municipalities) Date achieved 10/03/ /31/ /31/2013 Comments (incl. % Not achieved achievement)

8 (b) Intermediate Outcome Indicator(s) Indicator Indicator 1 : Value (quantitative or Qualitative) Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Number of municipalities that have been trained in the use of the new municipal simplified FMIS. Pilot of Harmonized Governmental Financial Information System (SIGAIF) installed Around 80% in some of municipalities, not approximately operational though. 1,200 small The Accounting municipalities Unit changed priorities to focus on developing the Government Accounting System (SCG) first. Date achieved 10/03/ /31/ /31/2013 Comments (incl. % Not achieved achievement) Indicator 2 : Functional design developed for the amplified FMIS that will cover the decentralized institutions in the Government. Value (quantitative or Qualitative) Non existent SIDAFF has a module designed for decentralized institutions. The Accounting Unit changed priorities to focus on developing the Government Accounting System first. Date achieved 10/03/ /31/ /31/2013 Comments (incl. % Not achieved achievement) Indicator 3 : Design of a conceptual model for integration of financial information in municipalities, states and federation. SIDAFF has a The Accounting module Unit changed Value designed for priorities to focus (quantitative Non existent integrating all on developing the or Qualitative) three levels of Government Government. Accounting System

9 first. Date achieved 10/03/ /31/ /31/2013 Comments (incl. % Not achieved achievement) Indicator 4 : Design and dissemination of a number of strategic evaluations (e.g. security policy, water policy, education). Value (quantitative or Qualitative) Non existent Strategic social and non-social evaluations carried out and disseminated through the portal. Three strategic evaluations were published in the Budget Transparency Portal in April Date achieved 10/03/ /31/ /30/2014 Comments (incl. % Partially achieved. achievement) Indicator 5 : Design of terms of reference for non-social program evaluations. Value (quantitative or Qualitative) Value (quantitative or Qualitative) Non existent Non existent Terms of Reference (ToR) for nonsocial program evaluations available. SHCP has advised on adapting ToR to conduct program evaluations other than social, and on revising the ToR of the Consistency and Results evaluations as well as the Cost Effectiveness for non social programs Date achieved 10/03/ /31/ /31/2013 Comments (incl. % Not achieved achievement) Indicator 6 : Development of a methodology and guidelines for better social program focalization. Methodology and guidelines for better social program focalization available. Not achieved Date achieved 10/03/ /31/ /31/2013 Comments (incl. % Not achieved achievement) Indicator 7 : Public programs portfolio review to identify program delays and development of methodology for ex ante screening.

10 Value (quantitative or Qualitative) Non existent Public programs portfolio review carried out and methodology for ex ante screening available. Not achieved Date achieved 10/03/ /31/ /31/2013 Comments (incl. % Not achieved achievement) Indicator 8 : Design and dissemination of an evaluation for the SED SED Important changes evaluation will take place Value carried out and fostered by the (quantitative Non existent disseminated PND, regarding the or Qualitative) through the SED s scope (i.e. portal. PGCM) Date achieved 10/03/ /31/ /31/2013 Comments (incl. % Not achieved achievement) G. Ratings of Project Performance in ISRs No. Actual Date ISR DO IP Disbursements Archived (US$ millions) 1 01/27/2009 Satisfactory Satisfactory /18/2009 Moderately Satisfactory Moderately Unsatisfactory /11/2010 Moderately Satisfactory Moderately Unsatisfactory /14/2010 Moderately Satisfactory Moderately Unsatisfactory /26/2011 Moderately Satisfactory Moderately Unsatisfactory /02/2011 Moderately Satisfactory Moderately Unsatisfactory /16/2012 Moderately Satisfactory Moderately Unsatisfactory /07/2013 Moderately Satisfactory Moderately Unsatisfactory /01/2013 Moderately Satisfactory Moderately Unsatisfactory /07/2013 Moderately Unsatisfactory 0.21

11 Unsatisfactory H. Restructuring (if any) Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO IP Amount Disbursed at Restructuring in US$ millions 11/15/2012 Y MS MU 0.00 Reason for Restructuring & Key Changes Made The restructuring aligned the PDO to a reduced Project scope and partial loan cancellation. If PDO and/or Key Outcome Targets were formally revised (approved by the original approving body) enter ratings below: Outcome Ratings Against Original PDO/Targets Moderately Unsatisfactory Against Formally Revised PDO/Targets Moderately Unsatisfactory Overall (weighted) rating Moderately Unsatisfactory I. Disbursement Profile

12

13 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. During the period, the Mexican economy experienced a balanced and broad-based economic expansion with an average annual gross domestic product (GDP) growth rate of 3.8 percent. A vigorous global economic expansion contributed to the enhanced growth performance, as the value of exports increased at a double-digit annual average rate. Price stability also contributed to a healthy growth of domestic demand, with improved purchasing power of wages and salaries, and an expansion of domestic credit and consumer lending. 2. In this context, the fiscal reform approved in September 2007 marked a crucial step in strengthening the country s fiscal accounts. This fiscal reform enabled the Government of Mexico (GoM) to raise tax collection rates, strengthen fiscal federalism, and decrease tax evasion. Building on the improved macroeconomic and fiscal framework, the GoM focused on enhancing the quality of public expenditures throughout the budget cycle, from planning to execution and evaluation. As a result, the Government established a results-based budgeting initiative, the Performance Evaluation System (Sistema de Evaluación del Desempeño, SED), for which it sought Bank support. 3. Furthermore, the Country Partnership Strategy (CPS) for Mexico called for strengthening institutions to improve their functioning and enhancing citizen perceptions of the public sector through a set of initiatives that included a move towards results-based budgeting. The higher-level objective of the Project improving the quality of public expenditures in Mexico was therefore fully consistent with the FY08-13 CPS. The Project was also aligned with two of the primary objectives of the National Development Plan (NDP), which aimed to improve regulation, management, the processes and results of Federal Public Administration to satisfy citizen needs in relation to the provision of public goods and services; and promote and guarantee transparency, accountability, access to information, and the protection of personal information in all spheres of government Original Project Development Objectives (PDO) and Key Indicators 4. The original objective of the Project was to strengthen the Borrower s capacity (through its federal departments and agencies) to provide decision makers and the public with rigorous, timely, user-friendly information on the efficiency and effectiveness of government organizations and their program expenditures. 2 1 Mexico Country Partnership Strategy FY , March 4, 2008, Report No MX. 2 There is a slight difference between the Original PDO in the PAD and the Loan Agreement (LA). The one used in the ICR is that of the LA. 1

14 5. The original PDO indicators were: (i) Performance information presented in budget requests from line ministries to SHCP and in budget submitted to Congress; (ii) Satisfaction of Congress and civil society organizations with the performance information made available via government websites (as measured through user surveys); (iii) Percent of programs with improved strategic design and reporting 3 ; (iv) Progression of organizations through the steps (levels) established for each Management Improvement Program (Programa de Mejora de la Gestión, PMG), marking increased levels of administrative capacity and efficiency. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 6. The Project had to be restructured (level 1) in November 2012 as a result of changes in national priorities and most importantly, due to the fact that some activities had been financed and implemented using local funds, in light of the loan s start up delays and lengthy process towards effectiveness. Therefore, the Government and the Bank agreed on reducing the Project scope, partially cancelling the loan and redesigning activities to better align them with Government priorities. 7. In order to align them with a reduced Project scope and partial loan cancellation, the PDO and its key indicators and the results framework were revised. The revised PDO is to strengthen SHCP s capacity to: (i) allow the use of standardized performance information of priority public programs during budget preparation; (ii) generate harmonized sub-national fiscal information; and (iii) disseminate budget performance information to the Borrower s citizens on a regular basis. The table below illustrates the change in key indicators. Table 1. Changes in PDO indicators Original PDO-Level Indicator (PAD 2008) Revised PDO-Level Indicator (restructuring 2012) 1. Performance information presented in budget requests from line ministries to SHCP and in budget submitted to Congress. (Revised) Performance information using a performance framework is generated and used during budget request discussions between line 2. Satisfaction of Congress and civil society organizations with the performance information made available via government websites (as measured through user surveys). 3. Percent of programs with improved strategic design and reporting. secretariats and SHCP. (Revised) An active network and a portal to inform society at large on a regular basis about budget performance, increasing transparency and accountability to citizens. (Revised) Percent of public programs for which there is performance information using a standardized performance framework. 3 There is evidence from previous analytic work supported by the Bank (Technical assistance to the SHCP on October 2007, corroborated by subsequent CONEVAL analysis) that the quality of strategic design of public programs is weak along several different dimensions (objectives, indicators, link to strategies). Using equivalent dimensions, the UED unit was supposed to be able to measure overall quality over time, using that previous work as a baseline. 2

15 4. Progression of organizations through the steps (levels) established for each PMG system, marking increased levels of administrative capacity and efficiency. (Dropped original and replaced with new) Percent of municipalities that have a simplified financial management system installed and working. Source: Proposed Project Restructuring of Results-based Management and Budgeting Loan MX, Main Beneficiaries 8. The primary target groups of the Project were the users of performance information, including line ministries, Congress and the general public. These main beneficiaries did not change when the Project was restructured. 1.5 Original Components (as approved) 4 Component 1.Management Improvement Program. 9. Provision of technical assistance and training for the design and implementation of selected management improvement and austerity systems, including, inter alia: (a) assistance in the design of the institutional module of the Borrower s Programa Especial de la Mejora de la Gestión en la Administración Pública Federal ; (b) assistance for technical groups established and to be established with respect to each of said management improvement and austerity systems; and (c) capacity building for actors responsible for the implementation of said management improvement and austerity systems. Component 2. Results-Based Management and Budgeting Integrated Information Systems 10. Provision of technical assistance and training to support the development of an integrated information system, including, inter alia, the development of the necessary institutional arrangements, procedures, and an associated electronic database to: (a) receive budget and other expenditure information from different sources; (b) organize and analyze said information; and (c) produce abridged, user-friendly reports with key messages to inform various stakeholders and improve transparency of budget execution. Component 3.Financial Management Strengthening for Results-Based Budgeting 11. Provision of technical assistance and training to strengthen the quality and integrity of financial data, including, inter alia: (a) modernization of the Borrower s public accounts (Cuenta Pública) and the underlying government accounting systems to strengthen the usefulness, integrity and transparency 4 The components are presented as per the Loan Agreement. For a more detailed description of the original components see the Project Appraisal Document Report No: MX. 3

16 of public accounting records and related accounting management systems for management decision making and public accountability; and (b) strengthening of the quality and timeliness of the Borrower s budget execution reporting. Component 4. Results-Based Management and Evaluation System Consolidation 12. Provision of technical assistance and training to: (a) address government s capacity limitations to carry out, contract and use evaluations of government programs; and (b) consolidate a system to evaluate the effectiveness of federal policies and programs through, inter alia: (i) (A) creation and consolidation of a Sistema de Evaluación del Desempeño in the Borrower s government; and (B) strengthening of the alignment of the Borrower s National System for Democratic Planning components, including, Vision 2030, National Development Plan, sector strategies and budgetary programs and the strengthening of a selected set of strategic indicators based on sharper program design, and ownership building of these indicators among the Borrower s line secretariats; (ii) strengthening the generation of performance data for the analysis of the Programas Federales; and (iii) implementation of results-based management approaches at the state level. Component 5. Monitoring and Evaluation 13. Provision of technical assistance and training to SFP and SHCP to develop the necessary organizational and evaluation capacity to monitor the progress and to evaluate the implementation of the Borrower s Sistema de Evaluación del Desempeño and related institutional reforms. 1.6 Revised Components 14. The November 2012 restructuring (level 1) introduced the following changes to the components: (i) (ii) (iii) Components 1 (Management Improvement Program) and 2 (Results-Based Management and Budgeting Integrated Information Systems) were cancelled as a result of changes in Government priorities and use of own resources; Component 3 (Financial Management Strengthening for Results-Based Budgeting) continued as originally designed with some changes in activities that focused on modernizing Government accounting and public accounts, as well as harmonizing and integrating budget execution reporting; Component 4 (Results-Based Management and Evaluation System Consolidation) was revised to update Project activities with respect to consolidating the 4

17 Government evaluation system and to include a fourth sub-component according to the following description 5 : the provision of support to the Borrower s fiscal transparency initiatives including: (a) the participatory design and implementation of an integrated fiscal transparency web portal, and (b) the design of guidelines and regulations to foster fiscal transparency in the Borrower s federal, state and municipal levels of government ; (iv) Component 5 (Monitoring and Evaluation) would continue with some changes in activities including cancellation of activities to be originally implemented by SFP. The revised description 6 is as follows: Provision of technical assistance and training to SHCP to develop the necessary organizational and evaluation capacity to monitor the progress and to evaluate the implementation of the Borrower's Sistema de Evaluación del Desempeño and related institutional reforms. 1.7 Other significant changes 15. The November 2012 restructuring (level 1) introduced these additional changes: (i) (ii) (iii) (iv) the restructured Project was to be implemented solely by the SHCP through its Subscretaría de Egresos (SE) eliminating the participation of the Public Administration Ministry or Secretaría de la Función Pública, SFP; the Project financing reflected the cancellation of US$ million of the loan amount (from US$17.2 million to US$4.4 million) and revisions to the costs by component, financing plan and expenditure categories; the procurement arrangements were changed to apply the Consultant and Procurement Guidelines of May 2004 as revised October 1, 2006 and May 1, 2010 to the restructured Project; this was to enable application of the Bank Guidelines that are harmonized with the then currently authorized policies and procedures for externally funded projects in the country; and the procurement plan and implementation schedule was revised as a result of the reduction in Project scope and partial loan cancellation. 16. With respect to Project financing changes, the total Project cost was reduced from US$31.9 million to US$6.7 million (excluding front-end fee) 7. In terms of the Project s financing plan, the reduction in total Project cost reduced both the counterpart fund contribution as well as the required IBRD loan amount (from US$17.2 million to US$4.4 million including the front-end fee). See Table 2 for details. 5 Amendment to the Loan Agreement dated November 29, Ibid. 7 See Annex 1 for more details. 5

18 Table 2: Summary of Project Total Costs and Estimated Financing Plan (US$ million) As approved (PAD) After restructuring Components/Activities Local IBRD Total Local IBRD Total 1. Management improvement program (PMG) 2. Development of an Integrated Information System for Results-Based Budgeting and Management (SISED) 3. Strengthening Financial Information for the implementation of Results Based Budgeting 4. Consolidation of the Performance Evaluation System and Results Based Budgeting 5. Institutional Strengthening and Program Monitoring and Evaluation TOTAL Source: Proposed Project Restructuring of Results-based Management and Budgeting Loan MX, Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 17. The quality at entry is viewed to have been moderately unsatisfactory. During preparation, the Project benefitted from synergies that emerged from activities organized by the Bank, such as the Performance Budgeting workshop that took place in Mexico in June 2008, which presented international experiences on the topic and became an important input for the initial design of the reforms. In addition, the two original counterparts, SFP and SHCP, also benefited from the economies of scale derived from implementing the results-oriented agenda towards a comprehensive reform including both management and budget. However, the design of the Project was complex; the potential lack of coordination between the two main counterparts (SFP and SHCP), identified as substantial risk at appraisal, materialized during the Project s initial stages and implementation. Moreover, it should be noted that moving from monitoring budget execution to results monitoring, and from compliance to results in public sector management via a more comprehensive program, such as the PMG, implied a big reform. Most countries in Latin America and in Asia as highlighted by one of the peer reviewers have struggled with this type of changes. As such, even though counterparts coordinated their efforts and defined their roles for the results-based management and budgeting reform during the Project s preparation, the significant delay in loan effectiveness and subsequent delay in implementation suggests that the coordination mechanisms needed for the Project were underestimated. 6

19 18. The PAD indicated that the Project met the criteria for readiness for implementation. 8 However, there were delays between the signing of the Loan Agreement and the declaration of effectiveness. The Project was approved by the Board on December 11, The Project s legal agreement was signed on April 21, 2009, and became effective on March 19, 2010 due to an unanticipated delay in internal GoM administrative processes specifically, the execution of the administrative agreements (Contrato de Mandato) 9 between the Borrower s Representative (Public Credit Unit in SHCP), the Financial Agent (Nacional Financiera, NAFIN), and the implementing agencies - SFP and SHCP. 19. Overall, the long time elapsed since the loan s approval and effectiveness (16 months), along with a challenging procurement environment involving multiple parties (i.e., in Mexico it typically involves participation of three to four parties - the implementing entity/ies, the financial agent and the Bank) and weak procurement capacities in the implementing agencies, made it impossible for the Government to start utilizing the loan proceeds until one and a half years after Board approval. 2.2 Implementation 20. The implementation of the Project is viewed to have been moderately unsatisfactory primarily due to the lack of physical progress under the Project. 21. Start-up delays, a complex project design that included a governance framework involving two implementing agencies, limited implementation capacity, as well as changes in Government counterparts 10, severely hampered Project execution. Moreover, within SHCP the reform and Project activities were not fully internalized or appropriated in particular by the operational teams. SHCP areas (Budget Direction, Evaluation Direction, and International Affairs) were not fully aligned conceptually regarding Performance Based Budgeting and its priorities and targets. These resulted in the Government requesting a partial cancellation of the Project (US$12.7 million equivalent to 74% of Loan amount) as early as August As of Project restructuring in November 2012, only the front-end fee for the IBRD loan of US$43,108 had been disbursed under the loan. Moreover, lengthy procurement processes, in some cases caused by changes in Project Government leaders and operational teams, not only affected Project implementation but also posed a risk to its continuity, triggering internal 8 Readiness for implementation means that the Project ideally should become effective within four months from Board approval and is able to evidence physical implementation including initial disbursement for project activities (not just front-end fee) within the first six months following effectiveness. 9 There were two specific effectiveness conditions for this operation:(a) the duly executed Contrato de Mandato by the respective parties; and (b) related legal opinions from the respective parties of the Contrato de Mandato. 10 In the SFP, there were 5 subsecretarios between 2008 and

20 Bank discussions on a possible loan closing during the process of restructuring in 2011/ Despite this lack of progress, the Government remained committed to the overall objective of the Project that supports a critical public sector reform, namely the introduction of results-based management and budgeting in the context of a continued effort to modernize and harmonize public financial management (PFM) rules and procedures in all levels of government in Mexico. This is evidenced by some activities being advanced and financed with local resources and which was the basis for the continued positive assessment of the achievement of the PDO Using its own resources, significant progress was made by the Government in pursuing the indicated PFM reform agenda particularly in the areas of results-based budgeting, performance evaluation, accounting harmonization, fiscal transparency and public procurement management. The most important achievements in these areas are presented below: 24. Results-Based budgeting and performance evaluation: The PBR-SED 12 reform led by the SE has successfully advanced the design and introduction of specific guidelines and instruments to gradually adopt a results and performance orientation in the Federal Government s budgeting process. By developing a conceptual model, the new system articulated two instruments aiming at incorporating the results and performance dimensions into the resource allocation procedures, i.e., the adoption of performance indicators for public sector programs and the implementation of systematic evaluation activities over public sector policies and programs. 25. Accounting harmonization: As part of the overall strategy to modernize public sector financial management practices in the country, the Government started an ambitious program towards the harmonization of public accounts and accounting practices across Federal and sub-national governments. Introducing results-informed management capabilities require adequate accounting, reporting and consolidation of information mechanisms as a sine qua non condition for success. Thus, based on timelines clearly established in legislation passed at the request of the Administration, efforts were deployed to modernize and harmonize accounting systems. 26. Fiscal transparency: The SE also achieved significant progress in the fiscal transparency agenda. After a broad consultation and participatory process involving specialized Non Government Organizations (NGOs) and experts, and in collaboration with the Access to Information regulatory entity IFAI, the SE launched a fiscal transparency Portal 13 that provides access to detailed information on the allocation and use of federal government s resources. The portal is now fully operational and has also included a citizens friendly application Presupuesto Ciudadano that not only facilitates 11 ISR DO rating was Moderately Satisfactory from late 2009 early PBR-SED stands for Presupuesto Basado en Resultados y Sistema de Evaluación del Desempeño. 13 For more details visit 8

21 access to budget information, but also describes the budgetary process in a very accessible way for citizens. 27. The SFP had also been advancing the public sector management modernization agenda complementing efforts in areas related to the improvement of public sector management practices, streamlining internal regulatory frameworks, and modernizing the Federal Government s public procurement system and procedures. 28. Progress achieved by both the SE and SFP using own resources was in line with the PDO and had received Bank s feedback and guidance. In the case of the PBR-SED effort, the Bank sustained technical dialogue during Project implementation support missions for those activities that should be carried out through the Bank loan. On the other hand, the PFM harmonization work was supported through both Bank implementation support activities and a parallel advisory services activity. 29. With these advances using local funds, a Government request for a partial cancellation was received on August 8, This was followed by a request in late 2011 to reduce the Project scope to reflect the advances and redesign of activities to better align them with current priorities. In this regard, the processing of the cancellation by the Bank was delayed pending validation of the final set of activities that would need to be dropped and those that needed to be incorporated under the restructured Project to ensure viability after restructuring. Moreover, the November 2012 restructuring was finalized after a careful re-assessment by Government counterparts and the Bank that took into account the potential impact of the political transition following the Presidential elections in July Specifically, the SFP indicated its preference to cancel its activities in the Project. 30. Moreover, soon after the Project restructuring, the new Administration took place in Among the first impact on the Project was the change in the executing unit from the Budget Office of the SE to the Performance Evaluation Unit (Unidad de Evaluación del Desempeño, UED) 14. This facilitated a better alignment of presidential priorities with sectorial plans and programs objectives/ targets. However, the political transition also resulted into a set-back in Project execution since the activities needed to be revalidated with the new counterparts. Actual Project activities implemented were minimal with total disbursements of less than US$400K. Furthermore, priority shifts in some critical results areas of the Project led to some results indicators not being achieved at all. This was particularly true for the roll-out of simplified financial management systems to small municipalities that could not be achieved because of a priority shift towards the development of the Government Accounting System first. 14 This evaluation unit reported to the Budget Office Director (UPCP). In October 2012, the internal Regulations of the SHCP were amended through a Decree; these reforms included the creation of the Performance Evaluation Unit with attributions to coordinate performance evaluations established in article 111 of the Budget and Fiscal Responsibility Federal Law (LFPRH). 9

22 31. Another factor affecting implementation was the existence of a similar operation supported by IADB. The IADB supported Performance Based Budgeting through a loan, but there was a lack of effective coordination between the two institutions. In this context, both Banks were almost competing and not aligned in technical aspects. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 32. M&E design. The results framework presented in the PAD provided for performance indicators for the PDO and intermediate results for each component. However, these were largely qualitative in nature and with only the desired target value at end of Project identified. No target values were identified for the time period between beginning and ending which made monitoring a challenge and perhaps more susceptible to optimistic assessments as evidenced by the Moderately Satisfactory ratings for PDO achievement throughout the life of the Project except for the last ISR. Progress towards achieving the PDO could have been more meaningfully and objectively assessed if target values were identified for the implementation period and not just at exit. Moreover, with mid-course target values, mid-course corrections could have been introduced earlier. 33. M&E implementation. During the second and third years of implementation, the Bank scheduled periodic high level meetings with and between Deputy Ministers to discuss the progress of the Project, but these meetings did not occur as planned and when they did, they could not provide guidance to the Project at the political or technical level needed. The PDO and intermediate indicators were adjusted during Project restructuring (level one) in 2012 to establish a better link with Project activities. While the target values were also identified only for Project exit, the indicators have improved to allow for a more objective if not quantitative assessment during implementation postrestructuring. The Bank regularly monitored progress under the Project through periodic implementation support missions and were also included in the implementing agencies progress reports. 34. M&E utilization. The PDO and intermediate results indicators, particularly with respect to the target measures or percentages, were closely monitored by the Bank team and reported in ISRs. The counterparts provided updates to the Bank on the results indicators usually at the conclusion of every implementation support missions. Postrestructuring, the Government also used the transparency portal to monitor most of the results indicators. However, more in situ monitoring effort is done in each ministry through their planning units instead of the UED at SHCP. 2.4 Safeguard and Fiduciary Compliance 35. No safeguard policies were triggered by the Project at design, an appropriate decision, as no such issues arose during implementation. 36. The key issues and risks concerning procurement for Project implementation were identified since Project appraisal; these included the lack of experience of the two implementing agencies in applying Bank procurement guidelines. The corrective 10

23 measures were agreed with the executing agencies and were mainly related to keeping enough trained personnel on Bank guidelines in the implementing agencies procurement units. However, limited procurement capacity continued to be an issue during implementation before and after restructuring which posed a substantial risk to the Project s success. 37. Risk for Project financial management was assessed as low and was considered to be satisfactory throughout Project implementation. Financial monitoring reports were submitted on time and were acceptable to the Bank. All minor inconsistencies found during supervision were properly addressed by the client and did not impact the controls of the Project. Annual audit reports for the Project were submitted accordingly and reviewed by the Bank, with auditors issuing their unqualified opinions. 2.5 Post-completion Operation/Next Phase 38. Continuing commitment to the development objective beyond the scope of the loan has been demonstrated through the decisions of actors including the UED to engage in related activities with their own funding. There are also positive signals that future Bank engagement with SHCP would include a focus on strengthening the UED as well as the M&E capacities of line ministries and sub-national governments. To this end, the lessons from this Project are expected to inform future engagements and could therefore help to consolidate and sustain the Close and Modern Government Program (Programa de Gobierno Cercano y Moderno, PGCM), 15 as well as the reform process of SHCP in the area of performance information, integrated financial management, use of this information for improving commitments that result from evaluations and for management and budget decision making. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 39. To achieve faster growth, reduce extreme poverty and promote shared prosperity, Mexico needed to improve public service delivery. Mexico s public service delivery is still hindered by expenditure inefficiencies at all levels of government: federal, state, and municipal. In this context, sound and informed budget decision making were critical to ensure that services were delivered efficiently, as well as in a transparent and accountable manner. 40. The GoM recognized this challenge in the NDP, which lays out a crosscutting program for a Close and Modern Government. The PGCM aims at establishing a results-based, efficient government, as well as developing monitoring and evaluation 15 See section 3.1 for more details on this cross cutting program included in the National Development Plan

24 mechanisms for improving performance and service delivery. Additionally, the program seeks to simplify procedures and regulations, allow for greater transparency and accountability, ensure an optimal use of public resources, and seize the potential of information and communication technologies. By focusing on enhancing the information on performance available for government entities and citizens, the Project s original design and implementation are fully aligned with the GoM s current strategy and continue to be highly relevant for Mexico. 41. As mentioned in the Project Appraisal Document (PAD), the Project was aligned with the CPS FY that called for strengthening institutions to improve their functioning and to enhance citizen perception of the public sector through a set of initiatives including results based-budgeting. The Project was also consistent with the World Bank thematic engagement set forth in the new CPS FY14-19, specifically with the third theme, Strengthening Public Finances and Government Efficiency. 17 The Bank strategy underscores the need for Mexico to improve public sector performance, including expenditure quality and equity, and aims at supporting better service delivery. Through this Project, the Bank has contributed to extending the monitoring and evaluation techniques of key public policies, strengthening the supervision of indicator achievements, and generalizing the follow up of intermediate results, establishing the foundations for a more efficient and transparent public administration. 42. As mentioned earlier, design of the Project was complex but it was key to enable the changes that needed to happen in order to pursue a results-oriented reform in budget and public sector management and effectiveness. With respect to implementation, albeit unsatisfactory in the end, it was relevant as it allowed for a continuing engagement and dialogue between the Government and the Bank on the critical public sector reform areas beyond the Project. 3.2 Achievement of Project Development Objectives 43. The overall rating for PDO achievement is Moderately Unsatisfactory. 44. Prior to the November 2012 restructuring, the achievement of outcomes is assessed in this ICR to be moderately unsatisfactory. While credit is due to the SHCP and SFP for their achievements using own resources, these did not enable the Project to sufficiently progress towards achieving its desired outcomes. Nevertheless, as discussed in Section 2, their achievements in advancing results based budgeting and performance evaluation, accounting harmonization and fiscal transparency, albeit without any loan disbursement except for the front-end-fee, were aligned with the PDO. 45. Post-restructuring, the achievement of outcomes with disbursement of approximately US$400K are assessed to be likewise moderately unsatisfactory for failure 16 Report No MX 17 Report No MX 12

25 to achieve several results indicators including the PDO indicator and related intermediate outcome indicators with respect to the roll-out of a simplified financial management system to small municipalities and the evaluation of public programs. Nevertheless, the following achievements after the restructuring are worth noting: (i) Performance information (PI) is now presented in a timely manner to Congress in a format that allows interpretation and meets their needs. The content includes outcome and output measures. (ii) The majority of the public programs (84.5%) have a robust design across key dimensions (objectives, indicators, baselines, links to strategies) enabling better reporting and evaluation. (iii) The Portal containing the PI on budget performance is available to citizens. In 2013, the UED developed more citizen friendly applications such as Cuenta Pública Ciudadana 2012 and 2013, as well as their English versions. These initiatives were mostly financed by component 4 of the loan. An improved version of the budget transparency portal is to be launched in June 2014, including new platform, features, etc. See figure below and Annex 2 for more details on outputs that contributed to achieve PDOs. Outputs. 1. Budget Transparency Portal 2. Established methodology and manuals that set the minimum technical requirements and guidelines for budget transparency and provide orientation for performance indicators 3. Developed Training Programs based on the challenges identified and on demands (i.e. logical framework matrix) Component 4. Consolidation of the performance evaluation system s (Sistema de Evaluación del Desempeño SED) institutional architecture and strengthening the system for evaluating policy and program effectiveness (Presupuesto Basado en Resultados PBR) Project Development Objective 2. An active network and a portal to inform society at large on a regular basis about budget performance, increasing transparency and accountability to citizens 46. Overall, the Project supported the introduction of better management practices, processes and tools in the Mexican public administration, including better and timely performance information, enhanced reporting and evaluation of public programs, and transparent and available budget information. The adoption of the Budget Transparency Portal, which was largely funded by loan funds, resulted into tangible outcomes in terms of generating information and enabled the country to maintain its Open Budget Index standing. 18 At the same time, the Project financed the training of officials through a 18 In 2012, Mexico has been one of the countries that provide significant budget information per the Open Budget Index. 13

26 series of activities including online courses, workshops and the development of a series of manuals and methodologies to collect, monitor and evaluate budget information. This now appear to have good results as evidenced by the work being continued and specifically evidenced with SHCP s presentation of the Guidelines to register, revise, update, schedule and monitor the Budget Programs Results Indicators Matrix (MIR) in the first semester of Efficiency 47. Project efficiency is low. The reduction in scope and the limited results achieved despite the restructuring and extension suggest that overall efficiency is much lower than expectations at design and at restructuring. The long time elapsed between preparation and the Project closing with minimal physical accomplishments and limited results vis-àvis the overall desired outcomes suggest there were opportunities missed which were partly alluded to in the Borrower s comments on the relevance of Project activities (particularly those of SFP) having been reduced given the delays. In monetary terms, from the beginning, a cost-benefit analysis was not considered to assess the impact of these institutional reforms, as problems in the quantification of intangibles and issues of attribution make it difficult to express their impacts. Moreover, given loan amount disbursed (US$373,684) vis-à-vis the time-overrun in implementation and the resources mobilized in the Bank for preparation and provision of implementation support (US$967,977) which are more than double the loan disbursement, it could be considered that the Project failed to be efficient. Nevertheless, the operation managed to achieve significant results in terms of increased transparency, good governance and accountability, as measured by greater availability and access to budget data and performance information (see Annex for more details). Further potential efficiency gains and cost savings are expected from these types of reforms when fully implemented. 3.4 Justification of Overall Outcome Rating Rating: Moderately Unsatisfactory 48. As discussed above, while the relevance of the objective remains high with the public financial management and results-based agenda at the federal and subnational levels remaining a priority, the achievement of the PDO could only be rated moderately unsatisfactory given the inefficient Project implementation and the non-achievement of significant outcomes. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 49. No direct poverty, gender or measurable impact on the population is attributable to the Project. However, the Project s impact on public sector management areas has potential indirect impact on reducing poverty through the effect of better resource 14

27 allocation and improved service delivery conditions, as well as by channeling generated savings more efficiently to national priorities. (b) Institutional Change/Strengthening 50. The M&E capacity of the UED was significantly strengthened in the latter part of the Project. From Project start to November 2012, the Project was executed by the Budget Office of the SE. Beginning December 2012 (when the current administration took office) the UED 19 took over Project execution. This facilitated a better alignment of presidential priorities with sectorial plans and programs objectives and targets. Technical assistance during implementation support missions by the Bank contributed to this, as well as feedback to the reform decisions provided by TTLs and experts members of the World Bank team. (c) Other Unintended Outcomes and Impacts (positive or negative) 51. No significant unintended outcomes or impacts were identified. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 52. No beneficiary survey or stakeholder workshops were conducted specifically for reviewing Project performance. 4. Assessment of Risk to Development Outcome Rating: Moderate 53. The general direction of the reforms supported by the Project is aligned with those of the Government s PGCM. As such, risks to the outcomes that have been achieved to date are considered to be moderate. While in the medium to long term, further advances in effectiveness building off Project activities will likely require additional investments in technology and process redesign, all of these are considered in the PGCM. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 19 This evaluation unit reported to the Budget Office Director (or UPCP). In October 2012 through a Decree the internal Regulations of the SHCP were amended; these reforms included the creation of the Performance Evaluation Unit (UED) with attributions to coordinate performance evaluations established in article 111 of the Budget and Fiscal Responsibility Federal Law (LFPRH). 15

28 54. The Project was intended to achieve challenging reforms in public management and results-based budgeting, building from efforts supported by the Bank during the beginning of the administration and Project preparation. The Project was prepared taking into account lessons learned from international experiences and taking advantage of the synergies between SFP and SHCP who were then working together towards a results oriented agenda. The Project was consistent with the federal government s objectives and the Bank strategy in Mexico. However, several shortcomings in Project design posed significant challenges to implementation. In particular, the failure to identify the challenges involved in coordinating amongst and with several federal government entities, and the overestimation of the counterpart s capacity to carry out complex procurement processes (both in SFP and SHCP) caused significant risks to the successful completion of the Project. As noted by the Borrower in their comments, Project implementation could have benefited from having a focal point for coordination as well from providing a venue to collaborate among all the stakeholders during Project preparation. The delay in declaring the loan effective (16 months from Board approval) suggests a significant under-estimation of the Project s readiness to be implemented. (b) Quality of Supervision Rating: Moderately Unsatisfactory 55. The initial implementation support phase of the Project could be characterized as overly optimistic and highly focused on technical aspects, with inadequate attention paid to operational constraints. On the positive side, in a context where the Government was moving fast with their reform agenda and the demands were changing, the Bank was able to sustain technical dialogue specifically on the PBR-SED and PFM harmonization efforts which were financed by local funds. On the negative side, the Bank failed to take advantage of the synergies in place between SFP and SHCP in the early stages of implementation. On top of that, the SFP faced high-level public officials turnover and the World Bank team failed to recognize these implementation risk flags or adjust to these changes in the context accurately. The Bank s implementation support team also suffered from lack of leadership at the beginning of implementation when the TTLs were changed 20. Hence, realistic assessments of Project implementation constraints early on could have facilitated a Project turnaround by addressing the changing dynamics in the reform. There were also indications that communications between the Bank team, fiduciary teams, financial agent and implementing agencies could have been better to improve Project implementation support and administrative processes. 56. By 2011, it became clear that the Project scope had to be reduced and Project activities redesigned to better align them with the incoming Government s priorities as 20 The Project had a total of three TTLs throughout the five years of implementation, 16

29 was detailed in the Government requests. However, the processing of the partial cancellation by the Bank was postponed to allow for a thorough assessment of viability and commitment from the newly elected administration (at the end of 2012). During this time, the Bank flagged potential issues and took action to solve them effectively throughout the last year of implementation. The new government had just taken office and continued to align its priorities; and as noted above, even though the Bank team engaged in intensive implementation support, procurement processes lingered for extensive periods given procurement capacity constraints within the implementing agency and as such only a few contracts were executed with less than 10% of the total restructured loan funds disbursed. Nevertheless, post-restructuring, a closer collaboration between the Bank team and the counterparts were established that not only help move the Project forward as well as contributed to the institutional strengthening of the UED. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Unsatisfactory 57. Based on the Bank performance in ensuring quality at entry, and the quality of implementation support, the overall Bank performance is rated as Moderately Unsatisfactory. A number of challenges that led to delays in Project start-up and execution were not fully identified at design, and communication between the Bank and the client could have been better during the initial part of implementation. However, this ICR recognizes the Bank implementation support team s significant efforts during the latter part of implementation to better address the procurement issues and to try to achieve the new Government s goals. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Unsatisfactory 58. Despite the fact that Project implementation suffered delays, the areas supported under this Project were part of the Government of Mexico s priority agenda with respect to improving results based management and budgeting as evidenced by the reform advances that took place using their own resources. However, the political commitment to the Project was weakened by the weak (if not lack of) coordination between SFP and SHCP and compounded by the turnover of government officials (5 subsecretarios held office in the SFP as the main counterparts for Components 1 and 2). Moreover, the Project was not fully internalized or appropriated in particular by the SHCP s operational teams. SHCP areas (Budget Direction, Evaluation Direction, and International Affairs) were not fully aligned conceptually regarding Performance Based Budgeting and its priorities and targets. SFP and SHCP s high-level commitment could have translated into proactive measures by ensuring that the Project operational units were progressing with Project implementation satisfactorily. 17

30 (b) Implementing Agency or Agencies Performance Rating: Moderately Unsatisfactory 59. The Project Implementation Units both at the SFP and SHCP were committed to the Project but had a low capacity in fiduciary aspects, especially procurement as identified in the risks at appraisal. During the last year of implementation, SHCP s implementing unit (within the UED) was very organized and engaged but lacked familiarity with the Bank s procurement guidelines that did not allow them to disburse most of the resources. Nevertheless, despite their capacity constraint, they were able to focus on limited activities that had significant contribution to the desired Project outcomes. With their limited implementation capacity, they were able to advance on the Budget Transparency Portal that enabled the Project to deliver on its objective of disseminating budget performance information to the Borrower s citizens on a regular basis. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Unsatisfactory 60. Considering the above arguments, the overall Borrower performance is deemed moderately unsatisfactory. 6. Lessons Learned 61. The instrument was not the best alternative for what was needed for these reforms. The mitigation measures in place did not lower the risks identified at appraisal. This Project benefitted the most from the technical assistance and feedback received both during preparation and the last years of implementation. In retrospect, it would have been better to have supported the Government s agenda with a Non-Lending Technical Assistance (NLTA) program or a Reimbursable Advisory Services (RAS) to support more effectively the type of reforms the Government was pursuing. The chosen instrument allowed the Bank to promote the technical dialogue with the authorities and amongst stakeholders, and also ensured agreements and commitments at the highest levels. However, these were not fully internalized or appropriated in particular by the SHCP s operational teams as discussed above. 62. Assessment of Risks - Coordination mechanisms were not enough. Although there were mitigation measures in place for this risk, the fact that there were two implementing agencies and a financial agent was more complex than initially assessed. Moreover, within each of the implementing agencies, coordination was not enforced thus affecting project implementation and disbursement performance. The designation of a focal point for coordination could have helped as noted by the Borrower. 63. Demand not well read in the case of SHCP. While Components 1 and 2 were very much in line with the Government s (SFP) agenda, both conceptual and technical; the 18

31 Components related to results-based budgeting (SE of SHCP) which included the Budget Office (UPCP) in the first stages and the UED in the last years were not aligned. This is largely reflected in the lack of disbursements throughout the Project. 64. Additional implementation support should be considered for projects that involve more than one implementing unit that are not familiar with large and complex procurement processes, such as some investments within this Project. For this Project, the need for additional support should have been anticipated at the beginning of Project implementation. While capacities at the Ministries were strengthened, this was not enough. Moreover, improved harmonization of procurement procedures between the Bank and GoM is a necessity as noted by the Borrower s comments. 65. Ensure alignment between Government processing and Bank required action. As reflected in the Borrower s comments below, timely action from the Bank with respect to issuing Bank feedback especially no objection is also a critical element for timely implementation. 66. Meaningful and objective results indicators are critical for monitoring complex reform initiatives. While reform initiatives do not always allow for quantifiable measures of success, this Project could have benefited from a results framework that provided for mid-course target values that could have informed counterparts and the Bank that mid-course corrections were warranted earlier. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 67. The Borrower provided very informative and pertinent comments and these have been taken into consideration in the assessment of overall performance of the Project, the Borrower and the Bank as well as in the lessons learned section. These comments are as follows: a) The initial objective of the GoM s Results based Management and Budgeting Project aimed at having the departments and federal organizations of the country provide decision makers and citizens in general with rigorous, timely and easy to access information regarding the efficiency and effectiveness of public expenditure programs. b) The GoM sought Bank support for the financing of the Project through an IBRD loan, with the objective of reaching the goals proposed under the five components that integrated the Project. Given the legal attributions conferred by the Federal Public Administration Organic Law, the Government mandated the two main ministries involved in the PBR-SED initiative (SHCP and SFP) with the implementation of the Project. 19

32 c) Having two implementing agencies was an obstacle to the successful execution of the Project, basically due to the complexity of coordinating the work of the two ministries and the lack of clarity in the Project objectives, as well as its scope. This was accentuated by the absence of a coordination focal point and the limited cooperation amongst both ministries, the Financial Agent and the Bank. Also, the pace of the execution of the Project did not satisfy Government expectations in terms of results. d) In addition, the delay between the signing of the Legal Agreement (April 21, 2009) and effectiveness (March 19, 2010) had also a negative influence in the implementation of the Project. During the first stages of Project execution, it was not possible to hire any consultancy and advisory services due to delays in the execution of Bank administrative processes such as no objections through the Financial Agent. As a result, the first consultancy services were only provided in e) Under these circumstances, one of the executing agencies, SFP, decided not to continue with the implementation of the Project, cancelling the total amount of resources allocated to its components. As the entity responsible for the PMG, SFP was counting on the Project to help advance that agenda, including the design of documents, tools and methodologies. The initial delays reduced the relevance of the components included in the Project from the perspective of SFP. f) Consequently, the Project was restructured in order to have only one executing agency, the SHCP. The SHCP used the loan resources to: hire, inter alia, the consultancy services to analyze and evaluate the strategic indicators of the budget programs; conduct training sessions with regards to PBR-SED; and implement the mechanisms to report the federal resources transferred to state entities and municipalities. These activities had a positive impact in the PBR-SED implementation. g) Under the restructured Project, the Financial Agent acted in a timely manner, which was key for the review of the documents needed to register processes and for obtaining no objections from the Bank with respect to the hiring of consultancy services, as well as disbursements. This also helped produce Project-required financial and progress reports on time. h) In the view of SHCP, Bank procurement and hiring processes, especially those requiring no objections, could be improved. The long time elapsed between Bank approval/rejection caused delays in the processing of contracts, which greatly affected the proper implementation of Project activities and as a result had a negative impact on its overall implementation. i) In terms of areas of opportunity, greater harmonization between GoM and Bank procurement procedures could be sought, even though there are already 20

33 documents streamlining Federal Government and World Bank processes. It would be also advisable to establish work programs on time and in accordance to reality, including clear hiring procedures that expedite processes so the products delivered can be used to influence the final results. Finally, it would be convenient to improve project preparation by working jointly and with better coordination and communication among involved stakeholders. 68. The Borrower also provided additional editorial comments in other sections of the ICR. These have been addressed in finalizing the document. (b) Cofinanciers Not applicable (c) Other partners and stakeholders 69. IADB supported Performance Based Budgeting through a loan, but did not work in coordination with the World Bank. Based on the interviews of stakeholders of the reforms, the Banks were seen as almost competing and not aligned technically. 21

34 Annex 1. Project Costs and Financing (a) Project Cost by Component (in US$ Million equivalent) Components Appraisal Estimate (US$ millions) Actual/Latest Estimate (US$ millions) Percentage of Appraisal Component Component Component Component * 8.70 Component Total Baseline Cost Physical Contingencies Price Contingencies Total Project Costs Front-end fee PPF Front-end fee IBRD Total Financing Required ** 2.90 Source: Informe de avance segundo semestre de 2013, UED - SHCP *Sum of Loan Disbursement of US$330, and Counterpart Fund Contributions of US$543,391. **May not equal due to rounding off. (b) Financing Source of Funds Type of Cofinancing Appraisal Estimate (US$ millions) Actual/Latest Estimate (US$ millions) Borrower International Bank for Reconstruction and Development Percentage of Appraisal 22

35 Annex 2. Outputs by Component As discussed in the main section of the ICR, the executed Project activities relate to Component 4. Below presents the outputs under said Component. Component 4 - Results-Based Management and Evaluation System Consolidation 1. The key outputs delivered by Component 4 of the Project are described below: a) Budget Transparency Portal (BTP): Since 2011, SHCP has had in place a Budget Transparency Portal ( on which performance information of the budget programs can be found, as well as public financial information. The main objectives and contents of this portal are describe in the table below: Table 2.1. Objectives and contents of the BTP Objectives More transparency in Public Resources Management Content Public Finance Statistics Transferred resources to the local governments Budget Information Quarterly Reports in Open Data formats Cuenta de la Hacienda Pública Federal Budget transparency towards citizens (friendly language) Citizen Budget Cuenta Pública Ciudadana or Public Accounts for Citizens Frequently Asked Questions Glossary Transparency by Thematic areas, Investment Projects focused on citizens interests Performance Information (indicators and goals) External/Independent Evaluations Aid and support for Natural Disasters Reconstruction Source: Final Report and Results Framework Matrix, UED, SHCP. April 21, The main strength of the portal compared to other countries is the fact that all the performance information is public, including external evaluations (more than a thousand) and MIRs (results indicators matrix) with indicators (757). There is also access to public investment information including those funded under special funds such as FONDEN (or Natural Disaster Fund). The UED has also reached out to NGOs and civil society to ensure that the information is useful for their purposes and 23

36 understandable for everyone. For instance, the information included in the citizen budget for 2014 explains in a friendly manner the following topics: Growth Total Budget Sources of revenue and debt Deficit Functional Classification of Expenditures (why the Government spends?) Administrative Classification of Expenditures (who spends?) Economic Classification of Expenditures (on what the Government spends?) Investment Projects Measures taken to reduce expenditures SED information such as percentage of expenditures with performance indicators Links to more detailed information b) Methodology and Manuals were developed, including: - Methodology to develop and implement budget transparency portals at the federal and local levels (technical and functional) - Guide to establish and monitor goals for federal entities and agencies of the Government - Training Manual to monitor Budget Program s objectives based on strategic indicators - Improved guide to design strategic and management indicators c) Various Training Programs have been and will continue to be conducted: - Online Training - Training for Trainers - Training based on identified challenges on MIRs (theory, practical, continued technical assistance, ad hoc workshops) - Training to improve the Single Format System or Sistema de Formato Único (SFU), i.e. information collection and registration as well as quality of information. 2. A full account of the outputs completed by the Project under component 4 is presented in the table below. These outputs were mostly financed by counterpart funds and contributed towards achieving the intermediate result indicator Consolidation of the performance evaluation system s (SED) institutional architecture and strengthening the system for evaluating policy and program effectiveness by supporting the targets specified below as well. Table 2.2. Completed Project outputs by source of financing Output Original Budget 24 Executed Budget Outputs financed with loan resources 346, ,884 Target supported Consultancy to prepare and facilitate the 31,000 24,259 Evaluation unit operational in SFP,

37 Project s selection and hiring processes (1) Methodological guide for the technical and functional implementation of the transparency web portal at the federal and subnational levels Development of minimum criteria and transparency guidelines for the federal and subnational governments Collection of feedback on PBR program and the SED 40,000 40,000 32,000 32,000 52,000 52,000 Training of trainers course 33,000 33,000 Consultancy for the analysis of budget strategic indicators for the 2013 budget cycle Consultancy for the maintenance of the Budget transparency portal Consultancy for the implementation of the SFU Consultancy for the conceptualization and analysis of the information produced by the SFU Consultancy to prepare and facilitate the Project s selection and hiring processes (2) Consultancy for the establishment of a monitoring and evaluation system for the Project s selection and hiring processes 18,000 18,000 18,000 18,000 18,000 18,000 25,000 25,000 52,000 37,500 27,000 28,125 Outputs financed with local resources 574, ,093 TOR for the carrying out of the activities to be conducted under components iii, iv and v of the Project 102, ,241 - External audit for loan 7622-MX 2,852 2,852 - Results-based budgeting online course (auditing and results-oriented indicators modules). 469, ,000 including an institutional performance evaluation system Evaluation unit operational in SFP, including an institutional performance evaluation system Evaluation unit operational in SFP, including an institutional performance evaluation system The SED unit manages the evaluation of non-social programs successfully Set of strategic indicators has improved after support and training provided to Government officials in charge of programs Evaluation unit operational in SFP, including an institutional performance evaluation system Evaluation unit operational in SFP, including an institutional performance evaluation system Set of strategic indicators has improved after support and training provided to Government officials in charge of programs Set of strategic indicators has improved after support and training provided to Government officials in charge of programs Evaluation unit operational in SFP, including an institutional performance evaluation system Evaluation unit operational in SFP, including an institutional performance evaluation system Set of strategic indicators has improved after support and training provided to Government officials in charge of programs 25

38 Annex 3. Economic and Financial Analysis 1. The overarching goals of the reforms supported by the Project aimed at increasing transparency and strengthening good governance and accountability in the Sistema de Evaluación al Desempeño (SED), which are difficult to measure. Therefore, the economic and financial analysis for this operation needs to go beyond quantitative estimations and rely on qualitative data. The assumption is that the support to the GoM s institution building efforts provided by the Project, notably in terms of results-based management and budgeting, has contributed to a more transparent and accountable governance framework. However, the outputs of the reforms carried out by the GoM, including the Programa de Mejora de la Gestión (PMG) and Presupuesto Basado en Resultados (PBR), as well as the SED and the sectorial strategies and specific programs, are not fully attributable to the Project or a single reform initiative, and their impact may not be apparent in the short-term. 2. Overall, the Project supported the introduction of better management practices, processes and tools in the Mexican public administration, including better and timely performance information, enhanced reporting and evaluation of public programs, and transparent and available budget information. The most important instrument in this regard is the Budget Transparency Portal, which was mostly financed with the loan resources. The adoption of this portal resulted into tangible outcomes in terms of generating information, since according to the Open Budget Index (OBI), Mexico has been one of the countries that provides significant budget information in their portal since Today most budget information is presented in Open Data format, which allows the private sector, NGOs and civil society in general to access what is relevant for them. In fact, online visits increased to 28.5% in 2013 compared to 2012, and as of the first quarter of 2014, there was an increase of almost 50% compared to the same period of 2013 and more than doubled compared to the first quarter of At the same time, the Project financed the training of officials through a series of activities, including online courses, workshops and the development of a series of manuals and methodologies. These initiatives were aimed at increasing the capacity of public officials to collect, monitor and evaluate budget information, and now appear to have achieved good results. At the end of 2013, out of the 981 budget programs included in the programmable budget (the total is 1,005), 854 programs have performance information, which is equivalent to 87% of the programmable budget. The work has continued in the first semester of 2014, when the SHCP presented the Guidelines to register, revise, update, schedule and monitor the Budget Programs Results Indicators Matrix (MIR). 4. The upward trend in the use of information by the public, along with the training received by officials, also suggests a positive impact of the Project in terms of creating an institutional context more conducive to efficiency. Although it cannot be quantified, it seems likely that the combined effect of greater accountability and increased capacity 26

39 will produce financial savings eventually, ultimately boosting the quality of public expenditures in Mexico. 27

40 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Responsibility/ Specialists Lending Pedro Arizti Senior Public Sector Specialist LCSPS TTL Rosa Maria Hernandez- Fernandez Senior Executive Assistant LCC1C Assistant Mariano Lafuente Public Sector Mgmt. Spec. LCSPS Management and HR Improvement Karla Soledad Lopez Flores Operations Assistant CTGBD Assistant Felix Prieto Arbelaez Senior Procurement Specialist LCSPT Jeffrey James Rinne Consultant PRMPS HR Management Fernando Rojas Consultant LCSPS Public Sector Maria Guadalupe Toscano Nicolas Public Sector Mgmt. Spec. LCSPS Manuel Antonio Vargas Madrigal Lead Financial Management Spec MNAFM Supervision/ICR Juan Carlos Alvarez Senior Counsel LEGES Pedro Arizti Senior Public Sector Specialist LCSPS TTL Dmitri Gourfinkel Financial Management Specialist LCSFM Financial Management Alberto Leyton Senior Public Sector Specialist ECSP4 TTL Xiomara A. Morel Senior Financial Management Specialist LCSFM Victor Manuel Ordonez Conde Senior Finance Officer CTRLN Jeffrey James Rinne Consultant PRMPS HR Management Tomas Socias Senior Procurement Specialist LCSPT Maria Guadalupe Toscano Nicolas Public Sector Mgmt. Spec. LCSPS Manuel Antonio Vargas Madrigal Lead Financial Management Spec MNAFM May Olalia Senior Operations Officer LSCPS Azul del Villar Baston Consultant LCSPS ICR Eguiar Lizundia Gonzalez Consultant LCSPS ICR (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle US$ Thousands (including No. of staff weeks travel and consultant costs) Lending FY FY FY Total:

41 Supervision/ICR FY FY FY FY FY FY Total:

42 Annex 5. Beneficiary Survey Results Not applicable 30

43 Annex 6. Stakeholder Workshop Report and Results Not applicable 31

44 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR 32

45 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Not applicable 33

46 Annex 9. List of Supporting Documents Project Appraisal Document on a Results-based Management and Budgeting Loan (Report No: MX), 2008 Mexico Country Partnership Strategy FY14-19 Mexico Country Partnership Strategy Proposed Project Restructuring of Results-based Management and Budgeting Loan (Report Number MX), Proyecto de Apoyo a la Gestión y Presupuesto para Resultados Préstamo MXBanco Internacional de Reconstrucción y Fomento. Informe de Progreso en la Ejecucion del Proyecto, Segundo Semestre SHCP, abril 21,

47 0