Document of The World Bank IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-47320) ON A LOAN IN THE AMOUNT OF US$ 9.

Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Document of The World Bank Report No: ICR00002035 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-47320) ON A LOAN IN THE AMOUNT OF US$ 9.8 MILLION TO THE FORMER YUGOSLAV REPUBLIC OF MACEDONIA FOR A SOCIAL PROTECTION IMPLEMENTATION PROJECT Human Sector Development Unit Department for South-East Europe Europe and Central Asia Region January 30, 2012

CURRENCY EQUIVALENTS (Exchange Rate Effective January 30, 2012) Currency Unit: Macedonia Denar (MKD) US$ 1.00 = MKD 46.75 MKD 1.00 = US$ 0.0214 FISCAL YEAR January 1 December 31 ABBREVIATIONS CAS Country Assistance Strategy CB MIS Cash Benefits Management and Information System CCTP Conditional Cash Transfers Project CEM Country Economic Memorandum CFAA Country Financial Accountability Assessment DPL Development Policy Loan EUR Euro FYR Former Yugoslav Republic GDP Gross Domestic Product HBS Household Budget Survey HD Human Development HIF Health Insurance Fund IBRD International Bank for Reconstruction and Development ICR Implementation Completion Report IFR Interim Financial Report ILO International Labor Organization IRI Intermediate Results Indicator ISR Implementation Status and Results Report IT Information Technology M&E Monitoring and Evaluation MAPAS Macedonian Agency for Supervision of the Capitally Funded Pension Insurance MIS Management Information System MKD Macedonian Denar Vice President: Country Director: Sector Manager: Project Team Leader: ICR Team Leader: MLSP Ministry of Labor and Social Policy NBRM National Bank of Republic of Macedonia NEB National Employment Bureau (= Employment Agency) NGO Non-governmental Organization PAD Project Appraisal Document PBG Policy Based Guarantee PCN Project Concept Note PDIF Pension and Disability Insurance Fund PDO Project Development Objective PERTAP Pension Reform and Technical Assistance Project PIU Project Implementation Unit PMU Project Management Unit PRO Public Revenues Office PSMAC Public Sector Management Adjustment Credit PSMAL Public Sector Management Adjustment Loan QER Quality Enhancement Review SPIL Social Protection Implementation Loan SSAC Social Sector Adjustment Credit SSO State Statistics Office SWC Social Welfare Center TA Technical Assistance US$ United States Dollar Philippe H. Le Houérou Jane Armitage Kathy Lindert Snjezana Plevko Rajna Cemerska

Implementation Completion and Results Report Table of Contents B. Key Dates... i C. Ratings Summary... i D. Sector and Theme Codes... ii E. Bank Staff... ii F. Results Framework Analysis... ii G. Ratings of Project Performance in ISRs... v H. Restructuring (if any)... vi I. Disbursement Profile... vi 1. Project Context, Development Objectives and Design... 1 1.1 Context at Appraisal... 1 1.2 Original Project Development Objectives (PDO) and Key Indicators... 3 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification... 3 1.4 Main Beneficiaries... 4 1.5 Original Components... 4 1.6 Revised Components... 6 1.7 Other significant changes... 6 2. Key Factors Affecting Implementation and Outcomes... 7 2.1 Project Preparation, Design and Quality at Entry... 7 2.2 Implementation... 10 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization... 12 2.4 Safeguard and Fiduciary Compliance... 15 2.5 Post-completion Operation/Next Phase... 16 3. Assessment of Outcomes... 17 3.1 Relevance of Objectives, Design and Implementation... 17 3.2 Achievement of Project Development Objectives... 18 3.3 Efficiency... 21 3.4 Justification of Overall Outcome Rating... 23 3.5 Overarching Themes, Other Outcomes and Impacts... 23 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops... 25 4. Assessment of Risk to Development Outcome... 25 5. Assessment of World Bank and Borrower Performance... 26 5.1 World Bank Performance... 26 5.2 Borrower Performance... 27

6. Lessons Learned... 28 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners... 29 Annex 1. Project Costs and Financing... 31 Annex 2. Outputs by Component... 32 Annex 3. Economic and Financial Analysis... 36 Annex 4. World Bank Lending and Implementation Support/Supervision Processes. 39 Annex 5. Beneficiary Survey Results... 41 Annex 6. Stakeholder Workshop Report and Results... 41 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR... 42 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders... 64 Annex 9. List of Supporting Documents... 64 MAP... 65

A. Basic Information Country: Macedonia, former Yugoslav Republic of Project Name: Social Protection Project Project ID: P074358 L/C/TF Number(s): IBRD-47320 ICR Date: 01/30/2012 ICR Type: Core ICR Lending Instrument: SIL Borrower: Original Total Commitment: Revised Amount: USD 9.80M Environmental Category: C Implementing Agencies: Ministry of Labor and Social Protection Cofinanciers and Other External Partners: GOVERNMENT OF MACEDONIA USD 9.80M Disbursed Amount: USD 9.45M B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 10/07/2003 Effectiveness: 09/01/2004 09/01/2004 Appraisal: 03/01/2004 Restructuring(s): 06/14/2010 Approval: 05/13/2004 Mid-term Review: 12/02/2006 02/02/2007 Closing: 06/30/2009 05/31/2011 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Borrower Performance: Moderately Satisfactory Moderate Moderately Satisfactory Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Satisfactory Quality of Supervision: Moderately Satisfactory Implementing Agency/Agencies: Moderately Satisfactory Overall Bank Overall Borrower Performance: Moderately Satisfactory Performance: Moderately Satisfactory i

C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Performance (if any) Potential Problem Project No at any time (Yes/No): Problem Project at any time (Yes/No): DO rating before Closing/Inactive status: No Moderately Satisfactory Quality at Entry (QEA): Quality of Supervision (QSA): None None Rating D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Compulsory pension and unemployment insurance 80 80 Other social services 20 20 Theme Code (as % of total Bank financing) Other public sector governance 17 17 Other social protection and risk management 33 33 Public expenditure, financial management and procurement 17 17 Social risk mitigation 33 33 E. Bank Staff Positions At ICR At Approval Vice President: Philippe H. Le Houerou Shigeo Katsu Country Director: Jane Armitage Orsalia Kalantzopoulos Sector Manager: Kathy A. Lindert Hermann A. von Gersdorff Project Team Leader: Snjezana Plevko Snjezana Plevko ICR Team Leader: ICR Primary Author: Rajna Cemerska-Krtova Rajna Cemerska-Krtova F. Results Framework Analysis Project Development Objectives The objectives of the project were to improve the effectiveness and efficiency of the social protection system through improved administration and long-term sustainability of the pension system and improved targeting and administration of cash benefits. ii

Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Indicator 1 : Improved financial solvency and administrative efficiency of the pension system. 85% (the PAD Value quantitative or Qualitative) 76.3% target value) instead of 95% was a typo error in the LA 92% Date achieved 05/13/2004 05/31/2011 05/31/2011 Comments (incl. % Over-achieved achievement) Indicator 2 : Improved targeting of social assistance benefits (in % by quintile) 3% increase in 5.4% increase in I - 17.49, II - 17.93, IIItargeted targeted social Value 11.64, IV - 12.20, V - social benefits that benefits that quantitative or 10.08, VI - 6.90, VII reaches the bottom reaches the bottom Qualitative) - 7.52, VIII - 5.82, IX - quintile of the quintile of the 4.68, X - 5.73 population population Date achieved 06/01/2004 05/31/2011 05/31/2011 Comments (incl. % Over-Achieved. achievement) (b) Intermediate Outcome Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Indicator 1 : Improved rate of collection of pension contribution Value (quantitative or Qualitative) 76.3% 85% 92% Date achieved 05/13/2004 05/31/2011 05/31/2011 Comments (incl. % Over-Achieved achievement) Indicator 2 : Increased number of pension benefits request processed within 30 days Value The data was not (quantitative available 90% 98.5% iii

or Qualitative) Date achieved 05/13/2004 05/31/2011 05/31/2011 Comments (incl. % Over-Achieved. achievement) Indicator 3 : Number of PDF/MLSP/MAPAS staff trained to use actuarial models of the pension system and conducting analysis of the pension system Value (quantitative or Qualitative) No pension model existed and there was no staff trained in pension modeling Six staff members in three different institutions are expected to be trained using the model and conducting pension policy analysis 7 experts trained during the project, 3 staff members remain (at PDIF) involved in actuarial modeling for the pension system Date achieved 05/13/2004 05/31/2011 05/31/2011 Achieved. There were seven staff who acquired training in actuarial analysis, Comments however, the three staff of the PDIF are licensed actuaries using the actuarial (incl. % analysis and pension system modeling for the purposes of pension policy achievement) evaluation and identification of policy options for the decision making Reduced overall administration costs of the disability pension assessment Indicator 4 : procedure Value (quantitative or Qualitative) MKD 5.3 million MKD 2.7 million less MKD 2.5 million Date achieved 05/13/2004 05/31/2011 05/31/2011 Comments (incl. % Achieved. achievement) Indicator 5 : Number of MAPAS staff trained to perform on-site and off-site supervision; NBRM Custody Unit (CIS) developed and capacities built in custodial practices Both MAPAS and CIS are fully operational; 8 staff members of Neither MAPAS nor the Value MAPAS trained in NBRM Custody (quantitative supervision and 5 Department existed at the or Qualitative) CIS members in beginning of the project NBRM Custody 6 Department trained in custody services Date achieved 05/13/2004 05/31/2011 05/31/2011 Comments (incl. % achievement) Indicator 6 : Achieved, although two staff members who obtained training, left MAPAS for other jobs Increased number of social assistance beneficiaries in lowest deciles iv

Value (quantitative or Qualitative) I - 17.04, II - 14.99, III- 10.91, IV - 10.83, V - 8.92, VI - 6.06, VII - 6.74, VIII - 5.58, IX - 4.07, X - 4.75, Total 8.98 5% more households in the bottom quintile of the population receiving social assistance benefits 1.47% and 3.81% decrease in the number of households in bottom quintiles I and II of the population receiving social assistance benefits Date achieved 05/13/2004 05/31/2011 05/31/2011 Comments (incl. % Not Achieved achievement) Indicator 7 : Reduction of social worker time spent on social benefit administration To be determined on the basis of the Value This was not measured at conducted analysis The target value has (quantitative the beginning of the of staff time spent not been set or Qualitative) project on social benefit administration Date achieved 05/13/2004 05/31/2011 05/31/2011 Comments (incl. % Not Achieved achievement) G. Ratings of Project Performance in ISRs No. Date ISR Archived DO IP Actual Disbursements (USD millions) 1 06/15/2004 Satisfactory Satisfactory 0.00 2 12/22/2004 Satisfactory Highly Satisfactory 0.54 3 06/03/2005 Satisfactory Satisfactory 0.62 4 06/01/2006 Satisfactory Satisfactory 2.03 5 09/13/2006 Satisfactory Satisfactory 3.45 6 04/03/2007 Satisfactory Satisfactory 4.03 7 01/31/2008 Satisfactory Satisfactory 5.08 8 12/24/2008 Satisfactory Satisfactory 6.14 9 06/30/2009 Satisfactory Satisfactory 6.53 10 08/31/2009 Satisfactory Satisfactory 6.88 11 05/18/2010 Moderately Satisfactory Satisfactory 7.10 12 12/25/2010 Moderately Satisfactory Moderately Satisfactory 8.94 13 06/13/2011 Moderately Satisfactory Satisfactory 9.38 v

H. Restructuring (if any) Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO Amount Disbursed at Restructuring in USD millions 06/14/2010 N MS S 7.10 IP Reason for Restructuring & Key Changes Made Extension of Closing Date, increase disbursement percentage of eligible expenditures to 100%, and reallocate funds within categories of Schedule 1 I. Disbursement Profile vi

1. Project Context, Development Objectives and Design 1. The Social Protection Implementation Project was approved on May 13, 2004. The Loan Agreement was signed on July 25, 2004 and became effective on September 1, 2004. The Project built on the Government s social protection and pension reform efforts, supported by World Bank-financed investment and development policy lending operations. The project was also expected to support the Government in meeting the policy conditionalities of the Public Sector Management Adjustment Loan (PSMAL), the adjustment operation prepared in parallel with the Project. 1.1 Context at Appraisal 2. The Former Yugoslav Republic of Macedonia (FYR Macedonia) was among the economically least developed of the former Yugoslav republics. Although its economy contracted even further in the years immediately following independence in 1991, the country made significant progress in macroeconomic stabilization and structural reforms during the 1995-1999 period. In that period, the economy experienced an average annual positive GDP growth rate of 2.9 percent, inflation was reduced to 2.2 percent, and fiscal deficit declined to 0.2 percent. These positive trends were interrupted by the Kosovo crisis (in 1999) and civil conflicts (in 2001) that resulted in a significant decline of industrial production and a severe deterioration of the fiscal balance. 3. At the time of appraisal in 2004, economic growth had slowed and investment was sluggish. The fiscal deficit of the general budget rose to 5.7 percent of GDP due to a significant increase in security-related expenditures, larger transfers to social funds, new public sector hiring, higher public wages and exceptional financing to compensate losses suffered in the wake of failed Ponzi schemes Although real GDP grew by 3.1 percent in 2003, significant challenges remained to achieve macroeconomic stability, a functioning market economy, sustained growth, and ultimately higher living standards for the people of FYR Macedonia. 4. At the time of appraisal, social assistance was characterized by weak administrative capacity and a complex and fragmented system of social cash benefits. Social transfers 1 at around 12.5 percent of GDP for most of the first half of the 2000s, before falling to about 11 percent in 2005, were modest compared to the South East Europe average of around 14 percent of GDP. 2 The system had developed in a piecemeal fashion, with little consideration given to linkages between programs or administrative efficiency. By 2002, there were as many as 0.5 million people (25 percent 1 Includes pension spending, the Health Insurance Fund s administrated social transfers, social protectionnon contributory transfers, and unemployment transfers. 2 Public Expenditure Policies in South East Europe, p. 20. 1

of the population) living in families that received some form of social assistance benefits. The efficiency and the effectiveness of those programs, however, remained low: programs were not well targeted; inclusion and exclusion errors were high. The 2005 Poverty Assessment found that social transfers were responsible for reducing the poverty headcount from 35 to 22 percent of population in 2003; it also found, however, that social assistance programs provided greater coverage to the non poor than to the poor 3. The assessment also found that even though the Government was making efforts to improve targeting efficiency of the means-tested social assistance program, these efforts were hampered by inadequate benefit administration and delivery, lack of proper monitoring of eligibility, and limited incentives for the beneficiaries to leave the system. 5. The pension system also needed reform. In 2002, total pension expenditures were 10.8 percent of GDP, and the Pension and Disability Insurance Fund (PDIF) required budget transfers of approximately 3 percent of GDP in addition to its direct contribution revenues. Moreover, the structural deficit of the PDIF was expected to grow in the future as a result of worsening system dependency ratios, generous benefit assessment and indexation rules, as well as the relatively easy access to occupational early retirement and disability status. To address these issues, the Government embarked on a comprehensive reform of the pension system. The reform included parametric policy changes, along with efforts to improve administration through improved system of contribution collection, record-keeping and internal controls, and strengthening of institutional capacity in the areas of strategic planning, business process development, human resource management and communication. In addition to the parametric adjustments to the public pension system, legislation was passed in 2000 to implement a structural reform introducing a mandatory funded ( second ) pension pillar. This pillar was designed to be mandatory for new entrants to the labor market and to be voluntary for persons who were already in the workforce. The implementation of a two-pillar system faced a number of significant challenges including creating the fiscal space necessary to accommodate the transitional deficit, creating adequate financial instruments for investment of the accumulated assets, and ensuring that the institutional framework established would be both cost-effective and consistent with sound governance principles and long-term financial return objectives. 6. The World Bank 2003-2006 Country Assistance Strategy (CAS) included three development objectives of direct relevance to this operation: (i) support to the public sector management reform program, (ii) building human capital, and (iii) protecting the most vulnerable. The CAS noted as important elements the need for improving the efficiency, effectiveness, quality of services and coordination of major social protection programs, including pensions, further institutional strengthening of 3 While in 2002 the social assistance transfers were received by 47 percent of non-poor beneficiaries, and by 53 percent of poor, in 2003 this ratio worsened with transfers were received by 51.2 percent non-poor and by 48.8 percent poor beneficiaries. Poor are people who are in the lowest income group (in the lowest quintile). 2

major institutions of the pension system and reforming the social safety net which had been lacking coordination at the system level. 4 7. The preparation and the design of the new investment operation the Social Protection Implementation Project - built on the achievements of the previous reform efforts of the Government in the areas of public sector management and social protection. Prior to the approval of the Project, the World Bank had supported FYR Macedonia by providing technical advice through several World Bank-financed investment projects as well as through adjustment (development policy) lending and Analytical and Advisory Activities 5. The project was expected to assist the Government in finalization of the social policy reform agenda supported under the PSMAL, the adjustment operation prepared in parallel with the Project. 1.2 Original Project Development Objectives (PDO) and Key Indicators 8. The development objectives of the Project were to improve the effectiveness and efficiency of the social protection system through improved administration and long-term sustainability of the pension system and improved targeting and administration of cash benefits. The development objectives as well as the performance indicators were the same across the main text of the PAD, Annex 3 of the Project Appraisal Document (PAD) (Results Framework and Arrangements for Results Monitoring) and the Financial Agreement. However, for regular monitoring, a subset of different indicators was selected and tracked in the Implementation Status and Results Reports (ISR). The indicators used for the purpose of this ICR are those used in the ISRs. 9. A full review of the Project documentation revealed that there was a minor discrepancy between the PDO as set out in the PAD and the PDO in the Loan Agreement: the term targeting was consistently referred to in the PAD, but was missing from the Loan agreement. The latter was identified during the December 2008 full review of the Project s documentation and was documented in the Sector Manager s comments filed in the Project s ISR no. 8, archived on December 24, 2008. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 10. The Project Development Objective was not revised. The PDIs were not formally revised, but they are stated differently in the PAD Annex 3 (Results Framework and Arrangements for Results Monitoring) and the ISRs. 4 Country Assistance Strategy for the Former Yugoslav Republic of Macedonia, August 14, 2003. 5 The Project built on these previous Bank operations and analytical work, including the FY 1998 Social Sector Adjustment Credit (SSAC), FY1999 Pension Reform Technical Assistance Project (PERTAP); FY99 Poverty Assessment, FY00 Public Expenditure and Institutional Review, FY02 Public Sector Management Adjustment Credit (PSMAC). FY03 Country Financial Accountability Assessment and FY03 Country Economic Memorandum. 3

11. In the Social Assistance component (Component 2), the intermediate indicators described in the PAD and intended for tracking the achievement of the PDO were not monitored during implementation - instead, a subset of indicators was selected for regular monitoring in the ISR. Although this practice was common at the time, it remained an obstacle for monitoring performance and assessing final results. The detailed findings linked to the adjustments made and the ICR team s assessments are elaborated in section 2.3 - Monitoring and Evaluation (M&E). 1.4 Main Beneficiaries 12. The direct beneficiaries of the Project under the Pension Administration component included the Ministry of Labor and Social Policy (MLSP), PDIF and its branch offices, the private pension sector's regulatory agency Macedonian Agency for Supervision of the Capitally Funded Pension Insurance (MAPAS), and the National Bank. 13. Under the Social Assistance component, the investments targeted a number of specific departments/units at MLSP such as the Social Protection Department, Child Protection Department, IT Department, Policy Evaluation Unit, Financial Department, Social Inspection Department, and the social welfare centers (SWCs). 14. In addition, the ultimate project beneficiaries were the households at the bottom of the income distribution and future pensioners. Taxpayers - through improved compliance and recordkeeping - would also enjoy the benefits of a more stable pension system, which would require fewer ad-hoc revenue or expenditure side adjustments to maintain its fiscal sustainability. 1.5 Original Components 15. The Project included three components: (i) Pension System Administration (hereinafter referred to as the Pension Administration component); (ii) Social Protection Program Support (hereinafter referred to as the Social Assistance component); and (iii) Project Management, Monitoring and Evaluation. COMPONENT 1: PENSION SYSTEM ADMINISTRATION (US$ 7.99 million) 16. This component aimed at improving the long-term sustainability of the pension system through improving the administration and efficiency of the existing pension system institutions; increasing the efficiency and effectiveness in collection of pension contributions through the introduction of monthly pension contribution recordkeeping on the individual basis; building regulatory and supervisory capacities in the newly formed MAPAS agency and custodial capacities in the National Bank of the Republic of Macedonia (NBRM); and improving public knowledge of the pension system through an extensive public education campaign. 17. Sub-component 1: Governance and management reform in the PDIF through implementation of sound business plans/processes and improved financial management 4

practices, improved human resource management, development and implementation of internal and external communication strategy for the PDIF, as well as strengthening policy and actuarial analysis capacities in the PDIF. It also supported the design and the implementation of an improved pension contribution collection system and the introduction of individual accounts for pension contributors. 18. Sub-component 2: MAPAS Development of: (i) the regulatory framework needed for proper functioning of the funded pension system; (ii) the design and implementation of the MAPAS business processes and MAPAS capacity building; and (iii) enhancement of public understanding of the funded pension system. 19. Sub-component 3: Capacity Building at MLSP for Pension System Analysis, Policy Development and Public Education, including: (i) identification of policy options and preparation of draft pension legislation; (ii) capacity building for pension system analysis and the development of future policy options; and the (iii) preparation and implementation of the integrated pension reform public education campaign, including organization of training, workshops and seminars, printing and distribution of the education materials, and other activities as needed. 20. Sub-component 4: Pension system custody (of second pillar pension assets) capacity development at the NBRM in: (i) identification of business processes and adequate organizational structure for the development of Custodial Department; (ii) preparation of rules and regulations related to custody over the second pillar pension assets, their evaluation, record-keeping and other custody-related activities; and (iii) provision of specialized training of the Custody Department staff on the issues relevant to the Department s activities. COMPONENT 2: SOCIAL PROTECTION PROGRAM SUPPORT (US$ 2.58 million) 21. This component supported the implementation of social assistance reforms and the development of technical and organizational capacities for policy formulation and mechanisms to improve resource management and service delivery in the MLSP and its strategy departments. 22. Sub-component 1: Policy framework for streamlined, cost effective and better targeted cash benefits, implementation of local social planning and deinstitutionalization of social work services, and establishment of inspection, supervision and quality assurance functions. In particular, the sub-component was aimed at supporting the formulation of social assistance policy; models for the decentralized management of social protection, interface between the cash benefit regime and labor market measures managed by the National Employment Bureau (NEB); design of technical and organizational options for reform and the formulation of operational structures for monitoring and reviewing cash benefits and social work services; and establishment of a consolidated inspection and quality assurance function. 5

23. Subcomponent 2: Information management systems and IT capacity strengthening of the institutions and departments in charge of administration of cash benefits, the delivery of social services and the interface between cash benefits managed by SWCs and active employment measures managed by the NEB. The sub-component was designed to ensure support for the design and implementation of business plans/processes related to the development of integrated system for administration of social protection programs; harmonization of data systems across key institutions and the development of IT data transfer systems and communication links between the MLSP and other institutions within the social protection system, including data protection and security system upgrade of the equipment and data communications networks; as well as training on sound information management practices and communication and information technology development. COMPONENT 3: PROJECT MANAGEMENT, MONITORING AND EVALUATION (US$ 0.51 million) 24. This component ensured effective administration and implementation of the project by supporting the operations of a Project Management Unit (PMU) which reported to the Minister of Labor and which was responsible for all the day-to-day project implementation activities, including procurement, disbursement, and accounting functions. 25. The Policy Unit in the MLSP and the PMU were jointly tasked with ensuring the effective monitoring and evaluation of project activities, achievements and outcomes during and after project implementation. The Policy Unit originally was staffed with two full-time staff and the project was aimed at supporting capacity building of this Unit through Government financing of one more full-time staff and through World Bank financing of two additional staff for an initial period of two years, to be transferred in mid-2006 to the Government budget. 1.6 Revised Components 26. Project components were not revised. There were, however, several reallocations of the Loan proceeds among the Project sub-components which are mentioned in the Section 1.7 below. 1.7 Other significant changes 27. There were two amendments to the Loan agreement: The first amendment, approved on January 26, 2009, extended the Loan s Closing Date by twelve months to June 30, 2010. The extension was approved to allow the Borrower to complete the development of the software for: (i) the Pension Payout, (ii) Financial Management Information System and Document Management System for the PDIF; (iii) core business functions of the MAPAS; (iv) the Cash Benefit MIS. It also allowed the procurement of IT hardware for the MLSP and the SWCs, and the networking equipment for establishing the linkages among the MLSP, SWCs, Employment Agency, and the PDIF. In addition, Schedule 1 of the Loan agreement was amended to reflect the reallocation of the 6

Loan proceeds within categories of expenditures, including increase in goods and consultant services, and reduction in training and incremental operational costs. The second amendment, approved on June 15, 2010, extended the Closing Date of the Loan by a further eleven months to May 31, 2011, to allow completion of the two critical software packages, (i) the Pension Payout software, and (ii) the software solutions for the Cash Benefit MIS, which were essential for the full accomplishment of the PDO. The amendment also included increasing the disbursement percentage of eligible expenditures to 100% due to the difficulties encountered by the Ministry of Finance to provide timely counterpart funds contribution to some activities; and reallocation of funds within categories including increase in goods and consultants services and reduction in training. The disbursement percentage for the incremental operating costs was not changed and it remained 25% from the World Bank Loan proceeds and 75% from the Government contribution. 28. The restructuring did not involve the introduction of new activities, or revisions of the project development objectives, outcomes, design, or scope. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry Rating: Moderately Satisfactory Soundness of background analysis 29. The Project preparation and Project s design including its objectives and defined components were based on extensive research, studies and policy discussions between the World Bank and the Government which took place over several years preceding the Project approval. The goals and means of the project reflected international best practice as adapted to local circumstances. The background analysis, used as a critical input into the Project s preparation, was grounded in the previous policy and technical assistance work supported by the Social Sector Adjustment Credit (SSAC), the Pension Reform Technical Assistance Project (PERTAP) and the Public Sector Management Adjustment Credit (PSMAC). 30. The support provided through the PERTAP was a critical factor which led to impressive achievements in the implementation of successive governments pension reform agenda. The PSMAC, as a one tranche operation, supported the medium-term strategy of the Government in public sector management, including the fiscal sustainability of the pension system. Other parallel analytical work and technical evidence came out of the assessments prepared by the World Bank Financial Sector Assessment Program team in 2003 which addressed issues related to the Macedonian financial markets as one of the key requirements for the implementation of the new pension system and the introduction of the second pension pillar. 31. The Project initially envisaged inclusion of reforms in social protection and health. While the reforms in both sectors were perceived important, they were at different stages of preparation, and this was seen as a major challenge for successful 7

preparation of the Project. The team, therefore, decided to seek guidance through a Quality Enhancement Review (QER). The QER was carried out on July 16, 2003. The QER commended the two sectoral teams for the decision to seek their guidance at an early stage of project preparation and highlighted the substantial risks associated with preparation and implementation of such a complex project. The QER proposed three alternatives: (i) a human development (HD) policy/adjustment loan in support of the second phase of the Public Sector Management Adjustment Loan (PSMAL); (ii) an HD investment loan focused on the introduction and application of modern information systems in social protection and health; and (iii) separation of the two sectoral components into two different operations: a social protection investment loan, focusing on information systems with linked technical assistance (TA); and a multi-sectoral (including health TA activities) TA loan linked to PSMAL II. Subsequently to the QER, at the Project Concept Note (PCN) Review meeting of October 7, 2003, the decision was taken to split the Project into two separate health and social protection projects. The total financing allocated for the joint project was divided between the two projects providing US$7.5 million the Social Protection Implementation Loan (SPIL). Since this amount could not cover all the activities that the Government was planning to implement in the areas of pensions and social protection, the World Bank management and the Government agreed to increase the loan amount to US$9.8 million. Quality of design 32. The Project preparation was supported by a Policy and Human Resource Development (PHRD) grant in an amount of US$397,320. The Grant financing allowed the team to contract high level experts who engaged in pension policy review and research, short-term and long-term pension system modeling, pension system administration, review of the available technical support and information and communication systems, development and staff training, etc. Social protection specialists were hired to help the Government to fix the multiple inconsistencies between the different databases used in the process of administration of the cash benefits, defining the eligibility criteria for the benefits which existed at that time, etc. Their outputs, reports and recommendations were based on the thorough review of the system of social protection benefits supported by the available statistics at that time and the related institutional, legal and regulatory framework. The Project s design succeeded in capturing the main issues and problems confronted by the Government to ensure improved performance of its functions in delivering programs and services in pensions and social assistance/welfare areas. The choice and definition of the indicators, though, did generate some issues around the design of the Project s results framework that are discussed in section 2.3 of this Implementation Completion and Results Report (ICR). 33. Alignment of the Project s design with the CAS (2003-2006): the Project design and specifically its PDO reflected one of the development objectives of the CAS which called for improving the efficiency, effectiveness, quality of services and coordination of major social protection programs, including pensions and required further institutional strengthening of major institutions of the pension system and reforming the social safety net which had been lacking coordination at the system level. 8

34. The Government was initially interested in a Project designed to reflect only the pension system reform requirements. Nevertheless, growing awareness about the need for initiation of social assistance cash benefit reforms led to inclusion of a social assistance dimension into the PDO. The Project s design ultimately reflected both dimensions, the pension and the social assistance, with attention initially, at the design stage, given to the Pension Administration component, at the expense of the Social Assistance component. 35. The design of the Pension Administration component followed a detailed road map of pension reform activities and related milestones, the so called Critical Path Analysis. The Project team used this as a management and supervision instrument throughout the Pension component preparation and also later during the component implementation. 36. The design of the Pension Administration component reflected properly the needs of the pension system. Administrative improvements were needed to support the already enacted parametric reforms and the planned structural reforms, i.e., the introduction of mandatory and voluntary fully funded pension pillars. Along with the development of a legislative framework for the funded pillars, the Project was to support design and development of a new system of collection of contributions and recordkeeping by the PDIF ahead of the introduction of the second fully funded pension pillar. Governance and management reforms aimed to improve collection performance and financial management at the PDIF required subsequent changes. The latter entailed reforms in PDF s accounting, budgeting and internal controls in compliance with international standards and revisions to the pension legislation. 37. The Project design also reflected appropriately the high priority needs of the Government to confront effectively the abuse of the disability pensions; thus the design included work on the disability insurance system modifications. Furthermore, the new second pillar legislation required strengthened regulatory capacity of the newly introduced institutions (such as Pension Supervision Agency) and further elaboration of the mechanics of the second pillar, including custodial functions, development of necessary rules and regulations, continued public education, etc., as well as a plan for financing transition costs, a wider range of investment instruments available for pension funds, etc. Lastly, the public education campaigns at different stages of Project implementation were planned to ensure that the public is well aware of the pension system reforms. The design of the Project reflected all these priorities accordingly. 38. In the Social Assistance component, too, the most serious challenges were identified and reflected adequately in the design of this component. These challenges included a complex set of issues related to tightening the benefits eligibility for improved targeting, reforms in the benefits administration and establishment of vertical and horizontal interfaces between the MLSP and the SWCs, and the MLSP and other key institutions in the system such as PDIF, Employment Office, Health Insurance Fund (HIF), Tax Office, etc. It also included reforms in the legal and regulatory framework governing the child benefits, improving inspection and quality assurance functions of the MLSP, improving the interface between the social assistance benefits and monitoring the employment status of beneficiaries, etc. 9

39. Notwithstanding the solid preparatory work on the design of this Project component, and the comprehensive approach to the social safety net reform, the ICR team assesses the design of this component as rather ambitious, from both design and implementation aspects. An issue which should have been better understood at the time of Project design was the considerable institutional capacity constraints faced by the Government. While the MLSP Pension team and the institutions of the pension system were relatively better staffed and had already gained experience in the system reform, the capacity of the MLSP social assistance team at that time was rather modest. It could not have been realistically expected that capacity constraints would disappear given the scarcity of trained specialists and financial constraints which limit government agencies' ability to attract experts. This design shortcoming haunted the project during its implementation and is largely to blame for the underperformance of the Social Assistance component. Lessons learned 40. The following lessons learned from previous operations were incorporated into the design of the Project: (i) the need for strong Government ownership and subsequent involvement of the respective Government agencies in the Project implementation; this played a crucial role in ensuring the Project s success and sustainability; (ii) the need for realism and flexibility of the implementation timetable supported by project management instruments such as a detailed critical path analysis; (iii) paying due attention and respect to the political economy of the planned pension and social assistance reforms through incorporating and executing public education campaigns; (iv) adherence to the parametric reforms in the pension system preceding the Project and development and application of subsequent pension modeling tools used for identification of the next generation of the pension system reform; (v) completion and operational readiness of the Project implementation arrangements during the Project preparation which ensured that implementation could start with no delays. Risk assessment 41. Most of the risks were well identified, and the subsequent mitigation measures were adequate. Nevertheless, the risk related to the Information Technology capacities of the client agencies to effectively design, procure and implement IT software solutions was under-rated. In this context, World Bank experience with IT procurement suggests that the length of time needed for implementation of the Project was also underestimated. Likewise, the capacity constraints, reflected in the lack of full understanding of the complexity of the social protection policy actions and interventions financed through the Project, were underestimated. 2.2 Implementation Rating: Moderately Satisfactory 42. Overall, the Project was successful in delivering most of the outputs anticipated under the Project s components, particularly in the case of the Pension 10

component. The delays in implementation which led to two extensions of the Project s closing date by 23 months were partially resolved, with both dimensions of the PDO successfully achieved. 43. The Project s ISRs rated the quality of implementation either Highly Satisfactory or Satisfactory. The only time when both the progress towards achievement of the PDO and the overall implementation progress were rated Moderately Satisfactory, was when the likelihood of achieving the PDO indicator of improving the targeting of social assistance was put at risk by the need to re-tender the selection of the developer for the strategically important Cash Benefit Management Information System (CB MIS) software. The contract with the first supplier for the CB MIS, after two extensions, was terminated in March 2010 in spite of the fact that the original contract completion date was November 15, 2009. The contract was terminated due to incapacity of the contractor to finish the work. The second attempt for the selection of the system developer that took place in May 2010 failed. Only the third selection was successful. Timely identification of procurement problems by the team, appropriate team s advice to the Government and subsequent actions taken by the Government reduced the risk of repeated problems with the third round of bidding and allowed for the selection of a contractor for the task. Nevertheless, the delays caused by the first two failed procurement attempts could not be fully compensated. The Cash Benefit MIS software could be neither completed nor tested prior to the Project closing date. The World Bank team and the Government agreed to transfer the remaining unfinished contractual obligations for the completion of the Cash Benefit MIS software to the new related investment operation the Conditional Cash Transfer Project (CCTP). Fortunately, soon after the closing of the SPIL Project the CB MIS was operationally accepted. The CB MIS is soon expected to start to deliver the anticipated outcomes related to improved administration efficiency and, ultimately, improved targeting of the cash benefits. Factors that positively influenced implementation 44. One positive factor was the strength of the Project Implementation Unit (PIU) managed by a Project Director who was a senior and experienced official of the MLSP. The PIU was the successor of the SPIL Project Preparation Unit with staff hired to ensure adequate preparation of the necessary project documents for the procurement of goods and services for the first year of the project so that project implementation could begin without any delay. With the Project of high strategic priority and strong government ownership, and the Project director and PIU with a good track record, the Project was able to get off to a strong start. 45. The effective implementation arrangements were reinforced by several additional structures established as standing working bodies representative of decision makers and other participating institutions in Project implementation, including: (i) Steering Committee for Pension Reform, which provided leadership and facilitated the cross-sectoral activities necessary for the successful implementation of the pension reform; (ii) Pension Reform Working Group, consisting of representatives of the MLSP, Ministry of Finance, NBRM and directors of the PDIF, MAPAS, and the PIU Pension Coordinator; and in later stages of Project implementation unit; (iii) Social Protection Working Group consisting of representatives of various MLSP departments and representatives of the Ministry of Finance and the State Statistics Office (SSO), 11

directors of the Employment Agency and Institute for Social Activities. The meetings of the working groups were devoted to assisting the PIU in implementation of different stages of contract management; in particular the working groups would review, comment and clear terms of reference for new upcoming tasks or review deliverables from ongoing tasks or tasks under completion, and the meeting notes were shared regularly with the World Bank team. Although, according to some ISRs, the meetings of the working groups were not that frequent throughout the lifetime of the Project, they remained a good instrument by which the Government extended its direct support to Project implementation. Factors that impeded implementation 46. Both Project components involved policy work and ensuing legislative action. For this purpose, technical assistance of international and local consultants was sought and, consequently, hired. In the case of the Pension Administration component, this work proceeded smoothly without any implementation delays. Within the Social Assistance component, implementation of technical assistance activities as inputs into legislative action required intensive efforts on the World Bank team s and the Government s side to ensure selection of the most optimal policy options and their translation into legislative action under a required timeframe. However, institutional capacity constraints in the MLSP delayed implementation. Eventually, the new Law on Social Protection was adopted with a twelve months delay. 47. The Project components also had a heavy focus on development of new business processes, software development to support these new processes and procurement and installation of IT hardware for their implementation. Some of these complex interventions hampered the Project implementation and led to delays which ultimately had to be resolved by allowing more time for implementation and extension of the closing date. In the case of the first large tender aimed at procuring a supplier to develop the Cash Benefits MIS, the originally selected company failed to deliver the software. The MLSP, terminated the contract with that supplier and made a decision to re-launch a new bid. The new invitation for bids was announced on May 21, 2010. The second ICB, however, ended with an attempt of misrepresentation of information in the process of the bid evaluation. This was revealed by the World Bank team and immediate corrective actions were taken while the case was reported to the World Bank s Integrity department. These delays required two extensions of the closing date of 23 months and the transfer of the remaining tasks for the completion and operationalization of the Cash Benefits MIS from the SPIL to the CCTP project. 48. In summary, while the implementation difficulties due to procurement and capacity constraints adversely affected implementation, there was continuously strong commitment at the top level of successive governments, at the level of the lead line ministry, the MLSP, and the Project Director and the PIU to facilitate communications among the various institutions and other actors and to implement the reform program. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 12