L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA Sep ,600, Original Commitment 26,700,

Transcription

1 Public Disclosure Authorized Independent Evaluation Group (IEG) 1. Project Data Report Number : ICRR Public Disclosure Authorized Project ID P Country Azerbaijan Project Name SOCIAL PROT DEV Practice Area(Lead) Social Protection & Labor L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA Sep ,600, Bank Approval Date 27-May-2008 Closing Date (Actual) 31-Aug-2015 IBRD/IDA (USD) Grants (USD) Public Disclosure Authorized Public Disclosure Authorized Original Commitment 26,700, Revised Commitment 20,822, Actual 19,083, Sector(s) Central Government(50%):Compulsory pension and unemployment insurance(30%):other social services(20%) Theme(s) Social Safety Nets/Social Assistance & Social Care Services(33%):Improving labor markets(33%):other social protection and risk management(17%):income Support for Old Age, Disability & Survivorship(17%) Prepared by Reviewed by ICR Review Coordinator Group Robert Mark Lacey Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives "The objective of the Project is to improve delivery of labor market and social protection interventions through strengthened institutions, enhanced institutional and human resources capacity and improved targeting of social safety net programs." (Financing Agreement, page 4, Schedule 1; Project Appraisal Document [PAD], page 12) For evaluation purposes, the objectives will be considered in two parts: (a) to improve the delivery of social protection interventions, and (b) to improve the delivery of labor market interventions, in both cases through strengthened institutions, enhanced institutional and human resources capacity, and improved targeting of social safety net programs.

2 b. Were the project objectives/key associated outcome targets revised during implementation? No c. Components The project had four components (PAD, pages 13-16, ICR Annex 1): Component A. Strengthening and development of labor market institutions, policies and programs and renewal of the National Employment Services (appraisal US$24.90 million, actual US$21.80 million). This component aimed at bringing labor market institutions, policies and programs in Azerbaijan up to modern standards. It was to finance technical assistance, hardware and software, services, training, vehicles and civil works. It would be implemented through the following three subcomponents: modernization of labor market institutions; renewal of the National Employment Services; and facilitating "school to work" transition for the young, and professional orientation and career counseling services for both youth and adult populations. Component B. Social safety net system and policy development and institutional strengthening and capacity building for knowledge-based policies and programs formulation and efficient and effective implementation (appraisal US$17.50 million, actual US$15.32 million). This component was to provide technical assistance, hardware and software, services, training, vehicles and civil works, and would be implemented through the following four subcomponents: establishment of a corporate Management Information System (MIS) into the Ministry of Labor and Social Protection of the Population (MLSPP) and evidence-based monitoring; strengthening operations and management of the targeted social assistance; restructuring of the National Disability Certification Service (NDCS); and piloting of social care centers and integrated provision of social protection interventions. Component C. Technical assistance (TA) for the pension system and policy development (appraisal US$3.1 million, actual US$0.35 million). This component was to provide technical assistance to the Government to: (i) perform a comprehensive analysis of the current pension system, including an actuarial analysis of its affordability and fiscal sustainability and an analysis of individual welfare consequences for different types of participants (formal sector employees, farmers, arrangements for informal sector workers, average-, high-, and low-income earners, etc.); (ii) formulate several options for the long-term development and reform of the pension system and model their fiscal and welfare consequences; (iii) based on the chosen option, prepare a detailed reform strategy in the form of a White Paper on the Pension System Reform; (iv) develop a detailed implementation/action plan; and (v) implement the reform. Component D. Project management (appraisal US$3.1 million, actual US$2.82 million): This component aimed to strengthen the capacity of the MLSPP to efficiently and effectively implement the proposed project, while adhering to the Bank s fiduciary requirements, particularly with respect to procurement, financial management and environmental standards. There were three Level 2 restructurings that resulted in changes to the components: 1. December 12, Component C (TA for the pension system) was cancelled (See Sections 4, 8 and 9 below). 2. April 4, SDR 1 million was reallocated from Unallocated Funds to Component A. 3. August 27, Activities related to the disability certification system and the development of social services standards (subcomponent B.3) were dropped and sub-component B.3 was cancelled. The reason was insufficient local capacity and lack of government commitment (ICR, page 6). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. Actual project cost was US$42.65 million (ICR, Annex 1), including front-end fees. The appraisal estimate was US$48.60 million. Most of the difference is explained by the cancellation of the activities related to pension reform, the disability system, and social service standards. The project team further confirmed that the difference is also explained by the front-end fees. Financing. Of the original IDA credit of US$25 million equivalent, US$19.08 million had been disbursed at closure. The balance was cancelled. There was no other external financing (ICR, Annex 1). Borrower contribution. The Borrower had contributed US$23.60 million by closure, slightly more than the appraisal estimate of US$22.90 million (ICR, Annex 1). Dates. There were two extensions to the closing date, both due to implementation delays. The first, on December 20, 2012, was from September 30, 2013, to December 31, 2014, and the second, on August 27, 2014, to August 31, The project closed on the latter date after a total delay of 23 months (36% of the original schedule). 3. Relevance of Objectives & Design

3 a. Relevance of Objectives The objectives were substantially relevant to the World Bank Group s strategy for Azerbaijan at appraisal. The third pillar of the Country Partnership Strategy (CPS) sought to increase the quality of, and access to, social services. The CPS also refers specifically to improved targeting of social protection as part of the Government s long-term program to reduce poverty and achieve the Millennium Development Goals. The objectives are also substantially relevant to the first of the two focus areas of the Country Partnership Framework, public sector management and service delivery. (The second focus area is economic competitiveness.) The objectives were aligned with the Government s priorities as defined in its National Employment Strategy and National Action Plan for Employment adopted in The strategy s proposed areas of action included reforming labor market institutions and policies, strengthening of the National Employment Service (NES), modernizing the vocational and training system, introducing the lifelong learning concept, and improving social protection of job seekers and unemployed citizens. The strategy also planned to promote employment of youth and women, disabled people, internally displaced persons, refugees, and other groups experiencing difficulties entering the labor market. The objectives were, and remain, relevant to country conditions. Since adopting a market economy, the Government had focused on streamlining and modernizing the social insurance and targeted social assistance administrations. Despite this, the social administration system at project appraisal was unresponsive to client needs, and the number of poor covered remained low. Coverage was estimated at 12.4 percent of the poor population in Less than 4 percent of the entire population benefited from targeted social assistance (TSA). The delivery of social protection interventions, institutions and staffing capacity, and the targeting of social safety net programs all required strengthening. The essential physical and technological infrastructure needed for a modern labor market and social protection system was not in place. Delivery was characterized by low service standards, lack of transparency, and inadequate policies, procedures, and standards. Rating Substantial b. Relevance of Design The project was designed to achieve its development objectives through equipping Azerbaijan with the essential architecture needed for a functioning modern labor market and social protection system. The activities supported by the first two components were generally well fitted to this approach and hence to the achievement of the development objectives. For example, activities were envisaged for modernization of the Labor Code and other labor laws; capacity building in Labor Inspectorates; development of improved occupational classifications based on ISCO-88/ISCO-08, as well as occupational and training standards (employer-relevant occupational skill certification); and strengthening capacity to enforce work and occupational safety standards. The renewal of the National Employment Service would be an essential tool for helping the unemployed and first-time entrants to access the labor market. Enhancing the targeting mechanism for social assistance would be necessary both for efficiency and in order to reach the poor effectively. Design aimed to offer a well-functioning environment for service provision through physical plant enhancements in employment offices and social care centers and through the acquisition of modern equipment such as teaching and physical rehabilitation aids. The design of the information technology (IT)-related activities to improve business processes within the Ministry of Labor and Social Protection of the Population (MOLSPP) and in its regional offices appropriately avoided focusing on complex and sophisticated products and services, opting instead to address critical infrastructure needs, common services, and strengthening general capacity to adopt and utilize improvements based on the use of IT. However, the addition, at the Government s request, of the activities related to pension reform (Component C) undermined the coherence of the design and also its relevance since, as the ICR (page xv) points out, this was only remotely if at all related to the project s development objectives. These activities also threatened to make the scope of the project s interventions too wide and to distract from its central purpose. In the event, Component C was dropped at the end of 2012, since the Government could not reach an internal consensus on a strategy and draft legislation for pension reform, which IDA had made a condition for supporting this activity. Design also missed the opportunity to learn from the experience of other oil-revenue-rich economies, such as Norway and Canada, in finding and applying efficient mechanisms for channeling funds to the population. Rating Modest

4 4. Achievement of Objectives (Efficacy) PHREVISEDTBL Objective 1 Objective To improve the delivery of social protection interventions through strengthened institutions, enhanced institutional and human resources capacity, and improved targeting of social safety net programs. Rationale Outputs A new computerized network connected 14 departments in MOLPP with the State Labor Inspection Service, Social Welfare Service, State Employment Service (SES), State Medical-Social Expertise, Rehabilitation Service, more than 70 regional offices across the country, and some other institutions and centers. A unique identification (ID) system was set up with an ID number for each citizen. The system allows MOLPP to analyze employees by age, gender, education, labor conditions, and wages, and conduct regional comparisons. The system can also be used by employers for their own human resources purposes and by employees to obtain employment verification letters and the like. As of project closure, more than 2.3 million people are registered in the system. As of June 2015, less than a year after the system was established, more than 40,000 people had accessed it. The system can be expanded to add more features in the future. A single, unified call center was established, servicing on average more than 430 calls per day. An online queuing management system was implemented for scheduling appointments, available through the e-government portal and the call center, improving responsiveness to citizens needs. A Citizen Reception Office was created, introducing a higher level of transparency and efficiency into the social services administration system. The new system prevents the gathering of large crowds of dissatisfied clients in the Ministry s lobby without appointments, guidance, or any information on whom to meet. As of project closure, an average of 138 persons were visiting the Citizens Reception Office daily. The project financed the implementation of a Labor Force Survey (LFS) in 2014 and of a Living Standards Measurement Survey (LSMS) in 2015 to increase the capacity within MOLSPP for conducting surveys. According to the ICR (page 14), "[these interventions] introduced the culture of using data and business processes for decision making for both institutions and citizens, which is probably the most significant outcome of the IT interventions supported by the project." Outcomes According to the ICR (page 14), as a result of the project, the MOLSPP s IT Department now has "impressive human resources and other capacities that can manage and implement the most complex projects. The general attitude toward the use of IT in daily business processes has positively changed and staff are more open to pursuing new ideas for the use of IT for further improvement of social services for citizens." Project-supported investments in IT, physical infrastructure, technical assistance and training have made it easier and quicker for clients to access the Ministry and social services as well as to track and receive their benefits. Concerns and complaints can now be lodged in a timelier manner. For its part, the Government and its agencies are better equipped to track and verify beneficiaries and to respond to their concerns more rapidly. However, due to lack of data at project closure, the ICR contains no quantitative estimates or examples of these outcomes. According to a beneficiary survey, 83% of users were satisfied with the improved social administration and employment services, exceeding the target of 70%. However, there was no baseline for the survey, so the level of satisfaction prior to the project is unknown. Moreover, the ICR contains almost no information about the methodology used for the survey sampling size, interviewing techniques and so forth which would have enabled a judgment about its reliability. The percentage of the very poor population covered by the Targeted Social Assistance (TSA) program was 17.6% in 2015 compared to an estimated baseline of 12.4% in The error of inclusion (people receiving TSA benefits but who are not poor) was only 3.8%. The error of exclusion (people who should receive benefits but do not) is by contrast high at over 80%. This, however, reflects the fact that the program is still small only 4.9% of the country s population benefits from TSA, whereas 9.1% are estimated to be poor enough to qualify for assistance.

5 Data for 2015 on the number of beneficiaries of safety net programs are only available for the first quarter. They indicate, however, substantial progress. For example, the number of beneficiaries of social safety net and female social safety net programs in the first quarter was 117,622 and 60,000 respectively, compared to 145,000 and 72,000 respectively in 2008 for the whole year. Rating Substantial PHREVISEDTBL Objective 2 Objective To improve the delivery of labor market interventions through strengthened institutions, enhanced institutional and human resources capacity, and improved targeting of social safety net programs. Rationale Outputs All of the outputs discussed under the first objective are also relevant to this one. In addition: A Labor Agreement System was established in July 2014, under the auspices of which 165,000 labor contracts had been registered by project closure. The eventual aim would be to register all significant labor contracts, estimated to be some two million. This system allows the employer to register all employees, tracking employment history and salaries, providing information on each employer and employee, and detailing social protection benefits for each employee. The system also enables employees to see all information on agreements, thus improving the transparency of employment contracts. The system is integrated into the e-government portal. The labor inspection system was improved by equipping inspectors with 215 tablets to support the input of information on employees and to document informal sector employment, and also with "compact suitcases" or mobile laboratories for on-site evaluations of conditions. 43 new module training programs and three new module training centers were developed to improve or adapt the skills of the unemployed. The centers were fully functional at project closure. Operational guidelines and manuals for a new menu of Active Labor Market Programs (ALMPs) were prepared and are in use. Participants in ALMPs are assisted in the search for employment, especially (but not exclusively) in the occupational field in which they have been trained. 245 occupational standards were adopted, 200 relevant training standards were prepared, and 63 qualification standards were developed. There were no occupational standards in place in Azerbaijan prior to the project s intervention. The project introduced and financed training for students by creating virtual apprenticeships in two universities in Baku for them to be able to gain practical and theoretical knowledge on how to run and manage business enterprises and to interact with people outside of Azerbaijan. Outcomes The results of the client satisfaction survey discussed under the first objective are also relevant to this objective. According to the Labor Force Survey, 45% of job seekers used the National Employment Service compared to the baseline of 25% in 2008, surpassing the target of 40%. (In 2014, there were 112,221 registered job seekers in the NES, while in the first half of 2015 this number was 115,810; 29,062 of them had the status of unemployed.) 62% of the registered unemployed in 2014 who participated in the ALMP were provided with jobs through the program. Data for the first half of 2015 show that 49% of the total registered unemployed were provided jobs. Both figures compare well with the 2008 baseline figure of 20%. Data for 2015 on the number of beneficiaries of unemployment insurance and other labor market programs are only available for the first quarter. However, they indicate substantial progress. For example, the number of labor market program beneficiaries in the first quarter was 12,915, compared with 46,467 for the year 2008 as a whole. The ICR (page 15) states that the Virtual Enterprise Program has been successful; however, little evidence is presented to support this statement.

6 Rating Substantial 5. Efficiency There was no ex-ante or ex-post cost-benefit analysis or cost-effectiveness calculation. The ICR (pages 16-17) points out that the project would have achieved economic gains through the introduction of cost-effective processes for the administration of the labor market and safety net programs, more efficient use of public resources through better targeting, and reduced transaction costs for beneficiaries. However, insufficient data were available at the time of ICR preparation to measure these. There were no cost overruns. However, there were operational and administrative inefficiencies reflected in two extensions to the closing date totaling 23 months, or 36% of the original schedule. The start of implementation was held up for political reasons, and there were further administrative delays during implementation (see Section 9 below for more details). Project management costs were 43% higher than estimated at appraisal. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Appraisal 0 ICR Estimate 0 0 Not Applicable 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated substantial and that of design modest. Although data are incomplete, there are sufficient indications that the intended outcomes of the improvements supported by the project were substantially achieved. Efficiency is rated modest, reflecting the lack of quantitative information on the project s benefits and significant operational and administrative inefficiencies. Overall, the project had moderate shortcomings and Outcome is assessed as moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating

7 The political risk is considered to be low. All project interventions (except the cancelled pension reform activity) requiring the backing of the law have been supported by necessary legislation. The project-financed improvements enjoy widespread public support, according to the beneficiary survey. They have also eased the task of those in government administering social services and the labor market. It is unlikely that any future administration would wish to reverse these gains. The administrative and institutional risks are also low. The project avoided the creation of new institutions and focused its capacity-building efforts on MOLSPP and other existing agencies. A large percentage of staff benefited from training and new technological tools. The information technology supplied was kept as simple as possible and did not embody complicated or sophisticated software likely to stretch unduly the capacity of the operators. The most important risk stems from the fall in international oil prices. As a result of this, the Ministry of Finance reduced MOLSPP s budget allocation in 2015 (the extent of the reduction is not mentioned in the ICR). The ICR (page 18) reports that at project closure, MOLSPP still had sufficient funds to operate all the programs supported by the project, but the duration of this state of affairs is unknown and depends to a large extent on the future course of oil prices. a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry The concept of the project was well aligned with Azerbaijan s strategic priorities. The reform of labor market institutions, policies and programs was regarded as an urgent challenge, critical for supporting the country s competitiveness and employment agenda. Design built on the previous Pension and Social Assistance Project, and was informed by findings of surveys that identified the need to improve the targeting mechanism of benefits for those in need. The Bank s preparation team was experienced in the social protection sector and had established a good professional relationship with the Government. The PAD identified a number of lessons learned from previous projects that were to be reflected in design, including: (a) taking account of local implementation capacity limitations; (b) ensuring project ownership in the implementing agency (in this case the MOLSPP), the Government as a whole, and the wider group of potential stakeholders through broad consultations during preparation; (c) the need for training during the preparation phase on Bank guidelines, safeguards, procedures, and fiduciary responsibilities; (d) keeping design simple with the least complex institutional and information technology arrangements possible; (e) bearing in mind the time needed to implement reforms, sometimes in the face of vested interests with delaying potential; (f) establishing adequate monitoring arrangements before start-up; and (g) identifying and implementing key actions early, in particular preparing bidding documents to the extent possible before effectiveness. The extent to which these lessons were applied was variable. The pension reform component was not originally included in the project and was added at the request of the Ministry of Finance at a very late stage in the preparation process. However, there was disagreement between the Ministry of Finance and MOLSPP about the inclusion. In particular, the Head of the Pension Fund was opposed to the activity. His position was eventually reinforced by the new Minister of Labor and Social Protection, who was appointed in late The ICR does not elaborate upon the nature of the disagreements. Broader and more careful consultations could have better informed the Bank s decision as to whether or not to add the component. As it was, the lack of full Government ownership led to the eventual cancellation of the activity. Similarly, the ICR (page 8) reports that "political issues around disability benefits were not considered" (again, without specifying what these issues were). This too led to the later cancellation of the relevant activities. The monitoring system was not fully set up during preparation, and M&E design featured a number of shortcomings (see Section 10 below), which limited the effectiveness of M&E activities during implementation. Although risks were well identified, mitigation measures proved insufficient in some cases. In addition to the pensions and disability issues already referred to, the risk related to the MOLSPP s capacity to implement a relatively complex operation was rated low and turned out to be high. Overall, these shortcomings are considered to be significant and Quality at Entry is rated Moderately Unsatisfactory. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The Bank team was equipped with the appropriate professional skills, and the ICR (page 19) reports that sound advice was given to the Borrower. Although there were three different Task Team Leaders, the transition between them was smooth and continuity was largely unaffected. Implementation support to the Project Implementation Unit (PIU) located in the MOLSPP took place on a constant basis and with frequent communication. The mid-term review was carried out on schedule and provided useful suggestions on how to improve progress

8 toward the achievement of the development objectives. In its summary of the Borrower s comments, the ICR (page 21) states that "the Borrower views the Bank as having been highly responsive during project implementation." The Bank team demonstrated flexibility in adapting implementation to changing conditions. This flexibility was necessary given that there were two changes of Minister at the MOLSPP during implementation, one in late 2009 (just before project effectiveness) and the other in late The first of the new Ministers showed diminishing interest in the project, and this was reflected in Supervision Report ratings. Throughout 2010 and 2011, progress in both implementation and towards development objectives was downgraded from satisfactory to moderately satisfactory, and from the beginning of 2012 to moderately unsatisfactory. The new Minister (the previous Director of the Pension Fund) appointed in late 2013 demonstrated considerably more interest in the project than his predecessor. This was reflected in Supervision Reports, where the rating of progress from February 2014 onwards was raised to moderately satisfactory, where it remained until closure. There were three restructurings, and all were appropriate. During the August 2014 restructuring, the Results Framework was updated to better reflect developments on the ground, including the introduction of a few additional activities of interest to the new Minister and his priorities, while remaining consistent with the project development objectives. There were, however, two moderate shortcomings: The cancellation of the pension component could have taken place earlier. As it was, the Bank spent several years trying to push forward a reform agenda that clearly had little Borrower buy-in. A number of significant procurement issues were not identified in Supervision Reports, but only during a procurement review mission in April 2013 (see Section 11 below). Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance The Government displayed a strong commitment to labor market and social protection reform, and saw the project activities as a continuation of previous efforts. However, the Government reached consensus on which components to include in the project only at a late stage in implementation. As noted above, while the Ministry of Finance insisted late in the preparation process on adding the pension component, the Head of the Pension Fund and the MOSLPP did not agree, resulting in a lack of ownership and eventual cancellation of the component. Project start-up was delayed because of political changes at the ministerial level. Throughout project implementation, the Government did not follow the timelines that had been agreed during project preparation, stated in the PAD, and confirmed during Negotiations. For example, the implementation of the M&E system was slow because of changes at ministerial level, and the Project Implementation Unit (PIU) faced the challenge of key staff leaving because the delay in credit approval and signing resulted in unpaid salaries and an increased level of uncertainty. Matters improved only after late 2013 with the appointment of a new Minister of Labor and Social Protection, who also named a new Executive Director for the PIU. After this, the pace of implementation accelerated, and several important results were achieved during the last 18 months of the project. In late 2014, the Government began to allocate significantly increased amounts of its own resources in an effort to complete project activities before closure. Overall, the shortcomings in Government performance are considered significant, leading to a rating of moderately unsatisfactory. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance The Implementing Agency was the MOLSPP, but day-to-day administration was in the hands of the PIU located in the Ministry. The staff of the PIU were well qualified and experienced in dealing with Bank operations. Nevertheless, morale and commitment were adversely affected by the ministerial changes that occurred in late 2009, shortly before effectiveness. This delayed effectiveness until ten months after Board approval and slowed implementation. Some problems were not the fault of the PIU. For instance, the labor market component suffered initial delays when some key contracts did not attract sufficient interest from bidders. Despite the difficult environment, the Unit performed its financial management functions satisfactorily throughout implementation (ICR, page 20). Procurement was less well handled.

9 Some issues led to an upgrade of the procurement risk to substantial following a post-mid-term-review mission. The PIU was able to address these issues within a seven-month period, after which the risk was adjusted to moderate again. The Unit s performance improved considerably after the appointment of a new Minister of Labor and Social Protection in late The last 18 months of the project were characterized by smooth administration and quick resolution of implementation issues. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The PIU, supported by the MOLSPP s M&E Department, was responsible for the operation of M&E. Design of the M&E framework had a number of shortcomings. Of the four outcome indicators, three did not have a baseline (including the level of user satisfaction as measured by beneficiary surveys). Moreover, the baseline of the only outcome indicator to have one (the coverage of the very poor population by Targeted Social Assistance) was incorrect and remained so throughout implementation. The ICR (page 10) reports that it was unable to recreate the baseline as reported in the PAD using LSMS data. In addition, there were 20 intermediate outcome indicators, making systematic data collection and analysis across such a large number of indicators difficult. b. M&E Implementation Some improvements to the M&E framework s weak design were made during implementation. Data were collected for two of the three missing outcome indicator baselines within the first twelve months of project implementation. There was, however, no attempt to supply the customer satisfaction indicator with a baseline. Some intermediate outcome indicators were dropped, but others added; there were still 19 at closure, meaning that data collection and analysis were only marginally simplified. As noted above, data collection during the first two or three years of implementation was slow, though this reflected issues within the Ministry rather than with the system per se. However, improvements financed under the project particularly in the IT field strengthened the Government s own M&E system, which is a critical step in enhancing sound and transparent policy decisions in the future. c. M&E Utilization The M&E framework was used to track project implementation and informed decisions concerning the three restructurings. M&E Quality Rating Modest 11. Other Issues a. Safeguards The project was expected to have a limited environmental impact and was classified at appraisal as Category "B" for environmental assessment purposes. Other than OP 4.01, no safeguard policies were triggered. OP 4.01 was triggered because of civil works to be financed under the operation: new electrical systems, bathrooms, windows, painted walls, and so forth. The ICR (page 11) reports that an Environmental Management Plan was prepared by the MOLSPP and that the draft was discussed with local officials and stakeholders. The Plan was disseminated in local newspapers, and consultative meetings were held with local officials, the media, school directors, teachers,

10 parents, and students. The ICR does not state whether or not any environmental issues arose during construction or if OP 4.01 was complied with. The project team subsequently informed IEG that there was full compliance. b. Fiduciary Compliance Financial Management. The ICR reports that financial management was satisfactory throughout implementation. "The project used a suitable automated accounting system, including planning and budgeting, had well qualified staff, and prepared accounts and reports in a satisfactory manner. Financial covenants as stated in the financing agreement were complied with, and quarterly financial reports, annual audited financial statements, and external audits were submitted on time. The external auditors opinions were not qualified. Financial management and disbursement-related control policies and procedures were documented in the project s operational manual" (ICR, page 11). Procurement. The ICR (page 11) states that "procurement was carried out in compliance with Bank procedures." Although the "highly qualified staff" in the PIU were in high demand by other agencies, "new staff with similar experience were hired, allowing for a smooth transition." However, the ICR also reports that a procurement post review mission (which took place in April 2013) identified a number of issues. Among these: a) the price adjustment formula, which is only applicable if the contract duration is more than 18 months, was also used for contracts that had a shorter duration; (b) for some awarded contracts the evaluation process was questionable; (c) the value of some contracts was increased during implementation without including the proper amendments such as a justification for the increase; and (d) items that did not have prior Bank approval and were not included in the procurement plan were nonetheless procured. As noted above, this caused the procurement risk to be elevated from moderate to substantial. According to the ICR (page 12), these issues had been resolved by the end of 2013, and the procurement risk was once again lowered to moderate. c. Unintended impacts (Positive or Negative) None reported. d. Other Ratings Ratings ICR IEG Outcome Moderately Satisfactory Moderately Satisfactory --- Risk to Development Outcome Modest Modest --- Bank Performance Moderately Satisfactory Moderately Satisfactory --- Borrower Performance Moderately Satisfactory Moderately Satisfactory --- Quality of ICR Substantial --- Reason for Disagreements/Comment Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons IEG draws three lessons from the experience of preparing and implementing this project:

11 Thorough and broadly-based consultations at the appraisal stage can help to establish government ownership and avoid the inclusion in the project of activities that do not enjoy a sufficient degree of political consensus. As a result of acceding too quickly to the Ministry of Finance s request to include a pension reform component, client ownership was undermined, and the Bank had to spend several years trying to push forward the pension agenda when there was no desire on the part of the Borrower to do so. More careful attention during supervision missions can lead to the early identification and rectification of procurement issues. In this case, the issues were not identified until a procurement post review mission took place in April 2013, four years after effectiveness. Focusing attention on existing institutions and on building their capacity has the double advantage of helping to ensure sustainability and reducing the complexity of project implementation. In this case, the creation of new institutions and major reorganization of existing ones were deliberately avoided. Similarly, the choice of relatively straightforward IT applications, rather than complicated and sophisticated technologies, facilitated the absorption and beneficial use of the technology by the Borrower s entities. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR is generally thorough and informative. The discussion of the achievement of objectives is clear and largely evidence based, with an appropriate distinction made between outputs and outcomes. There are, nevertheless, some unsupported assertions (for example, that the Virtual Enterprise Program was successful). The discussion and rating of efficiency is also adequate, given the absence of any quantitative estimates of value-for-money. The Lessons are evidence based and fairly complete, although a few important ones are not specified. A clear discussion in a single place of the political difficulties that led to the cancellation of the pension and disability component would have been useful. Instead, the story has to be patched together from a number of scattered statements, one of which does not appear until late in the document (the "Lessons Learned" Section). The discussion of procurement contains two self-contradictory paragraphs. There is no discussion of compliance with the safeguard policy that was triggered. In Annex 1, the figures in Table A do not add up, and the total financing requirement differs from the financing provided as shown in Table B. Physical and price contingencies are mistakenly included in actual costs, and they are, in any event, much too large. There are also differences between the project cost and financing figures in the Operations Portal and those in the ICR. The project team subsequently clarified the cost and financing figures, as reflected in Section 2d. a. Quality of ICR Rating Substantial