Mongolia: Social Security Sector Development Program

Validation Report Reference Number: PVR196 Project Number: 33335 Loan Numbers: 1836 and 1837(SF) November 2012 Mongolia: Social Security Sector Development Program Independent Evaluation Department

ABBREVIATIONS ADB Asian Development Bank M&E MECS MFE MSWL NDF PCR SDR TA monitoring and evaluation Ministry of Education, Culture, and Science Ministry of Finance and Economy Ministry of Social Welfare and Labor Nordic Development Fund program completion report special drawing right technical assistance NOTE In this report, "$" refers to US dollars. Key Words adb, asian development bank, means testing, mongolia, pcr, poverty reduction, sector development program loan, social insurance, social protection, social security sector development, social security system, technical assistance, validation, welfare services The guidelines formally adopted by the Independent Evaluation Department (IED) on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of IED management, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, IED does not intend to make any judgments as to the legal or other status of any territory or area.

PROGRAM BASIC DATA Project Number: 33335 PCR Circulation Date: Dec 2010 Loan Numbers: 1836 and 1837 (SF) PCR Validation Date: Nov 2012 Program Name: Social Security Sector Development Program Country: Mongolia Approved ($ million) Sector: Health and Social Protection ADB Financing: ($ million) ADF: Program Loan: 8.0 Project Loan: 4.0 Total Program and Project Costs: Program Investment Project Loans: Program Loan (SDR equivalent) 9.5 10.0 8.0 6.4 Actual ($ million) 5.7 10.4 5.7 4.0 Project Loan (SDR equivalent) 4.0 3.2 4.3 3.0 OCR: 0.0 Borrower: Program 1.5... Investment Project 2.0 1.5 Beneficiaries: 0.0 0.0 Others: 0.0 0.0 Cofinancier: Nordic Development Fund Total Cofinancing: Investment Project 4.0 4.6 Approval Date: 28 Aug 2001 Effectiveness Date: 8 Feb 2002 21 Feb 2002 Signing Date: 08 Nov 2001 Closing Date: Program Loan Project Loan Project Officers: Validators: Quality Reviewer: R. Schoelhammer K. Saleh R. Hiraoka L. Wu W. Walker J. Dimayuga, Senior Evaluation Officer, IED2 S. Thalakada, Consultant R. Sabirova, Evaluation Specialist, IED1 Location: Peer Reviewer: Director: 31 Oct 2005 30 Apr 2007 ADB headquarters ADB headquarters ADB headquarters ADB headquarters ADB headquarters C. Kim, Principal Evaluation Specialist, IED2 W. Kolkma, IED1 22 Nov 2007 9 Oct 2009 = data not available, ADB = Asian Development Bank, ADF = Asian Development Fund, IED 1/2 = Independent Evaluation Department (Division 1/2), OCR = ordinary capital resources, PCR = program completion report, SDR = special drawing right. A. Rationale I. PROGRAM DESCRIPTION 1. Between 1990 and 2001, poverty levels have risen to over 35%, with real unemployment estimated at over 17%. Among the most seriously affected were the elderly, disabled, children, and single parent families. The government adopted new policies to reduce poverty, and civil society became active in providing informal social security services. However, much of the formal social security system had shown itself to be increasingly ineffective. Reform was, therefore, necessary to ensure that the social security system could serve Mongolia s needs.

2 The top priorities were the maintenance and improvement of social welfare services for the most vulnerable. At the same time, the system needed to help the unemployed and underemployed find productive and sustainable jobs. Given the government s financial and technical capacity constraints, the Asian Development Bank (ADB) supported the restructuring of the social security system through the Social Security Sector Development Program (the program). 1 B. Expected Impact 2. The main expected impact of the program was poverty reduction and the increased security of vulnerable groups by creating a sustainable, cost-effective, and efficient social security system that would deliver adequate essential services to the poor. The reformed social security system was intended to target 870,000 principal beneficiaries, 72% of whom were living below the poverty line. The targeted direct beneficiaries of the investment project were 20,000 very poor people, including vulnerable children, the elderly, and people with disabilities; these were all groups that had disproportionately borne the brunt of the difficult transition period. C. Objectives or Expected Outcomes 3. The objective of the program, which was compatible with the sector goals, was to strengthen the social security system s ability to deliver essential welfare, insurance, and employment services to the poorest, very poor, and those close to the poverty line. The expected sector outcomes were to reduce poverty and increase security among vulnerable groups by increasing employment and providing essential social security services for those unable to work. The program s policy reforms were expected to improve social welfare services and benefits, employment services, and social insurance systems (unemployment, occupational injury, and disease insurance). 4. The program (comprising both program and project loans) was also expected to contribute to human development by providing employment-oriented education and training for the unemployed, vulnerable groups, and people with disabilities. It also sought to improve the quality of life for the poor who depend on social welfare services by enhancing existing services and developing community-based social welfare services that were more attuned to local needs and demands. The program s investment project was expected to help capacity building by strengthening the government s ability to oversee, coordinate, and manage social security services. D. Components and Outputs 5. The program loan was designed to support key reforms contained in the government s five-year Social Security Sector Action Plan (20002004), including the following: (i) improving the quality and coverage of social welfare services and benefits, (ii) developing pro-poor labor market policies, (iii) strengthening social insurance operations to increase compliance, and (iv) developing a social security sector master plan. Meanwhile, the outputs of the investment project, supported by the program, were the following: (i) improved social welfare services, including the upgrade of seven nursing homes and the National Center for the Rehabilitation of the Disabled, establishment of two pilot social security centers, and a survey of existing community-based activities; (ii) employment promotion, including the establishment of 25 employment training centers, 40 employment education centers, 6 regional methodological 1 ADB. 2001. Report and Recommendation of the President to the Board of Directors on Proposed Loans and Technical Assistance Grant to Mongolia for the Social Security Sector Development Program. Manila.

3 centers, 5 business incubators, and management information system and labor market analysis systems at the Labor and Social Welfare Service Office; and (iii) enhanced social insurance through improving the capacity of the State Social Insurance General Office and its branches, examining the feasibility of expanding social insurance to the self-employed and herders, launching a public awareness program, and training social insurance inspectors. While the program framework (report and recommendation of the President, Appendix 1), showed the number of targeted people under each of the outputs, it did not provide any monitorable baselines and target indicators. However, the PCR 2 in its Appendix 2 provides baseline indicators and actual performance indicators, which indicate an overall increase in the number of affected people against the baseline output targets. E. Provision of Inputs 6. The total cost of the reform program was estimated at $9.5 million, with $8.0 million to be financed by an ADB program loan and the balance by the government. The loan funds were to be disbursed in three tranches over 4 years. The first tranche of $2.0 million was to be disbursed upon loan effectiveness; the second of $3.0 million, 24 months after the first tranche; and the third of $3.0 million was disbursed 48 months after the first tranche. Actual disbursement totaled only $5.7 million equivalent in two tranches: (i) SDR1.6 million ($2.0 million equivalent) and (ii) SDR2.4 million ($3.7 million equivalent) once the policy conditions had been fully met. The third tranche of SDR2.4 million ($3.8 million) was canceled, with the government relinquishing its claim for that tranche mainly because of Mongolia s improved budgetary situation, the introduction of a number of universal benefits, and the adoption of means testing for social assistance. The cost of the project was estimated at $10.0 million equivalent, with $4.0 million equivalent to be financed by an ADB loan. The Nordic Development Fund (NDF) was to provide parallel financing of $4.0 million, with the balance of $2.0 million from the government. The actual cost of the project was $10.4 million. F. Implementation Arrangements 7. The Ministry of Finance and Economy (MFE) was the executing agency for the program loan and the Ministry of Social Welfare and Labor (MSWL) was the executing agency for the project loan. Both the imprest account of the ADB project loan and the special account of the NDF credit were administered according to MFE guidelines. At the beginning of project implementation, a project management unit (PMU) was established at MSWL, and a sub-pmu at the Ministry of Education, Culture, and Science (MECS). To increase coordination and cooperation between the MSWL and the MECS, the executing agencies, in consultation with ADB, merged the two units into one. Policy guidance was provided by a project steering committee. II. A. Relevance of Design and Formulation EVALUATION OF PERFORMANCE AND RATINGS 8. The PCR rated the program highly relevant at appraisal and completion. However, this validation lowers the rating of the program at completion to relevant, as the program lacked exceptional value-adding features in its design to justify the highest rating. 3 The program was designed to address the needs of the social protection sector and poor beneficiaries after the 2 ADB. 2010. Completion Report: Social Security Sector Development Program in Mongolia. Manila. 3 The Urban and Social Sector Division (EASS) of EARD disagrees with the downgrade in rating to relevance since the overall objectives and approach of the program were still, and continue to be highly relevant ex post.

4 political and economic transition of the 1990s, when the human development and economic stability of the socialist period suddenly collapsed. The government s determination to improve the social security system was reflected in its proposal to establish a Social Safety Net Sector Development Plan, which led to the formulation of the program. The development plan was provided through an attached technical assistance (TA) 4 for this loan. The program was also in line with the Poverty Partnership Agreement between the government and ADB. The program design was based on TA reports and extensive discussions between ADB and the government. A national workshop was held in April 2001 with a wide cross section of stakeholders, including the private sector and development partners, who provided feedback on the scope and activities of the program. 9. The policy work initiated under the program was significant and was followed by two subsequent ADB projects relating to safety nets for food security and social sector support. The emphasis on capacity building helped build ownership at all levels and ensure sustainability. However, the relevance of the program design was significantly reduced by the failure to complete the reforms as planned, since the intended targeted social welfare benefit system was dropped and a universal benefits system was chosen instead. This change in the government s position was brought about by an improved budgetary situation following the surge in copper prices, and this led to the cancellation of the third tranche of $3.8 million in 2007. B. Effectiveness in Achieving Outcomes 10. The PCR rated the program effective in achieving its outcomes. This validation supports the rating because the new legislation, together with the training, equipment, information sharing, and a fast-growing economy enabled the expected outcomes to be reached effectively. Details of the outcomes for each component are as follows: 11. Component 1: Social welfare services. The expected outcome was quantitative and qualitative improvements in social welfare services for up to 230,000 people, 70% of whom were poor, 20012005 and beyond. According to the Labor and Social Welfare Service Office, this target was nearly doubled to 442,083 by 2008. The qualitative improvements were also evident: (i) the new one-stop social security centers that had been piloted under the project were being replicated by the MSWL; and (ii) nursing homes had been upgraded. 12. Component 2: Employment promotion. This component sought to achieve quantitative and qualitative improvements in employment services for up to 143,000 people, 43% of whom were poor, 20012005 and beyond. Quantitative achievements were greater than expected with increases as follows: (i) persons employed through employment services increased to 328,547 during 20012005 (baseline 14,000 in 2000); (ii) persons receiving skills training and re-training increased to 155,354 during 20012005 (baseline 5,000 in 2000); and (iii) persons involved in public works increased to 129,922 during 20012005 (baseline 5,100 in 2000). This component contributed very significantly to the overall effectiveness of the project. The approach was participatory and hence enjoyed high levels of support, which resulted in all targets being met or exceeded. 13. Component 3: Social insurance. Against its expected output, this component achieved the following: (i) the number of newly insured persons in 20022006 reached 356,800; (ii) the State Social Insurance General Office s management, key staff, and information and 4 ADB. 2005. Technical Assistance Completion Report. Strengthening Policy for Social Security Reform. (TA 3709- MON). Manila.

5 communication technology specialists were trained in MIS management and operation; (iii) a local consulting team conducted surveys and projections of necessary funds, contribution rate, pension level, and other factors required to include the targeted population, particularly herders and self-employed, into a basic social pension insurance scheme; and (iv) public awareness program was launched to inform and educate the public about the advantages of the new social security sector reform and innovations. C. Efficiency of Resource Use in Achieving Outcomes and Outputs 14. The PCR rated the program efficient in achieving the outcomes and outputs. No economic internal rate of return was established during project preparation, so no reevaluation was possible. Primarily, delays were experienced in starting the community-based pilot projects, because the concept was new and both executing agencies took time to build required support and ownership. The program loan, which was expected to be implemented over 4 years, was extended once, with closure of the loan being delayed by about 2 years. The project loan was expected to be implemented over 5 years, but following three extensions, closure was delayed by about 2.5 years. The PCR stated that the project received its final extension so that it could be closed at the same time as the NDF loan. 5 The NDF loan was extended for construction of the Labor Exchange Bureau and the National Center for the Rehabilitation of the Disabled, which had not been planned at appraisal. The PCR also stated that the delay in starting the community-based pilot projects was because the concept was new and it took time to build consensus and ownership. 15. This validation downgrades the rating to less than efficient 6 considering the significant implementation delays and the non-utilization of the third tranche of the program loan. The delays in closing the program and project loans were considerable and the third tranche was canceled. This validation considers that the delays and cancellation caused inefficiencies in the delivery of the intended outcomes and outputs of the program and project. As a result, the loan resources were not used as efficiently as had been expected under the original program. D. Preliminary Assessment of Sustainability 16. The PCR rated the program likely to be sustainable. This validation supports that assessment. The program played a key role in forging a new vision for social protection and instituting fundamental reforms in social welfare, employment promotion, and social insurance. These reforms paved the way for the creation of legal frameworks, institutions, and budgetary support systems required for sustainable change. Adoption of the sector strategy paper was a major reform milestone. Activities under the program fundamentally transformed the structure of the sector, management of data, and organization of technical education and vocational training. In addition, capacity building created a cadre of trained specialists, many of whom have risen to decision-making levels and continue to accept ownership of the outputs from the project. 17. Areas such as employment promotion, social insurance expansion, and improvement of service standards had high levels of political support. However, key concepts relating to social welfare reforms, such as provision of community-based services and proxy means testing, were 5 EASS/EARD commented that the delays in closing over the last 2 years of implementation were due largely to agreeing to close at the same time as Nordic Development Fund so that a joint PCR mission could be undertaken and the ADB PCR could serve as a completion document for Nordic Development Fund. 6 EASS/EARD disagrees with the downgrade in rating to less than efficient since the project outputs have been achieved in a timely manner and achievements were more than the originally planned targets.

6 new and politically sensitive. The sustainability of the welfare benefits became questionable with the government opting to make them universal instead of only for the poor. However, this question of sustainability would not have arisen if the 2010 Draft Social Welfare Law, which targets welfare benefits and implementation of the proxy means test, had been enacted and enforced by government. E. Impact 18. The program had a significant impact. The anticipated impact was reducing poverty and increasing security among vulnerable groups by improving social welfare and social insurance services, and increasing employment and income-generating opportunities. Performance indicators in the design and monitoring framework were as follows: (i) a reduction in the number of long-term unemployed, (ii) a reduction in the number of people living under the poverty line, and (iii) increased participation in insurance for unemployment and occupational injury and diseases. Improvements in all these three areas, as outlined under item B (Effectiveness in Achieving Outcomes), have helped reduce poverty and increase security among vulnerable groups. As outlined under item D (Preliminary Assessment of Sustainability), the reforms led to the establishment of the necessary legal frameworks, institutions, budget arrangements, capacity building, education and technical training, and data management systems, which in the long-term, should contribute to achieving the overall objective of the program to reduce poverty and increase security among vulnerable groups. III. OTHER PERFORMANCE ASSESSMENTS A. Performance of the Borrower and Executing Agency 19. The PCR assessed the performance of the borrower (MFE), MSWL, and MECS satisfactory. This validation concurs with the PCR s rating. In 2006, the National Audit Board carried out a performance audit of loan projects, including the program, and judged implementation to be satisfactory. However, the borrower s decision to cancel the third tranche reflected a lack of commitment to and ownership of the key theme of the program, which was to target support for the poor and not to make the welfare benefits universal. B. Performance of the Asian Development Bank 20. The PCR rated the performance of ADB satisfactory. This validation concurs with the PCR s rating. ADB conducted nine review missions during 20022009, including an inception mission. The PCR noted the high turnover of ADB staff, and added that more stable staffing would have helped in taking into account the complicated nature of the policy reforms in the social security sector. The PCR also stated that implementation arrangements involving three different ministries were complicated and could have been better designed and coordinated. Based on hindsight, however, ADB could have made the program framework much stronger by obtaining the borrower s firm commitment at appraisal to target the welfare benefits only to the poor and not to make them universal, and by making that agreement a condition for release of the first tranche rather than the third tranche. With the cancellation of the third tranche, the spirit of ADB s entire assistance to support only the poor with welfare benefits was not attained. C. Others 21. Loan disbursement and procurement followed ADB guidelines. No adverse environmental effects were expected from the program.

7 IV. OVERALL ASSESSMENT, LESSONS, AND RECOMMENDATIONS A. Overall Assessment and Ratings 22. This validation rates the program relevant, effective in achieving its outcome, likely sustainable and significant in terms of impact, but less than efficient in achieving its outcome and outputs (see table). Overall, this validation concurs with the PCR s rating of the program as successful. OVERALL RATINGS Criteria PCR IED Review Reason for Disagreement/Comments Relevance Highly relevant Relevant The relevance of the program design was somewhat reduced by the failure to complete the reforms as planned, with the intended targeted social welfare benefit system being dropped and a universal benefits system being chosen instead (paras. 89). Effectiveness in achieving outcome Efficiency in achieving outcome and outputs Effective Efficient Effective Less than efficient Preliminary assessment of Likely Likely sustainability Overall assessment Successful Successful The efficiency rating is downgraded as the closing of the program loan was delayed by 2 years and the project loan by 2.5 years. Also, as the third tranche was canceled, the loan resources were not used as efficiently as had been expected under the original program (para. 1415). Borrower and executing Satisfactory Satisfactory agency Performance of ADB Satisfactory Satisfactory Impact No rating given Significant Refer to para. 18. but indicated a positive impact. Quality of PCR Satisfactory ADB = Asian Development Bank, IED = Independent Evaluation Department, PCR = project completion report. Note: From May 2012, IED views the PCR's rating terminology of "partly" or "less" as equivalent to "less than" and uses this terminology for its own rating categories to improve clarity. Source: ADB Independent Evaluation Department. B. Lessons 23. The main lesson identified was that social welfare reform is politically sensitive. Therefore, at the very early stage of preparing the program framework, the borrower s commitment to the key theme of the program needs to be carefully assessed. In addition, that commitment should be made a condition for program effectiveness and release of the first loan

8 tranche. Postponing the meeting of that condition to a later date or to the release of a later loan tranche might make it difficult to ensure that the government will adhere to its commitment during program implementation and on a sustained basis thereafter, as it happened in this program loan. The other key lesson was the need to set up an appropriate monitoring and evaluation (M&E) system at the time of preparing the program framework to facilitate collection and analysis of information necessary to evaluate the impact of the project. The PCR pointed out other lessons as well (footnote 1, paras. 7181, and Appendix 7). This validation supports all of them. C. Recommendations for Follow-Up 24. ADB should strengthen the preparation of program frameworks to help ensure achievement of the key themes of the programs that ADB supports. The program frameworks should also ensure that M&E systems are put in place to facilitate collection and analysis of necessary information to properly evaluate the impacts of projects. The PCR made a number of recommendations (footnote 1, paras. 8285), and this validation supports all of them. The East Asia Department needs to follow up on the government s enactment and enforcement of the 2010 Draft Social Welfare Law. IX. OTHER CONSIDERATIONS AND FOLLOW-UP A. Monitoring and Evaluation Design, Implementation, and Utilization 25. According to the PCR, the program did not have an adequate M&E system. This hampered proper assessment of the impact of the reform measures, such as an analysis of gender and socioeconomic impacts. This validation agrees with the PCR recommendation that a monitoring and evaluation expert should have been hired to put in place an appropriate M&E system to properly evaluate the impact of program reform measures and project activities. B. Comments on Project Completion Report Quality 26. The PCR satisfactorily meets both the guidelines on PCRs (Project Administration Instructions 6.07) and on preparing program performance evaluation reports. 7 It provides a balanced assessment of program outputs, including adequate explanation of noncompliance with targets. In general, there is good evidence and analysis of the program. The PCR lessons and recommendations are satisfactory, as the lessons drawn are thorough and relevant for future ADB operations. This validation rates the PCR satisfactory. C. Data Sources for Validation 27. The main data references include: (i) the report and recommendation of the President, (ii) the ADB PCR, and (iii) the 2008 country assistance program evaluation for Mongolia and the associated rapid sector assessment prepared by the Independent Evaluation Department. D. Recommendation for Independent Evaluation Department Follow-Up 28. A program performance evaluation report on the program is not necessary because of the good quality of the PCR evaluation. 7 ADB. 2009. Project Completion Report for Sovereign Operations. Project Administration Instructions. PAI 6.07. Manila.