Board of Directors IN.365-10 21 December 2010 Validation Report Rural Finance Project (Mongolia) (Loan 1848-MON) The attached Report is circulated at the request of the Director General, Independent Evaluation Department. The report is also being made publicly available. For Inquiries: H. Hettige, Independent Evaluation Department (Ext. 4102) C. Kim, Independent Evaluation Department (Ext. 4151)
Validation Report Reference Number: PCV: MON 2010-72 Project Number: 28201 Loan Number: 1848(SF) December 2010 Mongolia: Rural Finance Project Independent Evaluation Department
ABBREVIATIONS ADB Asian Development Bank CUSO FRC IED IRC ITC MOFE PCR PIO PMU RRP SCC SCU SDU TA credit union service organization Financial Regulatory Commission Independent Evaluation Department information and resource center information and training center Ministry of Finance and Economy project completion report project implementation office project management unit report and recommendation of the President savings and credit cooperative savings and credit union supervision development unit technical assistance NOTE In this report, "$" refers to US dollars. Key Words asian development bank, adb, borrower, executing agency, rural sector, savings and credit societies, economic growth, poverty reduction, credit line, training, monitoring, legal and regulatory reform, consultants, performance, relevance, outputs, outcomes, effectiveness, efficiency, sustainability, impact, lessons, recommendations Director: Team leader: Team members: H. Hettige, Independent Evaluation Division 2 (IED2), Independent Evaluation Department (IED) C. Kim, Principal Evaluation Specialist, IED2, IED J. Dimayuga, Evaluations Officer, IED2, IED R. Perez, Senior Operations Evaluation Assistant, IED2, IED In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, the Independent Evaluation Department does not intend to make any judgments as to the legal or other status of any territory or area.
PROJECT COMPLETION REPORT VALIDATION A. Basic Project Data PCR Validation Date December 2010 Project Number: Loan Number: 28201 1848(SF) Approved Actual Project Name Rural Finance Project Total Project Cost 15.00 ($ million): Country Mongolia Loan (ADB) ($ million): 8.69 0.97 Sector Finance Sector Total Cofinancing 0.35 ($ million): ADB Financing ADF: SDR6.854 million Borrower ($ million): 0.72 (equivalent $8.686 million) OCR: 0.00 Beneficiaries ($ million): 4.87 Cofinanciers Others ($ million): 0.38 Approval Date 25 Oct 2001 Effectiveness Date 29 Apr 2002 29 Jul 2002 Signing Date 29 Jan 2002 Closing Date 30 Jun 2009 09 Nov 2007 Project Officers Name B. Wilkinson B. Losolsuren Location Mongolia Resident Mission From 2001 To 2007 Validator: S. Thalakada, Staff Director H. Hettige, IED2 Consultant, IED2 Quality Control Reviewer/Peer Reviewer C. Kim, Principal Evaluation Specialist, IED2 J. Dimayuga, Evaluation Officer, IED2 = data not available, ADB = Asian Development Bank, ADF = Asian Development Fund, IED2 = Independent Evaluation Division 2, OCR = ordinary capital resources, PCR = project completion report, SDR = special drawing rights. B. Project Description (summarized from the report and recommendation of the President) 1 (i) Rationale. The project supported the development of a system of community-based savings and credit unions (SCUs), later referred to as savings and credit cooperatives (SCCs). These were intended to (a) increase access of the rural poor to financial services (savings and credit), and (b) stimulate the development of the financial sector over the short to medium term. At appraisal, most of the rural population did not have access to formal savings and credit; about 3% had access to credit services from a bank or government institutions. The financial sector faced many constraints in terms of liquidity and institutional capability. Delivery mechanisms in rural areas could not satisfy the demand for financial services, particularly from those without collateral, including the poor and small enterprises. The informal sector provided credit but at a high cost. Thus alternative formal financial institutions were needed to offer a range of financial services on a sustainable basis. The government recognized the need to revive the rural sector and saw microfinance playing a pivotal role. The establishment of an SCU network was regarded as a viable form of financial intermediation in rural areas. (ii) Impact. The project was expected to provide improved access for the rural population to savings and credit services through SCUs. Availability of and access to financial services had been identified as primary constraints to rural development and poverty reduction. The total membership of SCUs established under the project was expected to be 22,100: 12,000 in 60 provincial (aimag) center SCUs, and 10,100 in 270 district (soum) SCUs; making share contributions and savings amounting to $4.9 million. Saving facilities 1 ADB. 2001. Report and Recommendation of the President to the Board of Directors: Proposed Loan for the Rural Finance Project in Mongolia. Manila.
2 were expected to support capital accumulation by the poor, enhancing their ability to manage risks and to take advantage of economic opportunities. The project was expected to create new employment opportunities for about 16,500 people, as credit was used to develop and expand small businesses. Information and resource centers (IRCs) were to be established in pilot provinces, providing a wide range of business and skills training, and information services to about 29,000 people. Financial analysis of the credit union service organizations (CUSOs) and sample SCUs found that all such institutions would be viable within 34 years and could become sustainable in the long term. (iii) Objectives or Expected Outcomes. The long-term objective of the project was to develop a sustainable rural financial system that would contribute to both economic growth and poverty reduction. The project aimed to (a) increase the availability of and access to savings and credit services in rural areas and among the poor (22,100 members targeted by the end of the project), and (b) provide a focal point for rural communities to access information and resources to support economic activities to be financed by SCUs, and satisfy related information and training needs (a total of 4,000 persons to be trained). (iv) Components. The project had three components: (a) development of rural SCUs; (b) establishment of IRCs; and (c) capacity building in project management. The project was to be initially implemented on a pilot basis over 3 years in four provinces (Arhangay, Dundgovi, Dzavhan, and Sukhbaatar) (called Phase I) and expanded nationwide during the remaining 4 years of the project (called Phase II). The estimated project completion date was 31 December 2008 and the loan utilization period was to end on 30 June 2009. C. Evaluation of Design and Implementation (project completion report assessment and validation) (i) Relevance of design and formulation. The project was judged relevant at appraisal due to the diverse demand for financial services in rural areas, particularly among the poor. This validation concurs with such a judgment. Savings and credit were primarily rendered for emergencies, investments in business, household expenditure, education, and social and family obligations. The project was consistent with policies and plans of the Government of Mongolia and the Asian Development Bank (ADB). In September 2000, the government produced an action plan that, in the area of rural finance and poverty, aimed at (a) promoting microfinance activities to provide incentives for enterprise development, and (b) generating employment opportunities. In conjunction with the government, ADB developed a poverty reduction strategy for 20002005 that formed the focus of the ADB country operations strategy, and the related Partnership Agreement of March 2000. That strategy identified five core sectors, of which the finance and agriculture sectors were fundamental to the reduction of income-based poverty. Since Mongolia became a member of ADB in 1991, ADB has maintained a continuous dialogue with the government on its general reform program for transition, focusing on the agriculture and finance sectors. ADB strongly influenced the government to consider mechanisms for moving away from traditional approaches to rural finance. Accordingly, the government has pursued a long-term strategy to develop self-sustaining rural financial institutions. The steady growth and development of the SCU movement in Mongolia proved the appropriateness of that approach. The project was coordinated with other development partners working in the field of financial sector development. The project design was formulated by an ADB-financed project preparatory technical assistance, 2 taking into account the lessons identified by an ADB impact study 3 relating to the implementation of rural credit projects. (ii) Project outputs. Project implementation fell short of appraisal expectations. Implementation took place over about a year (February 2004May 2005), and only in the four pilot provinces, compared with the appraisal estimate of a 7-year implementation period, in the four pilot provinces during Phase I and nationwide under Phase II. As a result, about $7.7 million (89%) was cancelled from the ADB loan, 2 ADB. 2000. Technical Assistance to Mongolia for the Rural Finance Project. Manila (TA 3397, for $700,000, approved on 22 February). 3 ADB. 2001. Impact Evaluation Study: ADB s Rural Credit Assistance in Bangladesh, People s Republic of China, Indonesia, Nepal, Philippines, Sri Lanka, and Thailand. Manila.
3 reducing the loan amount to about $1.0 million (11%). The actual outputs compared with those envisaged at appraisal were outlined in detail in the project completion report (PCR). 4 These were as follows: (a) Development of rural savings and credit unions. The appraisal envisaged the setting up an SCU system, comprising a CUSO, a network of about 330 individual member-owned SCUs, and a supervision development unit (SDU) to mobilize savings and provide lending services to members in provinces and districts. The project was to provide a revolving line of credit of $4.9 million to the SCUs. The actual outputs during implementation were, however, disappointing. The credit line of $4.9 million was not used for various reasons (e.g., proliferation of unqualified SCCs, unviable operations of the SCC system, lack of legislative and supervisory framework, capacity deficiencies, and competition from the banking system in rural areas). The SDU did not become fully functional and was superseded in early 2006 by the Financial Regulatory Commission (FRC), which was supported by a subsequent program loan and became responsible for the regulation and supervision of the nonbank financial sector, including SCCs. As for the CUSO, its capacity for undertaking training was strengthened by the international project consultants. The appraisal estimated that $11.689 million, out of the total project cost of $15.0 million (78%), would be spent under this component but only $272,000 (1.8%) was spent by close of project. As of December 2008, the SCCs numbered 209 nationwide with a membership of about 24,700 (against the project target of 330 SCCs and 22,100 memberships). These SCCs were healthy due to a combination of FRC supervision, and training materials developed under the project. (b) Establishment of information and resource centers. The appraisal envisaged setting up four IRCs (one in each of the four pilot provinces), and making them fully operational, including full cost recovery, by the end of project implementation. The four IRCs were established in May 2004 but were called information and training centers (ITCs) due to a merger of the project implementation offices (PIOs) and the IRCs as a cost-saving measure. However, full operation and cost recovery was not possible in the 13-month period that the ITCs were in operation. The appraisal estimated that $1.157 million, out of the total project cost of $15.0 million (7.7%), would be spent under this component but only $292,000 (1.9%) was spent by close of project, mainly because of the above-mentioned cost saving measure. (c) Capacity building in project management. The appraisal envisaged establishing a project management unit (PMU) and PIOs under the Ministry of Finance and Economy for overall management of the project. The PMU and PIOs were set up and managed the project to the extent it was implemented. The appraisal estimated that $0.893 million, out of the total project cost of $15.0 million (5.9%), would be spent under this component but only $364,000 (2.4% of the total project cost) was spent by close of project. (iii) Program cost, disbursements, borrower contribution, and conformance to schedule (as relevant to project performance). The project cost was estimated at $15.0 million equivalent, consisting of $3.242 million (22%) equivalent in foreign exchange cost and $11.758 million equivalent (78%) in local currency cost. It was to be financed by an ADB loan ($8.686 million equivalent [58%]), the government ($0.715 million equivalent [5%]), and by beneficiaries and others ($5.599 million equivalent [37%]). The PCR did not quantify the actual project cost, which was unattainable from the authorities. Of the ADB loan of $8.686 million, only $0.966 million (11%) was disbursed to finance the three project components as outlined under project outputs (C, ii, ac); the balance of $7.720 million (89%) was cancelled. The PCR did not provide data on the contributions made, if any, by the borrower and the other beneficiaries. The project was implemented over about a year (from February 2004 through May 2005) only in the four pilot provinces, compared with the estimated 7-year period including a nationwide expansion, but financially closed in November 2007, about 2 years earlier than envisaged at appraisal, in June 2009. (iv) Implementation arrangements, conditions and covenants, related technical assistance, and procurement and consultant performance. Implementation arrangements generally followed those proposed at appraisal except that the SDU took longer than envisaged to establish and was ultimately superseded by the FRC. According to the PCR, compliance with loan covenants was generally satisfactory. However, as the project was cancelled before its anticipated completion date, only partial 4 ADB. 2009. Completion Report: Rural Finance Project in Mongolia. Manila.
4 compliance was achieved with most covenants. Those covenants related to the credit line were not met as it was cancelled without utilization. There was no associated technical assistance (TA); however, significant technical support and contracted consultants were used in the development of SCUs. Engagement of consultants was delayed due to lack of knowledge on the part of the Ministry of Finance and Economy (MOFE) of ADB procedures for their selection. According to the PCR, the consultants performed their tasks in a timely and effective manner. However, the draft SCC law prepared by project consultants was not considered by Parliament, which opted for another draft to be prepared by the FRC. (v) Performance of the borrower and executing agency. The PCR rated the performance of the borrower and the executing agency, (MOFE) partly satisfactory. This validation agrees with that assessment. Delays occurred in the selection of consultants and in making reimbursement claims. Also, the borrower and executing agency did not submit the PCR to ADB as required. Further, lack of project data (due to the dissolution of the PMU in 2005 and reorganization of the MOFE to the Ministry of Finance) made PCR preparation difficult for ADB. This validation considers that the government could have proceeded with the project rather than agreeing to its premature cancellation. The project could have continued with appropriate adjustments and reallocations of funds, which would have helped to develop a strong SCC network. However, the PCR suggests that this approach was probably rejected by the ADB review mission (October 2004) due to the various problems experienced by the SCCs during project implementation. This was a missed opportunity to resolve the weaknesses of the SCC system and facilitate the development of a sustainable rural financial system. (vi) Performance of the Asian Development Bank. The PCR rated ADB performance unsatisfactory. This validation supports that assessment. As stated by the PCR, the project, as formulated by ADB, had two design flaws: (i) the legislative and supervisory environments were too weak to operate a viable SCC system, as illustrated by the troubled SCC operations; and (ii) project funds were inappropriately allocated, favoring the credit line over much needed capacity building for proper understanding and operation of a rural SCC system. The project sought to address both issues during project implementation but was not effective. Firstly, it was overly ambitious to expect that the legislative and supervisory frameworks could be set up and made to operate effectively after project implementation had begun. Secondly, the funds and time allocated were inadequate to develop a legislative framework and conduct capacity building. In addition to these design flaws, ADB could have been more cautious in initiating premature project termination without attempting to make changes in the project parameters so as to meet the changed circumstances. That approach would have ensured greater loan utilization and greater attainment of the project objectives. ADB took the easiest course of action in cancelling the unutilized portion of the loan. Thus, an opportunity was missed to achieve the expected project outcome of a sustainable rural financial system to facilitate accelerated economic growth and poverty reduction. The considerable resources expended by the government and ADB in formulating, processing, and implementing this project were largely wasted. According to the PCR, ADB project supervision missions were infrequent, of insufficient duration, and did not take place at the critical juncture of project implementation; all of which led to the premature closure of the project. D. Evaluation of Performance (project completion report assessment and validation) (i) Relevance. The PCR described the project as relevant for the most part. This validation considers that the project was relevant at appraisal, as it sought to build a sustainable rural finance system to help meet the large need for those services. However, its relevance declined during project implementation for the following reasons: (a) the project design made implementation difficult; (b) the existing SCC system operated poorly due to lack of proper supervision; (c) many SCC members were disillusioned about the benefits of participation in such a system; and (d) competition with SCCs came from a new, vibrant, and privatized banking system that marketed savings and credit schemes successfully in rural areas. Under these circumstances, the government possibly lost faith and ownership in further implementation of the project, opting to get the right legal and supervisory framework in place before further implementation. Project consultants prepared draft legislation relating to SCCs that was not passed by Parliament, which instead opted for legislation covering a wider scope of activities. This led to the subsequent establishment of the FRC with wider powers to regulate and supervise the entire nonbank financial institutions sector,
5 including the SCCs and the capital market. Taking these factors into account, this validation rates the project partly relevant. (ii) Effectiveness in achieving outcome. The expected outcome of the project was the establishment of a sustainable rural finance system to help accelerate economic development and poverty reduction. But, the project fell far short of these expectations. The PCR did not rate the project under this. This validation rates the project less effective but bordering ineffective. None of the envisaged outputs were fully achieved, and loan funds were largely under-utilized as the intended nationwide roll-over of the pilot program did not take place. The expected number of SCCs could not be financed as the line of credit intended for that purpose was not utilized. Similarly, the promotion, training, and monitoring under CUSO, and the dissemination of information under the ITCs did not take place to the extent expected at appraisal due to under-utilization of loan allocations. Finally, the expected SCC regulation and supervision unit did not become operational and was ultimately superseded by the FRC. Meanwhile, the SCC system as a whole continued to be beset with viability and public image problems, which were expected to be resolved by the establishment of the FRC, bringing strong regulation and supervision. (iii) Efficiency in achieving outcome and outputs. The PCR did not rate the project under this criterion. This validation rates the project inefficient. Start-up of the project was held up by delays in selecting consultants. Also, the project was implemented over about a year compared to the 7-year period envisaged at appraisal. Only about 11% of the ADB loan was utilized and the balance (89%) was cancelled. The credit line, which formed the major portion of the ADB loan (55%), was not utilized and hence did not result in the establishment of the number of SCCs envisaged. Likewise, the proposed SDU did not become operational. Only the training and information dissemination components made some progress but these too were under-utilized. The outputs and outcomes achieved were insignificant and not commensurate with the resources expended by the government and ADB in processing and implementing the project. (iv) Preliminary assessment of sustainability. The PCR stated that the project did not continue long enough for a sustainability assessment to be made. This validation rates the project unlikely to be sustainable. The project was not implemented to the extent envisaged at appraisal. Two components were not implemented at all (line of credit and operation of the SDU), thus the question about their sustainability did not arise. The training and information dissemination components were partially implemented; both have some prospects for sustainability provided the FRC gives them sufficient support. As to the sustainability of the overall SCC concept in the country, a lot depends on four factors: (a) the extent to which the new private banks and nongovernment organizations are successful in penetrating rural areas and in meeting the need for financial services; (b) the enactment and enforcement of the SCC law under consideration by Parliament at the end of 2009; (c) the success of the FRC in providing rigorous regulation and supervision of SCC activities, resolving past difficulties, and making them viable; and (d) the extent to which public confidence in the SCC system can be restored. (v) Impact (both intended and unintended). The total membership of SCUs established under the project was expected to be 22,100: 12,000 in 60 provincial center SCUs; and 10,100 in 270 district SCUs; making share contributions and savings amounting to $4.9 million. Since the line of credit was cancelled, this number of qualified SCUs could not be established under project funding. Similarly, the expected employment of about 16,500 people was not created. Also, the project could not set up a regulatory and supervisory framework for SCCs as the related legislation proposed under the project was not passed by Parliament. Instead, Parliament opted for a more comprehensive and wide ranging law covering the entire nonbank financial institutions sector, which was under consideration at the end of 2009. Thus, the impact under these two components was negligible. The impact of the training and information dissemination on SCCs undertaken under the project depends on the sustainability of the SCCs and the extent that the FRC supports them. The PCR did not assess the impact of the project as it was implemented only over a short time and this was not long enough to allow for an assessment of impact. Based on these factors, this validation assessed the project s impact as negligible.
6 E. Overall Assessment, Lessons, and Recommendations (validation of project completion report assessment) (i) Overall assessment. The PCR rated the project unsuccessful overall. This validation supports this rating. The project had design flaws that made project implementation difficult. ADB s management of the project was not sufficiently competent to compensate for those flaws, and for changing circumstances. All this led to the project s premature termination. Thus, few of the outcomes and outputs envisaged at appraisal were achieved, making the project impact negligible. Under the circumstances, the project s returns were not commensurate with the resources expended on processing and implementation. (ii) Lessons. Two lessons could be learned from the experience of implementing this project: (a) The need to lay down a firm legal, regulatory, and supervisory foundation prior to embarking on promotion of financing institutions, such as SCCs. In this case, ADB attempted to establish the regulatory structure in parallel to project implementation, which did not work well. The two activities should have been better prepared, and sequenced as separate projects. The regulatory structure should have been supported by a comprehensive TA with adequate resources to prepare and promote the legal, regulatory, and supervisory framework, to have it passed by the legislature, and to ensure adequate institutional capacity to execute that framework. With the regulatory structure in place, the implementation should have proceeded with components for training, promotion, and institution development of SCCs. That approach would have had better prospects for success in establishing a sustainable rural finance system. (b) The need to enhance the capacity of ADB staff to quickly assess changing project circumstances and adjust project parameters to ensure greater use of ADB loan funds and better attainment of project objectives. The PCR identified some other important lessons. (iii) Recommendations. Three recommendations are made, in addition to the five recommendations of the PCR: (a) ADB should not initiate or agree to a premature termination of an approved project, unless as a last resort when all other options have been properly assessed and pursued. This did not happen in this case, as outlined in detail by the PCR (paras. 5257). The borrower, executing agency, and ADB had opportunities to reorient project parameters and to reallocate funds among the different components so that a greater portion of the loan could have been used for attainment of the project objectives. For example, funds could have been reallocated to expedite the establishment of the required regulatory and supervisory framework, undertake more training, and improve the infrastructure of the rural finance system (accounting and auditing policies and procedures, internal control systems, and management information system). Funds could even have been reallocated for use by the private banks that had successfully undertaken rural financial services (Mongol Post Bank, Khan Bank, and Xac Bank). Such action would have prevented the unwarranted and premature loan cancellation, and avoided the waste of considerable resources that were expended on loan processing by the government and ADB. (b) The capacity of the Mongolia Resident Mission in this case should be strengthened, particularly in the area of project supervision, to properly research, analyze, discuss, and negotiate with borrowers and executing agencies to make appropriate changes in project parameters to ensure greater use of loan funds and better attainment of project objectives. (c) ADB should follow up with the government on the present status of the SCC-related legislation that was under consideration by Parliament towards the end of 2009. This validation considers that a properly constituted SCC system could still build a niche role in providing financial services to the less privileged segments of the rural community not normally serviced by the formal banking system. The PCR made a number of important recommendations which were supported by this validation. F. Monitoring and Evaluation Design, Implementation, and Utilization (project completion report assessment and validation)
7 To help with monitoring and evaluation, a project framework was attached to the Report and Recommendation of the President (RRP) (Appendix 1) outlining the project monitoring mechanisms (project management reports, and ADB review and midterm missions). The MOFE was also expected to establish a project monitoring and evaluation system, acceptable to ADB, to allow comprehensive reporting of project input and output data (RRP, paras. 110112). The government submitted the quarterly reports to ADB on time and included relevant information. ADB fielded 11 missions between November 2002 and May 2007, including an inception mission, 2 disbursement missions, and 1 midterm review mission for project supervision. As pointed out by the PCR, these missions were infrequent, of insufficient duration, and failed to recognize warning signs or take action to ensure a greater use of loan funds and attainment of project objectives. G. Other The annual audit report to be submitted to ADB included the audit of the imprest account and statement of expenditures. ADB s anticorruption policy was explained to government officials. The project was not expected to cause any significant environmental concerns. Procurement followed ADB guidelines. H. Ratings PCR IED Review Reason for Disagreement/Comments Relevance Effectiveness in Achieving Outcome Efficiency in Achieving Outcome and Outputs Preliminary Assessment of Sustainability Relevant for the most part Not rated Not rated Not rated Partly relevant Less effective but bordering ineffective Inefficient Unlikely Relevant at appraisal but relevance declined during project implementation for various reasons. The project had design flaws that made implementation difficult. Also, the SCCs encountered problems and lost credibility as a suitable vehicle to provide a viable rural finance system. The banking system began to offer competition in rural areas as an alternative source of finance. Under these circumstances, the government appeared to lose interest in implementing the project further. The legal and supervisory frameworks for SCCs needed to be put in place before any further promotion. This was subsequently done to some extent with the establishment of the FRC. However, the required SCC law was still under consideration by Parliament at the end of 2009.
8 Borrower and Executing Agency Performance of ADB Partly satisfactory Unsatisfactory Partly satisfactory Unsatisfactory Impact Not rated Negligible Overall Unsuccessful Unsuccessful Assessment Quality of PCR Satisfactory I. Comments on Project Completion Report Quality The PCR was objective and would not require further independent review. J. Consideration for IED s Follow-Up Preparation of a project performance evaluation report may not be necessary. K. Data Sources for Validation RRP, PCR, and selected project administration documents
9 REGIONAL DEPARTMENT S RESPONSE TO THE PROJECT COMPLETION REPORT VALIDATION REPORT On 13 October 2010, the Director of the Independent Evaluation Division 2 of the Independent Evaluation Department (IED) received the following comments from the Financial Sector, Public Management, and Regional Cooperation Division of the East Asia Department. The validation report is well written. We appreciate IED for considering the interdepartmental review comments adequately. We also agree with and support the IED s rating reviews.