May Annual Evaluation Overview Report ab0cd. Project Evaluation Department

Transcription

1 May 2004 Annual Evaluation Overview Report 2004 Project Evaluation Department ab0cd

2

3 DOCUMENT OF THE EUROPEAN BANK FOR RECONSTRUCTION AND DEVELOPMENT ANNUAL EVALUATION OVERVIEW REPORT FOR 2004 PROJECT EVALUATION DEPARTMENT MAY 2004 For Official Use Only This document has a restricted distribution and may be used by the recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without the Bank's authorisation.

4

5 ANNUAL EVALUATION OVERVIEW REPORT 2004 TABLE OF CONTENTS LIST OF ABBREVIATIONS EXECUTIVE SUMMARY Page iii v 1. PERFORMANCE OF INVESTMENT OPERATIONS ASSESSED AGAINST THE BANK'S MANDATE Introduction Performance indicators Overall Performance Ratings Transition impact and environmental performance & change Financial performance of projects Enhancing the Bank s additionality in projects Conclusions based on evaluated projects in INSTITUTION BUILDING ACHIEVEMENTS THROUGH PROJECTS IN CENTRAL ASIA The context of institutional reforms Projects as instruments of institutional change Experience from recent evaluation in Central Asia Recommendations in respect of institutional change KEY FACTORS OF HIGHLY SUCCESSFUL AND UNSUCCESSFUL PROJECTS Introduction Main categories of factors affecting performance Relative importance of determining factors for project performance Factors more specific to failure/success Conclusions and recommendations Best practice and possible starting points as preconditions for appraisal of investment opportunities EVALUATION OF TECHNICAL COOPERATION (TC) OPERATIONS TC Evaluation Coverage Performance evaluation of TC operations TC-Related Evaluation Work in RECOMMENDATIONS IN EVALUATION REPORTS PREPARED UNDER PED S WORK PROGRAMME OF Recommendations from the evaluation special study on the TurnAround Management (TAM) Programme Recommendations in respect of the evaluation special study on the Russian Small Business Fund (RSBF) Programme Recommendations from other projects evaluated in FINDINGS FROM THE COUNTRY STRATEGY EVALUATION EXERCISE ON THE SLOVAK REPUBLIC Introduction Methodology Scope of the evaluation Overall assessment Recommendations for future strategies 32 i

6 ANNUAL EVALUATION OVERVIEW REPORT REVIEW OF FOLLOW-UP ON RECOMMENDATIONS OF SELECTED OPER REPORTS AND FROM PREVIOUS AEORS Recommendations to improve Environmental monitoring TIMS performance review AUDIT COMMITTEE S REVIEW OF EVALUATION REPORTS Evaluation reports discussed Review of recommendations as presented in the minutes of the above-mentioned meetings of the Audit Committee Other activities in respect of highlighting recommendations from evaluation reports 39 List of appendices ii

7 PROJECT EVALUATION DEPARTMENT ABBREVIATIONS AND DEFINED TERMS AEOR CEB CIS CHP CSU CO 2 DTM EAP EC EET EIRR EMS ESCO ETC EU FDI FI FIRR FSU GDP GEF IEA IFC IFI IFRS ISO JI KTC LC LLD MDB MEI MIS MoU MPF OCE OCU OECD OGC OM OPER PB PCR PED PMM Annual Evaluation Overview Report Countries of central and eastern Europe and the Baltic states Commonwealth of Independent States Combined heat and power Consultancy Services Unit Carbon Dioxide Deal Tracking Module Environmental action plan European Commission Energy efficiency team Economic internal rates of return Environmental management system Energy service companies Early transition countries European Union Foreign direct investment Financial institutions business group Financial internal rates of return Former Soviet Union Gross domestic product Global environmental facility International Energy Agency (of OECD) International Finance Corporation International financial institution International Financial Reporting Standards issued by the International Accounting Standards Board International Organisation for Standardisation Joint Implementation (under Kyoto Protocol) Kazakhtelecom Letter of credit Lessons Learned Database Multilateral development bank Municipal and Environmental Infrastructure Team Management information system Memorandum of Understanding Multi-project facility Office of the Chief Economist Official Cofinancing Unit Organisation for Economic Cooperation and Development Office of the General Counsel Operations Manual Operation Performance Evaluation Review Participating bank Project Completion Report Project Evaluation Department Project Monitoring Module iii

8 PROJECT EVALUATION DEPARTMENT PPF RO RSBF SEE SME Tacis TAM TC TCFP TI TIMS TMG TOR UNG XMR XMRA Post-privatisation fund Resident Office (of EBRD in a country of operation) Russia Small Business Fund Countries of south east Europe Small and medium-sized enterprises EC Assistance Programme for eastern Europe, the Caucasus and Central Asia Turnaround Management Programme Technical Cooperation Technical Cooperation Funds Programme Transition Impact Transition Impact Monitoring System TurnAround Management Group Terms of Reference National Corporation for Oil and Gas Industry Expanded Monitoring Report (investment operations) XMR Assessment CAPEX Ex-ante Ex-post capital expenditure before project signing at project appraisal after project signing at post-evaluation iv

9 EXECUTIVE SUMMARY OF THE ANNUAL EVALUATION OVERVIEW REPORT FOR PROJECT EVALUATION S FUNCTION The EBRD s Project Evaluation Department (PED) synthesises its overall findings, including the Bank s performance relating to its mandate, in this Annual Evaluation Overview Report (AEOR), thereby complying with its accountability obligations towards the Board of Directors and management. PED also contributes to preserving the corporate memory of the Bank, collecting lessons learned during evaluation of projects and carrying out special studies. 2. INVESTMENT PERFORMANCE JUDGED AGAINST THE BANK S MANDATE Performance ratings of investment operations. The EBRD has continued to select and develop projects that largely meet the Bank s mandate. During , only 54 per cent of the EBRD s evaluated projects achieved Successful-Highly Successful overall performance ratings, mainly reflecting the modest financial performance of the projects. By contrast, the Bank s projects scored much higher on transition impact (TI), with 75 per cent achieving a Satisfactory-Excellent rating for The fact that 26 per cent of the evaluated projects were given a disappointing Negative-Marginal rating reflects the fact that the Bank operates in difficult environments where many obstacles to transition remain. Further analysis of Unsuccessful projects in general industry and the banking sector indicated that poor corporate governance and management skills were recurrent factors, despite the Bank's efforts. Performance of country groups. South-eastern Europe continued to show a relatively stronger performance compared with other regions, in particular during the period It is now getting closer to Central Europe and the Baltic states whose performance has deteriorated, partly because of lower verified additionality (whereby it does not replace private sector financing). It is also noticeable that operations in the early transition countries (ETCs) recorded lower performance ratings than in other regions. The seven ETCs have suffered from political instability and face difficult transition challenges, mainly due to the small size of domestic and export markets, underdeveloped financial systems and public governance issues. There are, however, some encouraging signs of improved performance of evaluated projects in ETCs in recent years. Performance of sectors. The large disparity in performance between different industry sectors, seen in earlier AEORs, appears to be decreasing. Infrastructure operations still receive better overall success ratings, in comparison with projects in financial institutions and general industry. However, these differences are not as marked as in previous years. 2. INSTITUTION BUILDING ACHIEVEMENTS THROUGH PROJECTS IN CENTRAL ASIA The Bank s Transition Reports show that institutional reforms advance more slowly than the initial phase of trade and price reforms. The main reasons for this are the uncertainty concerning the outcome of the reforms, and the limited capacity within the state to implement them. While there are understandable reasons for the delays, the continuing lack of properly functioning institutions can create major business obstacles and reduce private investment opportunities. It is therefore essential that the project, being the main source of the Bank s intervention, carries institutional changes among its transition objectives. The report presents the outcomes of several evaluated projects in Central Asia, in which important institutional development experience was gained. A number of more generic conclusions are drawn with the purpose of enhancing the Bank s activities and improving project quality in early transition environments through: (a) better positioning of the project within other sector

10 activities; (b) improving the design of future projects through enhanced conditionality; (c) identifying relevant transition objectives; and (d) better identifying bottlenecks at sector level, especially in the area of institution building. 3. KEY FACTORS BEHIND HIGHLY SUCCESSFUL AND UNSUCCESSFUL PROJECTS PED has not to date conducted a formal statistical analysis of why projects either do very well or are unsuccessful. Short of relying on a formal list of factors, and matching their presence or absence in dedicated sections of evaluation reports on investment operations, the exercise undertaken and presented in this report is based on what has been de facto revealed in evaluation reports as the most recurrent causes of success and failures. The analysis focuses on recurrent performance factors of which three are internal to the project (financial, commercial, and institutional), and two others are external (effect of the business cycle and government). Bank handling of the process is added as a main factor both in project design and implementation. The main conclusion from this exercise is that the causes of success or failure are more internal to the project than external. This offers scope for the Bank to increase its likelihood of success and reduce the possibility of failure. The report presents recommendations and lists critical success factors for project preparation, which can be of use when preparing financing for business opportunities in early transition countries. 4. EVALUATION OF TECHNICAL COOPERATION (TC) OPERATIONS TC activities facilitate the EBRD s core investment operations and enhance the fulfilment of its transition impact mandate. In compliance with its responsibility towards the contributors to its Technical Cooperation Funds Programme, the Bank must focus equally on TC projects as it does on investments funded from the Bank s own resources. Accordingly, TC projects are subject to a mandatory self-evaluation process, in the form of Project Completion Reports (PCRs), and to an independent evaluation process on a sample basis. Since 1993, when PED started TC evaluation work, it has conducted 46 OPERs and 19 special studies on sectors and themes, covering numerous TC operations. The total volume of evaluated TC operations based on an OPER report, as a percentage of the volume of TC operations with a completed PCR increased to 21.8 per cent in 2003, from 18.9 per cent in 1998, when the PCR review and assessment work was introduced. If groups of TC commitments covered in special studies are included, the coverage ratio rises to 57.2 per cent. The experience gained from TC evaluation shows that the Bank has been improving the preparation and use of TC donor funds. This can be attributed in part to an enhanced internal process of preparing, approving and monitoring TC operations. The report also outlines the findings of the latest special study on PCR assessments. 5. RECOMMENDATIONS FROM EVALUATION REPORTS Chapter 5 presents recommendations from evaluation special studies on the EBRD s Turn Around Management (TAM) Programme, as well as on the Bank s Russia Small Business Fund. The chapter also highlights the recommendations from OPERs on investment operations prepared under the 2003 Work Programme. In 2003 the Audit Committee began to review evaluation reports on investment operations and evaluation special studies on a more systemic basis. It was believed that by regularly discussing evaluation reports in the Audit Committee, more attention could be given to the follow-up of recommendations by management, and it would also strengthen the lessons vi

11 learned uptake on behalf of the Board. It would further strengthen the accountability function of evaluation and provide an important challenge for PED to secure a high quality for its evaluation products. Chapter 8 highlights specific recommendations that were selected by the Audit Committee during discussions in 2003 on five OPER reports on investment operations. Comments by management on these recommendations are also included. Chapter 7 reviews the follow-up on recommendations from selected OPER reports and from previous AEORs. The chapter revisits recommendations from previous AEORs on environmental monitoring and notes positive follow-up actions taken by management in the past years, as well as including an additional recommendation on environmental monitoring. Having suggested the establishment of a transition impact monitoring system in the AEOR of 1999, the Transition Impact Monitoring System (TIMS) was approved in December 2002 and began to be implemented in January After just 15 months of implementation, it may seem premature to evaluate the performance of the new system, but it was nevertheless considered to be useful to assess the actions already taken, while the TIMS continues to expand and deepen on a daily basis. Based on participation in review meetings and the analysis of TIMS draft documents supporting the process, the preliminary evaluation findings of PED staff are positive. Already implemented to the maximum extent within the existing staff and time constraints of the Office of the Chief Economist (OCE), the process is well on its way to helping improve the structure of the projects (TI targets, covenants, risk mitigation) and addressing the problems as and when they arise, as well as raising the awareness of the Bank. To further improve the process, more time should be allocated by OCE staff to respond to the additional demands from TIMS and the Banking Department should give more emphasis to transition impact monitoring within this new framework. 6. FINDINGS FROM THE COUNTRY STRATEGY EVALUATION EXERCISE ON THE SLOVAK REPUBLIC As highlighted in Chapter 6, the transition challenges facing the Slovak Republic have changed considerably over the past decade. They required continuous strategic monitoring and the establishment of new medium-term strategic directions by the EBRD. The strategies were in general well adapted to the way the business environment and private sector government policies evolved in the Slovak Republic. The strategic changes were also implemented in a timely way, as highlighted by evolving priorities in the portfolio for the Slovak Republic from one strategy to the next. The level of transition impact achieved, however, was mixed during the 1990s. The corporations that the Bank supported for privatisation did not benefit from a proper protective and preventive legal framework for corporate governance, solvency and bankruptcy before mid The banking sector was also restructured with minimal banking supervision regulation and other prudential safeguards. The coordination with other IFIs on policy dialogue and structural reforms was constrained by the political conditions in the country. The lessons learned from this evaluation range from strategy design to implementation and impact. They lead to the key recommendations that the strategy documents should be even more focused on priorities for immediate interventions, rather than longer-term intended initiatives, and that they should be better designed to effectively involve policy dialogue at sector and national levels, in coordination with other IFIs. vii

12 1. PERFORMANCE OF INVESTMENT OPERATIONS ASSESSED AGAINST THE EBRD S MANDATE 1.1 INTRODUCTION The EBRD s Project Evaluation Department (PED) helps preserve the corporate memory of the Bank by evaluating projects, carrying out special studies and identifying lessons learned. PED also synthesises its overall findings - including the Bank s performance on its mandate in this Annual Evaluation Overview Report (AEOR), thereby complying with its accountability obligations towards the Board of Directors and management. To ensure optimal use of this corporate memory, PED assists the banking teams and others during the early stages of project preparation to use the relevant lessons learned from past experience. This process ensures that past experience is applied to the selection and design of future projects. The experience gained from the Bank s past performance and the generic lessons and recommendations presented in this report are thus available for the Bank s future strategic orientation. During 2003 an update of the evaluation policy was prepared and presented to the Board at the beginning of In addition, decisions were taken by management regarding the independence of the evaluation function. The box below gives further details. Evaluation Policy Review of 2004 and further enhancements of PED s independence The Evaluation Policy Review of 2004 which was approved by the Board in February 2004 reviews the evaluation practices and procedures as they have evolved over the years in the Bank and presents the necessary updates. The report is based on: (1) the integration of experience gained by PED during the six years of project evaluation since the last update; (2) more than 10 years of the Bank s learning experience; (3) changes in the Bank's modus operandi and its organisation; and, (4) enhanced harmonisation efforts in recent years with the other multilateral development banks (MDBs). Particular attention is given to: defining project evaluation distinctly from other functions, and establishing an approach to project evaluation which places critical importance on independent accountability and transparency and learning. The Review implements the good practice standards prepared by the MDBs evaluation cooperation group (ECG) on private sector evaluation and thereby contributes to the harmonisation process among MDBs. The independence of the evaluation function was further enhanced in 2003 through the implementation of a reporting structure in which the Corporate Director for Evaluation reports directly to the President while maintaining the reporting line to the Audit Committee of the Board of Directors. To maintain good contact with management on lessons learned the Secretary General is consulted by the Corporate Director for Evaluation on evaluation reports of projects and special studies. The Corporate Director also discusses with the Secretary General a draft of the AEOR and PED s Work Programme before finalisation of these reports. 1.2 PERFORMANCE INDICATORS By the end of 2003, 508 investment projects of the Bank s total portfolio of 1017 projects had reached a stage where they were ready for evaluation. Since the start of the Bank in 1991, PED has evaluated a total of 385 (or 76 per cent) of these investment projects. A well balanced sector and country coverage in the sample of evaluated projects has secured a broad representation of the overall portfolio of the Bank. Appendix 8, Section 10 gives further evidence of the size and representation of the sample of evaluated projects.

13 2 The evaluation performance indicators, which allow PED to assign the overall performance rating, are primarily based on the Bank's mandate to foster transition in its countries of operations. The relevant indicators consist of the following: Evaluation performance indicators 1 Mandate-related indicators Sound banking principle-related indicators Bank effectivenessrelated indicators Performance on transition impact Project and company financial performance The Bank's investment performance Environmental performance and change Fulfilment of project objectives Bank handling The Bank's additionality Overall performance rating The indicator boxes that are presented above in blue make up the indicators that define results on the ground and as such make up the transition outcome rating. 2 The evaluation of transition impact focuses on the broader effects that the project has on the sector and economy at large. Seven transition impact indicators, as used by the Bank during the screening and approval of projects, cover privatisation, competition, linkages to other sectors and skills transfer as well as the development of frameworks for markets, demonstration effects and corporate governance standards. PED assesses the short-term transition impact of a project that can be verified at the post-evaluation stage, as well as the longer-term transition impact potential that can still be realised. PED then reviews the risk of the project to realise its full transition potential. Appendix 7.1 presents the list of transition objectives that is used by PED and OCE when assessing transition impact ex-ante (before project signing) and ex-post (after project signing at post-evaluation). The transition matrices highlighted in Appendix 7.2 for each project evaluated in 2003 illustrate how PED deals with measuring ex-post realised transition impact, the longer-term transition potential still remaining and with the risk that full transition potential is realised during the life of the project. Appendix 8 gives details on the overall scores and the seven underlying performance rating categories for all evaluated projects. The evaluated operations referred to in this AEOR are, on the one hand, based on the postevaluation of a sample of evaluated projects undergoing an operation performance evaluation 1 2 Details on EBRD s Operation Performance Rating System at Post-Evaluation, with details on the benchmarks for each of the rating criteria are presented in Appendix 1 of EBRD s Evaluation Policy Review of 2004, which is posted on EBRD s website: Presenting evaluation findings based on results on the ground, i.e. transition outcome, makes the findings more comparable with other multilateral development banks (MDBs). See further details in Appendix 8, section 1.

14 3 review (OPER) and, on the other hand, on the assessment of expanded monitoring reports (XMRs), the self-evaluation reports prepared by operational staff. The OPER exercises are normally undertaken by PED after the investment has been realised, i.e. one year (from 2004 onwards, 18 months) after last disbursement of the loan and two years after last disbursement of equity. Another stipulation is that at least one year of commercial operation must have occurred with one year of audited financial accounts. 3 In 2003, a total of 18 projects ready for evaluation 4 were selected for an OPER exercise, based on a purposive sample 5 comprising 26 per cent of operations ready for evaluation. The XMR assessments (XMRAs) carried out by PED comprised a total of 30 projects (or 43.5 per cent of ready operations) and are selected on a random basis. Evaluation, therefore, covered a total of 69.5 per cent of projects ready for evaluation in 2003 (see Appendix 8, section 1). PED believes that the self-evaluation process, whereby operational staff prepare the XMRs, work well. This view is based on the quality of the XMRs prepared by the EBRD s banking department on all projects ready for evaluation. To further enhance this process, PED has started to organise XMR training sessions for operational staff who are set to prepare XMRs in the near future. Once a project has been evaluated, either through an OPER exercise or through an XMR assessment, there is no regular re-assessment of a project s performance in light of later events, although thematic and sector-oriented special studies may revisit some post-evaluated projects. A system of monitoring transition impact during project execution, recommended in AEORs of previous years and recently introduced by the office of the Chief Economist (OCE) is now in place and a process review of the transition impact monitoring System (TIMS) is presented in Section 7.3 of this report. 1.3 OVERALL PERFORMANCE RATINGS During , 54 per cent of evaluated operations were given Successful or Highly Successful ratings, 6 as shown in Table 1.1 with the overall performance ratings. A positive development was evident in 2003 when the Successful and Highly Successful rated projects reached 60 per cent. Cumulatively the outcomes remained relatively stable at 54 per cent during this period, as in 2001 and 2002 a score below 50 per cent was observed. Eight per cent of the projects scored Highly Successful overall, while 17 per cent were rated Unsuccessful per cent of evaluated operations were given Successful or Highly Successful ratings on overall performance for the period in The overall performance outcome may seem modest. However, many parts of the region in which the Bank operates remain risky from an investment perspective, and this continues to be true even for the more recent EBRD projects developed in advanced transition economies. In particular in these countries, where the Bank needs to secure its additionality in projects, it Appendix 10 includes a flow chart of the evaluation process relating to the evaluation of investment operations. In total, 19 OPER reports were produced. However, one of these was not an evaluation of a ready project but of the relationship between EBRD and one of its major clients. Therefore it does not count towards the coverage ration of projects ready for evaluation. Projects ready for evaluation on which an OPER report is prepared are selected on a purposive basis, i.e. projects are selected based on lessons-learned potential, risk for the Bank, a project s high profile, etc. Weighting by volume of investment yields better results with 64 per cent Successful or higher, 23 per cent Partly Successful, while the Unsuccessful ratings share is 13 per cent. While in the AEOR for 2001 no projects scored Highly Successful, in 2002 and per cent of the evaluated projects scored Highly Successful. In addition, the number or projects in 2002 with an Unsuccessful rating dropped to 14 per cent compared with 22 per cent in This rating declined further in 2003 to 9 per cent.

15 4 means that it must sometimes accept relatively high-risk projects as sponsors for lower-risk projects in these countries can readily attract financing from the market. In recent years the EBRD has further deepened the transition process in more challenging areas where unemployment is high and financial intermediation limited, thereby accepting additional credit risk. The average score on transition impact Satisfactory to Excellent is much higher than the above score on the overall performance (see Section 1.4). The relatively lower average rating on financial performance partly drives the outcomes on overall performance (see Section 1.5). In addition, experience has shown that projects scoring Satisfactory on transition impact tend to be associated with a rating of Partly Successful in the overall rating. Year of evaluation Table 1.1: Overall performance ratings distribution of evaluated projects, Unsuccessful 1 Partly Successful 2 Subtotal 1+2 Successful 3 Highly Successful 4 Subtotal % 29% 44% 44% 12% 56% 34 No. of evaluations % 42% 53% 36% 11% 47% % 20% 43% 53% 4% 57% % 22% 46% 46% 8% 54% % 24% 43% 40% 17% 57% % 31% 53% 47% 0% 47% % 38% 52% 42% 6% 48% % 31% 40% 54% 6% 60% % 36% 49% 40% 11% 51% % 29% 46% 45% 9% 54% % 27% 46% 46% 8% 54% % 27% 46% 44% 10% 54% % 27% 47% 45% 8% 53% % 29% 48% 44% 8% 52% % 29% 46% 46% 8% 54% 358 Chart 1.1 below shows the breakdown of overall performance ratings by country groups, for all investment operations evaluated since South eastern Europe shows a relatively stronger performance compared with other regions, in particular during the period (see also Appendix 8, section 2.3). It is getting closer to Central Europe and the Baltic states whose performance has deteriorated, partly because of lower verified additionality (see section 1.6) Chart 1.1: Percentage distribution of overall performance ratings on 369 post-evaluated investment operations in by country groups % of evaluated projects 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 17% 24% 8% 34% 49% 49% 33% 10% 8% 9% CE Europe & Baltics (177) SE Europe (61) 27% 26% 31% 36% 38% 8% 36% 52% 4% Russia (64) ETCs (42) Other CIS (25) Region Unsuccessful Partly Successful Successful Highly Successful Note: 16 regional projects are omitted

16 5 Chart 1.1 also shows that operations in the early transition countries (ETCs) 8, have lower performance ratings than in other regions. These seven countries have suffered from political instability and face difficult transition challenges, mainly due to the small size of domestic and export markets, underdeveloped financial systems and public governance issues. This very poor investment climate results in a low level of foreign investment and creates major challenges for EBRD investments. The relatively poor past performance of projects in early transition environments is discussed in Chapter 2 through selected case studies. The analysis in Appendix 8 shows, however, some encouraging signs of improved performance for ETCs in recent years. The above mentioned low average performance is mainly due to projects in the region evaluated before 2000, when only 21 per cent were rated Successful or better. Since then the overall performance of projects in the ETCs has risen to 52 per cent with these ratings, getting closer to the average across all regions. 1.4 TRANSITION IMPACT AND ENVIRONMENTAL PERFORMANCE & CHANGE Performance on transition impact Table 1.2 presents the performance ratings on the transition impact, applying the six-point rating scale 9 that was introduced in Of a total of 189 projects evaluated in , 75 per cent achieved Satisfactory-Excellent ratings. 75 per cent of evaluated operations obtained Satisfactory-Excellent ratings on transition impact for the period This score is a very important accomplishment which confirms the Bank s mandate compliance. However, 25 per cent of the evaluated projects obtained a rating of Negative- Marginal which shows that the Bank operates in difficult environments where many obstacles to transition remain. Evaluation Year/ Period Negative Table 1.2: Transition impact of post evaluated projects in Unsatis- Factory Marginal Subtotal Satisfactory % 10% 7% 22% 24% 40% 14% 78% 42 Good Excellent Subtotal No. of evaluated projects % 14% 14% 32% 25% 43% 0% 68% % 6% 16% 24% 26% 44% 6% 76% % 4% 15% 19% 29% 48% 4% 81% % 12% 12% 28% 23% 42% 7% 72% % 10% 13% 27% 24% 43% 6% 73% % 8% 14% 25% 25% 44% 6% 75% ETCs: Armenia, Azerbaijan, Georgia, the Kyrgyz Republic, Moldova, Tajikistan and Uzbekistan In April 1999 the five-point rating scale (High, Medium, Low, None, Negative) on measuring transition impact was changed to a sixpoint rating scale (Excellent, Good, Satisfactory, Marginal, Unsatisfactory, Negative). The same six-point rating scale is used ex-ante as well as ex-post (see Appendix 8, Section 3.2). As such, in 2000 PED started with a new time series in rating projects on transition impact. The trend observed over the past three years is comparable to the trend shown over the period when the five point rating scale was used.

17 6 Chart 1.2 presents the transition impact (TI) rating distribution by country groups of 177 projects 10 evaluated in Again, performance in the ETCs and other CIS countries (excluding Russia) 11 lags behind other regions, though not by as much as for the overall performance rating. The time-sequence analysis in Appendix 8, Section 3.3 shows that transition impact in ETCs has remained steady with per cent of projects rated Satisfactory or above for transition impact. This suggests that the projects introduced transition reforms, but that their sustainability and hence overall performance was affected by the economic and political climate. The region with the highest transition impact ratings remains south-eastern Europe, as in last year s AEOR. Chart 1.2: Percentage distribution of transition impact ratings on 177 post-evaluated investment operations in by country groups 100% 90% 4% 3% 6% 10% 9% 9% 15% 6% % of evaluated projects 80% 70% 60% 50% 40% 30% 20% 9% 34% 37% 22% 56% 15% 15% 42% 22% 28% 15% 22% 48% 39% Negative Unsatisfactory Marginal Satis-factory Good Excellent 10% 0% 4% CE Europe & Baltics (67) 9% 12% SE Europe (32) 6% Russia (33) ETCs (27) Other CIS (18) Region Note: Twelve regional operations are omitted. Chart 1.3 shows assigned TI ratings by sector. It seems that differences in results between sectors, observed in past years, have become much less apparent. This implies a reduction over time in the TI ratings for operations in infrastructure and financial institutions, and an improvement for operations in industry and commerce. Chart 1.3: Percentage distribution of transition impact ratings on 189 post-evaluated investment operations in by sector groups 100% 90% 80% 3% 3% 10% 3% 11% 11% 26% 14% 3% 25% Negative % of evaluated projects 70% 60% 50% 40% 30% 20% 28% 16% 26% 46% 48% 39% 58% Unsatisfactory Marginal Satis-factory Good Excellent 10% 0% 3% 3% Infrastructure - Municipal, power, energy efficiency and transport, excluding shipping; Industry and commerce - agribusiness, general industry, natural resources, property/tourism and telecommunications 8% 17% Financial Infrastructure Industry excl. Telecoms (12) Institutions (72) (31) Telecoms (74) Sector Twelve regional projects are omitted from the total of 189 evaluated projects for transition impact. Other CIS countries, excluding Russia, are: Belarus, Kazakhstan, Turkmenistan and Ukraine.

18 Environmental performance and environmental change The EBRD was established with a specific environmental mandate. Article 2, clause (vii) of the Agreement Establishing the EBRD encourages the Bank to promote in the full range of its activities environmentally sound and sustainable development. Environmental performance 12, which is not rated during project appraisal, is included in the ex-post assessments. Environmental performance in was rated Satisfactory or better in 80 per cent of cases. By major sectors, in terms of all past evaluated data, environmental performance was rated Satisfactory or better in 89 per cent of the cases for infrastructure, 78 per cent for industry and commerce (excluding telecoms) and 76 per cent for financial institutions (FI). Over the period , only 3 per cent of the projects evaluated have been rated Unsatisfactory in respect of environmental performance. Nonfinancial institutions tend to receive greater environmental focus during due diligence and result in better outcomes, even though these projects are held to a higher performance standard per cent of evaluated operations obtained ratings of Satisfactory or better on environmental performance for the period In respect of environmental change 14, 20 per cent of the evaluated projects rated Substantial or Outstanding, while 54 per cent achieved Some environmental change. The EBRD achieved its most impressive results for environmental change in Category A 15 projects (which on entry are judged to represent significant environmental liability), with 58 per cent of the projects were being rated as Substantial or Outstanding. This implies that the EBRD is best able to achieve its environmental mandate in projects that present the largest amount of environmental risk. Environmental change is a non-quantitative measure of the Bank s environmental footprint. PED considers these ratings to be positive, in particular, as the potential for environmental change is not a consideration at the time of project selection. Further details on environmental performance and change are presented in Appendix 8, Section FINANCIAL PERFORMANCE OF PROJECTS The share of investment operations that achieved ratings of Good-Excellent on project financial performance remained fairly stable and was at 56 per cent in Almost one-fifth of all project financial performance ratings, however, remain Unsatisfactory. The project financial performance was rated Good or Excellent in 56 per cent of the cases for the period Environmental performance of projects is measured by accumulating the environmental and health and safety performance indicators: environment being the status of the environment in the vicinity; health and safety: the way in which health and safety and respective risk assessment systems are effectively applied and the extent of compliance in this respect; pollution loads and energy efficiency: the extent to which the emissions are significantly lower that the regulatory limits; environmental management: the level of compliance with the agreed environmental action plan; public consultation and participation: whether the public consultation and participation has been carefully planned and organised with a responsible person in charge. The performance standards for FI sub-projects are largely local laws while non-fi projects are measured against EU or WB standards. The extent of environmental change (environmental impact) is measured as the difference between the environmental performance before the project started and its performance at the time of evaluation. The EBRD s environmental categories are defined on the web: Category A projects are those projects presenting the greatest environmental risk. The project financial performance by volume of investments is Good-Excellent for 66 per cent of the evaluated projects.

19 8 Evaluation data support the fact that financially sustainable projects are more difficult to achieve in the early and intermediate transition countries, including in Russia, where the systemic constraints and risks are high. The percentage of Good and Excellent is as low as 29 per cent of the cases in the ETCs (Appendix 8, section 5), showing the need to focus attention on the systemic constraints at sector level that are likely to directly affect the financial performance of the project (see Chapter 2). The risk was relatively high on average in general industry and in financial institutions, in particular in complex reform projects without a strong strategic sponsor. Positive financial performance, where the sustainability of a project is not threatened, is a necessary condition for transition impact to unfold, but it is not a sufficient condition as some financially successful projects can still score low on transition impact. Chapter 3 provides further analysis on factors affecting financial performance, as well as other aspects of project performance, for highly successful and unsuccessful projects. 1.6 ENHANCING THE BANK S ADDITIONALITY IN PROJECTS Since the establishment of the EBRD, the Bank s additionality in projects has been very good in the majority of cases. 17 On a cumulative basis, from 1993 to 2003, 62 per cent of the projects evaluated rated the Bank s additionality as Verified in All Respects, 28 per cent Largely Verified and only 10 per cent Verified in Part or Not Verified. The Bank s additionality in projects was rated Verified in All Respects and Largely Verified in 90 per cent of the cases for the period A closer analysis, however, shows deterioration over time, with the latter two ratings rising from below five per cent in the mid-1990s to 14 per cent in 2001, and as high as 24 per cent in While there was a slight improvement in 2003 (see Chart 1.6), the figures of recent years indicate that the Bank should improve its additionality in projects. Chart 1.6: Ratings on the Bank s additionality ( ) 100% 90% % of evaluated projects 80% 70% 60% 50% 40% 30% 20% 10% Not Verified Verified in Part Verified at Large Verified in All Respects 0% Year of evaluation 17 The Bank s additionality in projects is verified in terms of whether the Bank provides financing that could not be mobilised on the same terms by markets and/or whether the Bank can influence the design and functioning of a project to secure transition impact.

20 9 Further analysis in Section 4 of Appendix 8 shows that evaluated projects in central and eastern Europe and the Baltic states score amongst the lowest in respect of the Bank s additionality in the portfolio. This reflects the difficulty of identifying projects in which the Bank is additional in the most advanced transition countries. 1.7 CONCLUSIONS BASED ON EVALUATED PROJECTS IN 2003 In total 54 per cent of the evaluated projects in achieved Successful-Highly Successful overall performance ratings mainly reflecting the more modest financial performance of projects. Of evaluated projects in a total of 75 per cent scored positively on transition impact. This positive outcome on transition impact leads PED to the conclusion that the Bank continues to maintain a portfolio which largely meets the Bank s mandate. Regarding transition impact, the performance of the evaluated projects in the southeastern Europe (SEE) group of countries remains ahead of central eastern Europe and the Baltic states (CEB) and other groups, with a 65 per cent share of projects in the Good or Excellent bracket. Renewed attention should be given to additionality requirements in the design and implementation of projects in the regions advanced in transition, so that high scores on additionality maintains the overall project performance at the desired level. Continued focus should also be given to project financial performance in the ETC group to ensure that the current momentum of improvement in overall project performance is maintained. It is important to enable this group of countries to reach and maintain a level of performance comparable to the higher levels seen in other regions (see also Chapter 2). 2. INSTITUTION BUILDING ACHIEVEMENTS THROUGH PROJECTS IN CENTRAL ASIA 2.1 THE CONTEXT OF INSTITUTIONAL REFORMS The EBRD s Transition Reports distinguish between an initial phase of reforms which take priority in the early years of transition - mainly price and trade liberalisation - and a second phase which focuses on institutional reforms. Such reforms comprise competition policy and other regulatory policies, restructuring and privatisation of large-scale enterprises, the development of market based financial institutions, and reforms in the infrastructure sector. These reforms tend to be considerably more complex than initial reforms, require significant implementation capacity from the state, and can meet resistance from various interest groups. 18 The Transition Reports show that institutional reforms advance more slowly than the initial phase of the trade and price reforms. The main reasons for this are the uncertainty concerning the outcome of the reforms and the limited capacity within the state to implement them. 19 While there are understandable reasons for the delays, the continuing lack of properly See Transition Report 2002, chapter 2. See Transition Report 2001, chapter 2.

21 10 functioning institutions can create major business obstacles and reduce private investment opportunities. Institutional reforms in advanced countries are often prompted by external factors such as direct exposure to foreign markets or prospects of accession into the European Union. These factors are usually absent in early transition economies which have often remained inward oriented, relying on a few primary goods to generate export revenues. In addition, their domestic markets of early transition economies tend to be small, the local infrastructure is underdeveloped and there are often widespread deficiencies in governance, both public and private. 20 This is why proper use of Bank s instruments to successfully accomplish institution building in early transition economies has become so important. The projects should be carefully designed and selected to reflect the reality of weak institutions and poor business environment PROJECTS AS INSTRUMENTS OF INSTITUTIONAL CHANGE The Bank s mandate is to support the transition to a fully operating market economy in countries which are committed to the principles of Article 1 of the Agreement Establishing the Bank. At all stages of transition, one can find a need to build institutional components into projects. It is essential that the project, being the main source of the Bank s intervention, carries institutional changes among its transition objectives. In this Chapter the outcomes of several evaluated projects in which important institutional development experience was gained are presented. A number of more generic conclusions will be drawn with the purpose of enhancing the Bank s activities in early transition environments and improving project quality in that region. 2.3 EXPERIENCE FROM RECENT EVALUATIONS IN CENTRAL ASIA Introduction. Among the evaluation exercises that PED has undertaken in early transition economies over the last two years, the following investment operations in early transition environments in Central Asia provide important lessons in respect of institution building: Uzbekistan - Refinery project rated Successful. The authorities were reluctant to change the organisational (hierarchical) structure and related vested interests in the sector. The technical cooperation funding helped to bring in international financial rules and standards (IFRS) and to expose the staff of the refinery to modern management practices, but only little of what was learned could be used, as the company was not ultimately privatised. Uzbekistan - Power project rated Partly Successful. The Bank financed a few generators but added unrealistic institution building requirements. The leverage resulting from the amount of investment was not sufficient to trigger institutional changes. Kazakhstan Telecoms project rated Successful. The authorities created an independent regulatory body, they undertook to rebalance tariffs, but they did not privatise the company, contrary to expectations See A revised Approach and Action Plan for Early Transition Countries, EBRD SGS04-48, 26 Feb 2004 page 4. Same as above, page 10

22 11 Kazakhstan - Aktau Port rated Partly Successful. The harbour administration was expected to act as an autonomous entity and manage its assets according to commercial principles. The central government, however, imposed the subletting of lucrative petroleum loading facilities to another state entity on unfavourable terms. This policy endangers the port s ability to self-finance necessary future improvement and extension work, and possibly its debt servicing to the Bank (from port revenues) as well Evaluation findings and lessons learned. For each of the above mentioned projects, PED has selected the main lesson on institution building (reported in this section), together with a more detailed description of project experience and highlights of key evaluation findings that leads to the lesson. LESSON 1: The choice of the project within the sector, while aiming to achieve commercialisation and transition targets, is very important and can contribute to the fulfilment of the key institutional development target. Simply participating in some relatively minor improvements of an outdated (old Russian design) refinery situated in the most difficult location - concerning feedstock sources and product sales - may not give the Bank much leverage regarding sector reform targets in a difficult low transition environment. The Bank s investment in the natural resources sector in Uzbekistan took place in January 1997 in the form of a US$ 90 million loan to the National Corporation for Oil and Gas Industry (UNG), guaranteed by the Republic of Uzbekistan. In a parallel financing, JEXIM provided a similar loan amount for this project. These loans were used to install a desulphurisation unit (HDS plant) at the Fergana refinery to cope with high sulphur domestic oil, together with a number of other refinery modifications to improve the environmental and safety standard and increase the yield of the refinery. The project achieved some progress at the company level (reduction of environmental pollution, upgrade of technology, reduction of sulphur content, safety training and motivation of staff and management etc.) but the impact on the sector has proved to be limited and there is a high risk that the transition impact will not materialise due to the centralised decision making process. Major evaluation finding from the refinery project: The evaluation of this project concluded that the commercialisation and privatisation covenant of the Bank was not well conceived. There are almost no circumstances where a stand alone refinery could be successfully commercialised and ultimately privatised, due to the inherent lack of interest from the global oil companies and the difficulty of structuring a refinery as a financially viable stand-alone project. This is particularly the case in remote frontier locations with difficult access to both feedstock and to key markets for petroleum products. LESSON 2: The size of the project, illustrating to a degree the Bank s commitment to the sector, needs to be in proportion to the envisaged transition impact targets. Specific targets must also be supported by a well structured technical cooperation component with timely delivery of the goods in close coordination with the project implementation schedule.