Completion Report. Project Number: Loan Number: 2119/2120 December Azerbaijan: Urban Water Supply and Sanitation Project

Transcription

1 Completion Report Project Number: Loan Number: 2119/2120 December 2011 Azerbaijan: Urban Water Supply and Sanitation Project

2 CURRENCY EQUIVALENTS Currency Unit Azerbaijan manat (AZN) At Appraisal At Project Completion (as of 31 October 2004) (as of 31 December 2010) AZN1.00 = $ $ $1.00 = AZN4,905 AZN ABBREVIATIONS ADB Asian Development Bank ADF Asian Development Fund EIRR economic internal rate of return FIRR financial internal rate or return JSC joint stock company MFF multitranche financing facility OCR ordinary capital resources PIU project implementation unit PMU project management unit PPMS project performance management system SAWMA State Amelioration and Water Management Agency TA technical assistance TWUA town water users association WSS water supply and sanitation NOTES (i) (ii) The fiscal year of the government and its agencies ends on 31 December. In this report, "$" refers to US dollars. Vice-President X. Zhao, Operations 1 Director General J. Miranda, Central and West Asia Department (CWRD) Country Director O. Norojono, Azerbaijan Resident Mission, CWRD Sector Director R. Subramaniam, Urban Development and Water Division, CWRD Team leader Team members M. Davila, Urban Development Specialist, CWRD F. Huseynbeyov, Senior Project Officer, AZRM R. Seno, Operations Assistant, CWRD C. Soliman, Associate Operations Officer, CWRD A. Yusupov, Young Professional, CWRD In preparing any country program or strategy, financing any project, or by making any designation of or reference to a particular territory or geographic area in this document, the Asian Development Bank does not intend to make any judgments as to the legal or other status of any territory or area.

3 CONTENTS Page BASIC DATA I. PROJECT DESCRIPTION 1 II. EVALUATION OF DESIGN AND IMPLEMENTATION 2 A. Relevance of Design and Formulation 2 B. Project Outputs 3 C. Project Costs 3 D. Disbursements 5 E. Project Schedule 5 F. Implementation Arrangements 5 G. Conditions and Covenants 6 H. Related Technical Assistance 7 I. Consultant Recruitment and Procurement 8 J. Performance of Consultants, Contractors, and Suppliers 8 K. Performance of the Borrower and the Executing Agency 8 L. Performance of the Asian Development Bank 9 III. EVALUATION OF PERFORMANCE 10 A. Relevance 10 B. Effectiveness in Achieving Outcome 10 C. Efficiency in Achieving Outcome and Outputs 10 D. Preliminary Assessment of Sustainability 11 E. Impact 11 IV. OVERALL ASSESSMENT AND RECOMMENDATIONS 12 A. Overall Assessment 12 B. Lessons 12 C. Recommendations 13 APPENDIXES 1. Design and Monitoring Framework Status of Compliance with Loan Covenants Economic and Financial Reevaluation Overall Assessment 28 ii

4 BASIC DATA A. Loan Identification 1. Country 2. Loan Number 3. Project Title 4. Borrower 5. Executing Agencies 6. Amount of Loan 7. Project Completion Report Number B. Loan Data 1. Appraisal Date Started Date Completed 2. Loan Negotiations Date Started Date Completed 3. Date of Board Approval 4. Date of Loan Agreement 5. Date of Loan Effectiveness In Loan Agreement Actual Number of Extensions 6. Closing Date In Loan Agreement Actual Number of Extensions 7. Terms of Loan Special Funds Resources loan Interest Rate Maturity (number of years) Grace Period (number of years) Ordinary Capital Resources loan Interest Rate Maturity (number of years) Grace Period (number of years) 8. Terms of Relending (if any) Special Fund Resources loan Interest Rate Maturity (number of years) Grace Period (number of years) Second-Step Borrower Azerbaijan 2119 and 2120 Urban Water Supply and Sanitation Republic of Azerbaijan Loan 2119: Azersu Joint Stock Company Loan 2120: State Amelioration and Water Management Committee (formerly State Committee for Construction and Architecture) Loan 2119 SDR13,580,000 (ADF) Loan 2120 $10,000,000 (OCR) PCR:AZE July July October October December November February October 2006 Three 30 June 2011 TBD None 1% per year during the grace period, and 1.5% thereafter 32 8 London interbank offered rate (LIBOR-based) 0.75% commitment charge per annum % per year during the grace period, and 1.5% thereafter 32 8 Agdash and Goychay joint stock companies

5 iii Ordinary Capital Resources loan Interest Rate Maturity (number of years) Grace Period (number of years) Second-Step Borrower London interbank offered rate (LIBOR-based) 0.75% commitment charge per annum 25 6 Nakhchivan joint stock companies 9. Disbursements a. Dates Loan 2119 Initial Disbursement 26 September 2007 Effective Date 27 October 2006 Final Disbursement TBA Original Closing Date 30 June 2011 Time Interval Time Interval 56 months Loan 2120 Initial Disbursement 15 December 2006 Final Disbursement TBA Time Interval Effective Date 27 October 2006 Original Closing Date 30 June 2011 Time Interval 56 months b. Loan Amount (SDR) Last Revised Allocation Net Amount Available Category or Subloan Original Allocation Amount Canceled Amount Disbursed Civil Works Agdash 1,744,000 3,520, ,520,000 3,520,000 0 Goychay 1,225,000 4,977, ,977,000 4,977,000 0 Nakhchivan 0 1,562, ,562,000 1,562,000 0 Equipment and Materials Agdash 3,337, , , ,000 0 Goychay 2,178, , , ,000 0 Nakhchivan 372, Consulting Services Project Undisbursed Balance 1,582,000 2,686, ,686,000 2,686,000 0 implementation Management 1,195, contractor Incremental 255,000 90, ,000 90,000 0 Administration Interest Charge 275, , , ,000 0 Unallocated 1,417, Total 13,580,000 13,580, ,580,000 13,580,000 0 c. Loan Amount ($) Category or Subloan Original Allocation Last Revised Allocation Amount Canceled Net Amount Available Amount Disbursed Undisbursed Balance Civil Works 915,000 1,908, ,908,000 1,892,021 15,979 Equipment and Materials 7,474,000 7,508, ,508,000 7,179, ,933

6 iv Interest and 725, , , , Commitment Charge Unallocated 886, Incremental 270, , , ,759 Administration Total 10,000,000 10,000, ,000,000 9,524, , Local Costs (Financed) Loan Amount ($) 11,778,665 - Percent of Local Costs Loan Amount ($) 140,241 - Percent of Local Costs 4.70 C. Project Data 1. Project Cost ($ million) Loan 2119 Cost Appraisal Estimate Actual Foreign Exchange Cost Local Currency Cost Total Loan 2120 Cost Appraisal Estimate Actual Foreign Exchange Cost Local Currency Cost Total Financing Plan ($ million) Cost Appraisal Estimate Actual Implementation Costs Borrower Financed ADB Financed ADF (Loan 2119) OCR (Loan 2120) Other External Financing Total IDC Costs Borrower Financed ADB Financed: ADF (Loan 2119) OCR (Loan 2120) Other External Financing Total ADB = Asian Development Bank, ADF = Asian Development Fund, IDC = interest during construction, OCR = ordinary capital resources.

7 v 3. Cost Breakdown by Project Component ($ million) Loan 2119 Component Appraisal Estimate Actual Civil Works Equipment and Materials Consulting Services Project implementation Management contractor Incremental Administration Contingencies Interest and Other Charges during Construction Total a a Estimated dollar equivalent in SDR. Loan 2120 Component Appraisal Estimate Actual Civil Works Equipment and Materials Interest and Commitment Charge Contingencies Incremental Administration Total a a Final withdrawal application to be submitted by the executing agency. 4. Project Schedule Item Appraisal Estimate Actual Date of Contract with Project Implementation Consultants Q Q Completion of Engineering Designs Q Q Civil Works Contract Date of award Q Q Completion of work Q Q Equipment and Supplies Dates First procurement Q Q Last procurement Q Q Completion of equipment installation Q Q Start of Operations Completion of tests and commissioning Q Q Beginning of start-up Q Q4 2010

8 vi 5. Project Performance Report Ratings Implementation Period Development Objectives Ratings Implementation Progress From 31 December 2004 to 31 May 2005 S S From 30 June 2005 to 30 June 2006 S U From 31 July 2006 to 31 January 2007 PS U From 28 February 2007 to 30 September 2007 S S From 31 October 2007 to 31 January 2008 S PS From 29 February 2008 to 28 February 2009 S S From 31 March 2009 to 30 June 2010 S S From 1 July 2010 to 31 July 2010 S U From 31 August 2010 to 31 December 2010 S S PS = partly satisfactory, S = satisfactory, U = unsatisfactory. D. Data on Asian Development Bank Missions Name of Mission Date No. of Persons No. of Person-Days Specialization of Members a Fact-finding 24 May 11 Jun c, i, j, k Appraisal Jul i, j, k Inception b None Review Nov a Review Feb a, d Review Oct a, d Review 4 31 Mar 7 Apr b, e Midterm review 5 9 Oct b, d, g Review Feb b Review 6 c 5 9 Oct b, d Review May a, d Project completion review c 27 Oct 4 Nov a, f, h a a = senior urban development specialist, b = urban development specialist, c = counsel, d = project implementation officer, e = senior social development and resettlement specialist, f = young professional, g = associate project analyst (formerly assistant project analyst), h = associate operations officer, i = urban economist, j = principal urban development specialist, k = consultant. b No available data. c While in the field for the review of tranche 1 of loan 2571 for the Water Supply and Sanitation Investment Program, the mission also reviewed loans 2119 and 2120.

9 I. PROJECT DESCRIPTION 1. The Urban Water Supply and Sanitation Project for Azerbaijan was approved by the Asian Development Bank (ADB) in 2004 to address the poor water supply and sanitation (WSS) situation in the country. 1 Prior to the project, several regions of the country were beset with a range of problems including: outdated and dilapidated physical infrastructure, operational and technical losses, institutional weaknesses, and severe financial constraints. Most of the WSS systems in the country were in a serious state of disrepair. In secondary towns of Agdash and Goychay, service delivery was extremely poor. Social surveys showed that 95% of respondents rated water quality as inferior. Almost one-third of the respondents were receiving water only 1 or 2 days a week. Poor sanitation caused severe public health and environmental problems. Stagnant sewage and infiltration of sewage into the water supply system caused diseases and infections. 2. The objective of the project was to improve the quality, reliability, and sustainability of WSS services, and improve management of water and sewerage systems in Agdash, Goychay, and Nakhchivan towns. The physical infrastructure component of the project included comprehensive rehabilitation of the WSS systems, including water extraction and intake facilities, water supply and sewerage networks, and wastewater treatment. The institutional reform component included creation of joint stock companies (JSCs) in project towns; they would be entirely new independent entities and would not include any assets or staff of the existing sukanals (water and wastewater enterprises). This reform was to be accompanied by the introduction of a management contract, tariff reform, and creation of town water users associations (TWUAs). As originally planned, the project would provide 147,000 customers with potable water and reliable sewerage services with appropriate treatment for wastewater. 3. In 2008, the Government requested ADB to approve a change in scope to address a severe cost overrun between 2004 and 2008, with the price of steel, cement, pipes and petroleum products rising steeply and pushing the overall cost up by 220%. Specifically, the Government proposed to exclude the rehabilitation or construction of water distribution and sewerage systems, and to focus exclusively on water intake and transmission facilities in the three principal towns (Agdash, Goychay and Nakchivan). This approach would address intake related problems as fully as possible, instead of continuing with a piecemeal solution. The change resulted in: (i) civil works expanding by 40% from $12.1 million to $17.4 million, targeting water intake and transmission systems; and (ii) equipment and materials being reduced by about 40% from $14.7 million to $8.2 million. The Government requested that the WSS network and distribution needs be supported through a multi-tranche financing facility (MFF) 2 along with comprehensive system-wide reforms. Most of the physical components removed from the original project design have been included (often at higher specification standards) under the MFF. In parallel with the focused approach of building intake facilities, the reform elements of the project were also modified (see section II below). As a result of these changes, the design and monitoring framework (DMF) had to be amended to take reflect the revised inputs and activities, as well as revised output, outcome and impact performance targets. Appendix 1 presents both the original and revised performance indicators. 1 ADB Report and Recommendation of the President to the Board of Directors: Proposed Loans and Technical Assistance Grant to the Republic of Azerbaijan for the Urban Water Supply and Sanitation Project. Manila. 2 ADB Report and Recommendation of the President to the Board of Directors: Proposed Multitranche Financing Facility to the Republic of Azerbaijan for the Water Supply and Sanitation Program. Manila.

10 2 II. EVALUATION OF DESIGN AND IMPLEMENTATION A. Relevance of Design and Formulation 4. The initial design of the project was relevant, based on an assessment of the investment as well as reform requirements. The impact and outcome were consistent with the government s development strategy. The design of the project was consistent with the ADB country strategy and Strategy It focused on infrastructure investments combined with reforms, and included many best practice reform ideas including decentralization, creation of JSCs, management contracts, and civil society participation through the TWUAs. Onlending, rather than ongranting, of funds was expected to encourage financial independence of the water utilities. 5. However, the investment requirements turned out to be far larger than estimated; and the focus on and design of the reforms were ahead of their time. This was ADB s first project in WSS in the country. The reform design included sound ideas in principle, but the WSS sector in Azerbaijan was not ready for wide-ranging and rapid reforms. The investment related changes are outlined in para. 3. A few critical challenges on the reform front are outlined below. 6. Decentralization of WSS management. The project specified creation of decentralized JSCs. At the time of the project conceptualization, the government was centralizing water supply through the creation of Azersu, an integrated JSC to oversee WSS in the country, with the exception of Nakchivan. The water sector was close to collapse and the government did not want to further fragment the sector. Decentralization would have created weak entities with skeletal assets and very low capacity. Furthermore, full decentralization would have reduced economies of scale and made it more difficult to attract suitable expertise. Finally, under Azerbaijan law, municipalities were not allowed to own JSCs, and they had only a limited role in both governance and service provision. 7. Radical JSC design. The JSC design tried to address the concern of creating weak entities by providing new and rehabilitated assets, and replacing all staff of the existing sukanals with new staff and management under the JSCs. While new assets were constructed, the Project Implementation Unit (PIU) and Project Management Unit (PMU) were to manage the delivery of WSS services. However, the JSC design had several flaws; in particular, dismissing all staff would be politically unacceptable. The PIU staff were unlikely to be able to provide adequate services while operating an unfamiliar system. Finally, the assets provided under the project were insufficient to fully rehabilitate the existing system. 8. Introduction of management contract too early. At the time of appraisal, experience with public private partnerships in Azerbaijan was limited, and the country s previous experience had not been positive. Thus the government was reluctant to engage a management contractor. Also, management contracts for small towns, combined with an uncertain regulatory framework, were not likely to attract well-qualified international contractors. Risk of failure and subsequent reputational risks were high. 3 ADB Strategy 2020: The Long-Term Strategic Framework of the Asian Development Bank, Manila.

11 3 9. These design challenges were successfully addressed through the major changes in scope and implementation arrangements approved by ADB s Management in 2008 and 2009, which increased the relevance of the project to meet the WSS needs in Azerbaijan. In summary, the changes include: (i) (ii) Relaxing conditions for JSC formation and deferring the management contract. JSCs were formed as independent entities but retained existing staff and assets. They are still part of Azersu. The advisory technical assistance (TA) consultant reported that there was no willingness to implement the management contract. Revising the project investment plan. The scope of the investments was reduced to what was possible with available funds; the remaining components were moved to the MFF, which followed the project. Following the revisions, the project covered water intakes, office buildings, and essential equipment for operation and maintenance. 10. These revisions greatly reduced the scope of the project but made it more realistic. This, combined with the preparation of the MFF, recast the project as a first step in longer-term ADB engagement in WSS in Azerbaijan. Within that framework the revised project became very relevant to sector development. B. Project Outputs 11. Outputs as envisaged during appraisal and as per the changes in scope and implementation approved subsequently, and their status are given in Appendix 1. The removal of investments in distribution networks means that the project did not achieve projected outcomes, including increases in coverage, and reduction of nonrevenue water and service quality. Also, removal of investments in sewerage means that increased sewerage coverage and improved sewerage treatment were not achieved. The reduction in outputs was the result of somewhat optimistic initial cost estimates and cost escalation, which meant not all planned outputs could be achieved. C. Project Costs 12. The original project costs were greatly underestimated. The cost of the original project components increased by 218% from appraisal in 2004 to the project revision in The main reasons for the increase include the following: (i) (ii) (iii) Increases in construction costs. Gross domestic product growth was very high during (more than 30% in some years) as additional oil production capacity became available. This was accompanied by increases in prices in all sectors of the economy, including construction costs, which are estimated to have increased by around 150%. Optimistic project cost estimates at appraisal. The project cost estimates were somewhat over-optimistic, as the ADB resources available for Azerbaijan were limited in Fulfilling the original project scope with the available budget would have been difficult, even without cost escalation. Lack of information on the technical condition of existing assets. During appraisal, the project planned to rehabilitate some existing assets. However,

12 4 during the initial stages of implementation, the assets were found to be in very poor condition and needed to be completely replaced. While leading to a severe increase in costs, the problem surfaced after approval because of lack of information and relatively limited scope and time for engineering assessments during the project preparatory technical assistance. This is a relatively common problem for ADB projects (para. 66). 13. After the revisions in project scope, the project costs did not increase (Table 1). Table 1: Comparative Assessment of Project Costs by Category ($ million) Category Original Actual 1. Civil Works Water intake and structures Other civil works Sewage treatment plants Subtotal (1) Materials and Equipment Water transmission Water distribution Vehicles and equipment Sewer lines Subtotal (2) Consulting Services Project implementation (firm) Individual consultants Subtotal (3) Incremental Administration Interest Charges Contingencies Source: ADB Estimates 14. The economic internal rate of return (EIRR) and financial internal rate of return (FIRR) decreased because the reduced project scope led to decreased benefits, while the costs remained the same. However, since the original projects costs were underestimated, the original EIRR and FIRR were not realistic. Furthermore, the assets in Agdash and Goychay are not fully commissioned, because the existing distribution network cannot accept the increased water pressure and increased water flow will increase the leakage. 15. The increase of the local currency cost ceilings in November 2009 had no impact on the project. These increases were for civil works components with high local currency costs: (i) for loan 2119, from SDR5,228,000 to SDR8,911,000 due to the reallocation of funds to the civil works component; and

13 5 (ii) for loan 2120, from $1,592,000 to $2,978,000 due to the reallocation of funds to the civil works component. D. Disbursements 16. No disbursement schedule was prepared during appraisal. Significant disbursements only started in 2008, although they were planned in 2006 (according to activities in the project schedule). This was mostly due to delays in project effectiveness and procurement problems (paras. 20 and 22). Once project implementation started in 2008, both executing agencies were able to achieve full disbursement before project closing. The only lag in disbursements was a slight delay in purchase and installation of chlorination and laboratory equipment for Nakhchivan. This was due to revisions in designs for the water treatment plant under the MFF and not related to the project. 17. Both imprest fund and statement of expenditure disbursement methods were used. Both methods were effective and both executing agencies had no problems in using them. E. Project Schedule 18. The project experienced major delays on four fronts. 19. Delay in signing the loan agreement. The loan was approved in December 2004 but the loan agreement was not signed until October The delay was largely due to the late formal submission by ADB of the approved loan agreement to the government. The government did not initiate signing authorization until it received a formal letter with the approved loan agreement in mid Upon receipt of the loan agreement, the government issued the signing authorization in 3 months. 20. Delay in loan effectiveness. The loan was declared effective in October 2006, 22 months after loan approval and 17 months later the date originally planned, because of delay in the issuance of a legal opinion in English on the loan agreement. 21. Delay in PMU and PIU staffing. The PMU and PIU were fully staffed only in February 2008, 32 months later than originally planned. No clear reason was given for this delay, but it seems to be part of Azersu s initial reluctance to implement the project (para. 41). 22. Delay in procurement of design consultants. The consultants were recruited in December 2007, 14 months later than originally planned. The delay was due to Azersu s lack of capacity and unwillingness to implement the project as originally designed (para. 41). 23. The minor delay in procurement and installation of chlorination and laboratory equipment in Nakhchivan was due to technical revisions and delays in water treatment plant designs under the MFF. This delayed the project by about 6 months. F. Implementation Arrangements 24. As designed at appraisal, the project implementation arrangements were adequate. The project used a relatively standard arrangement with a steering committee providing overall supervision, a PMU based in the executing agency, and PIUs in target cities. The only peculiarity of the arrangements was that the State Committee for Construction and Architecture was originally the executing agency for Nakhchivan. Selecting this agency was not appropriate,

14 6 as it was not directly responsible for WSS. Changing the executing agency to the State Amelioration and Water Management Agency (SAWMA) improved the implementation arrangements since SAWMA was directly involved in WSS in Nakhchivan. 25. The actual implementation arrangements turned out to be quite different than designed. The steering committee provided nominal supervision for the project, met infrequently, and did not have a strong role in the project. The real drivers of project implementation were the two executing agencies: Azersu and SAWMA. 26. With Azersu, the real implementation arrangements were significantly different than designed. The PIUs were initially created, but relatively quickly absorbed into the PMU in Baku. This was a peculiar change but had no tangible effects due to the strong centralization for WSS. Furthermore, the PMU played a relatively minor role in project implementation. Azersu s own procurement and technical departments conducted most of the implementation. Again considering the centralized nature of Azersu, this change did not have a major effect on project implementation. However, if Azersu departments were to be responsible for implementation, this should have been made transparent. It is noteworthy that Azersu went through a major restructuring in 2010 and a new management has assumed charge. These changes have greatly facilitated WSS reforms and investments in Azerbaijan, including those supported under ADB interventions. The ongoing MFF tranches are being implemented in a satisfactory manner. 27. SAWMA project implementation arrangements worked as designed. Since SAWMA is based in Nakhchivan city, no separate PIU was required. 28. One particular problem in the design was that all consultant procurement was done through Azersu. This meant that SAWMA had to move at the same pace as Azersu at the start of the project, even if SAWMA wanted to move faster. Also the process created difficulties for SAWMA to communicate with and to access their design consultants based in Baku. This problem was rectified in the MFF implementation arrangement where SAWMA procures its own consultants. Other than this problem, the implementation arrangements delivered project outputs adequately. Other problems encountered during implementation were the result of performance of government agencies and not of the implementation arrangements. G. Conditions and Covenants 29. The list of covenants and status of compliance are presented in Appendix 2. The covenants were relevant considering the original project design. In particular, it was good that institutional reforms were prominently included in the covenants, even though institutional design was problematic. Despite these problems, most of the covenants were complied with including general, safeguards, and reporting covenants. However, due to project problems and revisions, compliance was delayed, covenants were waived, and some covenants were complied with but not as originally designed. 30. Covenants where compliance was significantly delayed include the following: (i) Formation of JSCs and establishment of PMUs and PIUs. Compliance with this covenant was delayed due to the lag in issuing the legal opinion on the loan agreement (para. 40). This was a procedural problem, which occurred only in this project. Later ADB projects were not affected. In retrospect this covenant was somewhat unrealistic because of government reluctance to implement the project according to the original design.

15 7 (ii) Hiring of design and project management consultants and appointing PMU director and staff. This was delayed due to low Azersu capacity and reluctance to start project implementation (para. 41). This covenant, however, was realistic and should have been complied with on time. 31. The revisions made to the project design meant that some covenants were either waived or were complied with but within the revised project scope: (i) (ii) (iii) (iv) Hiring a management contractor. This covenant was waived during the change in project scope. This was done because the institutional reforms under the project were reduced to more realistic aspects and the project focused on investments that led to the MFF loan. Considering the lack of information, reputational risks for the private sector, and premature introduction of the private sector into WSS, this covenant was not realistic. No transfer of old assets and liabilities to JSCs. This covenant could not be fully complied with because assets created through the project were not sufficient to replace all of the old assets. The old assets will be phased out as new assets are completed under the MFF. All of the water sector liabilities were written off in This covenant was quite unrealistic considering ADB s unfamiliarity with the sector before the start of the project and poor capital expenditure estimates. Independence of JSCs and asset ownership in Agdash and Goychay. The JSC operations are independent. However, Azersu still owns the assets, as municipalities cannot own assets under Azerbaijan law. This aspect of the covenant was not realistic since it contravened existing laws. Covenants relating to creation of the PMU and PIUs. The PMU and PIUs were created more to comply with conditions rather than as operating entities (para. 26). H. Related Technical Assistance 32. The technical assistance (TA) included with the project was meant to support institutional reforms envisaged in the project design. The TA was to assist the government in (i) developing JSCs, (ii) developing frameworks and procedures for closure of the sukanals, (iii) providing transaction advisory services to hire the management contractor, (iv) developing project management procedures and helping the PMUs, and (v) supporting public awareness campaigns on tariff increases and water conservation. 33. The TA outputs included discussion papers on customer service and complaint management systems, corporate development plans, organizational development and human resources, management contractor strategy, and liquidation of sukanals. Also, the TA provided training for JSC staff and suggested amendments in the existing charter for the Agdash and Goychay JSCs. 34. The TA did not provide any transaction advisory services, to hire the management contractor, and this was eventually removed from the project. Also, it did not assist the PMUs or support the awareness campaigns because these were not in effect during TA implementation.

16 8 35. Overall the TA is rated partially successful 4 because transaction advisory services were not carried out and the TA had a relatively limited impact due to understandable government reluctance to implement the reforms. Even the consultant noted the lack of interest in implementing the reforms without the actual physical investments. This means that the TA was implemented before the government was ready to accept its recommendations. However, during discussions, some of the Azersu staff indicated interest in implementing some of the recommendations in the future. I. Consultant Recruitment and Procurement 36. The recruitment of design and project management consultants was problematic. Azersu required about 18 months to complete the procurement, with significant gaps between stages of the procurement process. For example, preparation and submission of the shortlist took 6 months. Overall these problems were the result of Azersu s general reluctance to implement the project. Furthermore, the original estimated costs for the consulting package were too low due to significant price increases in Azerbaijan. The remaining four individual consultants and topographic survey consultants were procured without any issues. All consultant procurement was done according to ADB procedures. 37. The original procurement plan was significantly changed during the 2008 revisions and many components were removed. However, the procurement methods were not changed. The main civil works and equipment packages were procured using international competitive bidding. No significant procurement issues were encountered after the 2008 revisions. J. Performance of Consultants, Contractors, and Suppliers 38. Once project implementation was under way, all consultants, contractors, and suppliers under the project performed satisfactorily. The only minor issue was that SAWMA had difficulty working with the design and project management consultants due to geographic distance. The consultant posted some staff in Nakhchivan, and eventually the consultants and SAWMA managed to establish a good working relationship. K. Performance of the Borrower and the Executing Agencies 39. The borrower and executing agencies displayed two distinct and completely opposite levels of performance. At the start of the project their performance was very poor, but after the project revisions their performance improved greatly. 40. At the beginning of the project, the government took an exceptionally long time to make the loan effective and sign the loan agreement. The Ministry of Justice caused this major delay by refusing to provide a legal opinion on the loan in English. The justification for doing this is that other development partners, such as the World Bank, did not require this. The borrower refused to acknowledge the difference in procedures between different development organizations. It subsequently complied, and once the Ministry of Justice issued an opinion for the project, this problem was not encountered again in subsequent projects. 4 Asian Development Bank Technical Assistance Completion Report. TA 4465-AZE: Institutional Strengthening of the Water Supply and Sanitation Sector in Secondary Towns. Manila

17 9 41. At the start of the project, Azersu performance was poor. Azersu took an exceptionally long time to procure the consultants. The main reason given is that Azersu was not experienced with ADB procurement procedures. Yet this is in contrast to Azersu performance after project revision, when no procurement issues arose. Furthermore, Azersu took a very long time to create and staff the PMU and PIU. No clear reason was given for this delay, but it is probably because Azersu intended to manage the project implementation internally (para. 26). The sudden change in performance indicates that Azersu had the capacity to implement the project from the start. 42. The original project design appears to have resulted in significant discussion within Azersu and the government. Overall, people involved in the project indicated that the radical JSC design and hiring of the management contractor were controversial. Furthermore there was some confusion on how these reforms would align with government plans in the sector. More seamless sharing of such differences of position to ADB by the Government during appraisal or even initial implementation of the project would have greatly helped the project. 43. Following the project revision, no issues were identified with borrower or Azersu performance. The borrower gave all approvals and counterpart funds were available on time. Azersu managed to carry out all procurement without any major problems. Procurement documentation was good, and Azersu was responsive to ADB requests. 44. For SAWMA, project implementation started only after the procurement of design and project management consultants by Azersu. Before that, SAWMA could do very little on its own to move the project forward. SAWMA performance from the start of the project was very good. All SAWMA procurement and project activities were done on time. The documentation prepared by SAWMA was good. SAWMA was very responsive and established good communications and working relationships with the ADB team. 45. The capacity and capability of the two executing agencies were not assessed at appraisal. However, both executing agencies reported that their capacity increased as they learned from the implementation experience and from the consultants. Furthermore, the JSCs seem to have developed improved operating capacity. All the JSCs are reporting improvements in financial performance, and Nakhchivan is already providing improved service to its customers using the new facilities (para. 59). All the JSCs have stated they are committed to achieving financial sustainability and paying back the loans for the project as soon as the networks are commissioned under the MFF. This is a very commendable intention. 46. Initial borrower and executing agency performance was unsatisfactory. Yet, performance after project revisions in 2008 was satisfactory. This means that overall, borrower performance was less satisfactory, Azersu performance was less satisfactory, and SAWMA performance was highly satisfactory. 47. These ratings should come with a strong caveat that both the borrower and executing agencies are now capable of performing at a satisfactory level. In particular, Nakhchivan should be singled out for very good performance despite difficult initial conditions, and is used as a good example for other ADB projects. L. Performance of the Asian Development Bank 48. ADB performance on this project shows two distinct steps. Initial performance was less satisfactory but this was followed by pragmatic project revision and, afterward, satisfactory

18 10 implementation. However, overall, ADB s performance should be categorized as less satisfactory for reasons outlined below. 49. The early stages of implementation were not very effective due to frequent changes of project officers. This project changed project officers seven times and was handled by six different project officers. This means that it was difficult to get the necessary continuity to overcome the initial implementation obstacles. 50. ADB performance greatly improved after the project revisions in Implementation ran smoothly, with no major disagreements between ADB and the executing agencies. Both executing agencies commented that the ADB team was very responsive. 51. Other than the TA activities, ADB did not provide significant advisory services. This is an area that could have been improved. III. EVALUATION OF PERFORMANCE A. Relevance 52. The original project design, impact, and outcome were commendable. However, lack of knowledge and understanding of the sector led to an overambitious design that proved to be impossible to implement in practice. This, combined with increased project costs, almost led to the project being canceled at the end of In its original form the project could be classified not relevant, even though many ideas included in the project are relevant to ADB operations as a whole. 53. However, the ADB implementation team managed to revise the project so that it now fits within the greater reform and investment plan implemented under the MFF. Under this framework the project piloted a JSC model that is acceptable to government, started a policy dialogue with the government, and provided crucial investments in the target towns that will eventually lead to a reliable supply of potable water. As part of this approach, the project became much more relevant and in its final form is rated relevant. B. Effectiveness in Achieving Outcome 54. Most of the project outcomes specified in the design and monitoring framework were not achieved (Appendix 1). The specified outcomes were too ambitious for the size of the investment and ADB knowledge in the sector. However, the actions taken under the project accomplished some of the outcomes and provided the way for achieving the remainder of the outcomes under the MFF program. Overall the project is rated less effective. C. Efficiency in Achieving Outcome and Outputs 55. The reduction in scope of the investments and reforms for the same costs means that the project was not as efficient as estimated at appraisal. The original project EIRR was 21.8%, and EIRRs for each town were Agdash, 25.5%; Goychay, 24.8%; and Nakhchivan, 18.7%. The original project FIRR was 4.4%, and FIRRs for each town were Agdash, 3.6%; Goychay, 5.5%; and Nakhchivan, 4.4%. The project EIRR and FIRR were recalculated when the project was redesigned. The EIRR was 19.1% and FIRR was 4.98%.

19 The post-project EIRR and FIRR for Agdash and Goychay cannot be evaluated since the assets created under the project are not yet being used for delivery of water services, as the intake facilities need to be supplemented by concrete improvements in distribution networks. Therefore they have not yet started to generate economic benefit or financial revenue streams. The post-project EIRR and FIRR for Nakchivan were calculated to be 8.9% and -1.9 respectively, only for the investments made under this project. These low IRRs are due to the fact that the economic and financial benefits of intake and maintenance facilities are constrained by the lack of investments in the rest of the system. In practical terms this means that the water is still not potable, the unaccounted for water is high and coverage is relatively low. The investments in the rest of the system, which will be done under the MFF, will unlock the remaining economic and financial benefits of the investments made under this project. Executing agency efficiency was low before the change in scope but improved significantly afterward. Once project implementation was under way, the ADB team efficiency was very high. Overall the project is rated inefficient due to limited economic and financial benefits that can be realized from the investments at this time. Appendix 3 presents the economic and financial reevaluation undertaken for the project. D. Preliminary Assessment of Sustainability 57. Outputs produced by the project, after the 2008 revisions, built a foundation for longterm improvements and ADB involvement in WSS in Azerbaijan. The physical assets provided under the project, despite their currently limited utilization, are really a first step in the total renewal of the assets in the project towns. In Nakhchivan, the assets are fully operational and have led to improvements in the quality of service (para. 59). In Agdash and Goychay, the assets are sparsely used due to risk of leaks in the existing network. Also the reforms implemented under the project are first steps in creating sustainable utilities. In Nakhchivan, billing and collection is already improved. 58. Sustainability of these achievements is supported by the MFF, which followed the project. Under the MFF, renewal of the assets will be completed and the new assets will be capable of providing reliable and safe water supply. Furthermore, commitment to achieve financial sustainability is strong and the projections prepared under the MFF indicate that the JSCs will be able to cover operating expenses and even repay the capital costs. Overall the project results are most likely sustainable because they are part of the long-term plan to improve the WSS system. E. Impact 59. The impact of the project on consumers is limited. The outputs, after the 2008 revisions in the project scope, concentrated on laying the foundation for long-term changes in WSS rather than on providing direct improvements in customer service. The project increased the quantity of water available for consumers in Nakhchivan and 80% of the customers are receiving a 24- hour water supply. However, water is not yet potable, pending completion of the water treatment plant under the MFF. Also, more consumers are now being billed and paying for the water provided to them. However, beyond this, the consumers, in Agdash and Goychay, have not experienced any other beneficial or adverse impacts because project assets are not yet operational. This is because the increased flow and pressure from the intake facilities would cause increased leakage in the existing distribution system. It would lead to higher maintenance costs, due to increased number of repairs, and electricity costs, due to increased leakage of water which was pumped from the boreholes. These assets are expected to be fully operational once the network is completed under the MFF, after which they will have a significant positive

20 12 impact on consumers, as part of the new water supply system. The project included both environmental control and safeguard measures, which were implemented effectively. No longterm adverse environmental or social safeguards impacts resulted. Overall the project impact is rated moderate. IV. OVERALL ASSESSMENT AND RECOMMENDATIONS A. Overall Assessment 60. The original project design was not implemented as conceived. The original reform design was not done well and the investment funds were insufficient to support the original scope of the investments. Implementation was significantly delayed by government inaction. Also increases in project costs were large. In its original form, the project would be unsuccessful. However, on the verge of being canceled in 2008, the project was successfully recast as a first stage in long-term ADB involvement in WSS. The reform and project scope was reduced to a realistic level; after that project implementation was efficient. Assessed in isolation, the revised project could be seen as successful, were it not for the constraints on the benefits delivered by the project. 61. Most of the intended project impacts and outcomes were not achieved. However, they will be achieved, albeit in a slightly different form, through the MFF that follows the project. Also the economic and financial benefits of the project are limited and will be constrained until investments under the MFF are completed. Overall, considering both forms that the project took, the project is rated partly successful. Appendix 4 summarizes the assessment. B. Lessons 62. Four key lessons are identified from this project: (i) (ii) (iii) (iv) Understanding the sector environment in new member countries is important. When engaging in a new sector or country, understanding the key sector participants and their agendas, as well as developments in the sector is important. This will improve the project design and increase the likelihood that the project will have strong government support. Use phased reform programs. Radical reform programs, especially if they carry political risks, are likely to result in government resistance, unless the government strongly supports the concept. A phased long-term reform program is more likely to gain government support, especially if each stage is aligned with government objectives. Use conservative cost estimates and ensure sufficient funding. The funding allocation should be sufficient to cover project costs; if it is not, the project scope should be reduced or the funding allocation increased with no delay during implementation. Ensure the government understands the proposed project. Ensure that government agencies fully review and understand proposed projects. Reconfirmation with all relevant agencies is critical.

21 13 C. Recommendations 1. Project Related 63. Future monitoring, further action, follow up, and additional assistance. This project does not require any specific monitoring or follow-up because it is followed by an MFF. Therefore, the MFF is essentially the follow-up action and monitoring will be done through the MFF. Also the MFF provides additional assistance that will enhance the project s impact and sustainability. 64. Covenants. The existing covenants that were not waived should be maintained in their current form. 65. Timing of the project performance evaluation report. In this particular case, the project performance evaluation report should be deferred until the completion of the MFF. The project s improvements and impact will not be fully realized until the MFF is implemented. Then the project and the MFF performance can be evaluated as a whole, since they essentially form one continuous program. 2. General 66. The recommendations at appraisal include the following: (i) (ii) (iii) Use reforms that match the country s level of development and capacity. In the case of this project, use of public private partnerships, even in its basic form such as management contracting, was premature. Ensure design ambitions match funding allocations for countries and projects. Increase the scope of the engineering work at project preparation to improve the technical information available for project design. While it could result in higher project preparatory technical assistance costs and duration, it will avoid the problem of missing technical information which adversely affects projects. ADB teams should explore doing additional technical work through advanced action and/or retroactive financing. 67. The recommendations at the implementation stage include the following: (i) (ii) (iii) (iv) The team that processed the project should also implement the project at least for the first 1-2 years. This gives continuity to the project, ensures understanding of the project, and strengthens relationships with government counterparts. Headquarters and resident mission staff should work very closely in a seamless manner. Ensure sufficient training is provided, especially on procurement, for the relevant executing agency staff. Use individual international procurement experts to improve the quality of the executing agency s procurement documents and procedures.