Validation Report. India: North Karnataka Urban Sector Investment Program, Tranche 1. Independent Evaluation Department

Transcription

1 Validation Report Reference Number: PVR-451 Project Number: Loan Number: 2312 August 2016 India: North Karnataka Urban Sector Investment Program, Tranche 1 Independent Evaluation Department

2 ABBREVIATIONS ADB Asian Development Bank CDP community development plan DMF design and monitoring framework EIRR economic internal rate of return FIRR financial internal rate of return GAP gender action plan HSC house service connection IED Independent Evaluation Department km kilometer KSFES Karnataka State Fire and Emergency Services KSPHC Karnataka State Police Housing Corporation KUIDFC Karnataka Urban Infrastructure Development and Finance Corporation lpcd liter per capita per day MFF multitranche financing facility PCR program completion report PFR periodic financing request PMU program management unit ULB urban local body NOTE In this report, $ refers to US dollars. Key Words adb, asian development bank, gender action plan, india, mff, multitranche financing facility, operation and maintenance, sanitation, sewerage, water supply, validation The guidelines formally adopted by the Independent Evaluation Department (IED) on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of IED management, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. The final ratings are the ratings of IED and may or may not coincide with those originally proposed by the consultant engaged for this report. In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, IED does not intend to make any judgments as to the legal or other status of any territory or area.

3 BASIC DATA Project Number PCR Circulation Date 13 Aug 2015 Loan Number 2312 PCR Validation Date Aug 2016 Project Name North Karnataka Urban Sector Investment Program, Tranche 1 Sector and subsector Water and other urban Urban water supply infrastructure and services Theme and subtheme Inclusive social development Human development Safeguard categories Environment B Involuntary Resettlement B Indigenous Peoples B Country India Approved ($ million) Actual ($ million) ADB Financing ADF: 0.00 Total Project Costs ($ million) OCR: Loan Borrower Beneficiaries Others Cofinancier Total Cofinancing Approval Date MFF 2312 Signing Date FFA 2312 Project Officers IED Review Director 6 Dec Jan Oct Jan 2008 K. Tamaki K. Choe M. Tachiiri M. Fan T. Gallego-Lizon W. Kolkma, IED1 Effectiveness Date 8 May Apr 2008 Closing Date 31 Dec Jul 2014 Location ADB headquarters ADB headquarters ADB headquarters ADB headquarters ADB headquarters From Feb 2007 Feb 2008 Mar 2009 Jun 2011 Nov 2011 To Jan 2008 Feb 2009 Jun 2011 Nov 2011 Jul 2014 Team members M. Gatti, Principal Evaluation Specialist, IED2 G. Kilroy, Evaluation Specialist, IED1 M.J. Dimayuga, Senior Evaluation Officer, IED1 I. Green, Consultant ADB = Asian Development Bank; ADF = Asian Development Fund; FFA = framework financing agreement; IED1 = Independent Evaluation Department, Division 1; IED 2 = Independent Evaluation Department, Division 2; MFF = multitranche financing facility; OCR = ordinary capital resources; PCR = project completion report. A. Rationale I. DESCRIPTION 1. This project is Tranche 1 of the North Karnataka Urban Sector Investment Program, 1 a multitranche financing facility (MFF) comprising four tranches. 2 The urban sector and tourism investment proposals of the government s overall investment plan for North Karnataka were recognized as priority sectors and were allocated 33% ($862 million) of the total investment 1 Asian Development Bank (ADB) Report and Recommendation of the President to the Board of Directors: Proposed Multitranche Financing Facility to India for the North Karnataka Urban Sector Investment Program. Manila. 2 The MFF is a flexible financing instrument that enables ADB to provide program assistance by aligning the provision of financing with the project readiness and long-term needs of a client.

4 2 plan, including an Asian Development Bank (ADB) contribution of $270 million in the MFF. The planned impact of the MFF was improvement in the people s quality of life in 25 urban local bodies (ULBs) by providing basic urban infrastructure, which is particularly lacking in North Karnataka compared to the rest of the state. The MFF aimed to increase economic opportunities and growth in these towns, thereby reducing imbalances in the state. Tranche 1 was designed primarily to support water supply and sanitation infrastructure improvements in three ULBs in the towns of Haveri, Hospet, and Raichur. 2. The North Karnataka Urban Sector Investment Program and the executing agency, the Karnataka Urban Infrastructure Development and Finance Corporation (KUIDFC), benefited from the experience of previous ADB assistance to the state in the southern and coastal areas. 3 The program was aligned with the policies and sector priorities of the national government of India, the state government of Karnataka, and that of ADB. 4 The MFF modality was considered appropriate because the government s overall investment program of $2,652 million for North Karnataka provided a clear and long-term strategic context and the MFF sets out a long-term partnership for the sector. 5 In addition, the MFF allows the flexibility required for the investment program to support the participating 25 ULBs with different needs and absorptive capacities. 3. The first periodic financing request (PFR) identified subprojects and components, including sanitation and water supply infrastructure in 3 ULBs, firefighting equipment in 23 ULBs, tourism infrastructure in 6 ULBs, geographic information systems and other hardware in 21 ULBs, 5 consultancy packages, incremental administration costs, and the KUIDFC building. 6 B. Expected Impacts, Outcomes, and Outputs 4. The project completion report (PCR) noted that a separate design and monitoring framework (DMF) for the first tranche was not created until years after Board approval for Tranche 1, and 3 years after its loan signing. 7 It was argued in the PCR that this was the first pilot MFF in the urban sector and as guidelines for MFFs were not clearly developed at that time, it was not possible to prepare a DMF for the first tranche. However, it is noted that the logical framework approach has been used by ADB since the early 1990s and became a mandatory attachment to all project documents in Tranche 1 was a project that needed to be designed using the logical framework approach, albeit within the framework of the long-term MFF. By 2007, there were guidelines 8 for the use of the DMF in MFFs, hence, a DMF for this project could have been provided at an earlier stage before loan effectiveness. 9 3 ADB Report and Recommendation of the President to the Board of Directors: Proposed Loan to India for the Karnataka Urban Development and Coastal Environmental Management Project. Manila; ADB Report and Recommendation of the President to the Board of Directors: Proposed Loans and Technical Assistance to India for the Karnataka Urban Infrastructure Development Project. Manila. Both were implemented by the KUIDFC. 4 Government of India Tenth Five- Year Plan, ( )Delhi; Government of Karnataka Urban Development Policy for Karnataka (Draft); ADB Strategy 2020: The Long-Term Strategic Framework of the Asian Development Bank, Manila; ADB Country Partnership Strategy: India, Manila. 5 Government of Karnataka High Power Committee Report for Redressal of Regional Imbalances. Bangalore. 6 A PFR is a formal request to ADB by the client to finance a tranche under an MFF. 7 A DMF was requested from the government in 2011 (aide memoire of the consultation mission, October 2011). It was also noted that where relevant, as part of the indicators, the DMF should indicate the (i) number of beneficiaries, (ii) % increase in serviced areas, and (iii) baseline information. The DMF should also indicate any assumptions or risks associated with the tranche. 8 ADB Project Performance Management System Guidelines for Preparing a Design and Monitoring Framework, 2nd ed. Manila. This document notes that the primary MFF document is the Framework Financing Agreement with a DMF in Schedule 2 covering the overall MFF. A PFR is a formal request from the counterparties to finance a project under the MFF. A DMF will be prepared for each PFR. This will show the details of the

5 3 5. The DMF provided in the PCR neither reflected the project s original scope prior to the cost escalations nor the further changes in scope when sewerage subprojects and house connections were transferred to Tranche The DMF also failed to present all project outputs. Project outputs under Tranche 1 of the MFF were as follows: (i) implemented investment program for institutional development; (ii) implemented investment program for nonmunicipal infrastructure (equipment), (iii) implemented investment program for sanitation sewerage, and (iv) implemented investment program for water supply. The DMF is silent on a fifth output, which is listed in the PFR and comprised almost a quarter of project funds to support, among others, the construction of an office building for the KUIDFC. 6. In view of the above, the DMF presented the impact of Tranche 1 as overall improvement in the quality of life in project ULBs due to improved urban infrastructure and services, related increase in economic opportunities and growth in North Karnataka, and reduced imbalances between North Karnataka and the rest of the state. The impact and outcome of Tranche 1 were not linked to the agriculture, industry, and tourism sectors. The project outcome was described as improved basic urban services and sustainable infrastructure services for three investment program ULBs. 7. The gender action plan (GAP) and the community development plan (CDP) activities promoted project receptivity and increased awareness on issues related to water, sanitation, and hygiene. Awareness related to hygiene, water, and waste management, along with the enhanced capacities of female community members and elected women representatives to oversee the quality of implementation of physical works, have contributed to long-term capacity development within the sector and to achieving the objectives of the overall MFF. 8. Since 40% of the original project budget was for the sanitation component, the completion of which has been deferred, together with the planned water supply household connections, which was also deferred, it may not be possible to do a meaningful evaluation until these activities are completed under Tranche 4. C. Provision of Inputs 9. At appraisal, the total cost of the Tranche 1 project was estimated at $51.4 million with ADB financing of $33.0 million (64.2%) and a Government of Karnataka contribution of $18.4 million (35.8%). At completion, ADB cost only amounted to $23.4 million (with a canceled amount of $9.6 million), while the government reduced its expenditure to $12.03 million. By component, the breakdown of the ADB cost at appraisal and completion, respectively, is as follows: (i) sanitation $11.6 million and $4.6 million, (ii) water supply $4.3 million and $4.7 million (increased allocation), (iii) nonmunicipal infrastructure $5.6 million and $1.99 million, (iv) institutional development $1.1 million and $0.0, and (v) investment program assistance facility $9.4 million and $12.3 million. Overall, the project expenditure was $35.4 million compared to the appraisal estimate of $51.4 million a 31% reduction in planned expenditure (paras ). investment and the linkage to the overall MFF, i.e., the outputs in the overall DMF are the respective projects. Each output statement becomes the outcome of the respective project to be funded by a PFR. 9 The logical framework approach is not only useful for project monitoring, it is also the framework for project design and should have been available at the time of Tranche 1 approval (January 2007). 10 The DMF continued to reflect the house service connections and sewerage works as outputs. The transfer of these activities to Tranche 4 is noted in the Achievements column of the DMF, but these changes should have been reflected as revised outcome and output performance targets.

6 4 10. Despite advanced action, the recruitment of program consultants was delayed by almost a year (completed in December 2007), and this resulted in the delay of contract awarding. Civil work packages were not awarded until Procurement for civil work packages was challenging because it was difficult to get larger, more competent contractors for projects in interior towns where dispersed works were involved. Although Tranche 1 was effective on 25 April 2008, engaging the apex and the implementing nongovernment organizations was delayed for almost 3 years. 11. A bridging technical assistance of $400,000 as specified in the approved MFF was not utilized in Tranche 1. One of the purposes of the technical assistance was to help prepare private sector participation contracts, however, no such contracts were to be financed under Tranche 1. D. Implementation Arrangements 12. As planned, the State of Karnataka, acting through the KUIDFC, was the executing agency for the program. The implementing agencies under Tranche 1 were (i) the respective ULBs of Haveri, Hospet, and Raichur for the water supply and infrastructure subprojects; 11 (ii) the Karnataka State Police Housing Corporation (KSPHC) for the nonmunicipal infrastructure component; 12 and (iii) the Directorate of Municipal Administration and the KUIDFC for the institutional development component. KUIDFC is also responsible for the investment program assistance facility. The program management unit (PMU) comprised the head office of the KUIDFC in Bangalore, in coordination with the regional office in Hubballi. The PMU was responsible for program coordination among stakeholders, monitoring, implementation, and reporting. It was headed by the managing director of the KUIDFC, acting as program director at the head office. The regional office was headed by the executive program director. In addition, four divisional offices each headed by a divisional program director were established in Balagavi, Ballari, Hubballi, and Kalburgito and these supervised the implementing agencies under the MFF. The program consultant teams were based in the divisional offices. 13. For the MFF, a state-level empowered committee provided oversight for the entire investment program and decided on major issues, including counterpart funding, implementation bottlenecks, land disputes, and special procurement. District-level program steering committees were formed in each of the 13 districts to monitor the implementation of subprograms and institutional reforms in the ULBs, and to coordinate approvals and clearances among various agencies. Five tender evaluation committees were also set up to evaluate tenders. 14. According to the PCR, the KUIDFC complied with 20 of the 31 loan covenants, including covenants with delayed compliance. Five of the loan covenants were deemed not applicable and the remaining six were either classified as partially complied with or will only be met when the infrastructure is fully implemented and operating. Partial compliance was mostly on site management, housekeeping, and health and safety. The PCR reported that the KUIDFC is now giving these issues more attention. Improvements in bill collection efficiencies were applied late, hence, additional time has been allowed. Volumetric tariffs for water and tariffs for sewerage will be implemented only after water meters were installed and the house service connections (HSCs) were provided in Tranche 4 as the completion of these activities are fundamental to achieving the program s overall expected outcomes. 11 The Karnataka Urban Water Supply and Drainage Board was not the implementing agency since there were no bulk water supply schemes included in the project. 12 The Karnataka State Fire and Emergency Services (KSFES) was initially the implementing agency for this component but since the KSFES is only a service provider this was later changed to the KSPHC. However, onlending was to the KSFES while actual implementation was carried out by the KSPHC.

7 5 II. EVALUATION OF PERFORMANCE AND RATINGS A. Relevance of Design and Formulation 15. The project was rated relevant by the PCR. On the positive side, it was consistent with the sector and country partnership strategies of the government and ADB at appraisal and continued to be relevant throughout implementation. The project was also appropriately focused on North Karnataka, which is the least-developed region. As noted in para. 2, the MFF modality was appropriate since it allowed flexibility to support the participating 25 ULBs, each with different needs and absorptive capacities. 16. Despite the project s positive strategic alignment and appropriate use of modality, the project design had some drawbacks. First, the link between the promotion of economic growth a key objective of the government s strategy and the investments in urban infrastructure was not reflected in the DMF. Second, the risk in implementation capacity was mistakenly expressed as an assumption (that trained and qualified staff will be retained in the PMU) and not as a risk. Third, the outcome statement and the associated performance indicators and targets in the Tranche 1 DMF were inadequate or lacking. Taken together, these observations comprising strong strategic relevance but with some design flaws contribute to a rating of relevant. B. Effectiveness in Achieving Project Outcomes and Outputs 17. The PCR rated the project less than effective. A number of issues affected the overall assessment of the project s effectiveness in achieving the expected outcomes. First, although the MFF modality has the flexibility to move project activities to subsequent tranches, the deferment of sewerage works (40% of the original project budget) and water supply household connections should have been reflected in the DMF with revised outcome statements and associated revised performance targets. As the DMF currently stands, over half of the project outcomes have yet to be achieved. Second, although bulk water supply has improved, the envisaged outcome cannot yet be completely achieved because, as the PCR stated, the last mile of connectivity in terms of HSCs and metering could not be included in the tranche (PCR, para. 36). Finally, although the introduction of a volumetric tariff has been instituted by a government order for water supply, this will only become effective with the introduction of metering and household connections planned for Tranche Efforts were made to compensate for the weak project design (which led to cost escalations and reduction in scope), especially to ensure that priority 13 water supply and sanitation subprojects activities could be implemented. Duplications with other programs 14 were eliminated and subprojects with a low degree of readiness were transferred to subsequent tranches. 15 As the sewerage works could not be delivered under Tranche 1, 16 these were transferred to Tranche 4 in 2013 at the request of the government. 19. The project, however, has a number of successful outputs that have contributed to the overall outcomes and should contribute to the wider MFF outcomes in due course. On water 13 Baseline studies identified water and sanitation to be top priorities in the ULBs. 14 For example, the procurement of $1.1 million of geographic information system and other hardware were discontinued to avoid duplication with the World Bank s Karnataka Municipal Reform Project. 15 For example, one package of firefighting equipment and the tourism packages were transferred to Tranche 2 and some institutional development activities were taken up by the state government. 16 At loan closing, 39% of the original loan allocation for this component had been disbursed and the completion of works under Tranche 1 stood at 67% for Haveri, 32% for Hospet, and 22% for Raichur, including the completion of km of networked sewers.

8 6 supply, civil works for bulk, treatment, and distribution networks have been successfully completed in the three project towns, with additional treatment capacity of 9.08 million liters per day (100.08% of target), 256 km of new pipe networks (102.4% of target), and rehabilitation of 9.14 kilometers (km) (30.05% of target) of existing pipe networks. As a result of the project interventions, access to water supply has improved with increased average water availability in these three towns. For households with existing connections and for the 1,464 newly connected households, water availability has increased from 100 liters per capita per day (lpcd) to 125 lpcd in Hospet, from 110 lpcd to 125 lpcd in Raichur, and from 30 lpcd to 60 lpcd in Haveri. In addition, 2,175 people in Haveri and Hospet who are dependent on standposts (connected to the main distribution network), have also benefited from increased water supply. The full benefits of the water supply subprojects will not manifest until the house connections, metering, and volumetric charging are completed under Tranche Through the loan covenants, the project supported reforms that would increase revenues to improve the financial base of the ULBs. For example, a self-assessment system and/or capital value system for property tax has been undertaken in all project towns to improve the efficiency of tax collection. During , the tax collection efficiencies were 82.2% for Raichur, 60.4% for Hospet, and 71.0% for Haveri. Water tariff collection efficiencies are still low with the collection efficiencies of 21.5% for Raichur, 26.5% for Hospet, and 32.3% for Haveri. These collection efficiencies have dropped since appraisal. 17 It was found that water tariff collection efficiencies had a direct correlation with the political will to improve collection efficiencies. 21. Although not appearing in the DMF, the gender mainstreaming and the gender strategy, participation strategy, and activities conducted under the GAP with the help of nongovernment organizations led to the much-needed awareness on hygiene, water, and waste management; and the willingness to pay for house connections (both water supply and sewerage) and volumetric tariffs In summary, notwithstanding several positive achievements, over half of the expected outcomes have yet to be achieved and consequently, this validation assesses Tranche 1 less than effective. C. Efficiency of Resource Use in Achieving Outcomes and Outputs 23. The PCR rated the project less than efficient. The main reasons for the rating were process inefficiencies. The original loan closing date was extended from 31 December 2011 to 23 July 2014 as implementation delays were encountered due to (i) slow start-up associated with the recruitment and mobilization of program consultants; (ii) delays in obtaining permits from local authorities and rights-of-way; (iii) difficult and limited access for pipe laying (including rocky strata), especially for sewerage works; (vi) lack of proper due diligence by program consultants during design preparation, which necessitated revisions in drawings; (v) procurement delays such as retendering of packages due to lack of technically qualified bids and high bid prices; (vi) an extended monsoon period in 2010; and (vii) unrealistic construction schedules limited to 18 months, which was found to be inadequate. As a consequence of these delays, the sewerage works and drinking water household connections were transferred to Tranche 4 resulting in a $9.3 million cancellation of the ADB loan, representing a 31% overall 17 At appraisal, the efficiencies were 68% for Raichur, 78% for Hospet, and 51% for Haveri. 18 GAP activities have helped to (i) increase the project implementation capacity of 119 counselors, including 29 women, by conducting three training programs; and (ii) build the gender capacity of 237 Karnataka slum clearance boards and ULB staff by providing training on gender-related aspects of the project.

9 7 reduction in planned expenditure. These factors raise the question of whether the project s readiness was sufficient at approval. 24. Disbursements were slow during the first 3 years, amounting only to $10.55 million (32% of total loan allocation and 45% of the total loan utilization). Reasons for the slow disbursement included (i) procurement delays arising from the re-tendering of packages resulting in delayed start of works; (ii) slow utilization of funds due to poor engineering design (necessitating design revisions), delays in preparing bidding documents, and poor contractor performance leading to delays in execution of works (PCR, paras ); and (iii) delays in timely submission of claims by contractors and subsequently increasing reimbursement requests. 25. The reevaluated economic analysis demonstrates economic internal rates of return (EIRRs) for the water supply subprojects of 26.43% for Raichur, 17.63% for Haveri, and 26.60% for Hospet. These EIRRs were lower than the appraisal values 46.28% for Raichur, 19.32% for Haveri, and 35.75% for Hospet as a result of cost and time overruns and a reduced coverage of project beneficiaries. The reevaluated EIRRs continue to be above the economic opportunity cost of capital (12%) and the subprojects are economically viable. It is difficult to rate the project on these EIRRs alone as they represent just one of the planned components and comprise only 14% of the planned program budget, with work on household connections, metering, and volumetric charging still to be completed. This validation also raises the issue of whether spending 24% of the project budget on a new building for the KUIDFC, which is not included in the DMF, was an efficient use of available funds, and whether it contributed to the overall expected outcome. Based on the above combination of process delays and limitations in the EIRRs provided, this validation assesses Tranche 1 less than efficient. D. Preliminary Assessment of Sustainability 26. In spite of achievements that contribute to the project s sustainability, the PCR rated the project less than likely sustainable due to concerns over operational sustainability. Water supply subproject designs have enhanced sustainability by augmenting water supply through the connection of seasonal and perennial water sources to meet supply shortfall during summer months. Outreach activities conducted under the CDP have increased awareness on the benefits of improved sanitation and hygiene and are expected to lead to an uptake in HSCs. Such commitment will be critical for the sustainability of the infrastructure that is being delivered under the MFF. 27. However, the financial internal rate of return (FIRR) for the water supply subprojects was negative during appraisal and this continues to be the case because of the poor water tariff collection efficiencies. 19 The low water tariff collection efficiency, coupled with low personnel capacity 20 at the ULB level, implies that ULBs currently lack sufficient capacity to adequately operate and maintain the assets and/or infrastructure that have been created. ULBs are also relying on external support from the KUIDFC and contractors through a defects liability period to meet their regular maintenance requirements. The KUIDFC has agreed to extend its full support through the stronger involvement of divisional offices until the operations and maintenance of the water supply subprojects under Tranche 4 are taken over by private contractors. However, this does not confirm sustainability, thus a project performance evaluation 19 If the effect of the devolution of funds from the state government to the ULBs is taken into account, then the FIRR shows positive results, thus, indicating that the government grants are helping to bridge the financial gap of meeting the regular operation and maintenance expenditures. 20 It is noted in the PCR that project ULBs have a small team of engineers (3 4), and only one is fully dedicated to water supply.

10 8 after the completion of the whole MFF would be useful to monitor sustainability and other aspects (para. 43). 28. The revenue base of the three ULBs has gone up significantly in nominal terms since fiscal year (FY) , with an annual compounded growth rate of 17%. However, the cash flow projections for revenue from water charges at appraisal could not be realized in FY , as the actual collection efficiency was low compared to the projections for all three ULBs. The property tax collection efficiency in FY ranged from 60% to 82% and can be further improved. The collection efficiency in water tariffs, which were in the range of 21% 32% for FY , can be substantially improved. In line with the State Finance Commission recommendations, 10% of the government s revenue receipts are shared with the ULBs. Overall, this validation assesses Tranche 1 less than likely sustainable. E. Impact 29. The PCR rated the project s impact significant. The project s intended impact was (i) the improvement of urban infrastructure and services, resulting in an overall improved quality of life in project ULBs and increase in economic opportunities; and (ii) growth in North Karnataka and reduction in imbalances between North Karnataka and the rest of the state. The impact indicators focused on economic growth. No indicators on quality of life were included in the DMF although increases in water supply in terms of higher liters per capita per day will undoubtedly contribute to improved quality of life. The per capita real income comparison between the assisted districts and the state average was selected as the impact indicator. Although the per capita incomes have increased in the assisted districts, it is not possible to identify how far this is attributable to the project interventions with the data available (net district domestic product at current prices). 30. A range of quality of life indicators could have been used including the reduction in the incidence of waterborne diseases. The project undertook a calculation of the poverty impact ratio, i.e., the proportion of project net benefits accruing to the poor for the three towns together and the findings could also have been used in the program s DMF. 21 Community development activities were highly successful, resulting in beneficial socioeconomic outcomes for participants, over 88% of whom were women. The CDP activities have promoted project receptivity and increased awareness on water, sanitation, and hygiene issues. These have contributed to an increased uptake of house connections (although less than 10% of originally envisaged connections), including among female-headed households by conducting project-related awareness campaigns. 31. The project was categorized as effective gender mainstreaming. The GAP evaluation presented in the PCR suggested that the GAP was not adequately designed, as it lacked baseline data and targets and used vague indicators and terminology. 22 It was also not aligned to the DMF, thus, making it difficult to assess the linkages of the GAP activities to the program outputs. 32. Although the achievements of the project are acknowledged and these will undoubtedly contribute to achieving the intended impact, there is not enough information available in the PCR or in the background documents to assess improvements in the quality of life in project ULBs and the related increase in economic opportunities and growth in North Karnataka, therefore, the reduced rating is considered more appropriate. Given the unfinished elements of 21 The poverty impact ratio for the three subprojects in all three towns ranged from 33% (Haveri) to 43% (Hospet). 22 GAP activities were conducted under the CDP component, financed under Tranche 2. The project GAP comprised 13 core activities spread across three focus areas (project design, project implementation and monitoring, and institutional development) and 11 objectives.

11 9 the water supply subprojects (household connections, volumetric water tariff, and metering) and the deferred sewerage works, the overall impact of the project is currently uncertain. Overall, this validation views the project impact less than satisfactory. 23 III. OTHER PERFORMANCE ASSESSMENTS A. Performance of the Borrower and Executing Agency 33. The PCR rated the performance of the borrower and executing agency (the KUIDFC) satisfactory. The borrower provided support to improve the performance of consultants and contractors. 24 The executing agency incorporated lessons learned from previous projects, particularly those relating to land acquisition 25 and to obtaining clearances in advance from other line departments. However, it appears that the performance of the implementing agencies was less than satisfactory due to a lack of capacity in project implementation and inadequate knowledge of contract management. In addition, frequent changes in key personnel affected project continuity and implementation progress. Overall, this validation assesses borrower and executing agency performance less than satisfactory. B. Performance of the Asian Development Bank 34. The PCR rated ADB performance satisfactory. Project preparation did not enjoy the benefit of using a DMF and problems were encountered in cost estimation. As the PCR noted, a higher degree of due diligence on the part of ADB could have helped to adequately foresee start-up complications and mitigate the effects of poor consultant and contractor performance. ADB, however, responded to the evident lack of capacity during implementation of the executing and implementing agencies by conducting seminars and trainings on the MFF modality and procurement and disbursement-related issues. 26 Throughout implementation and with the support of 14 supervisory missions, ADB also strongly pushed for better contract management and followed up on ULB financial reforms. Therefore, this validation considers the performance of ADB satisfactory. IV. OVERALL ASSESSMENT, LESSONS, AND RECOMMENDATIONS A. Overall Assessment and Ratings 35. Overall, the project was rated less than successful by the PCR and this validation s revised ratings do not lead to a change in the overall rating. Despite the significant reduction in scope and the issues with the DMF, the project remained relevant in addressing sector development constraints in a least-developed region in Karnataka. The project is less than effective because the envisaged outcome of Tranche 1 cannot be completely achieved with the deferment of some components to Tranche 4. A combination of process delays and the limitations in the EIRRs provided render the project less than efficient. The project is less likely to be sustainable as a result of poor capacity and poor tariff collection efficiencies at the ULB level. The 23 Beginning May 2016, IED adopts the ratings terminology of the April 2016 Guidelines for the Evaluation of Public Sector Operations on development impacts. In this guideline, a less than satisfactory rating coincides with the moderate rating the term used before April In particular, the KUIDFC organized contractors conferences to acquaint them with ADB guidelines and project details, such as land availability, status of clearances from line departments, and others. 25 An example is assessing land requirements before the detailed project report stage and getting land allotments before tenders are called. 26 Two seminars on ADB disbursement and procurement procedures were conducted (jointly with KUIDFC) in Hubballi and Ballari in February 2010 under ADB Technical Assistance to India for Knowledge Management and Capacity Building. Manila.

12 10 DMF was not available until 2011 and, even in 2011, was deficient in many aspects. The project design failed to adequately recognize risks, especially for failed contracts, poor contract management, and municipal capacity. Fortunately, the project has been able to leverage the MFF modality to deliver project outcomes across later tranches of the facility. Overall Ratings Criteria PCR IED Review Relevance Relevant Relevant Effectiveness in Less than Less than achieving outcome effective effective Efficiency in achieving Less than Less than outcome and outputs efficient efficient Preliminary assessment Less than likely Less than likely of sustainability sustainable sustainable Overall assessment Less than Less than successful successful Impact Significant Less than satisfactory Borrower and executing agency Satisfactory Less than satisfactory Reason for Disagreement and/or Comments Although the achievements of the project are acknowledged to contribute to achieving the intended impact, information available is not enough to assess improvements in the quality of life among ULBs and the related increase in economic opportunities and growth in North Karnataka (paras ). The project suffered from the implementing agencies lack of capacity in project implementation and inadequate knowledge of contract management. Frequent changes in key personnel affected project continuity and implementation progress (para. 33). Performance of ADB Satisfactory Satisfactory Quality of PCR Satisfactory Refer to para. 41. ADB = Asian Development Bank, DMF = design and monitoring framework, IED = Independent Evaluation Department, PCR = project completion report, ULB = urban local body. Note: This report uses the ratings terminology of the ADB Guidelines for the Evaluation of Public Sector Operations. Manila. Source: ADB Independent Evaluation Department. B. Lessons 36. This validation agrees with all the lessons identified in the PCR, particularly on the early commencement of community development activities. Additional consideration could be given to the following, which address particular problems encountered: (i) Cost estimation most of the reasons provided for the cost escalation were predictable, such as (i) higher bid premiums for civil works packages relative to engineering estimates, (ii) bidder concerns over timely payments (as the implementing agencies were small municipalities), (iii) difficulties in mobilizing resources for interior towns, and (iv) more thorough review of possible price risks was required. The frequent fluctuations in the rupee dollar exchange rates during the project design period ( ) may not have been so predictable but adequate contingency planning could mitigate this risk.

13 11 (ii) Frequent changes of key government personnel including program consultant specialists and project-level engineers affected project continuity. Thus, ways to retain key government personnel for the project implementation period and beyond needed to be agreed with the government while excessive consultant staff turnover could carry more stringent contract penalties. C. Recommendations for Follow-Up 37. This validation agrees with all the recommendations identified in the PCR. In addition, it stresses the need to prepare a project using the DMF as a tool for project design and for project monitoring. The adequacy of data sources specified in project DMFs could also be examined in PCRs, along with other data sources used or which could have been used. V. OTHER CONSIDERATIONS AND FOLLOW-UP A. Monitoring and Evaluation Design, Implementation, and Utilization 38. A DMF could have been prepared earlier than It should have been linked to the DMF for the entire MFF. The DMFs for each tranche should have a theoretical linkage with the overall DMF i.e., projects under individual tranches of MFF are monitored in the same way as stand-alone projects with DMFs tracking inputs, activities, outputs, results, and outcomes. These should show the details of the investment and the link between the tranches and the overall MFF. It is expected that the outputs in the overall DMF will be the tranches' outcomes. The Independent Evaluation Department notes that the ADB Operations Manual now states in the OM Section D14/BP that a design and monitoring framework (DMF) will be prepared for the entire MFF and for each tranche The program performance monitoring system was established after a delay of 4 months. A limited collection of project information (based on a review of documents provided by the KUIDFC dated May 2012) was compiled, updated, and used for quarterly progress reports. 40. A rich source of information was available during the GAP assessment. This, however, was an assessment made for the PCR and not intended for a monitoring system, which could have benefited project implementation. Although it was not available for the first tranche, it is noted that subsequent tranches have established a GAP monitoring system. B. Comments on Program Completion Report Quality 41. This validation finds the quality of the PCR satisfactory. The PCR is clearly written, consistent with the PCR guidelines, and provides adequate discussions on the key features of the project. Methods of recomputation of the EIRR and FIRR appear sound, although limited to drinking water supply subprojects. Lessons provided were adequate but could have addressed key project weaknesses, particularly in a project that experienced a number of significant problems. C. Data Sources for Validation 42. This validation is based on reviews of the PCR, the report and recommendation of the President, loan review mission reports, and the government s PCR. 27 ADB Multitranche Financing Facility. Operations Manual. OM Section D14/BP. Manila.

14 12 D. Recommendation for Independent Evaluation Department Follow-Up 43. It is recommended that this project is revisited during the preparation of the PCR for Tranche 4 so that a more meaningful evaluation that captures the deferred activities on sewerage and water supply connections can be carried out. Given that Tranche 4 is also the last tranche of this MFF, it may be more practical to carry out an evaluation on a PCR for the whole MFF. Similarly, a project performance evaluation report should only be carried out at least 2 years after the close of the MFF to allow the evaluation of outcomes.