Validation Report October 2017

Transcription

1 Validation Report October 2017 Kazakhstan: CAREC Transport Corridor I (Zhambyl Oblast Section) [Western Europe Western People's Republic of China International Transit Corridor] Investment Program (Tranche 3) Reference Number: PVR500 Project Number: Loan Numbers: 2697 and 8251

2 ABBREVIATIONS ADB Asian Development Bank CAREC Central Asia Regional Economic Cooperation COR Committee of Roads CPS country partnership strategy CSC construction supervision consultant DMF EIRR GDP ha design and monitoring framework economic internal rate of return gross domestic product hectare JICA Japan International Cooperation Agency JSC jointstock company km kilometer MFF multitranche financing facility MID Ministry of Investment and Development MOTC PCR PFRR PMC PPMS Ministry of Transport and Communications project completion report periodic financing request report project management consultant project performance monitoring system NOTE In this report, $ refers to US dollars. Director General Deputy Director General Director Team Leader M. TaylorDormond, Independent Evaluation Department (IED) V. SalzeLozac h, IED N. Subramaniam, Sector and Project Division, IED M. Gatti, Principal Evaluation Specialist, Thematic and Country Division, IED The guidelines formally adopted by the Independent Evaluation Department (IED) on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of IED management, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. The final ratings are the ratings of IED and may or may not coincide with those originally proposed by the consultant engaged for this report. In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, IED does not intend to make any judgments as to the legal or other status of any territory or area.

3 PROJECT BASIC DATA Project Number PCR Circulation Date 15 Sep 2016 Loan Numbers 2697 and 8251 PCR Validation Date Oct 2017 Project Name CAREC Transport Corridor I (Zhambyl Oblast Section) (Western Europe Western People's Republic of China International Transit Corridor) Investment Program (Tranche 3) Sector and Transport Road transport (nonurban) Subsector Strategic Agenda Inclusive economic growth Safeguard Environment A Categories Involuntary Resettlement B Indigenous Peoples C Country Kazakhstan Approved ($ million) Actual ($ million) ADB Financing ($ million) ADF: 0.00 Total Project Costs OCR: Loan Borrower Beneficiaries Others Cofinancier JICA Total Cofinancing Approval Date 15 Nov 2010 Effectiveness Date 13 Feb Jun 2011 Signing Date 15 Dec 2010 Closing Date 31 Dec Apr 2015 Project Officers IED Review Director Team Leader E. Kwon S. Mitra R. Idei Z. Wu A. Galiev Location ADB headquarters ADB headquarters ADB headquarters ADB headquarters ADB headquarters N. Subramaniam, Director, IESP M. Gatti, Principal Evaluation Specialist, IETC From Nov 2010 Oct 2011 May 2012 Mar 2013 Jan 2014 To Oct 2011 May 2012 Mar 2013 Jan 2014 Apr 2015 ADB = Asian Development Bank, ADF = Asian Development Fund, CAREC = Central Asia Regional Economic Cooperation, IED = Independent Evaluation Department, IESP = Sector and Project Division, IETC = Thematic and Country Division, JICA = Japan International Cooperation Agency, OCR = ordinary capital resources, PCR = project completion report. *Team members: L. Hauck (Senior Evaluation Specialist), F. De Guzman (Senior Evaluation Officer), L. Supangco (Consultant). A. Rationale I. PROJECT DESCRIPTION 1. In 2010, data showed that Kazakhstan s economy had grown rapidly for several years, with the country s annual gross domestic product (GDP) averaging over 10% annually during From 2008, however, GDP growth dipped below 10% per annum as the effects of external shocks on resource prices and the international financial crisis affected the economy. Despite this dip, Kazakhstan s economic growth was expected to result in a corresponding increase in road freight transport (in tonkilometers) traffic by 10% per year and in the number of vehicles by 5% per year. 2. Kazakhstan s vast territory, low population density, mineral resources scattered across the country, and its strategic location between Europe and Asia results in high dependency on

4 2 the transport system, with the roads as a key factor. In the late 2000s, much of the road network was in poor condition, with about 60% of the republican (national) roads requiring major rehabilitation and proper maintenance. The feeder road network that served the rural population was also not fully developed and in poor condition, hence, providing poor service. The road sector had longstanding operational and institutional bottlenecks. Revenues from transit were low and crossborder procedures were inefficient. Road sector planning was weak, affecting investments and project development. Management shortcomings created inefficiencies, high costs, and bad governance. 3. A transport sector strategy was developed to improve and expand the network over the short to medium term. 1 The strategy outlined the objectives, quantitative and qualitative targets, and operating principles. The underlying theme was to make the road system larger, better, more accessible, more affordable, more efficient, safer, and more environmentally friendly with greater integration between different modes, better cost recovery, and more outsourcing to the private sector. To implement the transport strategy, the Government of Kazakhstan sought financial assistance from the Asian Development Bank (ADB) and other development partners. 4. In November 2008, the ADB Board of Directors approved a $700 million multitranche financing facility (MFF) loan to support a $1.48 billion transport corridor construction and rehabilitation. 2 The ADB s portion of the corridor would be cofinanced with the Islamic Development Bank with $414 million, the Japan International Cooperation Agency (JICA) with $150 million, and the government with $216 million. These will contribute to the completion of the Western Europe Western People s Republic of China International Transit Corridor, which had an estimated total cost of $6.7 billion (including the $1.48 billion works supported by ADB). The European Bank for Reconstruction and Development, the World Bank, and the private sector were each financing sections of the 2,715 kilometers (km) of road to complete the corridor by the end of Given the breadth and scope of the investment requirements and capacity constraints, the MFF modality was considered more appropriate than the traditional investment project modality. It also provided flexibility to the government as it could easily modify the amount and timing of ADB contributions under each tranche. Financing could be linked with technical, safeguards, legal, and other projectreadiness filters and allow ADB to enter into a mediumterm partnership with the authorities and its cofinancing partners, and in the process create stability and credibility at the level of the financing plan. All conditions required for the use of an MFF were in place: strategic context, road map, policy framework, investment program, and financing plan. In November 2010, a periodic financing request report (PFRR) was approved for tranche 3 of the MFF 3 a $173 million loan from ADB to finance 118 km of road along the Zhambyl Oblast Section 1 Decree of the President of the Republic of Kazakhstan Transport Sector Strategy of the Republic of Kazakhstan up to Astana. 2 ADB Report and Recommendation of the President to the Board of Directors: Proposed Multitranche Financing Facility and Administration of Loan to the Republic of Kazakhstan for the CAREC Transport Corridor I (Zhambyl Oblast Section) [Western Europe Western People s Republic of China International Transit Corridor] Investment Program. Manila. 3 Prior to Tranche 3, in October 2009, a PFRR was approved for Tranche 2 of the MFF, which was completed and evaluated. Relevant documents include (i) ADB Periodic Finance Request Report: CAREC Transport Corridor I (Zhambyl Oblast Section) [Western Europe Western People s Republic of China International Transit Corridor] Investment Program Proposed Tranche 2 in Kazakhstan. Manila (MFF 0024KAZ); (ii) ADB Completion Report: CAREC Transport Corridor I (Zhambyl Oblast Section) [Western Europe Western People s Republic of China International Transit Corridor] Investment Program (Tranche 2) in Kazakhstan. Manila (Loan 2562); and (iii) IED Validation Report: CAREC Transport Corridor I (Zhambyl Oblast Section) [Western Europe Western People s Republic of China International Transit Corridor] Investment Program (Tranche 2) in Kazakhstan. Manila.

5 3 of the Central Asia Regional Economic Cooperation (CAREC) transport corridor. 4 Other cofinanciers included the government ($40 million) and JICA ($68 million). Activities under the third phase of the MFF were expected to cost $281 million. In September 2016, a project completion report (PCR) was prepared for tranche 3 of the MFF, which is the subject of this validation. 5 B. Expected Impacts, Outcomes, and Outputs 5. The expected impact of both the MFF and tranche 3 was sustainable economic development. At appraisal, the MFF had three impact indicators (GDP growth, increased value of export/import trade, and transport and distribution sector outputs), whereas the Tranche 3 PFRR had only two (GDP growth and export/import growth). Achievement of these indicators is discussed under the section on Preliminary Assessment of Development Impact (paras ). 6. The expected outcome of both the MFF and tranche 3 was the development of an efficient transport network in Zhambyl Oblast. However, the MFF s report and recommendation of the President (RRP) included an additional expected outcome of improved capacity for transport sector institutions. The design and monitoring framework (DMF) of the tranche 3 PFRR listed five indicators, while the RRP s DMF of the MFF had 10 indicators. The following were the four common outcome indicators of the two DMFs: (i) freight transport cost of the cargo value reduced from 10% to 5%, (ii) average traffic volume increased from 4,000 to 7,000 vehicles per day, (iii) average travel time between the southern Kazakhstan border and Otar through Zhambyl Oblast reduced from 10 hours to 6 hours, and (iv) road accident rate reduced from 0.3 fatalities to 0.1 fatalities per km. The fifth outcome indicator in the PFRR s DMF was to have benefited 8,000 people in the project area. 7. The expected output of the project was to have reconstructed 118 km of road sections of the CAREC Transport Corridor 1 in Zhambyl Oblast by This comprised (i) the Otar Blagoveshchenka section with the construction of a new 80 km road with oneway, twolane asphalt concrete pavement, and a fourlane embankment at km ; (ii) the Korday Karasu section with the upgrading of the existing 10.3 km twolane asphalt concrete road, and the construction of a new 7.4 km twolane asphalt concrete bypass road. A third output the Kulan Bypass (km ) section is the JICAfunded component of the project, and was to be administered by ADB. Under this output, the existing 5.2 km road was to be upgraded to a fourlane cement concrete road, and a new 14.8 km fourlane cement concrete bypass road was to be constructed. Achievement in these indicators is discussed in the section on Effectiveness in Achieving Program Outcomes and Outputs (paras ). C. Provision of Inputs 8. Tranche 3 was approved on 15 November 2010, with loan effectiveness planned for 13 February 2011, and loan closing planned for 31 December Actual loan effectiveness was on 15 June 2011, after two extensions on the effectivity date. The delay was due to the Parliament s lengthy procedures in ratifying the loan agreement. 4 ADB Periodic Finance Request Report: CAREC Transport Corridor I (Zhambyl Oblast Section) [Western Europe Western People s Republic of China International Transit Corridor] Investment Program Proposed Tranche 3 in Kazakhstan. Manila (MFF 0024KAZ). 5 ADB Completion Report: CAREC Transport Corridor I (Zhambyl Oblast Section) [Western Europe Western People s Republic of China International Transit Corridor] Investment Program (Tranche 3) in Kazakhstan. Manila.

6 4 9. At appraisal, the total project cost was estimated at $281 million $ million (86.5%) for civil works, $6.87 million (2.4%) for consulting services, and $31.10 million (11.1%) for contingency. ADB was to provide $173 million to finance the 97.7 km road section and consultancy for construction supervision services. JICA was to provide 6.36 billion (equivalent to $68.33 million) to finance a 20 km road section to be administered by ADB under the Accelerated Cofinancing Framework Agreement. The government will provide $40 million as counterpart funding. 10. At completion, the actual total project cost was $ million or 4.2% lower than expected. Actual ADB financing was $ million or 4.8% lower than estimated at appraisal, while the actual JICA financing was $68.33 million, which coincided with the loan amount. The actual government counterpart funding was $36.31 million or 9.2% less than estimated. The lower project cost at completion was due to the lower cost of construction supervision consultancy services and nonutilization of contingency costs. 11. At appraisal, it was estimated that 108 personmonths of international and 370 personmonths of national consulting services were required for the construction supervision of civil works on road sections financed by both ADB and JICA. However, the PCR did not provide the actual number of international and national consultants mobilized for the project at completion. D. Implementation Arrangements 12. At appraisal, the Ministry of Transport and Communications (MOTC) was to be designated executing agency, with the Committee of Roads (COR) as implementing agency. MOTC was to establish a team to manage the MFF program, to be headed by a project director; assisted by a team of MOTC staff in the areas of engineering, finance, legal, and procurement; and supported by the project management consultant (PMC) and construction supervision consultant teams. 13. The institutional arrangement during implementation was as envisioned at appraisal. The government reorganized its ministries in August 2014, replacing the MOTC with the Ministry of Investment and Development, which absorbed the functions of the MOTC. These institutional changes did not affect the implementation arrangements as the COR remained the implementing agency until project completion. The COR s deputy chairman was the Tranche 3 director and was assisted by selected COR staff with expertise in engineering, finance, legal, and procurement and the Zhambyl Oblast Road Department as COR s representative in the field sites. The Zhol (road) laboratories monitored laboratory test results, while the PMC assisted in managing the project. The construction supervision consultant (CSC) administered contracts and supervised the works progress, quality, and timeliness. 14. The KazAvtoZhol jointstock company (JSC) provided technical support to the COR and was involved during project implementation in By 2014, its role was limited to monitoring safeguards compliance. Subsequently, the Zhol (road) laboratory was the COR s representative in the field. The government was reorganized on 6 August 2014 and established the Ministry of Investment and Development (MID), which absorbed MOTC s functions. The COR, however, remained as the implementing agency until project completion. 15. This validation notes that a community liaison group was created to implement the grievance address mechanism for the investment program as a whole, including the project. The group s coordinator recorded 50 projectlevel complaints from local residents during implementation. These included environmental issues, resettlement issues not identified under the land acquisition and resettlement plan, and worksrelated issues (e.g., crossings for livestock

7 5 and agricultural machinery, sidewalks, and access roads). The mechanism was effective in resolving complaints and providing an avenue for interaction, and in coordinating the collaboration among stakeholders to resolve issues. All issues were addressed, with the PMC s grievance database helpful in monitoring progress toward resolving complaints. 16. This validation finds the implementation arrangements suitable and built on the expertise and experience gained from implementing previous ADB and foreignfunded projects. Even with the government s reorganization, the implementation arrangements were kept intact to ensure smooth project implementation. 17. Of the 52 loan covenants, most were complied with. Two were partly complied with and one was not complied with. One covenant that was partly complied with was due to the failure to mobilize the CSC prior to commencing the works. The other covenant involved CSC s failure to submit the project performance monitoring system (PPMS) report. Until the CSC was mobilized, a representative from Zhambyl Oblast COR served as the interim engineer. The project performance audit was the loan covenant that not complied with. However, the regular review missions, quarterly and monthly monitoring reports, annual financial audit reports, and eops project performance indicators helped to monitor project performance in the absence of this audit. II. EVALUATION OF PERFORMANCE AND RATINGS A. Relevance of Design and Formulation 18. The PCR rated the project relevant as it addressed the poor road network coverage, poor quality of road assets, impassable roads especially during winter, and road safety issues, among others. It noted that the project was aligned with the government s transport sector strategies, which prioritized the improvement of the republic road network. It also indicated that the project was consistent with the ADB Strategy 2020, Country Partnership Strategy (CPS), and the CAREC program. 19. The PCR noted that the project design was the same from appraisal to completion. Variations in the works during implementation included additional guardrails, sheltered bus stops, and crossings for cattle and agricultural machinery. This validation considers the design changes as minor and these were the government s response to community needs that surfaced in the community consultation meetings held during implementation. 20. This validation notes that the CAREC Corridor 1 Europe East Asia fulfills the east west traffic function, it runs from the People s Republic of China westward to Europe, and is a priority investment project under the CAREC Transport and Trade Facilitation Strategy. The project s DMF did not provide for an impact or outcome indicator that estimated crossborder traffic along this section of CAREC Corridor 1, and was focused primarily on the impact on Kazakhstan s international trade. Even without stating it, the project is an important link in crossborder trade among the countries linked by CAREC Corridor Given that the project is aligned and consistent with Kazakhstan s transport strategy, ADB s Strategy 2020, CPS, CAREC program, and with the relevance of the project design, this validation rates the project relevant. B. Effectiveness in Achieving Project Outcomes and Outputs 22. The PCR assessed the project effective. With the outcomes, the project fully achieved four

8 6 of the five envisaged outcome indicators of (i) increased average traffic volume, (ii) reduced travel time, (iii) improved road safety, and (iv) number of project beneficiaries. For the first indicator, the annual average daily traffic for the project roads reached 7,212 vehicles per day in 2015, which is above the 7,000 vehicles per day threshold set in the RRP. For the second indicator, travel time between Otar Blagoveschenka Kulan decreased from 10 to 6 hours, which translates to a 37% reduction in travel time cost for the three road sections. For the third indicator, the PCR noted that 2015 police data showed that road crashes declined by 50% from 0.3 fatality/km to 0.16 fatality/km by For the fourth achieved indicator, over 20,000 road users benefited from the roads, although it was unclear whether these beneficiaries were residents of the project area as targeted in the DMF. 23. The achievement of the fifth outcome indicator is more difficult to ascertain as the PCR did not use the same unit of measurement as the RRP. Whereas the RRP indicated that freight transport costs should be reduced from 10% to 5% of cargo volume, the PCR indicated that savings for freight vehicles in terms of vehicle operating costs per kilometer was at least 16% lower than at baseline. The PCR, however, candidly indicated in several places such data discrepancy problem. It even mentioned measurability and data availability of indicators in its section on lessons. 24. On outputs, the PCR noted that km of roads were completed and complied with the targeted international roughness index of less than 3. The completed roads resulted in a reduced distance of 20 km between Otar to Blagoveschenka, and a 2.5 km reduction between Korday to the Kyrgyz Republic border. 25. The project was classified under environmental category A. The MOTC prepared an environmental impact assessment that complied with ADB requirements under the Safeguard Policy Statement (2009). The project was classified as involuntary resettlement category B since civil works will require additional land for rightofway needs. A total of hectares (ha) of permanent land were required, of which ha were agricultural and 3.26 ha were commercial. For temporary use as base camps, ha of land were required as storage areas and for other construction activities. The government allocated T295 million from the state budget to compensate 107 verified affected people for lost land and structures 98 of these were households and nine legal entities and provided with relocation allowances. 26. The project was classified under category C for indigenous peoples as it did not affect anyone that fitted ADB s definition of indigenous peoples. 27. Based on the above discussions and noting that all outputs and the majority of outcomes were achieved, this validation rates the project effective. C. Efficiency of Resource Use 28. The PCR rated the project efficient. The ADB loan closing date was extended by a year to complete minor, unanticipated works variations. This did not affect project cost as the loan amount was sufficient to complete the project and had, in fact, generated savings for the ADB loan. All intended project outputs were achieved, and with the completed outputs, envisaged outcomes were achieved (para. 24). 29. The PCR recalculated the project s economic internal rate of return (EIRR) using the same methodology as at appraisal. The recalculated EIRR was 23.3% for the three road sections as a whole, which is higher than the 16.6% estimated at appraisal for the whole project. The EIRR for

9 7 the Kulan Bypass was 13.5%, 45.0% for the Korday Karasu section, and 25.1% for the Otar Blagoveschenka section. All the recalculated EIRRs exceeded the 12% hurdle rate for economic viability. This could be attributed to the higher vehicle volume and lower construction costs. The sensitivity analysis results showed the project to be viable for all scenarios considered. 30. One drawback of the PCR was that it did not differentiate between domestic and international vehicle traffic. The disaggregation of vehicle origins and destinations would have provided a clear understanding of traffic demand sources, more meaningful than just the aggregated traffic data. As noted, this was not a result of the PCR mission missing on available data, but due to the disaggregated data not being available at the time of the mission. Even with this lack of disaggregation, this validation finds the traffic forecast to be reasonable. 31. Since project costs were lower, this validation concurs with the PCR that it would have a positive effect on the recalculated EIRR. There was a 1year delay in project implementation, but the cost component of the delay was minimal, since according to the PCR, only minor works were left for completion. 32. Based on these discussions, with project outputs achieved at a lower cost, and the higher vehicle traffic forecasts, this validation assesses the project efficient. D. Preliminary Assessment of Sustainability 33. The PCR rated the project likely sustainable. The PCR noted several positive institutional and structural features of the project, including (i) strong state ownership of the project road; (ii) KazAvtoZhol JSC, which has a legal mandate to operate and maintain the roads and with qualified staff with technical expertise on road development, construction, and maintenance; (iii) roads designed for axleload of 13 tons, which is higher than Kazakhstan s 10 tons per axle regulated limit; (iv) the existence of load control posts or devices in Almaty and Aktai, Kazylorda, and Merke which will extend the service life of the roads; and (v) the existence of seven road maintenance depots at Korday, Merke, and Otar near the project site and others at Aksuyek, Akyrtobe, Shu, and Taraz in Zhambyl Oblast. 34. On road maintenance, the PCR noted three the following positive aspects: (i) the positive experience with the tolled 211 km (Astana Schshuchinsk) shows that the toll collected is sufficient to finance the maintenance of this road; (ii) sections of the CAREC Corridor are planned to be toll roads; and (iii) for the nontolled roads, the government continues to provide the budget for road maintenance following its definition of demand for road maintenance and repair as per established norms in Kazakhstan However, the PCR noted that state budget for road maintenance was still likely to be needed to complement the toll revenues, if toll rates do not provide sufficient revenues to maintain project roads. Moreover, the PCR did not provide a detailed financial evaluation of road maintenance funding in Kazakhstan to determine if financial resources with and without the tolling option will be sufficient. The business plan of KazAvtoZhol JSC should have been analyzed to determine the validity of its toll revenue assumptions. 36. Based on these, particularly on the issues affecting the funding for road maintenance, this validation assesses the project less than likely sustainable. 6 Footnote 39 of the PCR mentions government allocation for road maintenance during The KazAvtoZhol JSC and the COR have a state order agreement for a budget allocation for road maintenance.

10 8 III. OTHER PERFORMANCE ASSESSMENTS A. Preliminary Assessment of Development Impact 37. The PCR assessed project impact significant based on the project s initial contribution to regional cooperation and to the local economy. The improved roads are expected to stimulate trade between Kazakhstan and its neighbors. The PCR indicated that the project road section to Karasu, the truck parking area near the border gate, and the lifting of customs controls at all Kazakhstan Kyrgyz Republic border crossing points are likely to increase border traffic. On socioeconomic impact, the PCR pointed out that local government authorities and residents confirmed that the project roads improved travel convenience, access to markets, social service facilities, business areas, and employment opportunities and income. Cafés, small shops, and fuel stations at both ends of the Kulan Bypass had increased employment opportunities and incomes. By 2020, Kazakhstan s GDP is expected to grow by 60% from the 2010 baseline and its export and import to increase by 30%, also from The PCR noted that the project provided employment opportunities and income to local residents, directly employing 1,821 local personnel and workers. Construction aggregates and supplies were procured locally, generating employment and income in the local market. The project also generated rental income from the temporary use of land for base camps, storage areas, and other construction activities. 39. The PCR s impact assessment could have attempted measuring the project s impact performance indicators as set out in the DMF. Analyses should have determined whether Kazakhstan s GDP, and export and import growths are trending toward achieving the 2020 targets. Data for these performance indicators should be available from government institutional sources. Given that expected project impacts under tranches 1, 2, and 3 are the same, the attribution of GDP, and import and export growth by tranche would not be possible. Although the facility completion report is where this subject would be treated in detail, initial indications of whether the impact indicators were on track and would be met should have been reported in the PCR. 40. The PCR s narrative on development impacts (PCR, paras ) was mainly qualitative and little quantitative data was presented. Citing local government and resident interviews are insufficient to prove that the project achieved the desired impacts, especially when the number of respondents was not even indicated. While the project has the potential to generate significant impact, this still needs quantitative information for validation. 41. The PCR described the presumed impacts of the project, but presented insufficient data to support its assessment. Given this lack of evidence, this validation has not rated project impact. B. Performance of the Borrower and Executing Agency 42. The performances of the MOTC and COR were assessed satisfactory by the PCR. This was in spite of the lengthy government procedures to declare the loan effective, thus, requiring two extensions of the loan effectivity date. The government reorganization in August 2014, which absorbed the MOTC into the MID, did not affect project implementation as the COR kept its role as implementing agency. Most loan covenants were complied with, particularly on ensuring that counterpart financing was adequate and available when required. The COR had implemented a financial management system with separate records for the ADB and JICA loans.

11 9 43. The COR implemented the project in accordance with ADB guidelines on safeguards, anticorruption, disbursements, procurement of works, and recruitment of consultants. It has had extensive experience implementing ADBfinanced projects and other projects with international financing along CAREC Corridor 1. The executing and implementing agencies maintained regular communications with the ADB project team, which facilitated implementation and enhanced project management. The COR chaired the community liaison group, which was responsible for the project s grievance redress mechanism. 44. Given its relationship and long experience in implementing ADBfinanced projects, including tranches 1 and 2, the COR performed well and implemented the project at costs lower than estimated without sacrificing the quality of the roads infrastructure. This validation assesses the performance of the executing and implementing agencies satisfactory. C. Performance of the Asian Development Bank and Cofinanciers 45. The PCR rated the performance of ADB satisfactory. ADB had worked effectively with the Ministry of Finance and MOTC in strengthening project readiness through assistance in preparing the due diligence reports on safeguards and economic analysis. During project implementation, ADB provided timely support to MOTC/MID and COR by providing guidance in resolving implementation issues. ADB promptly responded to requests for reallocation of loan proceeds and in the review of contract variations to avoid delays, including the timely extension of the loan closing date. It provided coaching to COR staff on procurement, consultant recruitment, project administration, contract administration, disclosure, and disbursements. ADB monitored implementation progress and resolution of issues through 16 review missions and site visits. Communication and coordination among ADB, MID, and COR were considered effective. 46. At project preparation, implementation, and completion, ADB and JICA coordinated effectively. As administrator of the JICA loan, ADB carried out project appraisal, procurement and disbursement review, and other aspects of project administration. At the same time, ADB provided regular updates to JICA on project progress, reallocation of loan categories, and contract variations. ADB reviewed and endorsed claims for the JICA loan component, and facilitated the efficient release of JICA payments. 47. Based on these, this validation finds ADB performance satisfactory. IV. OVERALL ASSESSMENT, LESSONS, AND RECOMMENDATIONS A. Overall Assessment and Ratings 48. The PCR and this validation rated the project relevant, effective, and efficient. Both the PCR and this validation assess the project relevant since it was aligned with Kazakhstan s transport strategy, the ADB Strategy 2020, CPS, and the CAREC program. The project was well designed except for the declared project impact, which was the same for all three tranches. This may subsequently cause impact attribution issues during the conduct of postproject evaluation on the three tranches. This validation and the PCR concur that the project was effective as all outputs and outcomes were achieved. The PCR, however, did not use the same performance measurement unit for reduced transport cost for freight, which resulted in comparability issues. The PCR and this validation rate the project efficient. In spite of the 1year delay in implementation, the project was completed below estimated cost and maintained its economic feasibility. This validation assesses the project less than likely sustainable a notch lower than the PCR s rating of likely sustainable. This assessment is mainly due to the issues on how to fund

12 10 the road maintenance, which does not appear to be fully assured. Having to recourse to the state budget for road maintenance appears likely to be needed to complement the envisaged toll revenues. 49. The PCR and this validation both assess the project successful. B. Lessons 50. This validation agrees with the four lessons identified by the PCR. However, this validation notes that some of these lessons regarded unforeseen events as inherent to any project. These included consultant turnover and associated delays, and improving bordercrossing infrastructure and facilities which are usually included in similar projects but not in this case. Since minor changes in design are considered unavoidable, this should be provided for in project contingencies. In addition to these identified lessons, this validation finds two additional lessons to improve project sustainability: (i) undertake a more detailed financial evaluation of road maintenance funding to determine if financial resources are sufficient with and without the tolling option, and (ii) the business plan of the operating and maintenance company should be better analyzed to determine the validity of toll revenue assumptions. C. Recommendations for FollowUp 51. This validation finds the first PCR recommendation on covenants appropriate, but does not agree with the second recommendation for IED to undertake a project performance evaluation report. It also recommends an ADB followup of the implementation of the PPMS, especially the data collection for measuring and updating project performance indicators on a periodic basis. The data should be disaggregated for each of the tranches to avoid confusion during postevaluation. On the socioeconomic impacts, these are to be measured through surveys, although this was not stated in the DMF. The executing and implementing agencies should see to it that data collection under the PPMS is sustained even after project completion. Overall Ratings Validation Criteria PCR IED Review Reason for Disagreement and/or Comments Relevance Relevant Relevant Effectiveness Effective Effective Efficiency Efficient Efficient Sustainability Likely to be sustainable Less than likely sustainable Overall assessment Successful Successful Preliminary assessment of impact There are still unresolved issues surrounding the funding for road maintenance. Significant Not rated The PCR described the presumed impacts of the project, but presented insufficient data to support its assessment. Borrower and Satisfactory Satisfactory executing agency Performance of ADB Satisfactory Satisfactory Quality of PCR Satisfactory See para. 53. ADB = Asian Development Bank, IED = Independent Evaluation Department, PCR = project completion report. Note: From May 2012, IED views the PCR rating terminology of "partly" or "less" as equivalent to "less than" and uses this terminology for its own rating categories to improve clarity. Source: ADB Independent Evaluation Department.

13 11 A. Monitoring and Reporting V. OTHER CONSIDERATIONS AND FOLLOWUP 52. The COR complied with the loan covenant on monitoring and reporting. The COR monitored and evaluated project performance through the PPMS. Project performance monitoring reports consisted of monthly and quarterly progress reports prepared by the PMC and the CSC, safeguards (environment and land acquisition and resettlement) monitoring reports, and annual financial audit reports. The progress reports included updates on road and work safety. The PCR presented the project achievements in detail. B. Comments on Project Completion Report Quality 53. This validation finds the PCR quality satisfactory, with the project data presented in a clear and candid manner. Despite this positive assessment, an area that could have been better treated was the recalculation of the EIRR. The PCR should have been consistent in the unit of measurement used for the outcome performance indicators, rather than use a new unit. C. Data Sources for Validation 54. The data sources used in this validation include the RRP, PCR, backtooffice reports of loan review missions, CPS, Kazakhstan s Transport Strategy, and the CAREC Transport and Trade Facilitation Strategy D. Recommendation for Independent Evaluation Department FollowUp 55. No followup is required.