Public Disclosure Authorized Independent Evaluation Group (IEG) 1. Project Data Report Number : ICRR0020001 Public Disclosure Authorized Project ID P100580 Country Ukraine Project Name ROADS & SAFETY IMPROVEMENT Practice Area(Lead) Transport & ICT L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-76770 31-Dec-2012 500,000,000.00 Bank Approval Date 07-Apr-2009 Closing Date (Actual) 30-Nov-2014 IBRD/IDA (USD) Grants (USD) Public Disclosure Authorized Public Disclosure Authorized Original Commitment 400,000,000.00 0.00 Revised Commitment 380,351,122.80 0.00 Actual 380,351,122.80 0.00 Sector(s) Rural and Inter-Urban Roads and Highways(99%):Public administration- Transportation(1%) Theme(s) Trade facilitation and market access(46%):infrastructure services for private sector development(46%):injuries and non-communicable diseases(8%) Prepared by Reviewed by ICR Review Coordinator Group Fang Xu Victoria Alexeeva Christopher David Nelson IEGPS1 2. Project Objectives and Components a. Objectives The project development objective as stated in the project Loan Agreement is to improve the condition and quality of sections of the M-03 Road, and to increase safety on roads ( Loan Agreement page 5 and Project Appraisal Document page 5). b. Were the project objectives/key associated outcome targets revised during implementation?
No c. Components Road Rehabilitation of about 120 kilometer section (Borispil-Lubny) of the Kyiv-Kharkiv-Dovzhansky highway (M-03) (estimated cost US$298.5 million, actual cost: US$344 million). This component supported the rehabilitation/ strengthening of about 120 km section of the M- 03 highway, including safety measures for the vehicles (i.e. signaling, lighting in critical sections, crash barriers, etc.,) and for local inhabitants (i.e. pedestrian crossings, parking, and special facilities). Road safety improvement (estimated cost US$99.0 million, actual cost: US$33.6 million). The component was to finance about 110 black spots on public roads including safety measures for vehicles and local inhabitants such as vertical and horizontal signaling, reflectors (cats eyes), rumble strips, and modern crash barriers on the most critical points on public roads. Capacity Building (estimated cost US$1.5 million, actual cost: US$1.4 million). This component would finance the advisory services, training and equipment to enhance road management according to international practice. The project was restructured four times: The first restructuring occurred on July 19, 2011. The changes included reassigning resources from component 2 to component 1 to finance additional 20km road rehabilitation and reduced the number of black spot to be improved from 110 to 25 as a result of the lower than expected interest from the construction industry to execute the road safety black spot works under the project Component 2. The targets of some intermediate outcome indicators were changed as well at this restructuring. The second restructuring was approved on October 24, 2012 to extend the project closing date to December 31, 2013 to enable the completion of civil works awarded after the first restructuring. The third restructuring was on December 18, 2013 to extend the closing date from December 31, 2013 to September 30, 2014. The main objective of this restructuring was to allow for the completion of the 32 km long rehabilitation and upgrading contract on the M03 road under the Project s first component as the progress of this contract did not advance as expected because of the financial solvency issues of the contractor. The fourth restructuring was on September 29, 2014 to extend the closing date to November 30, 2014 to finalize the execution of the road rehabilitation and upgrading contract on M-03. The main reason for the last two extension was the bankrupt of a contractor for this specific civil work on M-03. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost: the original estimated project cost was $500 million. The total actual cost was US$80 million less than planned mainly due to the reduction in the number of black spots to be improved under component 2 (US$19.6 million), the cost saving on road rehabilitation work under component 1, and about $60 million less Value Added Tax payment from the Government. Financing: At closing, US$ 19.6 million remain undisbursed. There was an outstanding balance of US$ 176,360.40 on the Designated Account. The grace period for payments was extended to allow for additional disbursements Borrower Contribution: The original contribution from the borrower was US$100 million for project activities related Value Added Tax. The actual contribution from the borrower was US$41.3 million. Dates: The project was extended for a total of 23 months, from January 1, 2013 to November 30, 2014. 3. Relevance of Objectives & Design a. Relevance of Objectives This Road Rehabilitation and Safety project was the Bank s first project in the transport sector in Ukraine. The project development objectives were relevant to the country s circumstances and development priorities at appraisal stage. The Country Partnership Strategy (CPS) for Ukraine (FY07-11), proposed a two-pillar framework of support with the first pillar aiming at improving Ukraine s competitiveness through investments in public sector infrastructure (in particular transport and energy efficiency). The PDO remains closely aligned with the CPS for the Ukraine for FY12-16, which has one of its outcomes as improved road connectivity and safety. The Government had set the objective of halving the accident rate by 2010 as part of their quest to achieve EU levels of road safety. It is worth to mention that the Second Road Rehabilitation and Safety had the same development objectives as the first one. These two projects would provide continuous contribution to the country priority. Original Rating
High b. Relevance of Design The project s intervention had infrastructure improvement in a key corridor and public roads with special focus on incorporating safety features into the civil works, which were highly relevant to the project development objective of improving road condition and quality and increasing road safety. The activities supported by the third component focused on road assets management including mitigating the risks associated to lack of maintenance, these activities were not directly associated to the achievement of the PDO, but would contribute to the sustainability of the project development outcomes. It is noted both by this review and by the ICR that institutional strengthening measures, road safety regulations and enforcement and road user behavior change are critical for road safety improvement, however, the project s investment in these areas was very limited. The ICR (page 7) explained that the importance of engaging the borrower in those areas were well noted by the project team at preparation stage, but at that time, the Government was reluctant to use loan proceeds for institutional capacity activities and the implementing agency was not mandated to implement road safety campaigns to address high risk road user behavior. With such a design, the project s contribution to the road safety improvement at the national level would be limited and positive road safety results may not be sustainable. These limitations were being addressed by the following projects by including more activities on institutional strengthening and road user behavior change. Original Rating Substantial 4. Achievement of Objectives (Efficacy) PHREVISEDTBL Objective 1 Original Objective Improved condition and quality of sections along the M-03 highway. Original Narrative Main outputs under the project included: 150km road section on M-03 highway was rehabilitated with good quality, this was 30 km more than originally planned. 11 as against originally planned 110 black spots were improved. Bidding Documents for Pilot Output Performance Road Contract were prepared and successfully bid. Outcome: The road rehabilitation and upgrade of M-03 highway from a two-lane road (one way each direction) into a four lane carriageway for most of the alignment with higher standard directly led to the improved condition and quality of these sections along the M-03 highway. The road condition measured by the International Road Roughness Index (IRI) on these road sections was reduced to 1.16 at project closure from 5 or more prior to the project. Original Rating Substantial PHREVISEDTBL Objective 2 Original Objective Increase road safety. Original Narrative The road upgrade on the M-03 highway directly contributed to the reduced number of crashes between Boryspil and Lubny, from 248 per year before the project to 10 per year at project closing. The International Road Assessment carried out as part of the activities of the road safety component, showed that the works performed under the road safety component (component 2) were of good standard and had a positive impact in the improvement of safety in the roads. While the project intervention had positive impact on road safety improvement at the project area, it has limited impact on road safety improvement at national level because of very little project investment on road safety
policy, regulation, enforcement and no intervention on road users behavior change. The statistics of the World Health Organization showed a sharp decrease in the number of fatalities per 100,000 in population in Ukraine between 2007 and 2009. The annual fatalities per 10,000 vehicles also declined to 1 at national level at project closing. But those were more a result of the adoption of more stringent EU regulations rather than the result of the project intervention. Original Rating Modest 5. Efficiency At project appraisal stage, cost benefit analysis was carried out for the road rehabilitation component, which was about 75% of the project cost. The main benefits were the saving of vehicle operating cost and the saving from improved road safety on M-03 highway. The US$298.5 million investment yielded an Economic Internal Rate of Return 22%. The savings in vehicle operating costs contributing about 50 percent of total benefits, and those from the avoidance of crashes another 40 percent (PAD, Annex 9). At project closing, ex-post cost benefit analysis was conducted for the same road rehabilitation component which comprised about 82.5% of the actual total project cost. As the unit cost for the road rehabilitation works was lower, the traffic volume was slightly higher and the road condition as measured by IRI was better than the original estimate, the actual EIRR of the road rehabilitation and upgrade intervention was 28%, which is higher than what was estimated at the appraisal stage (22 percent). This review noted that the actual unit cost for black spot elimination was much higher than the original estimated ( $33 million for 11 black spots elimination vs. $99 million for 110 black spots elimination). The team clarified that the actual intervention on the corridor with black spots was more comprehensive after the project restructuring and that may explain why the unit cost seems to be much higher. The project ERR is satisfactory; in terms of cost effectiveness, a number of activities/ outputs were not completed as planned under Component 2 by project closure. There was also a 23-month delay in project completion. The overall project efficiency is assessed as substantial Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Appraisal 22.00 ICR Estimate 28.00 75.00 Not Applicable 82.50 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project objectives were highly relevant to the country priority and the implementation of the project activities were intended to directly lead to the achievement of the first sub-objective though the project s design could not ensure the achievement of the second sub-objective of improving the road safety at the national level. At project closing, the first sub-objective was substantially achieved and the road safety objective was modestly achieved. The project efficiency was overall substantial. The project outcome is assessed as being moderately satisfactory because of the moderate shortcomings with the project efficacy. Despite a Level 2 restructuring that included minor changes to the key associated outcome targets, IEG has chosen not to undertake a Split evaluation as the changes did not make a material difference to the achievement of the project against the PDO.
a. Outcome Rating 7. Rationale for Risk to Development Outcome Rating The main risk to the development outcome is the insufficient funding for road maintenance. The ICR (page 20) commended that the road maintenance was mostly carried out in house which usually was more costly. With the project s assistance, the first performance based contract for maintenance was awarded and is expected to be the first step for improved management of maintenance. The ICR also stated that the Government of Ukraine was contemplating advancing the decentralization agenda including introducing private participation in the road sector to increase funding sources for routine and periodic maintenance, and improve the quality assurance processes. If successfully implemented, these initiatives would contribute to the improved road maintenance and sustainable development outcome. However, at current moment, the road maintenance remained to be an issue as there was little progress from the government side in assuring enough resources for the operation and maintenance of the road sector especially when the country is now faced with serious economic and security challenges as its conflict with Russia remains unresolved. The overall risk to development outcome is assessed as being high a. Risk to Development Outcome Rating High 8. Assessment of Bank Performance a. Quality-at-Entry The project was prepared based on strong and comprehensive analytical work on Ukraine transport sector carried out during 2005 and 2006 (ICR page 5). At project preparation stage, the Bank team secured a grant of USD$700,000 to finance the capacity building of Ukravtodor as the implementing agency and the preparation of project civil work activity related documents such as feasibility studies, initial environmental and social assessments, engineering, design, and bidding documents. These helped ensure the project quality and the readiness for implementation. The quality at entry could have been improved in three areas: 1) the insufficiency of the institutional capacity component which was noted at the preparation stage, but the client was reluctant to borrow to finance advisory services. As such, the project s contribution to the road safety improvement at the national level was limited;2) The second sub-objective of improving road safety at the national level rather than at the project area was beyond the reach of the project given the limited amount of the project resources and the geographical coverage of the project, the project development objective could have been better phrased. 3) There were some shortcomings with the M&E design (more details refer to the M&E section). Quality-at-Entry Rating b. Quality of supervision The Bank team regularly carried out supervision missions twice a year to visit the project site to check the quality of the civil works and to work with the clients in person to resolve the issues that the project had encountered. The team was responsive to clients requests and engaged with the client during the project implementation period. The ICR (page 22) noted that after the second restructuring, in 2012, the Bank s supervision of the project progress was affected by the political situation in the Eastern and Southern part of the country. The bank team could have done a better job by updating the project development objectives and the project M&E framework when it turned out that this project would have limited impact on the road safety improvement at the national level especially when the black spot treatment component was reduced after the first restructuring. Quality of Supervision Rating Overall Bank Performance Rating
9. Assessment of Borrower Performance a. Government Performance The ICR (page 22) noted that the Government was committed to the project throughout the project cycle despite the political and economic challenges that the country experienced during the project implementation. The government s commitment was demonstrated by its compliance with all legal covenants and ensuring the compliance with the Bank safeguards policy. However, the Government was inconsistent in providing the counterpart funds for the Value Add Tax (VAT) payment; the late payment of VAT compounded with the contractors financial and cash flow issues negatively affected the project progress. Ultimately, the project was extended several times in order to allow for enough time to complete all works under the project. Government Performance Rating b. Implementing Agency Performance The implementing agency refers to Ukravtodor, the single implementing entity responsible for the project implementation. The implementing agency managed to deliver the large amount of civil works with good quality. The specific issues (ICR, page 22) that the agency had in implementing the project included: slow processing and late submission of documentation to the Bank for action which affected the implementation progress, examples included a number of no-objections request to the Bank after actions were taken and very late requests for extension of closing dates; low interest in implementing the soft component such as capacity building activities; and delays in the civil works implementation. Implementing Agency Performance Rating Overall Borrower Performance Rating 10. M&E Design, Implementation, & Utilization a. M&E Design The M&E design included two PDO indicators to measure the achievement of two project development objectives. The M&E framework also had intermediate outcome indicators and output indicators to measure the project s achievement and progress. At outcome level, the indicator on road safety (fatalities per 10,000 vehicles) was defined at national level rather than at the project area was too broad for the project intervention. The IRI is a good indicator for measuring the road condition, but the M&E framework could be improved by including more indicators to measure the improved road quality and condition. b. M&E Implementation The ICR (page 13) noted the implementing agency submitted detailed, periodic implementation reports, however, those reports were more output orientated by focusing on the project outputs and progress of contracts execution and expenditures than outcome orientated by monitoring progress toward the project s end results. The reliability of the road accident statistics was also a concern especially when the information was not regularly collected. c. M&E Utilization Since, the M&E information was more output focus, the utilization of M&E to monitor and assess the project outcome was limited. M&E Quality Rating Modest
11. Other Issues a. Safeguards Environment: The Project was an Environmental Category B (partial Assessment) Project, the World Bank Environment Assessment Policy OP/BP 4.01 was triggered. At project preparation stage, the Environmental Management Plan (EMP) was prepared which contained summaries of the mitigation plan. A detailed monitoring program to validate the effectiveness of the mitigating measures was included in the EMP. A corridor-specific Environmental Assessment, documenting baseline conditions and environmental risks was prepared and two Environmental Impact Assessment were developed based on Ukrainian legislation covering two regions in Ukraine (Kyiv and Poltava Regions). In addition, stakeholders consultation was conducted and the suggestions from the local communities were incorporated into the EMP ( PAD page 20). The mitigation measures specified in the EMP together with the suggestions from the local communities were reflected in the bidding documents and engineering design for the project civil works. The ICR (page 14) reported that throughout the project, Environmental Assessment ratings were highly satisfactory (ICR page 14). Social: The road rehabilitation and upgrade of road section on M-03 highway did not involve involuntary resettlement, the black spot elimination would involve land acquisition and resettlement. The Bank Involuntary Resettlement Policy OP/BP 4.12 was triggered. At project preparation stage, the number of people to be affected was not clear, hence, a draft Land Acquisition and Resettlement Framework (LARF) was prepared by Ukravtodor and was made public. The ICR (page 14) reported that when the actual civil works started, given the significantly reduced scope of work performed under the black spot elimination activity, minor relocation of businesses around the project area was satisfactorily performed in accordance with the RPF and specific Resettlement Action Plan. The ICR (page 14) reported that the involuntary resettlement has been rated Highly Satisfactory throughout most of the life of the project. b. Fiduciary Compliance Procurement: The project procurement was managed by the Project Implementation Unit with a small unit staffed with procurement specialists. The project procurement was mainly carried out through International Competitive Bids, leading to cost savings which allowed more civil works to be performed under the project. There were occasions that the preparation of procurement related documentation was slow, in some cases no-objections were requested after activities were carried out. Nevertheless, the project procurement management was rated as satisfactory throughout the project. No mis-procurement was reported by the ICR (page 15). Financial Management: The Financial Management (FM) Assessment completed in December 2008 concluded that the project FM capacity was sufficiently developed and the project satisfied the minimal financial requirements. At least nine measures were put in place to mitigate the risk of misuse of Bank funds (PAD, page 16). Proper training was provided to the relevant staff, and proper internal controls were applied for the project. During project implementation, the financial reports and audits were presented to the Bank on time and with an acceptable quality. However, there were delays in the implementation of an automated accounting system. The implementing agency, Ukrdorinvest, eventually, adopted an automated accounting system. Throughout the project implementation period, the project financial management was rated as or above. No ineligible expenditure was reported by the ICR(page 15) c. Unintended impacts (Positive or Negative) N/A d. Other --- 12. Ratings Ratings ICR IEG Outcome --- Risk to Development Outcome Substantial High Reason for Disagreements/Comment The high economic and security challenges that the country is now faced with as a result of the on-
going military conflict with Russia. Bank Performance --- Borrower Performance --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The main lessons generated by the ICR generated are summarized below: 1. An integrated road safety approach would improve the road safety more effectively. Investments in infrastructure together with substantial measures to strengthen institutions and enforcement as well as change road user behaviors would be more effective in improving the road safety than the infrastructure investment only. 2. The Bank should consider financing 100 percent of project cost where the country macroeconomic risk was identified as high or substantial and the government budget capacity was limited, so as to avoid the situation that the counterpart funding could not be provided in a timely manner, resulting in delays in project implementation. 3. Consultations with stakeholders during the design phase could contribute to the improved design of civil works and the compliance with safeguard policies. For this specific project, consultation with the local communities was carried out both at the engineering design stage and during the implementation stage, the feedbacks from the local communities were reflected into the engineering design, by doing so, the design was improved and the compliance with safeguard policies was enhanced. 14. Assessment Recommended? No 15. Comments on Quality of ICR This ICR was well written. It is concise with clear story line. The ICR was prepared with deep understanding of the sectoral issue and good project evaluation knowledge. This ICR could benefit from providing more information on the project development objective of improved condition and quality on the section of M-03 highway besides providing the International Roughness Index information. By doing so, the achievement of the project development objective could be more substantiated. a. Quality of ICR Rating Substantial