Solomon Islands: Second Road Improvement (Sector) Project

Transcription

1 Validation Report Reference Number: PVR-477 Project Number: Grant Numbers: 0175, 0176, 0177, 0207, 0210 December 2016 Solomon Islands: Second Road Improvement (Sector) Project Independent Evaluation Department

2 ABBREVIATIONS ADB Asian Development Bank CPIU central project implementation unit DMF design and monitoring framework EIRR economic internal rate of return EU European Union km kilometer LBES labor-based, equipment-supported m meter MID Ministry of Infrastructure Development NTF National Transport Fund PCR project completion report PMCBU project management and capacity building unit RRP report and recommendation of the President SIRIP Solomon Islands Road Improvement (Sector) Project NOTE In this report, $ refers to US dollars. Key Words adb, asian development bank, lessons, performance evaluation, project completion report, road maintenance, road rehabilitation, solomon islands, validation The guidelines formally adopted by the Independent Evaluation Department (IED) on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of IED management, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. The final ratings are the ratings of IED and may or may not coincide with those originally proposed by the consultants engaged for this report. In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, IED does not intend to make any judgments as to the legal or other status of any territory or area.

3 PROJECT BASIC DATA Project Number PCR Circulation Date 18 Sep 2015 Grant Numbers 0175, 0176, 0177, 0207, 0210 PCR Validation Date Dec 2016 Project Name Second Road Improvement (Sector) Project Sector and subsector Transport Road transport (nonurban) Theme and subtheme Economic growth Widening access to markets and economic opportunities Safeguard categories Environment B Involuntary Resettlement B Indigenous Peoples C Country Solomon Islands Approved ($ million) Actual ($ million) ADB Financing ADF: (Grant Total Project Costs ($ million) 0175) OCR: 0.00 Grant 0175 (ADB) 0176 (European Union) 0177,0210 (Australia) 0207 (New Zealand) Borrower Beneficiaries Cofinancier Approval Date 0175/0176/ /0210 Signing Date /0207/0210 Project Officers IED Review Director European Union Australia New Zealand 12 Nov May Dec Nov Aug 2010 R. Adhar D. Ling P. Indrawansa Others Total Cofinancing Effectiveness Date /0207/0210 Closing Date /0177/0207/0210 Location PLCO ADB headquarters PLCO B. Nguyen, Officer-in-Charge, IED2 28 Feb Feb Oct Mar Mar 2012 From Nov 2009 July 2012 June Mar Dec Aug Sep Dec 2013 To Jul 2012 Jun 2013 Dec 2013 Team members S. Palle-Venkata, Evaluation Specialist, IED2 Y. Ono, Senior Evaluation Specialist, IED2 B. Prakash, Consultant J. Supangco, Consultant ADB = Asian Development Bank, ADF = Asian Development Fund, IED2 = Independent Evaluation Department, Division 2, OCR = ordinary capital resources, PLCO = Pacific Liaison and Coordination Office in Sydney, Australia; PCR = project completion report.

4 2 I. PROJECT DESCRIPTION A. Rationale 1. Six large islands, dozens of smaller islands, and hundreds of islets and atolls make up the 28,000 square kilometers of Solomon Islands. At the time of appraisal, more than 80% of the 560,000 total population was rural, living in widely dispersed villages of a few hundred persons each. Infrastructure and services, especially in the rural areas, were inadequate. The road network reached only about a quarter of the rural villages, of which about 60% were on Guadalcanal and Malaita. Sealed or graveled roads were confined to Guadalcanal and Malaita, which have the only significant inland roads and a segment of the secondary road in the Western province. The other six provinces had short coastal sections of gravel or earth roads near the provincial centers. Lack of maintenance has left provincial and secondary roads in the rural provinces badly deteriorated, no longer maintainable and urgently needing rehabilitation. 2. To achieve pro-poor economic growth and improve the social and health indicators in Solomon Islands, these call for a revitalized and decentralized rural economy. Poor roads constrained the movement of people and goods, and discouraged productive rural economic activity. Transport infrastructure and services needed improvement to provide reliable access to domestic and international markets for rural produce and commodities. The country was also vulnerable to periodic climate disasters including flooding. In emergencies, the country s roads require high levels of urgent maintenance and repairs to minimize disruptions and continue to serve the population. 3. The Asian Development Bank (ADB) had earlier implemented the Solomon Islands Road Improvement Project (SIRIP) 1 to improve the efficiency and effectiveness of the national road network. The Second Road Improvement (Sector) Project (SIRIP II) 2 was approved in 2009 as a sector project and was to run in parallel with SIRIP. B. Expected Impacts, Outcomes, and Outputs 4. The anticipated impact of the project was economic growth, social development, and reduction of poverty. The impact indicators were a 20% increase in job opportunities and a 20% increase in household incomes in the project area. The project completion report (PCR), however, indicated a targeted 25% increase in both indicators. The PCR also added 2015 as the target year for attaining the impacts. 5. The envisaged outcome of the project was improved road transport for economic and social activities. The outcome indicators were (i) subproject roads not passable by four-wheeldrive vehicles to be fewer than 4 days per year; (ii) 100% of subproject road length in fair or good condition; (iii) 25% reduction in travel time for a typical trip to markets, schools, and health facilities; and (iv) 25% increase in public transport services available in project areas. The report and recommendation of the President (RRP) did not provide in its design and monitoring framework (DMF) the target year for achieving the outcome indicators. 1 ADB Report and Recommendation of the President to the Board of Directors: Proposed Asian Development Fund Grant to the Solomon Islands for the Road Improvement (Sector) Project. Manila. (Grants 0048/0049/0050- SOL, approved on 8 August, cofinanced by ADB [$0.35 million], the Government of Australia [$6.10 million], and the Government of New Zealand [$9.75 million]). 2 ADB Report and Recommendation of the President: Proposed Asian Development Fund Grant and Administration of Grants for the Second Road Improvement (Sector) Project in Solomon Islands. Manila.

5 3 6. The expected project outputs include, first, an improved network of national, provincial, and secondary roads as shown in 20 kilometers (km) of roads rehabilitated and upgraded, and 30 major crossings repaired or reconstructed to climatic adaptability conditions. The second set of expected outputs is a sound and sustainable road maintenance policies and practices established as indicated by (i) rehabilitated roads maintained annually for the duration of the project, (ii) road maintenance policies established, (iii) road maintenance training provided, and (iv) private contracting system institutionalized. The third set of expected outputs is enhanced participation of communities and women in road maintenance as indicated by (i) at least 25% of maintenance contracts are with communities in the project area, (ii) at least 20% of road maintenance contractors are women, and (iii) contractors are encouraged to employ at least 33% of women in wage labor. C. Provision of Inputs 7. At appraisal, the estimated project cost was $24.0 million, exclusive of taxes and duties. The estimated civil works cost of $18.5 million comprised $16.8 million for road rehabilitation, $0.65 million for labor-based, equipment-supported (LBES) activities, and $1.0 million for road maintenance. Consulting services were estimated at $1.45 million inclusive of design and supervision, project management, and capacity building services. Contingencies were estimated at $4.1 million. The Solomon Islands Government requested a grant of $15.0 million from the Asian Development Fund of ADB to help finance the project costs. The Government of Australia, through the Pacific Region Infrastructure Facility, provided a grant cofinancing of $4.5 million while the European Union (EU) extended a grant cofinancing of $3.3 million equivalent. All cofinanced grants were to be administered by ADB. The government was to provide $1.2 million as counterpart funds. 8. At project completion, the total project cost reached $33.5 million, $9.5 million (39.7%) higher than the appraisal estimate. The PCR indicated that the increase in project cost was due to the (i) increase in road rehabilitation cost from $16.8 million to $28.8 million due to the appreciation of the New Zealand dollar against the US dollar, the impact of the 2010 floods on the scope of the West Guadalcanal subproject, and the expansion of the scope of the Malaita subproject to address climate adaptation needs; and (ii) increase in the cost of consulting services to $2.5 million due to the need to extend and provide additional consultancy services (PCR, para. 26). The project actually rehabilitated 63 km of roads as against 20 km targeted at appraisal and the repair or construction of 34 major crossings and 50 minor crossings also increased when compared to the appraisal target of only 30 major crossings. Three packages of additional financing increased Australia s contribution to $10.9 million (32.7% of the total project cost), while the government s contribution increased by $3.0 million equivalent for a total of $4.2 million (12.4% of the total project cost). 9. At appraisal, a total of 53 person-months of international and 202 person-months of national consulting services were envisaged. At completion, 85 person-months of international and 275 person-months of national consultants were utilized. This increase was attributed by the PCR to the provision of procurement, contract management, community development, and safeguards services, among others, which this validation largely finds to be due to the additional works outside the project s original scope. 10. The project was classified Category B for involuntary resettlement with less than 0.02 square km acquired for the project. In Makira, land was acquired for two bridges, while in West Guadalcanal, land was needed for the approach road to Sasa bridge which were donated by landowners in the communities following the required procedures. The North Malaita

6 4 subproject did not require any land acquisition. 11. The project was classified Category C for indigenous peoples based on the ADB Policy on Indigenous Peoples (1998) and as confirmed through the social assessments undertaken. 12. The project was classified Category B for environment. The project did not have any significant adverse environmental impacts that could not be mitigated or managed. The government approved, and ADB endorsed, the initial environmental examination for each subproject. The project management and capacity building unit (PMCBU) worked with the contractors to prepare the environmental management plans to ensure compliance with the initial environmental examination. The contractors' compliance was monitored and reported through quarterly and semi-annual safeguards monitoring reports. The PMCBU provided capacity building to the Ministry of Infrastructure Development (MID) and the Environment and Conservation Department staff so they could undertake the monitoring. D. Implementation Arrangements 13. Implementation arrangements for SIRIP II remained as envisaged at appraisal and were unchanged from those established under SIRIP. The executing agency was the MID and a project steering committee included representatives from ADB, the EU, and from the governments of Australia and New Zealand. The PMCBU, under MID s director of Transport Infrastructure Management Services, and the central project implementation unit (CPIU), were responsible for the day-to-day project implementation and reported to the Permanent Secretary of MID. The PMCBU was supported by various consultants, administrative staff, and accounting services. This validation notes that the institutional arrangements should have considered MID s lack of technical staff to be seconded to the PMCBU. At appraisal, this deficiency in MID s technical staff resources was recognized, but it was still expected to provide the required staff to the PCMBU. In fact, an ADB technical assistance report 3 had noted this same issue. National consultants filled positions that should have been occupied by MID staff and had seemed inappropriate as the national consultants benefited from the capacity building provided under the project. 14. The government and the executing agency substantially complied with 21 of the grant covenants. However, the auditing of the project accounts up to 31 December 2013 was still ongoing at PCR preparation due to the delay in reconciling and closing of project imprest accounts and the limited availability of the project accountants. 3 ADB Proposed Technical Assistance to the Solomon Islands for Preparing the Domestic Maritime Support Project and Technical Support Program (Financed by the Japan Special Fund). Manila. Lack of staff seconded to ADB projects has been noted. This appears to be due to the (i) level of priority assigned by Ministry of Infrastructure Development (MID) management in allocating training time for staff and time for carrying out regular duties; (ii) high ratio of donor-supported projects as against actual number of staff that could be assigned to projects for on-the-job training; and (iii) lack of detailed training plan that clearly states learning objectives, evaluation mechanism, and time frame to allow supervisor and the trainee to assess their progress while carrying out regular tasks. See MID Training Needs Assessment & Proposed MID Management, Transport Infrastructure Management Services (TIMS) and Corporate Services Training Plan Linking Training and Supervision Plan (TSP) and Transport Sector Development Project (TSDP) Training Requirement.

7 5 II. EVALUATION OF PERFORMANCE AND RATINGS A. Relevance of Design and Formulation 15. The PCR rated the project design relevant. It noted that the project design was based on the development needs identified by the government and did not require adjustments during implementation. The project s impact and outcomes remained relevant at completion as noted in the government s current National Development Strategy ( ) and supported by the Country Partnership Strategy for Solomon Islands, and Strategy 2020 of ADB. 16. The PCR further noted that the LBES was aligned with the available resource capacity in the country for creating business opportunities for small and medium national contractors, and was appropriate for the efficient and effective maintenance of transport assets in remote locations. The LBES supported gender and development goals and provided women with opportunities to acquire productive assets and improve economic empowerment in their communities. 17. This validation notes that the project supported the implementation of the government s National Transport Plan, and was aligned with the New Zealand/Solomon Islands Programme Strategy ( ). 5 Contrary to the PCR s claim that the project design did not require any adjustment during implementation, this validation notes the substantial additional outputs, 6 which were not originally included and necessitated changes in the project DMF as identified by the midterm review mission for the project. 18. The scope of the Malaita subproject was expanded by adding three bridges to address climate change adaptation needs and substantial additional outputs were added. Since this is a sector project, flexibility is available for subproject components. However, in return, the sector project needs to highlight sector policy reforms. The RRP proposed capacity building, community participation and LBES maintenance, and climate change adaptation for the policy reforms. However, it is not clear from the PCR how these policy reforms were introduced widely in the sector. 19. The project was fully aligned with national policies and ADB strategies at appraisal and at completion. Based on the above, this validation assesses the project relevant. 4 Solomon Islands Government, Ministry of Infrastructure and Development National Transport Plan, Honiara. 5 New Zealand/Solomon Islands Government New Zealand/Solomon Islands Programme Strategy ( ).Honiara. 6 The additional outputs included the following: (i) increase in the sealed length of the road from 10 km to 17 km in West Guadalcanal and the unsealed length from 10 km to 14 km; (ii) need to elevate North Malaita road by 200 millimeters to prevent the road from overtopping; (iii) increase in the number of bridges, culverts, and wet crossings rehabilitated with protection works from 30 to 44, including the construction of Poha Bridge; (iv) increase in the length of coastal protection works for North Malaita subproject from 0 meter (m) to I,800 m; (v) increase in the length of concrete surface on existing steep road sections at Malu in North Malaita from 0 m to 500 m; (vi) increase in river training and retardation walls under West Guadalcanal subproject from 0 m to 800 m; and (vii) inclusion of realigned and reshaped channels in the Tomba, Doma 2, and Kohimarama rivers in West Guadalcanal. Items (ii) (vii) are climate change adaptation measures. In addition, the target achievements for the impact indicators increased from 20% to 25% (RRP, Appendix 1 and PCR, Appendix 1).

8 6 B. Effectiveness in Achieving Project Outcomes and Outputs 20. The PCR rated the project effective as it achieved its planned outcomes. It noted that (i) the improved roads remained passable for all vehicles during floods; (ii) the number of nonpassable days decreased to only 0.2 days from 4 days per year in Makira and West Guadalcanal, and from 11 days per year in Malaita to only 0.2 days at project completion. The average travel speed increased from 25 km/hour before the project to 45 km/hour at completion. The average travel time to the nearest health clinic decreased from 1.1 hours before the project to 0.3 hours at completion a 70% decrease in travel time. 21. By mid-2012, commercial transport trips in the project area had increased by 71% compared with 2007 data. Travel time between Honiara and Selwyn College decreased from the 3-hour boat journey before the project to 1 hour by road at completion. Public transport improved with nine buses servicing 3 4 return trips per day. The cost of transport had decreased by as much as 50%. An ancillary benefit was the increase in enrollment at Selwyn College from 470 in 2013 to 560 in In the surrounding communities, there had been some increased market activity. 22. A survey was conducted in Malaita following the completion of civil works and showed that access to basic needs had improved as people have increased access to water tanks and private household water supply. 23. Under community participation and labor-intensive activities, employment opportunities increased. This also increased the number of national contractors as LBES capability was generated. National and community-based contractors were trained to undertake road maintenance, while women participated in greater numbers, with women contractors gaining skills through on-the-job training. A total of nine maintenance contractors were trained and a small-scale private sector contracting setup was established in Central, Isabel, and Western provinces. This validation notes that LBES contracts maintained 47 km of roads in these provinces annually. All nine LBES contracts were with contractors from local communities. Of the nine trained contractors, three were women who managed the jobs. A further 40% of wage jobs were filled by women. 24. This validation notes that planned outputs at appraisal were exceeded as more civil works were added during implementation. While about 30 km of road were targeted for rehabilitation and upgrading during appraisal, the project completed 63.6 km (105%). From 30 major crossings to be repaired or reconstructed to adapt to climate change at appraisal, 35 major crossings and 50 new crossroad pipe culverts were built to climate change adaptation standard. These additional outputs were included in the revised DMF of the PCR, along with the following completed works: (i) about 3,000 m of coastal protection works and 18 km of sealed roads; and (ii) about 1,200 m of river training and retardation walls, and reshaping and realigning of river channels. 25. This validation notes that the outcome indicators should have included traffic on roads, as the indicators provided in the DMF do not illustrate how the infrastructure is used. Apart from the traffic counts in Kakabona, the RRP and the PCR did not provide traffic estimates, making it difficult to assess whether the expected traffic was achieved. Nevertheless, since the increase in traffic in Kakabona was substantial (32%) and given that project outputs and outcomes were fully achieved, this validation assesses the project effective.

9 7 C. Efficiency of Resource Use in Achieving Outcomes and Outputs 26. The PCR rated the project efficient based on the economic reevaluation conducted. Three months after the completion of the Makira subproject, a recomputation of the economic internal rate of return (EIRR) was made, which showed an EIRR of 12.3% compared to the 11.25% at appraisal. For the West Guadalcanal subproject, the economic reevaluation showed an EIRR of 35.3% compared to the 25.0% at appraisal. For the North Malaita subproject, an EIRR of 18.7% was recomputed compared to the 11.9% at appraisal. The project was to be implemented in 2 years, but was extended by more than a year due to additional works at West Guadalcanal and North Malaita, and delays by the contractor for North Malaita. The PCR indicated that the recomputation of the EIRRs were done within a very short period after project completion and finds it too short a time for changes in vehicle traffic and road user and community behavior to be immediately visible. The PCR, thus, considered that the previous assumptions on traffic growth remain. The PCR also noted that the comprehensive traffic counting was not conducted at completion. Only one site, Kakabona, had a 2013 traffic count, while the other traffic count stations covered in the appraisal did not have updated traffic counts. These count stations are Tamboco, Aruligo, Visale, and Selwyn College. 27. The methodology used for the economic reevaluation was the same as at appraisal. For the PCR, the economic reevaluation analyzed the case with climate change and the case without climate change. The computation was made based on with-project and without-project scenarios. Vehicle operating costs and travel time savings are normally counted as the benefits for road rehabilitation projects. However, the PCR counted (i) travel time savings, (ii) reduction of lost income from road closures, (iii) reduction of lost cargo output from the region due to road closures, (iv) increased cargo output through reduced freight costs, and (v) health benefits associated with reductions in road closures. The benefits created by reducing road closures are unique. This approach is challenging but understandable given the situation that roads in Solomon Islands are largely affected by harsh weather. However, the methodology needs to be explained in more detail and needs to be developed in a more sophisticated way with open discussion. The PCR economic reevaluation did not consider project implementation delays. All capital costs should be split into years for construction, considering that each province took more than 2 years to implement (PCR, Appendix 3). 28. In terms of process efficiency, reaching an agreement with the EU on cofinancing arrangements took about 3 years (PCR, footnote 7) and resulted in the extension of project completion by 18 months. 29. The EIRR recalculation results largely depends on traffic, but traffic counting at completion was weak and future traffic estimates are not clear. The recalculated EIRR in the PCR is 12.3% in the without climate change case. Since sensitivity analysis was not conducted, this validation questions whether the EIRR exceeded the 12.0% threshold given the expected risks. In the with climate change case, the PCR recalculated the EIRR under the condition that opportunities for road closures will likely be more significant in the without-project scenario. Although the PCR should provide more explanation of the methodology, the EIRR may be higher than 12.0% in the with climate change case. Based on the above, this validation assesses the project efficient. D. Preliminary Assessment of Sustainability 30. The PCR rated the project most likely sustainable, noting that the project outputs were designed to maintainable standards and specifications using local materials and methods, and

10 8 with minimal adverse effects on the environment. It further noted that the assets rehabilitated and improved under the project, including previous investments, are being sustained through regular maintenance contracts managed by MID and implemented through CPIU. These maintenance contracts are financed through the government s recurrent budget and the National Transport Fund (NTF). Staff trained by PMCBU in supervision and monitoring of the contracts have been assigned to the CPIU. 31. The PCR noted that the performance of LBES contractors was satisfactory as the quality of their work improved with enhanced training. However, it also noted that further capacity improvements were needed for the national contractors. There was no detailed analysis of contractors capacity as it was considered beyond the scope of the PCR. It was unclear whether they had the technical knowhow, equipment, and financial capacity to undertake the required maintenance works. The PCR did not present evidence on local contractors capabilities and that of the local communities and their role in road maintenance. However, it was noted that LBES was initiated under SIRIP and was being implemented in other externally assisted projects and in MID s own recurrent maintenance programs. The design specifications for roads were such that they could be maintained using local materials and by methods available in Solomon Islands. However, financial sustainability is a major issue. 32. While the government s recurrent budget and NTF are sources of funds for road maintenance, the PCR did not evaluate the adequacy of these funding sources nor identified other sources of funding to ensure that the required routine and preventive road maintenance requirements are undertaken. The utilization of the road asset management system as it pertains to the programming and allocation of financial resources to road maintenance was not explained or linked. The risk that the required road maintenance requirements would not be undertaken is, therefore, high. The government already faced challenges with regards to (i) resolving the scarcity and unreliability of budget allocations to the sector, (ii) recognizing that maintenance has higher priority than new works, and (iii) developing asset management capability from planning to implementation (RRP, Appendix 3). ADB project documents subsequent to SIRIP II indicated that problems continue to persist in regard to the funds for maintenance, and capacity of the CPIU. Based on the foregoing discussions, this validation assesses the project less than likely sustainable. E. Impact 33. The PCR did not rate the project s impact. This validation notes the PCR s claim on the early signs of increased agricultural production and household incomes supported by the postconstruction surveys undertaken (PCR, Appendix 8). The assessment of project impacts and benefits of the Makira subproject was included under the SIRIP completion report and the results showed that (i) the number of households engaged in copra had increased from 46% to 53% and those involved in cocoa had increased from 43% to 60%, (ii) the income from these products had also increased 48% for copra and 88% for cocoa, and (iii) a switch from boat to road-based transport and more frequent trips to the market and community centers was confirmed. 34. For the West Guadalcanal subproject, the baseline survey sampled 139 households and 200 households for the post-construction. Results showed an increase in traffic levels and public transport compared to the baseline. The total household expenses on fuel and fares for truck or bus had increased, while boat fares and users had decreased. Also noted were (i) a reduction in the average time taken to reach specific facilities and services such as schools, health centers, hospital, local and main markets, police stations, and banks; and (ii) decreased

11 9 travel times using public transport, privately owned vehicles, or walking. More households were producing and selling vegetables, fruits, coffee and cocoa, livestock, poultry, and other products resulting in increased incomes. 35. For the Malaita North Road subproject, the baseline and post-construction survey samples were 165 and 185 households, respectively. Results showed that fares paid had increased, while travel time had decreased. People had started to travel by available road transport, although many still walked. The shift in transport preference was affected by the road condition as rated by the respondents. The result showed that 95% of households were producing and selling products regularly to earn income. The percentage of households earning regular cash income before the subproject was 19.4% and 22.1% after the project. The average income earned had increased between 21.7% and 37.6%. Results showed that 53.8% of shops along the road section had started operation after the subproject. 36. Since this is a sector project, the PCR needs to assess how the project made impacts on the sector. However, overall, this validation considers project impact satisfactory. 7 III. OTHER PERFORMANCE ASSESSMENTS A. Performance of the Borrower and Executing Agency 37. The PCR rated the performance of the borrower and executing agency satisfactory. The government was committed to the project and complied with most of the grant covenants. Its financial share increased from $1.2 million to $4.2 million or 12.4% of total project cost. The MID played a vital role in project implementation and ensured continuous consultation and disclosure of information to affected persons by implementing a project communication plan and establishing a formal grievance redressal committee for effective community representation. While the MID was unable to mobilize the required number of technical staff to be seconded to the project (para. 13), it assigned six of its engineering staff to PMCBU for on-the-job training in project planning, feasibility studies, engineering designs, procurement, and contract supervision and management. Based on the above, this validation assesses the performance of the borrower and the executing agency satisfactory. B. Performance of the Asian Development Bank 38. The PCR rated the performance of ADB satisfactory. ADB carried out a fact-finding mission, an inception mission, a midterm review mission, and 7 grant review missions. ADB processed the EU financing 8 and the additional cofinancing provided by Australia and New Zealand expeditiously to proceed with emergency flood recovery works. ADB managed the administration of several grants received at various times during implementation. The in-country presence of ADB through its development coordination office provided close liaison with the executing agency and its development partners. In-country representatives of Australia, New Zealand, and the EU participated in all joint supervision missions and were available to provide support to MID and the PMCBU during implementation. This validation assesses ADB performance satisfactory. 7 Beginning May 2016, IED adopts the ratings terminology of the April 2016 Guidelines for the Evaluation of Public Sector Operations on development impacts. In this terminology, a satisfactory rating coincides with the significant rating that was used before. 8 See

12 10 IV. OVERALL ASSESSMENT, LESSONS, AND RECOMMENDATIONS A. Overall Assessment and Ratings 39. The PCR rated the project successful. This validation finds the project relevant. It was effective as it achieved its expected outputs and outcomes. It was economically viable, therefore, it was rated efficient. This validation assesses project sustainability less than likely sustainable because there is a high risk that the required road maintenance works would not be undertaken. Overall, this validation assesses the project successful. Overall Ratings Criteria PCR IED Review Relevance Relevant Relevant Effectiveness in Effective Effective achieving outcome Efficiency in achieving Efficient Efficient outcome and outputs Preliminary assessment Most likely Less than likely of sustainability sustainable sustainable Reason for Disagreement and/or Comments The overall situation on whether government allocation for road maintenance was sufficient or not was not fully analyzed. The capacity of local contractors in road maintenance was not evaluated although MID was using the modality used by other externally assisted projects and that of its own projects (paras ). Overall assessment Successful Successful Impact Not rated Satisfactory Borrower and executing Satisfactory Satisfactory agency Performance of ADB Satisfactory Satisfactory Quality of PCR Less than satisfactory Refer to para. 45. ADB = Asian Development Bank, IED = Independent Evaluation Department, MID = Ministry of Infrastructure Development, PCMBU = project management and capacity building unit, PCR = project completion report. Note: This report uses the ratings terminology of the April 2016 Guidelines for the Evaluation of Public Sector Operations. Source: ADB Independent Evaluation Department. B. Lessons 40. The PCR identified five lessons from the project. From this validation s viewpoint, an important lesson would be, at the outset, a thorough evaluation of the executing agency capacity s, especially in providing the required counterpart staff. This is to maximize the knowledge and skills transfer from the consultants to the national staff, and to enable the national staff to manage future projects and maintain existing infrastructures. Under the project, MID was unable to mobilize the required number of counterpart staff resulting in the national consultants assuming their roles in the project.

13 A second lesson is that the financing requirements for the project should have been firmed up from the beginning considering that there were other cofinanciers. It took some time before the EU actually signed the contribution agreement with ADB and this was identified by the midterm review mission as one issue for the project. While it was anticipated that the EU would sign the agreement by June 2011, it was signed only in This caused delays in the implementation of specific subprojects to be funded out of the EU contribution. C. Recommendations for Follow-Up 42. Aside from the recommendations given under the PCR, this validation further recommends that ADB closely monitors the performance of the private contractors and local communities participating in the preventive and routine maintenance of the road projects. This should also include determining, on a regular basis, the sufficiency of government financing for road maintenance. 43. This project included noteworthy value additions including (i) climate change adaptation investments, and (ii) a community engaged in LBES maintenance. Since this experience could be valuable for future projects, outcomes of the value additions could be analyzed in the future. Since EIRR calculations including the with-climate change case will be relevant for future projects with climate change adaptation components, the methodology could be further developed. V. OTHER CONSIDERATIONS AND FOLLOW-UP A. Monitoring and Evaluation Design, Implementation, and Utilization 44. The PCR referred to the baseline surveys and used data from these surveys to measure project achievement of outcomes and impacts. However, it was noted that the DMF did not mention the target years for which impacts and outcomes were to be achieved. The DMF itself was revised during implementation as additional outputs were included. It was not explicitly stated whether the project had a fully functional project performance management system, although there were extensive baseline and post-construction survey data available and used in the PCR. B. Comments on Project Completion Report Quality 45. This validation rated the PCR quality less than satisfactory. The PCR could have provided more discussions on the ratings for the evaluation criteria by utilizing the substantial data given in the appendices. The economic reevaluation (i) lacked reliability of traffic counting and future estimates of traffic since full traffic counts were not conducted due to resource limitations, (ii) lacked in-depth explanation of the methodology used in the estimation of benefits especially for health benefits, and (iii) did not consider the actual period in the disbursement of investment costs. The PCR did not analyze comprehensively the sufficiency of government financial resources from the NTF in relation to the operation and maintenance budget for roads, and the technical capacities to determine whether the road projects could be maintained sustainably. C. Data Sources for Validation 46. Data sources for this validation included the (i) PCR and RRPs for the Second Road Improvement (Sector) Project (Solomon Islands) (Grants 0175, 0176, 0177, 0207, and 0210)

14 12 and SIRIP; (ii) country partnership strategies; (iii) National Transport Plan of the Solomon Islands; (iv) National Development Strategy ( ) for the Solomon Islands; and (v) Solomon Islands: Country Partnership Strategy ( ), Solomon Islands: Interim Country Partnership Strategy ( ), Solomon Islands: Country Strategy and Program ( ), Solomon Islands: Country Operations Business Plan ( ), and the ADB Approach to Assisting the Pacific ( ). D. Recommendation for Independent Evaluation Department Follow-Up 47. This validation recommends that a project performance evaluation report be prepared after 2018 to allow sufficient time for project impacts to be visible and measurable. This evaluation concurs with the PCR recommendation that the availability of project data should be validated as the grant covenants required the recipient to submit annual monitoring and evaluation survey reports from 2015 to 2017.