IMPLEMENTATION COMPLETION AND RESULTS REPORT (TF ) ON A GRANT IN THE AMOUNT OF (US$16.0 MILLION) UNDER THE TO THE ISLAMIC REPUBLIC OF PAKISTAN

Transcription

1 Public Disclosure Authorized Document of The World Bank Report No: ICR Public Disclosure Authorized IMPLEMENTATION COMPLETION AND RESULTS REPORT (TF ) ON A GRANT IN THE AMOUNT OF (US$16.0 MILLION) UNDER THE Public Disclosure Authorized MDTF FATA EMERGENCY RURAL ROADS PROJECT (ERRP) TO THE ISLAMIC REPUBLIC OF PAKISTAN FOR A MDTF EMERGENCY ROADS RECOVERY PROJECT Public Disclosure Authorized Transport and ICT Global Practice South Asia Region May 18, 2016

2 CURRENCY EQUIVALENTS (Exchange Rate Effective October 1, 2012) Currency Unit = Pakistan Rupees (PKR) PKR 1.00 = US$94.91 US$ = PKR 1.00 FISCAL YEAR July 1 June 30 ABBREVIATIONS AND ACRONYMS ACS ADB B/C Ratio BP CE CPR CPS CQS DA DFID DNA DPD EA EIRR EMP EPA ESSAF ESFP ESMPs FATA FBS FIDIC FIU FODP FS FY GEF GoB GoKP GOP GPS Additional Chief Secretary Asian Development Bank Benefit Cost Ratio Bank Procedure Chief Engineer Contraceptive Prevalence Rate Country Partnership Strategy Selection Based on Consultants Qualifications Designated Account Department For International Development Damage Needs Assessment Deputy Project Director Environmental Assessment Economic Internal Rate of Return Environmental Management Plan Environmental Protection Agency Environmental and Social Screening and Assessment Framework Environmental and Social Focal Point Environmental and Social Management Plans Federally Administered Tribal Areas Fixed Budget Selection International Federation of Consulting Engineers Field Implementation Unit Friends of Democratic Pakistan Financing Strategy Fiscal Year Global Environment Facility Government of Balochistan Government of Khyber-Pakhtunkhwa Government of Pakistan Global Positioning System i

3 GRC Grievance Redressal Committee HQ Headquarters IA Implementing Agency IBRD International Bank for Reconstruction and Development ICB International Competitive Bidding ICR Implementation Completion Report IDA International Development Association IDP Internally Displaced Persons IFC International Finance Corporation IFRs Interim Financial Reports Km Kilometer KP Khyber-Pakhtunkhwa LCS Least Cost Selection M&E Monitoring and Evaluation MDTF Multi Donor Trust Fund NCB National Competitive Bidding NDMA National Disaster Management Authority NER Net Enrollment Rate NGO Non-Governmental Organization NPV Net Present Value OP Operational Policy ORAF Operational Risk Assessment Framework PAP Project Affected Persons PaRRSA Provincial Rehabilitation, Reconstruction and Settlement Authority PC Planning Commission PC-1 Planning Commission Performa 1 PCNA Post Conflict Needs Assessment PD Project Director PDO Project Development Objective PMU Project Management Unit QCBS Quality and Cost Based Selection QCS Quality Control Specialist RP Resettlement Plan RE Resident Engineer RVP Regional Vice President SA Social Assessment SAR South Asia Region SDO Sub Divisional Officer SPP Simplified Procurement Plan SSS Single Source Selection TOR Terms of Reference UN United Nations WB World Bank W&SD Works and Services Department XEN Executive Engineer ii

4 Senior Global Practice Director: Practice Manager: Project Team Leader: ICR Team Leader: Pierre Guislain Karla Gonzalez Carvajal Zafar Iqbal Raja Rakhi Basu iii

5

6 PAKISTAN Federally Administered Tribal Areas Emergency Rural Roads Project (FATA ERRP) Table of Contents A. Basic Information... v B. Key Dates... v C. Ratings Summary... v D. Sector and Theme Codes... vi E. Bank Staff... vi F. Results Framework Analysis... vii G. Ratings of Project Performance in ISRs... viii H. Restructuring (if any)... viii 1. Project Context, Development Objectives, and Design Key Factors Affecting Implementation and Outcomes Assessment of Outcomes Assessment of Risk to Development Outcome Assessment of Bank and Borrower Performance Lessons Learned Comments on Issues Raised by Borrower/Implementing Agencies/Partners Annex 1. Project Costs and Financing Annex 2. Outputs by Component Annex 3. Economic and Financial Analysis Annex 4. Grant Preparation and Implementation Support/Supervision Processes Annex 5. Beneficiary Survey Results Annex 6. Stakeholder Workshop Report and Results Annex 7. Summary of Borrower s ICR and/or Comments on Draft ICR Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Annex 9. List of Supporting Documents Annex 10. MAP iv

7 DATA SHEET A. Basic Information Country: Pakistan Project Name: FEDERALLY ADMINISTERED TRIBAL AREAS EMERGENCY RURAL ROADS PROJECT (FATA ERRP) Project ID: P Grant Number(s): TF PK ICR Date: ICR Type: Core ICR Lending Instrument: Grant Borrower: The Government of Pakistan Original Total Commitment: US$16 million Disbursed Amount: US$16 million Revised Amount: Environmental Category: B Implementing Agencies: FATA Works & Services Department Cofinanciers and Other External Partners: B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 08/16/2012 Effectiveness 12/10/ /10/2012 Appraisal: 09/06/2012 Approval: 11/19/2012 Midterm Review: 01/25/2015 November 2015 Closing: 6/30/ /12/2015 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Borrower Performance: Moderately Satisfactory High Moderately Satisfactory Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately satisfactory Government: Satisfactory v

8 Quality of Supervision: Moderately Implementing Unsatisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately satisfactory Performance: Performance: Satisfactory Satisfactory C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Performance (if any) Potential Problem Project at any time (Yes/No): Problem Project at any time (Yes/No): DO rating before Closing/Inactive status: No No Quality at Entry (QEA): Quality of Supervision (QSA): None None Rating Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) TA Roads & Highways 73% 73% JB Other Social Services 18% 18% BV Public Administration Transportation 9% 9% Theme Code (as % of total Bank financing) 58 Conflict Prevention and Post-Conflict Reconstruction 100% 100% E. Bank Staff Positions At ICR At Approval Vice President: Annette Dixon Isabel M. Guerrero Country Director: Illango Patchamuthu Rachid Benmessaoud Practice Manager/Manager: Karla Gonzalez Carvajal Karla Gonzalez Carvajal Project Team Leader: Zafar Iqbal Raja Zafar Iqbal Raja ICR Team Leader: Rakhi Basu ICR Primary Author: Rakhi Basu vi

9 F. Results Framework Analysis Project Development Objectives (from Legal Agreement) From the Emergency Paper: The Project Development Objective (PDO) is to increase year round access for rural population in the conflict hit areas covered by the Project. Revised Project Development Objectives (as approved by original approving authority) The statement of PDO remained unchanged throughout the life of the Project. (a) PDO Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) vii Formally Revised Target Values Actual Value Achieved at Completion or Target Years The Project Development Objective (PDO) is to increase year round access for rural population in the conflict hit areas covered by the Project. Indicator 1 (a) % increase in the share of rural population with all season access; Value (Quantitative or 0 50,000 Qualitative) Date achieved 2/10/2016 Comments (including % achievement) Indicator 2 Value (Quantitative or Qualitative) Baseline was not available at approval due to security situation and no target value was set. 50,000 population (actual value at completion) with access to all weather road is based on actual data collected by the client at project closure. 100 % increase in number of motorized trips taken by beneficiaries living along the rural roads No motorized traffic 340 vehicles per hour Date achieved At ICR Baseline was not available at approval due to security situation and no target Comments value was set. However, there is evidence as captured by traffic data at the end of (including % the project data that the Project improved rural access and mobility through achievement) construction of all-weather paved roads on existing non-motorable earthen tracks. (b) Intermediate Outcome Indicator(s) Component 1: Rural roads constructed Indicator Baseline Value Original Target Values (from approval documents) Actual Value Formally Achieved at Revised Completion or Target Values Target Years Indicator 1 Rural Roads Constructed (Kilometers) Value (Quantitative or Qualitative) Date achieved 12/01/ /14/ /10/2015

10 Comments (including % achievement) 100% achieved. This was confirmed by in the third party monitoring report. Details of road sections completed are in Annex 2. Component 2: Project Management Satisfactory performance of construction supervision and contract administration Indicator 1 and environmental and social safeguards activities. High quality Value Consultant services construction and (Quantitative or procured compliance with Qualitative) safeguards policies Date achieved 05/31/ /10/2015 Comments (including % achievement) The consultants performed their duties adequately. Works were delivered. There were no complaints from local population. G. Ratings of Project Performance in ISRs No. Date ISR Actual Disbursements DO IP Archived (US$, millions) 1 05/15/2013 Satisfactory Satisfactory /08/2013 Satisfactory Moderately Satisfactory /15/2014 Satisfactory Satisfactory /15/2015 Satisfactory Moderately Satisfactory /25/2015 Satisfactory Moderately Satisfactory H. Restructuring (if any) I. Disbursement Profile viii

11

12 1. Project Context, Development Objectives, and Design 1.1 Context at Appraisal 1. The provinces of Balochistan and Khyber-Pakhtunkhwa (KP) and the Federally Administered Tribal Areas (FATA) are strategically located along the Pakistan- Afghanistan border. The region presents the challenges of arduous mountainous terrain with equally difficult access. By 2008, most of this region was under Taliban control. In May 2009, the Government of Pakistan (GOP) launched major military operations to root out the local pockets of militants support in the KP Province and FATA. The project was prepared shortly after that in 2012 at a time when this lawless border region was recovering from political unrest notably a military offensive between Pakistan s military and Talibans. 2. FATA is amongst the most poorly developed parts of the country. During the last two hundred years of colonialism, the region became a buffer zone in the struggle between global powers in Central Asia to the North and those in the Subcontinent to the South and was, as a result, further isolated and remained severely under-developed. The transport sector in the crisis area of FATA solely depends on the road network. Accessibility provided by the road network is limited road density is low (0.24 km/square km) and FATA does not compare favorably with rest of Pakistan (0.32 km/square km). The lack of infrastructure and social services is thought to be a contributing factor for extremism in the region. This is in large part a result of the unique administrative arrangements. The FATA region is governed under a colonial-era legal framework based on tribal structures and is outside the regular constitutional framework that prevails in the area. This has impeded development planning to systematically target those most in need of such interventions. Traditional governance structures had been, by and large, uprooted by the conflict, making the area vulnerable to militancy. 3. The situation was made more challenging by the long-standing social inequities and deprivation among the population in the northwest frontier region circumstances resulting from decades of weak governance and inadequate social, economic, and physical infrastructure and service delivery systems. These destabilizing factors were reflected in, inter alia, widespread unemployment; poorly controlled armed groups and arms circulation; depleted or damaged infrastructure; a rise in criminal and interpersonal violence; and an increasing illicit economy. 4. The military operation caused widespread displacement (around 2.7 million people) disrupted lives, livelihoods, and the provision of normal public services. About 7 percent of displaced families moved to camps, the rest occupying schools, public buildings and living with host families mostly in FATA, Swabi, and Mardan. In the face of the crises that confronted KP and FATA, the federal and provincial governments embarked on rapid postcrisis recovery and reconstruction efforts in collaboration with international and national humanitarian agencies. 1

13 5. The GOP quickly mobilized a strong international response as evidenced by the Friends of Democratic Pakistan (FODP) at the April 2009 Tokyo Conference where over US$5.0 billion in additional assistance was pledged for Pakistan. The GOP also launched various assessments for strategic medium to long-term support for the region. A Damage and Needs Assessment (DNA) for KP and FATA was completed 1 in November 2009 with the Asian Development Bank (ADB) and World Bank support covering the areas first affected by the GOP s action to combat the militants, and a subsequent Post Crisis Needs Assessment (PCNA) was completed in October 2010 for the entire area of KP and FATA with the assistance of ADB, European Union (EU), United Nations (UN) and WB. The PCNA assessed and quantified the short and medium term social and economic needs of the region and provides the underpinning for a pragmatic, coherent, sequenced long-term peace building strategy that is aimed at delivering the following agreed vision within 10 years. 6. The PCNA did not identify individual projects, just priority areas that needed to be addressed which included nine sectors spread across three regions KP, FATA, and Balochistan. The GOP and donors, at a summit meeting on September 25, 2009 requested the World Bank to establish a Multi Donor Trust Fund (MDTF) in order to provide for a coordinated financing mechanism for donor support of areas affected by the crisis. The rationale for the World Bank s involvement as fiduciary administrator was its experience in transport infrastructure projects in post-conflict environments. Given its worldwide experience and expertise in institution building in conflict-affected areas, the World Bank was uniquely positioned to manage the MDTF. 7. To address the nine priority sectors, MDTF operations were spread across eleven projects and designed to be programmatic, for example, they could be piloted at a small level, but had the built-in capacity to ensure scalability, replication, and rolling out once its effectiveness has been proven. In the case of KP, FATA, and Balochistan, the three conflict regions, this became particularly necessary given that each has a unique set of circumstances and institutional arrangements. A total of US$145 million was available for eleven projects, the result being small project sizes, with US$16 million for the FATA ERRP. 1.2 Original Project Development Objectives (PDO) and Key Indicators 8. The Project Development Objective (PDO) is to increase year round access for rural population in the conflict-hit areas covered by the Project. 9. The key PDO indicators were: (a) percentage increase in the share of rural population with all season access and, (b) percentage increase in number of motorized trips taken by beneficiaries living along the rural roads. 1 In 2012, the Bank conducted a comprehensive Balochistan Development Needs Assessment which has guided development projects in the province. 2

14 10. To achieve the PDO, the project focused on connecting 15 to 20 villages with each other through 50 km of all-weather motorable paved rural roads in Bajaur and Orakzai districts, two of the most isolated areas in FATA with high incidence of rural poverty. The expected outcome was to increase year round access to economic opportunities and social services (mainly health and education) for poor rural population in these districts. 11. For measurement of the output, length of roads constructed (km) was identified as the key intermediate indicator. The anticipated results would support Governmental efforts to help maintain minimal economic activity and improve social and political stability in FATA. The Project was implemented by the Works & Services Department (W&SD) of FATA. 12. Under normal conditions, the PDO indicators would have been adequate for the purpose of measuring the achievement of the PDO. However, because of the post-conflict conditions, data was not available which resulted in no baseline data, nor viable targets, and thus the results framework was fundamentally weakened. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 13. The Project Development Objective was not revised. 1.4 Main Beneficiaries 14. The Project would benefit about 50,000 people as later established through government data and stated in the Borrower s ICR. These represent the primary target groups who are the rural population in Bajaur and Orakzai agencies. Bajaru is the smallest agency in FATA with a total area of 1,290 square km and population is estimated at about 900,000. Orakzai Agency is the second smallest Agency with a total area of 1,538 square km and has a population of approximately 450,000. In particular, the project increased accessibility to the remotely located areas of Kulala and Kharkano in Bajaur Agency and Largri Tan. Dop Sana, Dargai, Shamim Jan Killi, Palosia, Hakim Killi, Ghakhi, SharoDoghal in Lower Orakzai Agency. These populations would benefit from improved travel time, lower transport costs, and accessibility to district markets, hospitals, schools and district headquarters. Prior to the project, the majority of the villages were connected through fair weather non-motorable earthen tracks, which were 0.5 to 1.5 meter wide, with most journeys made on foot and entailing frequent load carriage on head or back. 15. Other beneficiaries included the Works and Services Department (W&SD) department whose technical capacity in project design and monitoring was improved. In addition, contracting firms benefited from enhanced capacity through working with supervision consultants in areas like project management, materials testing, environment and social safeguards, etc. 1.5 Original Components 16. The project was comprised of two components: 3

15 17. Component 1: Infrastructure Building. This consisted of civil works, including: (a) Construction of about 50 km of rural roads (42 km in Barjaur and 8 km in Orakzai), including related structures; and (b) Associated relocation of utilities, land acquisition and resettlement. 18. Component 2: Project Management. This included support for: (a) Detailed Engineering Designs, Economic Analysis, Environmental and Social Safeguards Instruments, Bidding Documents, (US$0.175 million); (b) Contract Administration and Construction Supervision (US$0.475 million); (c) Environmental and Social Safeguards (US$0.05 million); and (d) Other project management activities, such as procurement, financial management, audits, etc. (US$0.4 million). The objective of this component was to strengthen the foundation of Component 1 for sustainable management of the project. 1.6 Revised Components 19. There were no revisions to the original components. 1.7 Other Significant Changes 20. Extension of project closing dates. The project closing date was extended twice to a total of six months. The first extension from the original closing date of June 2015 to October 2015 was required to allow sufficient time for completing the works which were delayed due to the poor capacity of the contractor. The second extension, from October 2015 to December 2015 was necessary to address adverse weather condition which had further delayed the project. 2. Key Factors Affecting Implementation and Outcomes 21. Implementation of the project was confronted by a number of challenges soon after effectiveness, which is not surprising given the emergency nature of the operation in a remote, post conflict situation centered on complex geographic, ethnic, security, and political fault-lines. The project was confronted by a delay in procurement soon after effectiveness due to unsuccessful competitive bidding. The original plan incorporated basic and straightforward procurement packaging of 50 km of rural roads into 7 civil works packages. Competitive bidding using National Competitive Bidding (NCB) method was initiated for construction of seven rural roads (three in Bajaur Agency and four in Orakzai Agency). Both pre-bid and bid opening sessions were boycotted by local contractors while others chose to stay away. The local contractors boycotted as they did not fulfill the financial qualification requirements for larger NCB packages and this boycott discouraged other contractors from the catchment area from participating. Non-participants reportedly attributed it to tribal enmities which made it almost impossible for a contractor of one geographical region to work in another. 22. A second round of a competitive bidding was seen as unlikely to yield a different result, so direct contracting, and repackaging of seven contracts into sixteen allowed the procurement to take place. Repackaging was mindful of the capacity of local contractors and the formal and informal geographical divisions of tribes. 4

16 Terrorist attacks and sectarian violence further delayed implementation, and so did unforeseen bad weather. There was resistance from local communities with regard to contractors who belonged to a different ethnic group. The contractors therefore were picked based on their ethnic and religious background. 24. There were equal challenges with the hiring of PMU staff, and the project suffered from lack of management continuity. The human resource base in FATA was severely depleted and finding a senior project director for the PMU proved to be difficult. The project therefore started with a deputy director from the UK whose unfamiliarity with the rules and regulations of Pakistan further delayed project start-up. During the life of the project, directors were changed four times as none of them wished to continue working in FATA. This added to the disruption and hampered smooth implementation Project Preparation, Design, and Quality at Entry 25. Soundness of background analysis. The relevance of the FATA ERRP was substantial and was embodied in the Post Conflict Needs Assessment (PCNA) that was carried out for the entire area of KP and FATA. Drawing on extensive stakeholder consultations, the PCNA identified key crisis drivers political, economic, geo-strategic & insecurity, and social related and the consequent priority areas that needed to be addressed which included the following nine sectors to facilitate achievement of the above four strategic objectives: (a) governance, (b) rule of law, (c) social protection, (d) education, (e) health, (f) infrastructure (comprising energy, transport and water supply and sanitation), (g) non-farm economic development, (h) agricultural and natural resources, and (i) strategic communications. The PCNA assessment itself was completed through an institutional structure led by the Strategic Oversight Council (SOC) chaired by the Prime Minister. 2 The SOC continues as the strategic apex, with the KP and FATA Secretariats represented by the respective Additional Chief Secretaries, over and above the Chief Minister, the Governor other political, civil, and military representatives. This clearly indicates strong participation from the government in the project identification. 26. The project was designed as an emergency intervention to support the immediate recovery, and peace-building effort in the country. The selected roads were the most critical routes to connect conflict affected areas and remove isolation of the population, thereby improving the well-being of un-served and underserved low income communities in Orakzai and Bajaur, and helping to bring cohesion to the country a key objective of the FATA Sustainable Development Plan of , which includes a rural roads program costing US$140.0 million. The project was considered to be timely, was delivered in an impressively short period of time, and is considered to have contributed to reducing insurgency in FATA. That said, undertaking these eleven projects in a conflict region with poor institutional capacity could be seen as ambitious, and monitoring them was a challenge. 27. Lessons learned and reflected in project design. Lessons from previous emergency infrastructure projects were taken into account during preparation, as were a 2 Supported by a Steering Committee and PCNA Secretariat 5

17 number of other sources The successful implementation of the project, particularly the Resettlement Action Plan (RAP) can be attributed to a good understanding of local-social structures and ability to work with local leaders, a key lesson reflected in other post-conflict projects. Other lessons incorporated in the project included keeping design simple, such that there were only two components (one of which was managing the other), ensuring government commitment by choosing the rehabilitation of those roads identified by the FATA authorities, and strengthening implementation capacity in post crisis scenarios through outsourcing and dedicated staffing for the Project. However, a key lesson not factored into project design and implementation was rapid rotation of staff on the client side which presented a big challenge for continuity and has been highlighted in other post conflict projects (such as NERAP in Afghanistan). 28. Risks and mitigation measures. The majority of identified risks focused on the delay in either delivery of the physical infrastructure due to security, fiduciary, safeguards or the enabling policy environment. The identified mitigation measures for these risks were well thought through and the project was implemented in compliance with the World Bank's guidelines for fiduciary, procurement, social and resettlement. However, four issues arose during implementation that had not been adequately addressed during preparation, (a) Weak contractor capacity was not identified. (b) Failure to achieve the PDO and monitor outcome and output indicators was identified in the ORAF as a key risk, however the risk mitigation measures (simple and flexible design with manageable components and a single implementing agency) although valid, were inadequate to address the risks. Additional mitigation measures might have taken an output approach (simple to carry out and providing useful and continual feedback about progress during implementation) which had already been tried and tested under previous projects. (c) In the absence of baseline data, target values could not be set and the key risk was not being able to monitor outcome, the key weakness in a project with several achievements. The mitigation measure does not identify the unavailability of baseline data and secondary sources or proxies were not explored. (d) Lack of environmental compliance was due to the failure to appoint environmental specialists within the PMU, the supervision consultants and contractor, as defined in the ESIA/ESMP. (e) Frequent change of PMU staff. The project director of the PMU was changed four times and it proved challenging to find and retain staff. No mitigation provided. 29. Assessment of project design. The Project financed the construction of priority rural roads infrastructure in the Bajaur and Orakzai Agencies of FATA. The design was 6

18 informed from the results of a community survey in Pakistan which showed a strong correlation between social indicators and access The overall project management responsibilities were entrusted to the Project Management Unit (PMU) at W&SD located in Peshawar, supported by a Deputy Project Director and Finance Manager/Accounts Officer. The PMU functioning as the Project Secretariat supported the Implementation Steering Committee. The PMU liaised with the representatives of Political Agents of Bajaur and Orakzai Agencies and were responsible for internal/external processing of all approvals including PC-I, procurement and implementation of civil works, procurement and management of consultant services. The implementation at the field, was done by Field Implementation Unit (FIU) within each of the divisional offices of W&SD in Bajaur and Orakzai Agencies, headed by the respective Executive Engineer (XEN) and supported by Sub Divisional Officer (SDO) and Divisional Accounts Officer (DAO). This arrangement, together with other hiring of supervision consultants, contributed to ensuring local ownership and strengthened institutional capacities of the W&SD. 31. The overall design had an inherent weakness in its scope,that is, attempting to address a key issue in rural transport, poor accesses while only targeting 50 km in a network of 6000 km. This was partly a result of the sharing of funds among eleven projects and partly due to a general lack of resources, for which the team cannot be faulted. 32. The design of the components was focused on achieving technical quality. The majority of the villages are connected through fair weather non-motorable earthen tracks, which are 0.5 to 1.5 meter (1.5 to 5.0 feet) wide. The objective was to make the roads motorable and high quality with minimum maintenance required, as there was no component on road maintenance. The project therefore continued the trend in Pakistan that has continued to focus on road paving construction at the expense of road maintenance. Although it should be noted that W&SD receives an annual maintenance grant which is distributed to its divisional offices based on the ratio of total length of roads under their jurisdiction, and therefore some capacity building on road maintenance would be expected. 2.2 Implementation 33. The project was confronted by a number of challenges soon after effectiveness, which is possibly not surprising in an emergency operation in such a fluid context. 34. Project components. Although, as mentioned, all the risks may not have been sufficiently addressed, the design of the physical components was simple and satisfactory and responded to the emergency situation of the country and needs of the sector. The geographical spread was limited to two agencies: Bajaur Agency with a total area of 1,290 Square Km and Orakzai Agency with a total area of 1,538 Square Km. The institutional objective/component was limited and could have included some capacity building for W&SD, particularly in road maintenance to ensure sustainability of the investments. 3 Rural Access and Mobility Study in Pakistan: A Policy Note, World Bank

19 35. Inadequate contract slicing. The project was designed under OP 8.00 Rapid Response to Crises and Emergencies and used every allowable flexibility which ultimately resulted in satisfactory outputs. Although the project experienced a rocky start, with the initial failed bid under NCB of seven rural roads where the procurement process was boycotted by the construction industry, the World Bank subsequently learned the lessons regarding the financial capacity of the local contractors. The challenge was meeting the local contractor liquidity requirements; the contractors were not in a position to mobilize on site without some financial assistance. The borrower therefore waived the bid security requirement. However, it was still not possible to mobilize contractors on site without providing some monetary advance. Therefore, it was arranged that 20 percent of the mobilization advance was to be given in four tranches against: (a) availability of unconditional bank guarantee equal to the amount of advance; (b) pledging of equipment and; (c) the endorsement of a Political Agent. 4 The project design was adjusted and the seven civil works contracts were re-packaged into 16 small contracts. A direct contracting process was followed, and contracts were proposed for award on the basis of assessed financial capacity of contractors comparable to the contract package. 36. Implementation capacity constraints. Although attempts were made to mitigate capacity constraints by establishing a dedicated PMU, the PMU struggled to find and retain qualified personnel. This contributed to the delays in implementation. 37. Security situation and adverse weather. As already noted, delays in implementation required the closing date to be extended twice, from June 2015 to October 2015, and from October to December The reasons for the delays included delays poor capacity of construction contractors, and adverse weather that delayed the asphalt paving, which needed temperature well above 0 degrees. Security factors also significantly delayed the project. Although security measures were planned during project preparation to mitigate risks, incidents still occurred such as two suicide attacks while the project was being implemented. 38. Innovative approach. The project addressed a number of implementation issues with several innovative solutions in procurement, FM, and construction management. These included: (a) Designing a solution to address the sectarian issues. In FATA, where there is ethnic and religious split, the communities would only allow contractors who belonged to the community and of the same ethnic background to construct the roads. An equitable selection process was therefore designed that kept these sensitivities in mind and which allowed the roads to be constructed. 4 Each tribal agency is administered by a political agent, a public servant. As part of his administrative functions, the political agent oversees the working of line departments and service providers. He is responsible for handling intertribal disputes over boundaries or the use of natural resources, and for regulating the trade in natural resources with other agencies or the settled areas. The political agent plays a supervisory role for development projects. 8

20 (b) To meet the liquidity requirements of the contractors, security performance was waived and instead of a Bank guarantee (which was not available to contractors), contractors were allowed to pledge their equipment. (c) Slicing large contract packages into smaller ones and procured through direct contracting avoided collusion. If the sliced packages were to be procured through open competitive bidding, risk of collusion and consequent higher price would have been likely. (d) Personal cheques were issued for the first time in FATA to project-affected people allowing smooth and fast resettlement. 2.3 Monitoring and Evaluation (M&E) Design, Implementation, and Utilization M&E Design 39. The M&E design was perhaps the weakest link in the project. The PDO indicators (a) percentage increase in the share of rural population with all season access and, (b) percentage increase in number of motorized trips taken by beneficiaries living along the rural roads were relevant and the first indicator was a core sector indicator. However, no baseline was available for either indicator nor was a secondary or proxy indicator identified. 40. Actual progress of PDO and intermediate outcome indicators were not monitored in the ISRs and nor in the limited Aide Memoires during implementation. The Mid Term Review (MTR) was carried out in November 2015, a month before the revised project closing date and the PDO was rated as Satisfactory since the outcome indicators were likely to be achieved, as stated in the MTR report. This clearly indicates there were significant shortcomings in implementation and the absence of a clear and meaningful vision of success. 41. The focus of the project was in achieving the outputs and the M&E framework focused completely on monitoring whether physical outputs were produced. In that regard, the intermediary indicators could have been better designed and more closely monitored. The first intermediary indicator, Infrastructure Building was straightforward and measured kilometers of roads built. The second intermediary indicator was project management, Satisfactory performance of construction supervision and contract administration and environmental and social safeguards activities. The target value for this indicator at the end of the project was High quality construction and compliance with safeguards policies. While project management would not typically be considered a valid indicator, there might be some latitude given unusually challenging conditions under which the project was carried out. That said, even granting this latitude, if such an indicator was to have more significant validity, it could have been strengthened by including more details such as the monitoring of consultant reports that assessed the quality of construction, financial audits, etc. 42. Since the task team was not allowed to visit FATA due to security issues, monitoring was done through supervision consultants who sent monthly progress reports. 9

21 These reports were focused mostly on physical and financial progress and identified major issues related to physical progress. MDTF did third party monitoring of five of its projects including the FATA Emergency Rural Roads Project. 43. It probably cannot be overstated that many of these issues, such as M&E and supervision and implementation issues in general, can only be evaluated within the extreme context under which the project was prepared and implemented. During preparation, the FATA region was just emerging out of insurgency and remained highly fragile in security and many other aspects. The project was prepared and approved under OP Under this policy it was essential to keep project design simple. In line with this, km of roads constructed (the intermediate output indicator) could have reasonably been set as the single key indicator of the project as a proxy to measure the project outcome. 2.4 Safeguard and Fiduciary Compliance 44. Environmental management. The project was prepared under OP 8.00: Rapid Response to Crises and Emergency which stipulates that the project could follow simplified procedures regarding social and environment safeguard policies. It was a Category B project and none of the potential impacts were large scale, unprecedented or irreversible in nature, and could be addressed with the help of an appropriately designed and effectively implemented mitigation plan. The OP required the World Bank to prepare an Environmental and Social Screening and Assessment Framework (ESSAF) for the entire Balochistan/NWFP/FATA MDTF. The framework defined the assessment requirement for 11 projects. The ESSAF then defined safeguard assessment requirements for individual projects under MDTF. In compliance with these requirements, an ESA was prepared by the Government for the FATA ERRP project. 45. The Government of Pakistan through the FATA Secretariat conducted an environmental and social impact assessment (ESIA) for the project in accordance with the local regulatory requirements as well as the WB safeguard policies and in pursuance with the ESSAF. The ESIA, which also included an environmental and social management plan (ESMP), identified the potential environmental impacts associated with the project activities. The key potentially adverse environmental impacts of the project included soil and water contamination caused by inappropriate waste disposal at the construction sites and also at the construction camps, air quality deterioration caused by the road construction activities and operation of heavy machinery as well as asphalt plant, occupational health and safety risks for the construction workers and also for the local communities, and clearing of natural vegetation and cutting of trees for road construction. During the road operation, the key potential impacts include safety risks associated with the vehicular traffic on the road and maintenance activities. The ESIA proposed appropriate mitigation and preventive measures to minimize if not eliminate all of the potential impacts of the project briefly described above. The ESIA also proposed monitoring mechanism to ensure that the identified mitigation measures are adequately implemented. The ESIA also proposed institutional mechanism for the ESIA implementation. 46. The project engaged independent environmental consultants to conduct Effects Monitoring in pursuance with the monitoring requirements defined in the ESIA/ESMP. 10

22 The project also commissioned third party validation (TPV) toward the end of the construction activities. 47. Weaknesses. The most significant non-compliance was that the project failed to establish institutional arrangements for environmental management as defined in the ESIA/ESMP. These arrangements included appointing/engaging environmental specialists within the PMU and also with the supervision consultants and contractor. Apparently, this lapse lead to other non-compliance particularly complete disregard to the environmental reporting requirements defined in the ESIA. Since the construction sites were inaccessible to the World Bank s task team because of the security concerns, lack of environmental progress reports meant that no information was available to the task team regarding ESIA compliance. 48. The TPV report also confirmed these weaknesses and highlighted that no institutional arrangements were made for environmental management, almost no mitigation measures were implemented at the sites, no environmental monitoring was conducted, and no environmental trainings were conducted for the contractor as well as consultant s staff. However since the TPV was conducted toward the end of the project when most of the construction activities had already been completed, no corrective measures could be implemented to address the assessment findings. Resettlement 49. The project was in compliance with World Bank s resettlement framework. A detailed LARP (Land Acquisition and Resettlement Plan) was developed for this activity and its third party validation has been conducted and a comprehensive report has been developed to assess the level of fulfillment of this plan. 50. Based on detailed review of the LARP document itself and the findings of the validation activity, it has been observed that the entire land acquisition activity has been conducted within the WB land acquisition and resettlement framework and it has been ensured that the entire land acquisition process has been conducted in a most transparent and fair manner to ensure the rights of the land owners are accorded. All land procured for the project activity was agricultural in nature with no residential land being procured and thus no resettlement activity of the communities in the project area felt necessary. The entire process of land acquisition, from the identified of the rightful owners of the effected properties to subsequent detailed consultations and determination of the compensation rates for these Project Affected Persons (PAPs) and processing and disbursement of payments has been properly documented and has been completely transparent. 51. Detailed discussions with the PAPs to obtain their feedback regarding the land acquisition process were also conducted and they expressed their general satisfaction with the timely processing of their claims and issuance of payments. However, a few of the land owners mentioned that it would have been better if the rates compensated for their lost land had been higher considering the continuous inflation. 11

23 52. The framework was prepared on the basis of the Land Acquisition Act and the World Bank Operation Policy on Involuntary Resettlement. The overall objective was to mitigate adverse project impacts on people s livelihood, and assist them in improving or at least restoring their livelihoods. 53. The following principles were followed in developing resettlement plans: (a) Project affected people losing access to portion of their land or other productive assets with the remaining assets being economically viable were entitled to compensation at a replacement cost for that portion of land or assets lost to them. Compensation for the lost assets was made according to the following principles as unanimously decided / agreed: (i) (ii) (iii) (iv) (v) (vi) Replacement land with an equally productive plot, cash or other equivalent productive assets, Materials and assistance to fully replace solid structures that were demolished, Replacement of damaged or lost crops and trees, at market value, Other acceptable in-kind compensation, In case of cash compensation, the delivery of compensation were made in public that is, at the Community Meeting, In case of physical relocation, provision of civic infrastructure at the resettlement sites. (b) Project affected people losing access to a portion of their land or other economic assets rendering the remainder economically non-viable were given the options of compensation for the entire asset by provision of alternative land, cash or equivalent productive asset, according to the principles in (a) i- iv above. 54. Basic principles followed for resettlement plan. To meet the requirements of the WB Involuntary Resettlement Policy and guidelines, the following basic principles were implemented for the preparation of RAP: (a) Assurance of no adverse impacts on Project Affected Persons (PAPs), (b) Assurance of compensation to the PAPs where the adverse impacts were unavoidable, (c) Identification of vulnerable groups and their assistance to improve living standard, 12

24 (d) Assurance of equivalent cash compensation to the PAPs for their affected land, structures and other assets (tube well) falling within the RoW, (e) Assurance of one-time lump sum assistance equal to one month's inflation adjusted official poverty line, in addition to other admissible compensation / allowances, (f) In addition to cash compensation for the structure, the PAPs were allowed to salvage their building material, (g) Assurance of compensation/assistance has been made in accordance with the World Bank Policy before taking possession of the acquired land and properties, (h) Establishment of appropriate grievance redress mechanism at field and project level to ensure speedy resolution of disputes, (i) Eligibility of all the PAPs was ensured for compensation and livelihood / income restoration irrespective to possession of land title. However, the proof of title was ensured for payment of compensation for land, (j) All compensations were paid at replacement value for assets without deducting depreciation and salvage value, (k) All PAPs and stakeholders were consulted to improve the design of project, and for preparation of safeguards documents throughout the execution phase of the project. 55. Financial management. The FM risk was assessed Substantial, as this was the first World Bank financed project in FATA. To mitigate the FM risks, the project used a comprehensive financial management manual. In addition, a qualified and experienced Accounts Officer was hired who was supported by existing staff of Budget and Accounts Section at the Office of Chief Engineer, W&SD and the Divisional Accounts Officer (DAO) of Bajaur and Orakzai Agencies. Separate books of accounts, on cash basis, were maintained by W&SD for the Project activities using the New Accounting Model (NAM). 56. A notable achievement of the Project was the implementation of a national financial management system which allowed transactions to be displayed in national government system in real time. Initially there were some delays, which were soon resolved. This was the first experience of FATA with such a sophisticated FM system and resulted in raising of both awareness and capacity. Audit reports and Interim Financial Reports were submitted to the Bank in a timely manner. In accordance with Bank s fiduciary policies; as related to procurement and financial management, there were no unresolved fiduciary issues at project completion and, the implementing agencies complied with financial management covenants. However the audit performed by the office of Auditor General of Pakistan has raised certain issues; which are referenced primarily to government s own fiduciary systems. These issues are under process for disposal as per Federal Government s stipulated rules of business. 13

25 57. There were also exemplary achievements in financial management with regard to land acquisition. When the project started, there was no revenue record of ownership as this was an unsettled area. The client developed SOPs for land acquisition payment on the advice of World Bank through which land acquisition was done. Under the close guidance of the World Bank s FM team, the FATA used the SOPs to acquire the land and till to date there has been no complaint of land acquisition. 2.5 Post-completion Operation/Next Phase 58. Maintenance of the roads improved under the project. Keeping in mind the negligible maintenance budget and low capacity of the government, the roads have been designed such that limited maintenance is required at least for five to seven years. The infrastructure rehabilitation part of the Project was completed successfully but there is a lack of progress in areas relevant to the Project s long-term sustainability. Maintenance of the region s road network so far has been minimal. Continued financing is essential for the sustainability of the roads component. The World Bank has advised the FATA to make budgetary provision for road maintenance. Next Phase Project 59. After this emergency phase of development, the next project, if and when this should occur, should have a more expanded scope and ensure there is adequate financing. Fifty kilometers of roads, although a notable achievement within its context, will still have limited impact within a 6000 km network, where rural roads in fragile areas provide a lifeline for the community. Capacity building, particularly on maintenance, safeguards and monitoring should be a key area of focus for any follow up project which will build up the sustainability of the sector. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design, and Implementation Relevance of Objectives: High 60. The Project Development Objective (PDO) was to increase year round access for rural population in the conflict-hit areas covered by the Project. This objective strongly supported the country s priority when the Project was prepared and continues to be relevant to the Country Partnership Framework (CPF) for The Project contributed directly to two of the priority areas: (a) developing essential infrastructure that would help improve livelihoods, kick-start economic activity, communications and social services; and (b) a facilitating future MDTF projects. The CPF seeks to address sources of fragility and conflict as a priority particularly in KP, FATA, and Balochistan. The engagement will continue to focus on increasing employment and livelihoods opportunities and improvements in delivery of basic services in these regions. Relevance of Design and Implementation: Substantial 14

26 61. The project was designed with the objectives that were relevant at the time (and continued to be relevant) and was responsive to the needs of FATA region in the early post conflict period. The core concept design was extremely relevant, in that it set out to respond to the country s immediate challenges of improving access in conflict affected, isolated areas, thereby improving connectivity, and facilitating peace in post conflict FATA. The design was clean and simple and facilitated quick implementation of the civil works by local contractors. 62. The project implemented through the PMU with the help of Field Implementation Units (FIUs) achieved its planned physical targets on time. A six-month delay in a fragile and conflict situation with no cost escalation and full disbursement is a substantial achievement. Mechanisms for extensive consultation with tribal elders and political agents were set up which ensured smooth implementation particularly for sensitive land acquisition and procurement issues. Third party monitoring framework and reporting structures for safeguards and resettlement was well established, although some of the environmental safeguards were not complied with. 63. As mentioned elsewhere, the results framework, primarily the indicators, was undoubtedly the weakest part of the design. It was not well defined, lacking key baselines and targets and thus it could not provide an exact measure of the PDO. Given the emergency nature of the intervention and the limited available information, the focus in defining the results framework on outcomes, where baselines were not available, rather than measurable outputs, was flawed. However, because of the exceptional conditions under which this project was implemented, the single most important feature responsible for achievement of the objectives was the design of the project concept the simplicity and clarity of the activities and the implementation arrangement under which they were carried out. 3.2 Achievement of Project Development Objectives 64. The Project Development Objective (PDO) is to increase year round access for rural population in the conflict-hit areas covered by the Project. 65. The achievement of the PDO has been rated Moderately Satisfactory. The achievement of the PDO is generally evaluated against a baseline and the target values of the project s indicators and secondarily by taking into account project impacts on the affected populations the beneficiaries. In the absence of baselines, it is not possible to estimate the numerical achievement of the two PDO indicators share of population with all season access and number of trips. The supervision documentation does not identify secondary or proxy data which was available at the time of project preparation. An ISR in November 2015, a month before project closure does identify a baseline value of 30 percent for share of population with all-weather road access based on a 1998 census and a 2005 Rural Access & Mobility study conducted by World Bank. However, on review of the source data it was concluded that the data was not relevant due to a number of reasons that include: (a) the actual population which was used for the calculation of percent was not reported; (b) the project was prepared in 2012, almost seven years after the study, (not to mention the intervening conflict and its impacts), and the baseline was not adjusted to 15

27 account for the time difference or conflict impact; and (c) the target area in the 2005 report is not well defined and cannot be directly correlated with the two districts where the project was implemented. 66. While the progress on the two PDO indicators was not (and could not) be monitored, the physical output of the project was well monitored and the intermediate indicator regarding progress of output properly designed in terms of kilometers of roads constructed was confirmed. The fundamental output of the project was fully achieved, mostly on time and on budget. 67. Further, the expected outcome for the second PDO indicator ( increase in number of motorized trips taken by beneficiaries living along the rural roads ) was achieved although, not captured in the results framework. A traffic count survey (in Table 1 below is a proxy for indicator two) conducted at the end of the project showed the following results: Table 1. Traffic Count Survey Category Base Case 2012 Year 2016 Traffic Per Hour Passenger Car Mini Bus 0 88 Agricultural tractor 0 86 Total hourly Note: Data was collected over an eight hour period (there is usually no traffic after sun down) period for a week. 68. The emergency paper (PAD) states that there was almost no or negligible traffic on the majority of the roads in 2012, as these roads were not motorable. Thus, the baseline was effectively zero. The 2016 survey done by the PMU indicates substantial growth. 69. If one were to evaluate the project based on impacts on the lives of the population benefiting from the project, there is significant evidence of project benefits, although again, not quantifiable. The third party monitoring report observes the following: 70. Kharkano to Kulala, Length Km. The previous road was a dust and dirt track of 2.0 m to 3.0 m in width. The constructed road brought many positive impacts and is very important because it serves approximately half of the population of Tehsil Salarzai, Utman Khel and some population of Khar. These areas were deprived from communication facilities and facing great hardships. After construction of the road better communication facilities are available to the inhabitants of the area. The land on both side of the road is highly fertile on which people grow cash crops such as Wheat, maize, barley, and vegetables and the road links them to the market. Furthermore, the road is helpful for the inhabitants of the area to get connected to the education hub and health facilities at Head Quarter of the Agency. 71. Dag Qilla Bypass Road, Length Km. The Road connects Khar-Zulam Road with Khar-Nawgai road. The previous road was a dirt track and missing in parts at various points. After construction, this road serves as Khar By pass Road and the traffic load on the busy Bazaar of Agency Head Quarter has been reduced to great extent. The impacts of 16

28 road as reducing traffic, timely reaching the business market, health facilities and education institutions is a great contribution for the area. Moreover, the road is also contributing in reducing morbidity and mortility rates as connecting people with health facilities and reducing road accidents. 72. Road from Bado to Serisar, Length Km. This road starts from Khar-Zulam Road at K.M 14 and ends at Ghari Shamozai area of Barang Valley. The NAS constructed this road in as shingle road but later on the road conditions were very poor and the inhabitants of the area were facing great hardships. After blacktopping of this road the population of Tehsil Utman Khel and Barang are benefited by having better communication facilities. The people of the area are now able to bring their daily products to the nearby markets to the agency headquarter Khar and Timargara District Dir. The road enabled People to access better education and health facilities with quick and reliable communication. 73. Road from Largri Tan to Dop Sana, Length Km. This road is starts from Main Kadda-Kurez Road to Dop Sana. There was no blacktop (BT) road before this and the people of the area were using dirt tracks. After construction of this road, the communication problem of the people has been resolved. This road enabled the inhabitants to bring their crops/vegetables and other commodities to the local and nearby markets. This is the only stretch of road in the project area that is located near a forest cover area, but the alignment does not affect the forest cover as per observation during monitoring. The proposed road is constructed along with the existing dirt track and around 750 m of the road is extended clear of the forested area, near the agricultural lands. 74. Road from Dargai to Shamim Jan Killi, Length 1.68 Km. This road starts from Main Kadda Kurez Road and lead to Shamim Jan Killi via Dargai. There was no BT road before this and the people of the area were using dirt tracks. After construction of this road, the communication problem of the people has been resolved. The inhabitants of the area are now connected to the market and linked with health and education facilities. 75. Road from Palosia to Hakim Killi Khawari, Length Km. This road also starts from main Katcha Packa to Mitta Khan Palosia road via Khawara Baba Lamso to Hakim Khan Killi. The law & order situation in the area was normal but no better communication facilities were available in the area. On completion of this road, the inhabitants of the area are facilitated with better communication services and their economic conditions improved. 76. Road from Dargai to Sharo Doghal, Length Km. This road starts from main Kadda-Kurez Road via Ghakhi to Sharo Doghal. There was no black top road in the area and the people were using dirt tracks. This is a backward area. After construction of this black top road, the socio economic conditions of the people improved to great extent as they the road improved the access to markets, better health and educational facilities. 77. Based on the above, efficacy is rated Moderately Satisfactory. 3.3 Efficiency 17

29 Efficiency in achieving the project objectives in terms of EIRR is Highly Satisfactory. 78. This project involves the civil works of seven different rural roads constructed in FATA. Out of these seven roads, three have been constructed in Bajaur Agency and four have been constructed in Orakzai Agency. The total length of these seven rural roads is about 50 Km. (42 Km in Bajaur Agency & 8 Km in Orakzai Agency) The aim of the project was to improve connectivity in Bajaur and Orakzai Agencies of FATA through construction of these all-weather-paved rural roads on existing non-motorable earthen tracks. 79. The project s primary output, the construction of 50 km roads in conflict-hit areas, was achieved. The existing conditions of the tracks at project preparation was very poor with a width of about 0.5 ~ 1.5 m only. At completion, the width of these tracks were increased to 3.65 m with 1.5 m shoulders. 80. An economic analysis of the investments made under this grant has been carried out to provide an ex-post evaluation of the benefits and costs attributed to these investments, according to relevant World Bank Guidelines. The methodology used follows standard practice of comparing life cycle road agency and user costs with and without project, using Net Present Value (NPV) and EIRR at 12 percent discount rates as decision criteria. The Highway Development and Management Model of the World Bank, HDM-4 has been used for analysis. 81. Project economic benefits have been classified as Tangible and Intangible benefits. Intangible benefits are those which cannot be assigned any monetary value. Tangibles benefits are quantifiable. Reduced vehicle operating costs and travel time costs have been taken as a measure of project economic benefits resulting from improved road geometrics due to project implementation resulting in increase in travel speeds for various types of vehicles. The high economic returns (25.8 percent) compared to project preparation stage (19.05 percent) and the sensitivity analysis conducted provided an indication of the economic robustness of the Project. 82. Table 2 provides the comparison of major economic indicators with those derived at the project preparation stage. The project roads have achieved high rate of returns (EIRR), benefit costs ratio (B/C) and net present value (NPV) compared to project preparation stage. Table 2. Comparison of EIRR with Project Preparation Stage Parameters ICR Stage Project Preparation Stage 5 Net Present Value Rs. mil Benefit-Cost Ratio EIRR% Analysis of 5 Roads was carried out at project preparation stage while at ICR stage analysis of all the 7 roads has been carried out. 18

30 83. Summary of the economic indicators with different discount rates are as follows while detailed methodology is attached as annex. : Table 3. Economic Analysis Results Parameters Discount Rate (%) Discounted Benefits Discounted Cost Net Present Value Benefit-Cost Ratio Economic Internal Rate of Return 25.8% 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 84. The overall rating is based on the relevance of the objectives, the efficacy, or extent to which they were achieved, and the efficiency in doing so. Project objectives remained relevant to the current strategic priorities of Pakistan, World Bank and MTDF. Outputs delivered were largely satisfactory. The Project achieved high economic efficiency. The project has demonstrated its economic efficiency as its actual average EIRR of 25 percent at project completion and exceeds the EIRR of 19 percent at appraisal. The M&E reporting was poor, and outcomes could not be assessed thus reducing achievement of objectives to moderately satisfactory. A traffic count survey done in 2016 by PMU as a proxy indicator to measure the increase in increase in number of motorized trips taken by beneficiaries living along the rural roads (the second PDO outcome indicators) show an increase in vehicle along the rehabilitated roads. 85. MTDF had concluded that the project had positive economic and social impacts on the lives of many people, as noted below, not captured within the results framework. A key objective of the project was to improve economic activities and better connectivity of the strategically important locations and aid in the peace and security building process. The following observations by the PMU and MDTF surveys indicate the following: It is estimated that 50 kms roads would serve about 50,000 people living in the influence area of the road. Better connectivity to the strategic locations in the settled areas and Bajaur Orakzai Agencies and to the remotely located areas of Kulala and Kharkano in Bajaur Agency and Largri Tan. Dop Sana, Dargai, Shamim Jan Killi, Palosia, Hakim Killi, Ghakhi, SharoDoghal in Lower Orakzai Agency. Improved linkage of Bajaur and Orakzai Agency to the provincial and national highways with significantly reduced travel time to Peshawar. 19

31 The project roads are generating economic opportunities through improved market access. The economic activities include faster transportation of the agriculture produce (cash crops) and goods. The project has facilitated the improvement in security situation by allowing logistic support required by law enforcers and giving security forces greater control in these areas Forces have made pickets on paved roads after its construction and since the construction, there has been no attack by militants. A 3 km bypass to Barjaur city center has eased traffic congestion in the busy Bazaar of Agency Head Quarter. The road also gives quicker access to the main hospital and bus stand in the district 86. As a result of the project interventions, (a) two of the dispute affected districts have now all-weather access to essential social services and market places for the first time; (b) journey times and travel costs have reduced markedly due to improved roads - facilitating trade and commerce between regions; (c) children now have access to school throughout the year; (d) Enrollment in vocational training for women have increased in various centers in the region, (e) other MDTF financed projects like economic revitalization benefitted from cross fertilization where market access was provided on the roads built under this project allowing women entrepreneurs in the SME sector to flourish; (f) awareness and knowledge of professional construction project management knowledge, tools, skills and techniques have been built. 87. By combining the relevance, achievement of PDO, and efficiency the project is rated moderately satisfactory in spite of the many positive project achievements. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 88. Poverty impact. The upgrading of the roads has contributed to improvement in livelihoods of the residents through better road accessibility. 89. Improved access to market. The project was delivered in areas where transport plays critical role in improving the livelihood of the communities. The improvement of roads in these areas, reduced travel time and costs and improved livelihood through better access to market. The project also enabled the provision of all-year transport services for the population living within the influence areas of the improved road corridors, providing lower transport cost, creating employment opportunities and boosting local economy. 90. Improved health care practices by the local residents. The improvement of road resulted in frequent visits to health centers and health posts. 91. Increase in school enrollment. School enrollment has increased along this road corridor. 92. Gender aspects. It was estimated that majority of the beneficiaries in the project area are women, since most men in the villages have left in pursuit of better income 20

32 opportunities. The MDTF survey indicate that increased business opportunities for women in the SME sector as women can now take their wares to the market easily. Given the isolation of the project area prior to realization of the project, improved transport enabled youth and women to have better access to social services such as hospitals and schools. 93. Social development. The project was able to provide better access to social services such as schools, health centers, and markets which improved development of business activities and livelihood of people. The project also facilitated the peace building process and improved cohesion of the country by breaking the isolation of conflict-affected communities. The immediate socio-economic impact has been positive on the restoration of travel and livelihoods in areas served by the improved roads, with visible increases in traffic volumes on the key roads on which improvements have been carried out. (b) Institutional Change/Strengthening 94. This was the first engagement FATA had with a multi-donor agency and works and services department became aware of best practices in many areas. In FM, there was streamlining of land acquisition and compensation, and for the first time cheques were issued directly in the name of the landowner. Though FATA had their own FM system, they learned to prepare interim unaudited FM reports and became familiar with a more sophisticated financial management system. In procurement, awareness of internationally accepted procurement rules will allow smoother implementation of other donor funded projects. In Environmental and Social Management, W&SD s capacity to conduct environmental assessment was expanded for preparing and monitoring social impact assessments, and resettlement and RAP for road projects. 95. There was significant technical capacity building on road design and standards. Prior to the project, the works and services department did in house design for standard earth work. Through this project they were introduced for the first time to standards and specification for detailed design of bituminous roads. 96. The project assisted MDTF to build capacity of its M&E Directorate. The M&E directorate was defunct when the governance project funded by MDTF started in FATA. Through GIS mapping of the project roads with the help of Peshawar University, the M&E Directorate built its capacity and are now is possession of a powerful M&E tool which can be used across projects. (c) Other Unintended Outcomes and Impacts (positive or negative) 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 97. Not carried out for this project. 4. Assessment of Risk to Development Outcome Rating: High 98. There are two major risks that might affect the Project s development outcome. 21

33 99. Sustainability of assets. Inadequate capacity and funding for maintenance planning poses a challenge to ensure sustainability of the investment. In the short run, the rehabilitated roads may be in good physical condition but without continuous proper maintenance, the sustainability of the roads in the long run will be a challenge. Continued commitment by the Government is needed in financing the maintenance of the rehabilitated infrastructure Security Risk. The security situation in the project districts have improved since the construction of the roads by allowing logistics support required by law enforcers and giving security forces greater access and control in border areas. Although the improved security cannot be attributed solely to improved roads, the improve access definitely was a contributing factor to the reduction in insurgency. There has been no incident of terrorism since the roads were built. However, the region continues to be a conflict area and when conflict occurs, the infrastructure could be destroyed or damaged and remains a concern. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 101. The Project was well prepared from the beginning as an emergency operation The MDTF recommendation and Government of Pakistan s priorities together with the World Bank s experience in post conflict countries served as a strategic plan and guided the identification and preparation of the project. The technical expertise of the World Bank was indispensable for project preparation in the post-conflict environment with extremely low client capacity on the technical and organizational level The design was relevant and appropriate in addressing key issues in connectivity in conflict affected areas of Bajaur and Orakzai and improving implementation capacity constraints of implementing agencies at the time. There was no cost overrun largely due to the 10 percent cost escalation built into the contract packages above the market price. However, the planned outcome and key indicators in the results framework were not practical though relevant to the project activities. There were no existing baseline indicators and target value at completion was not identified. The project team was aware in the difficulties in collecting data for PDO indicators but no corrective action such as restructuring was taken. Given that this was an emergency project to be implemented in a short time frame, the indicators should have been better aligned with proper monitoring of progress during implementation. (b) Quality of Supervision Rating: Moderately Unsatisfactory 103. The World Bank performance during implementation was moderately unsatisfactory. The team was technically very strong with a good skill mix, however, due to security situation the team could not physically supervise the project and had to rely on supervision consultants and third party monitoring. However, the team acted could have 22

34 been more proactive in identifying issues and threats to the achievement of the PDOs and finding opportunities and solutions The PMU visited the World Bank s office in Islamabad every three months and the World Bank team visited the FATA secretariat in Peshawar regularly. However, the supervision and related documents were insufficiently detailed and with no action plans for next steps. Performance ratings for achieving the project outcome given in the Implementation Status Reports (ISRs) were unrealistic, and sufficient attention was not paid to the project s likely development impact. The MTR was done a month before the project closed where the livelihood of achieving the PDO is still rated achievable in the absence of monitoring indicators. The project could have been restructured to better align the PDOs to the results framework, but this was not done. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 105. The overall Bank performance is rated Moderately Satisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Satisfactory 106. The FATA Government, showed a sense of ownership toward the Project and was committed in completing the construction and rehabilitation works and thereby achieving the outputs. The Ministry was proactive and committed to completing the project on time. The Government agreed to use effective and transparent procurement procedures in a region that was unaccustomed to this, and to establish an independently working PIU. (b) Implementing Agency or Agencies Performance Rating: Satisfactory 107. The Department of Works and Services, played a critical role in coordinating various activities particularly resolving issues with contractors. Given its limited capacity and being new to working with the World Bank, its overall performance is rated as satisfactory. However, at the PMU level, the project director was changed several times which impacted continuity of the project. This was mainly due to the security situation in FATA and no project director wanted to continue for a long time. Replacements were not easy to find which impacted implementation. (c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory 108. All of the activities were completed. There was a six month delay due to bad weather and weak capacity of contractors, but given that this was the first World Bank project for the FATA government, the difficult political situation, the challenges of staffing the PMU, the weak capacity, the government made a tremendous effort to get familiar with World Bank practices in record time and deliver the project with little delay the overall performance of the Borrower is rated satisfactory. 23

35 6. Lessons Learned 109. The project had to overcome hurdles of reoccurring insecurity, low capacity, failed procurement, and challenging construction tasks before delivering results. The key lessons drawn from the project are summarized as follows: 110. In procurement, one size does not fit all. In addition to examining formal structures, rules, and institutions, understanding traditional authorities such as chiefs, aura and authority of political agents, and militancy drivers is equally important. Clearly, these cannot be easily spotted by examining the formal rules and regulations of a system. There is merit in linking procurement assessments to political economy structures which may help you identify perhaps second best but workable solutions where first best optimal arrangements are not feasible or realistic to achieve. In this case, a procurement packaging approach which is accounted more for tribal fractures and existing capacities of local contractors rather than a market s ability to respond or aggregated for economies of scale worked pretty well. In new procurement framework, World Bank has not only opened up to the principle of best-fit solutions it plans to institutionalize them. Learning from the packaging approach used in this project, practitioners can account for comparable economic, social, and political fractures that may be impede a procurement process Emergency projects need separate results framework. It is inappropriate for an emergency project in an insecure environment to implement a traditional outcome based results framework. An output approach (simple to carry out and providing useful feedback about progress during implementation) easy to carry out and interpret should be recommended in conflict, post-conflict recovery situation. The intangible benefits sometimes are far higher and cannot he captured through traditional indicators. Transport projects in conflict countries which are socially sound should go beyond traditional indicators to capture the full social value of the project Finding and retaining PMU staff in a country emerging from decades of civil war is challenging and impacts project continuity. The project director was changed four times along with other key staff and hindered implementation. On hindsight, it is perhaps best not to set up a separate PMU but train WS&D staff to implement the project. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners Please see Annex 7 for detailed comment from Borrower. No issues have been raised by the Borrower. However, the suggested lessons have been noted. a) Borrower/Implementing agencies (b) Cofinanciers/Donors (c) Other partners and stakeholders (for example, NGOs/private sector/civil society) 24

36 Annex 1. Project Costs and Financing (a) Project Cost by Component (in US$, millions equivalent) Components Appraisal Estimate (US, millions) Pak Rs. Actual/Latest Estimate (US$, millions) Percentage of Appraisal A. Infrastructure Building (a) Civil Works Component ,345,560, (b) Relocation of Utilities, Land Acquisition and Resettlement ,574, B. Project Management (c ) Consulting Services ,086, Incremental Operating Cost ,465, TOTAL ,601,687, (b) Financing Source of Funds MDTF for Crisis Affected Areas of NWFP/FATA/Balochistan Type of Cofinancing Appraisal Actual/Latest Estimate Estimate (US$, millions) (US$, millions) Percentage of Appraisal

37 Annex 2. Outputs by Component 1. The subject project has two components comprising of the following: Component 1: Infrastructure Building 2. This component includes about 50 kms of motorable rural access roads with associated relocation of utilities, land acquisition, and resettlement whose construction have been completed. Following roads have been constructed with related structures under the project and the defects liability period of one year of these roads have been started from 12th December 2015: (a) Bajaur Agency (i) (ii) (iii) Kharkano to Kulala Road ( Km): The existing road was a dirt track of 2.0 m to 3.0 m in width which starts from Khar-Zulam Road and ends at Kulala-Arang Valley. The Construction of this road was extremely necessary because it served approximately half the population of Tehsil Salarzai, Utman Khel and some population of Khar. Bado to Serisar Road ( Km): The road was resurfaced (blacktopped) as the original road constructed as a shingle road by in was in an un-satisfactory condition and the inhabitants of the area were facing great hardships. After blacktopping of this road the population of Tehsil Utman Khel and Barang gets benefit by having better communication facilities. Dak Qilla Bypass Road (2.832 Km): This Road connects Khar-Zulam Road with Khar-Nawgai road. The old road was mostly dirt track and missing in parts. After construction of this road, it now serves as Khar By pass Road and the traffic load on the busy Bazaar of Agency Head Quarter has been minimized. (b) Orakzai Agency (i) (ii) Largari Tan to Dop Sana Road (1.650 Km): There was no blacktop (BT) road and the people of the area were using dirt tracks. After construction of this road, the transportation problems of the people have been solved. They are now able to bring their crops / vegetables and other commodities to the local and nearby markets. Dargai to Shamim Jan Killi Road (1.333 Km): This road starts from Main Kadda Kurez Road and lead to Shamim Jan Killi via Dargai. There was no BT road and the people of the area were using dirt tracks. After construction of this road, the transportation problem of the people has been solved. 26

38 (iii) (iv) Palosia to Hakim Killi Khawar Road (2.255 Km): This road also starts from main Katcha Packa to Mitta Khan Palosia road via Khawara Baba Lamso to Hakim Khan Killi. On completion of this road, the inhabitants of the area have been facilitated with better communication facilities and their economic conditions will be improved. Dargai to Ghakhi to Sharo Doghal Road (1.955 Km): This road starts from Main Kadda Kurez Road and lead to Shamim Jan Killi via Dargai. There was no BT road and the people of the area were using dirt tracks. After construction of this road, the communication problem of the people has been solved. They are also be able to bring their crops/vegetables and other commodities to the local and nearby markets. Component 2: Project Management (a) Detailed Engineering Design, Economic Analysis, Environmental and Social Safeguards Instruments, Bid Documents, and PC-I (US$0.175 million): This sub-component financed a professional firm engaged for preparing detailed engineering design, economic analysis, environmental and social safeguards documents, bid documents, Planning Commission Proforma I (PC-I). (b) Contract Administration and Construction Supervision (US$0.475 million): This sub-component financed a professional firm engaged as the Engineer responsible for Contract Administration and Construction Supervision. The firm would be fully empowered as the Engineer in accordance with International Federation of Consulting Engineers (FIDIC) stipulations. (c) Environmental and Social Safeguards (US$0.05 million): This sub-component has financed costs associated with independent monitoring of environmental parameters and 3 rd Party Validation of implementation of the EMP/RAP activities. (d) Other Project Management Activities (US$0.4 million): This sub-component provides support to cover salaries, allowances, per diem of staff hired for the project, operation, and maintenance cost of office buildings, office equipment, bank charges, advertising expenses, temporary lodgings, and vehicles for the purposes of the Project. 27

39 Annex 3. Economic and Financial Analysis A. Introduction 1. This project involves the civil works of seven different rural roads constructed in Federally Administered Tribal Areas (FATA). Out of these seven roads, three have been constructed in Bajaur Agency and four have been constructed in Orakzai Agency. The project has been executed with the financial assistance of World Bank administered Multi- Donor Trust Fund (MDTF). The total length of these seven rural roads is about 50 Km. (42 Km in Bajaur Agency and 8 Km in Orakzai Agency) The aim of the project was to improve connectivity in Bajaur and Orakzai Agencies of FATA through construction of these allweather-paved rural roads on existing non-motorable earthen tracks. 2. The World Bank is currently preparing the Implementation Completion and Results Report (ICR) of FATA ERRP. An economic analysis of the investments made under this grant has been carried out to provide an ex-post evaluation of the benefits and costs attributed to these investments, according to relevant World Bank Guidelines. In this regard, an economic analysis of investment in a roads infrastructure sub-project completed under this project has been carried out. 3. The Project Development Objective (PDO) is to increase year round access for rural population in the conflict-hit areas covered by the Project. The project aimed to achieve this objective by improving connectivity in the Bajaur and Orakzai Agencies of FATA through construction of all weather-paved rural roads on existing non-motorable earthen tracks. The objectives of the project are to improve the existing rural roads through an efficient road network, reduction in travel time and vehicle operating costs, reduction in the fatal accidents and have generated economic opportunities and thus reducing the poverty by engaging local personnel and enhancing market activities. 4. Keeping in view, all the aspects of Hydrology, Traffic and Future requirements etc., the improvement and rehabilitation of the roads have been carried out. This economic analysis has been carried out with the completion costs and updated figures of the traffic for year 2016 after the construction and improvement of the road network. B. Methodology 5. The methodology used follows standard practice of comparing life cycle road agency and user costs with and without project, using Net Present Value (NPV) and EIRR at 12 percent discount rates as decision criteria. The Highway Development and Management Model of the World Bank, HDM-4 has been used for analysis. The main assumptions about maintenance and improvement strategies with and without project are discussed below. 6. For the existing roads being earthen tracks, the roughness level of existing road is taken as 20 IRI, only routine maintenance (do nothing) is provided for without project case. In recognition of the prevailing maintenance, standards in roughness levels are allowed to deteriorate up to the maximum value. 28

40 7. For improvement strategy (with project case), one treatment has been tested with improvement and rehabilitation of roads with improved drainage provisions with asphaltic concrete surface. The initial treatment will give IRI levels of 3.0 with Asphalt wearing Course. The maintenance strategy assumes that after 10 years, a 50 millimeters (mm) overlay will be added with Asphaltic Concrete to reduce the IRI back to 3. It is assumed that after improvement, routine maintenance will also be carried out and will include potholes being completely patched, ditch cleaning will be carried out on regular basis, any safety measures disturbed will be refurbished, and grading of shoulders will be carried out at regular intervals especially after the monsoon. For the overlay purpose (periodic maintenance), schedule maintenance strategy (after 10 years) has been adopted. C. Road Data 8. The condition. Traffic data, costs etc. of the existing roads has been taken from the Project Completion Report of the subject project. General Road data is given in table 3.1 below: Table 3.1. Road Data Area/Tow n Road Class No. of Road Section s Terrain Lengt h Km AADT Traffi c (2014) AADT Traffi c (2016) Conditio n Operatin g Speed Roads in Bajaur and Khar Agencies Rural Road s 7 6 Hilly/Rollin g 50 Nil 3778 Earthen Tracks of Width m Km per hr. 9. Traffic surveys have been carried by the Project Management Unit on the road sections from dawn to dusk for 4 days. The classified traffic count is shown in table 3.2. Table 3.2. Traffic Data for Project Road (Year 2016) 7 Description Car Wagon, Truck 2xl. Total Motorized Pickup Tractor Road in Bajaur and Orakzai Agencies At the project preparation stage analysis of 5 roads was carried out. 7 Data used from the PC-I of the subject project. 29

41 Traffic Growth Rate Assumptions 10. Following graph shows the growth of total number of Vehicles on Road by category for the past 22 years in Pakistan from 1991 to The ACGR (Annual Compound Growth Rate) from this data is shown for each category of vehicle 8 : Figure 3.1. Growth of Vehicles on Road in Pakistan Vehicles on Road in Pakistan ( ) Others Total (LCV+HCV) 8% 1% Tankers Water 5% Tankers Oil 5% Mcy/ Scooter 9% Motor Car 11% Tractor 7% M.Cab / Taxi 8% Trucks 5% Buses 5% Ambulance 4% Station Wagons 7% Pickup 8% Jeep 5% D.Van 5% Motor Rick. 5% 11. JICA did a study of traffic growth rates in Their projections, based on historical data, were a growth rate of 6.8 percent for passenger vehicles and 9.5 percent for cargo vehicles. Over the period from , GDP increased by 5.5 percent per annum. The elasticity of passengers traffic with respect to GDP was determined to be 1.18 and for cargo road traffic The coefficient of correlation (R 2 ) was found to be Data for GDP for the last 61 years ( ) shows that on average the growth rate has been increasing at a rate of 4.98 percent per annum which would indicate a traffic growth rate of 5.5 percent and 8.6 percent for passengers and freight traffic respectively. 12. The elasticity of traffic growth with population and industrial production has also been studied. In a 1983 JICA study traffic growth was shown to have a direct elasticity of 2.25 with the growth in population. Thus with a historical growth rate of 2.99 percent in population, the overall traffic should have grown at a rate of 6.73 percent annually. The growth of goods traffic with industrial growth showed an elasticity of Thus with a historical growth in production of approximately 6 percent, cargo traffic should increase at 6.54 percent annually. In 1995 JICA conducted another study, which recommended growth rates from 1997 to 2006 of 5.9 percent for passengers and 5.5 percent for goods. 8 Economic Survey of Pakistan, The Study on National Transport Plan in Islamic Republic of Pakistan, JICA, May, 1983 second in March 1988, third in February

42 GDP Growth in the Country 13. Historical data for GDP shows that on the average GDP has grown at 4.98 percent per annum over sixty years. In the 1980 s the GDP growth was at an average of 6.4 percent per annum and in the 1990 s at 4.6 percent per year. It has been found that the historical trend in the number of vehicles in Pakistan fits well with the trend in GDP. Economic growth as measured by GDP, in the last 13 years averaged below 4.5 percent which is lower than the past decades. This has been due to many reasons including a low economic activities and external and internal political situation. Following figure gives the historic GDP for the last 13 years. Figure 3.2. Historical Growth of GDP in Pakistan 14. Historical growth of the GDP in last 13 years is remained around 4.41 percent. In particular, the future growth rates for the project have been based on the detailed analysis recommended in the study sponsored by the World Bank for NHA Pre-feasibility Study for National Highway N-5, 1998, Louis Berger Int l Inc., USA. As per the study, it can fairly be assumed that the realistic scenario for overall future growth can be estimated at 1.0 elasticity w.r.t GDP and the pessimistic and optimistic scenarios as 0.8 and 1.2 respectively. Adopted Growth Rates 15. For this study, we have assumed a realistic/pessimistic estimate of elasticity of 0.9 with respect to historic GDP of Pakistan considering the roads fall in FATA and have assumed an overall growth rate of 4.0 percent. In relation to growth in passenger traffic, GDP has been found to have an elasticity of however, in FATA, we have assumed elasticity of 0.9 for passenger traffic also. Thus with a historical GDP growth of 10 JICA, Final Report, Pg. 144, (Avg. figure),