Pakistan: Road Sector Development Program

Transcription

1 Validation Report Reference Number: PCV: PAK Project Number: Loan Number: 1891 December 2010 Pakistan: Road Sector Development Program Independent Evaluation Department

2 ABBREVIATIONS ADB Asian Development Bank ADF Asian Development Fund IED Independent Evaluation Department MOC Ministry of Communications NHA National Highway Authority OCR ordinary capital resources PCR program completion report RRP report and recommendation of the President RSDP Road Sector Development Program TA technical assistance NOTE In this report, $ refers to US dollars. Key Words pakistan, adb, asian development bank, highways, roads, lessons, independent evaluation department, performance evaluation, sector development program Director H. Hettige, Independent Evaluation Division 2, Independent Evaluation Department (IED) Team leader R. Bolt, Advisor, Office of the Director General, Southeast Asia Department Team members R. Lumain, Senior Evaluation Officer, Independent Evaluation Division 2, IED C. Roldan, Assistant Operations Evaluation Analyst, Independent Evaluation Division 2, IED In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, the Independent Evaluation Department does not intend to make any judgments as to the legal or other status of any territory or area.

3 PROGRAM COMPLETION REPORT VALIDATION A. Basic Program Data PCR Validation Date: December 2010 Program Number: Loan Number: (program loan component only) Approved Actual Program Name: Road Sector Development Total Program Cost 83.2 Program ($ million): Country: Islamic Republic of Pakistan Loan/Grant ($ million) Sector: Transport (Roads Transport) Total Cofinancing ($ million): and Information and Communications Technology ADB Financing ADF: 0.0 Borrower ($ million): ($ million): OCR: 50.0 Beneficiaries ($ million): Cofinanciers: None Others ($ million): Approval Date: 19 December 2001 Effectiveness Date: 2 May Jun 2002 Signing Date: 1 February 2002 Closing Date: 30 Jun Jun 2007 Project Officers: Name: H. Carlsson H. Masood T. Nakazaki S. Muramoto O. Norojono Location: ADB headquarters ADB headquarters ADB headquarters ADB headquarters ADB headquarters From To Validator: Quality Control/ Peer Reviewers: T. F. Jones III, Consultant R. Bolt, Advisor, Office of the Director General, Southeast Asia Department 1 R. Lumain, Senior Evaluation Officer, IED2 Director: H. Hettige, IED2 ADB = Asian Development Bank, ADF = Asian Development Fund, OCR = ordinary capital resources, PCR = program completion report. B. Program Description (summarized from the report and recommendation of the President) 2 (i) Rationale. Policy reforms were critically needed to increase the efficiency of Pakistan s road sector at the national and provincial levels. Reforms initiated at the national level provided the framework for overall sector development involving (a) increased capital investments to develop an efficient transport network; (b) an efficient internal network to support regional cooperation; (c) increased resources and improved institutional mechanisms to address inadequate road rehabilitation and maintenance; (d) strengthened road network institutions in professional and management skills, and assistance in establishing an effective policy and regulatory enabling environment; (e) initiatives to improve traffic management and road safety; and (f) support for measures to control excessive axle loads and vehicle weights. In this context, the Asian Development Bank was to finance a portion of the adjustment costs associated with the national road sector policy reforms. (ii) (iii) Impact. The Road Sector Development Program (RSDP) was to improve management of the provincial and rural access roads network. Policy and institutional changes in the National Highway Authority (NHA) would result in the effective delivery of services in the road sector. This was then expected to help reduce rural poverty and contribute to national economic growth. Objectives or expected outcomes. The expected outcomes were (a) preservation of road assets, (b) governance and institutional development, (c) operational efficiency improvements, (d) technical 1 Formerly with the Independent Evaluation Department at the time this validation was carried out. 2 ADB Report and Recommendation of the President to the Board of Directors: Proposed Loans to the Islamic Republic of Pakistan for the Road Sector Development Program. Manila (Loans 1891-PAK,1892-PAK, and PAK(SF), for $200 million, approved on 19 December).

4 2 efficiency improvements, (e) road safety, (f) private sector participation including involvement of knowledge-based industries, and (g) poverty reduction and community development. Specific program targets include (a) improved network condition through better institutional management and maintenance practices, (b) improved local government capacity to become involved in the management of road networks within their jurisdiction, (c) reduced burden on public sector resources by alternative financing mechanisms, (d) improved access and lower transport costs, and (e) reduced incidence and severity of road accidents. (iv) Components and/or outputs. The RSDP comprised two components to address structural inefficiencies in the national and provincial road network: national road sector policy reform, and a provincial sector development project. On the former, the RSDP was to support the continuation of the reform process with the following elements: (a) rationalizing staffing; (b) developing the NHA Code, financial manual, and standard operating procedures; (c) enhancing staff accountability in financial management; (d) prioritizing investment based on economic principles; (e) self-financing maintenance needs through toll receipts; (f) preserving assets through an axle-load control program involving installation of weigh stations and enhancing enforcement procedures; (g) ensuring transparency in procurement processes; and (h) adopting a comprehensive approach to road safety, including public awareness and stakeholder consultations. C. Evaluation of Design and Implementation (program completion report assessment and validation) (i) Relevance of design and formulation. Conception of the RSDP was generally appropriate. The Asian Development Bank (ADB) assistance comprised three loans using a sector development program modality. One at $50 million equivalent (Loan 1891-PAK) was for development of the policy reform program, while the second and third (footnote 2) were for the improvement of provincial roads in Pakistan s Sindh Province. This validation covers the program completion report (PCR) on the national policy component of the RSDP, which covered the implementation of the national road sector reform program. 3 The PCR adequately assesses the relevance of program design and formulation. The policy reform component of the RSDP (and its timing) is very relevant. Its stated aim and purpose are consistent with the government s development strategy and the country assistance program of ADB for Pakistan from 2001 to This ADB-financed policy loan was mainly intended to support continuing efforts of the NHA to adapt business processes to those of contemporary road agencies. 4 Its program interventions were built around a comprehensive set of five policy themes involving (a) transport policy, (b) road sector resource management, (c) road asset preservation, (d) institutional efficiency improvement, and (e) road safety. These themes, which were consistent with the policy themes of the national policy reform program, were highly relevant and remain so. While the design and formulation of the RSDP was suitable, there was room for improvement. In particular, the PCR frankly comments that the policy loan should have been implemented with accompanying technical assistance (TA), particularly for the transport policy and road safety themes (section C[iv]). (ii) Program outputs. The accomplishments of the RSDP policy loan relative to the five themes above are summarized in Table 1 (see also PCR, paras ). The final status of associated policy actions (for tranche releases) relative to the national policy reform program policy matrix are described in Appendix 4 of the PCR. Of the twenty-four policy actions for tranche release, twenty (83.3% of total) were either complied or generally complied with. All policy actions for transport policy (1), road sector resource management (8), and road asset preservation (5) themes were carried out. For the institutional efficiency improvement theme, three were complied with while a fourth is judged to be generally complied with. Implementation of the road safety theme was an issue, with only two of 3 ADB Completion Report: Road Sector Development Program in the Islamic Republic of Pakistan. Manila. The closing date for Loans 1892 and 1893 was extended from 30 June 2007 to 30 June An ongoing national policy reform program largely reflected the commitment of the NHA to change and adapt. It is aimed at building the capacity of the NHA and transforming it into an efficient service provider to road users (i.e., to ensure safe and comfortable travel on the national highways).

5 3 the six tranche release conditions either fully complied with or generally complied with. 5 Two were partly complied with, while the balance was not complied with because of lack of resources. Policy Theme Transport Policy Road Sector Resource Management Road Asset Preservation Institutional Efficiency Improvement Table 1: Assessing Program Outputs Assessment Achieved. Transport policy reform involved the development of a long-term national transport policy framework that will combine strategic vision and ensure sustainability. According to the PCR, policy actions were carried out and objectives achieved. Achieved. The policy s objectives are being achieved and actions have been implemented. The NHA has prepared a long-term financial business plan for for approval by the government. To help ensure that national highway maintenance is sustainable, most of the national highways have been brought under a tolling regime, which generates revenues for this purpose but this is still inadequate. The toll rate is under review with a view to help increase revenues to the needed level. Generally achieved. Some progress has been achieved with the enactment of a national highway safety ordinance. The National Highway and Motorway Police was created under this ordinance. Of 112 planned vehicle weighbridges, 42 have been constructed and are operational with noticeable effect on the motorways. Further actions regarding axle weight control are being considered for implementation in the future. Generally achieved. These were focused on the NHA, which operates as a complete business entity that executes two areas of operations: (a) development of the national highway network, and (b) maintenance and upgrading of the developed network. The NHA outsources functions such as (a) maintenance, (b) construction, (c) toll collection, and (d) design and feasibility studies. The PCR relates that policy actions have been carried out and objectives achieved, while the institutional reform and corporate strengthening program are under way. The NHA initiated rightsizing and reduced staff in 2003 and later increased it in accordance with increasing work demands. Road Safety Partly achieved. The National Highway Safety Council was established in 2001 as part of the road safety theme, but it has become a dormant organization without a permanent secretariat because funds were not available. A road safety cell was established and road design safety audits were introduced as required by external funding agencies. Efforts by the National Highway and Motorway Police were partly responsible for the reduced number of road accidents. Inadequate funding retarded some policy actions from being implemented, as did a lack of commitment at higher government levels and a lack of coordination among institutions involved. NHA = National Highway Authority, PCR = program completion report. Source: Independent Evaluation Department staff based on Asian Development Bank Completion Report: Road Sector Development Program in the Islamic Republic of Pakistan. Manila. (iii) Program cost, disbursements, borrower contribution, and conformance to schedule (as relevant to project performance). As this was a program loan, it was disbursed in two tranches based on fulfillment of tranche conditions set out in the national policy reform program policy matrix (section C[ii]). As the ADB loan portion was fixed at loan approval, the total amount was disbursed with two adjustments. A total of $486,172 was debited to ADB as the front-end fee, while the total loan value (denominated in Y6,079.3 million or $50 million equivalent) increased to $50,763,300 because of depreciation of the US dollar versus the yen. Since the loan was to be used to pay for approved imports, none of the loan went directly to financing the reform program. The total appraised program cost was estimated at $83.2 million (PCR, Table 1). The PCR does not report on the actual costs incurred by the government, as much of this would have been in kind. The loan was approved on 19 December 2001 and closed on 30 June Although disbursement of the first and second tranches was delayed, the loan was closed as originally scheduled. The first 5 Three policy actions (on the road safety theme) required for the release of the first tranche were reportedly not fully met at program completion. On the other hand, an ADB communication with the government on the status of the national reform program matrix (dated August 2002) do indicate these were complied with at the time. To some extent, the release criteria seem to have been subject to meeting release schedules rather than tranche release requirements.

6 4 tranche in the amount of $34,065,602 equivalent was disbursed on 28 June 2002, the day after the loan was made effective. Loan release was delayed by 4 months compared with the original program period. 6 The second and final tranche was disbursed in two parts. The first part in the amount of $15,000,000 equivalent was disbursed on 14 November 2003, while the second part in the amount of $1,210,986 equivalent was disbursed on 28 November 2003, thus completing full disbursement under the program. There was a 17-month delay in disbursing the second tranche because of delayed government action on release conditions. (iv) Implementation arrangements, conditions and covenants, related technical assistance, and procurement and consultant performance. As identified at appraisal, the program was implemented through the executing agency, the Ministry of Finance (MOF), with the support of the Ministry of Communications (MOC) and the NHA. In the conditions and covenants section, section II.G of the PCR states that the delays in disbursing the two tranches were due to the government s delay in completing requirements for loan effectiveness (footnote 6) and completing the second tranche policy actions. Conditions for tranche release are given in Appendix 5 of the PCR. A review of the covenants reveals that only 1 of the 18 program-specific covenants was partly complied with. The partly complied with covenant involved the provision of funds in a timely manner for carrying out the program. Funds were not always readily available, particularly for implementing the road safety actions. The rest of the covenants were either complied with (8 covenants) or generally complied with (9 covenants). From this validation s point of view, the assessment of generally complied with seems to mean that while attempts were made to be in compliance, the actual compliance was less than the intended spirit of the covenant. For example, coordination among agencies was often poor, target dates were often missed, actions met at one point in time were not continued, and some actions were not initiated during the program period. There was no TA attached to the loan and no consultants were required. In hindsight, suitable TA should have been included, particularly for the transport policy and road safety themes (section C[i]). For example, transport policy TA was later sponsored upon government request, and this contributed to the national transport policy document (PCR, para. 59). 7 Procurement of the various goods and materials purchased by the government followed normal government procurement procedures. (v) Performance of the borrower and executing agency. In general, the program loan was implemented as conceived. ADB s program performance ratings for development objectives and implementation progress from June 2002 to June 2007 were satisfactory. The performance of the borrower and the MOF was satisfactory in terms of using the loan amount for the purchase of supplies and materials. The three agencies involved (MOF, MOC, and NHA) performed satisfactorily in establishing and implementing policy reforms. The NHA remains committed to continuing improvements in institutional efficiency and reforms in the management of the national road network (PCR, para. 63). As explained above, only one of the 18 loan covenants was partly complied with. Eight of the loan covenants were complied with, while the balance was judged generally complied with (section C[iv]). To a certain extent, this observation implies that some of the agencies responsible for implementing the program were not fully committed to instituting reforms over the 5-year implementation period. Several factors also detract from borrower performance. First, implementation of some of the road safety components had not had the expected impact and sustainability. This was partly caused by lack of funds. In general, the PCR rightfully points out that the borrower did not fully analyze the resources 6 The report and recommendation of the President stated that the period of the policy loan was from January 2002 to June The PCR indicated that the original schedule of first tranche release was February 2002 and the second tranche release on June Compliance with the requirements for loan effectiveness, as a condition for the first tranche release, was delayed. 7 ADB Technical Assistance to the Islamic Republic of Pakistan for Transport Policy Support. Manila (TA 4400-PAK, for $290,000, approved on 30 September).

7 5 required for implementation of some components (PCR, para. 58). These were started but could not be continued because of lack of funds. Their continuity is now dependent on establishing a stable source of funds to finance the components. Second, some basic problems in the sector related to poor interministerial coordination and lack of needed legislation prevailed during implementation. From a short-term perspective, this validation agrees with the PCR s overall performance rating of satisfactory. However, in the longer term, the government must ensure that the policy reforms are continued for sustainable operations. As cited above, continued implementation of some of the reforms have already suffered from lack of funds (section D[iv]). (vi) Performance of the Asian Development Bank. The PCR assessment of ADB performance is contradictory. In para. 33 of the PCR, it states The Program was administered by different officers in ADB and monitoring could have been better. Although the tranche conditions were satisfied, ADB s review could have been more periodic to avoid partial compliance actions by the EA. Para. 67 follows these up with The program administration and review by ADB should emphasize close monitoring of the executing agency and ensure continuity of staff in the administration team. The PCR goes on to emphasize that ADB provided assistance in a timely manner, which helped to facilitate actions taken by the government agencies in complying with tranche conditions required for the release of loan tranches (PCR, para. 33). This leaves questions as to whether ADB provided adequate assistance during program implementation, especially in terms of field supervision. While ADB headquarters support appears to have been sufficient to ensure that project management issues were addressed, it is unclear from the PCR whether field-level dialogue was adequate. The PCR assessed ADB performance satisfactory. This validation considers ADB headquarters-based administrative support to have been adequate before and after tranche release. However, supervision missions fielded were seemingly not sufficient. During the 5-year implementation period, ADB undertook only three review missions led by two different staff members. 8 This validation considers this level of supervision inadequate for a program. The program loan mission leader had no part in program implementation. Overall, taking into consideration the program s timely completion and successes in a challenging reform process, this validation agrees with the PCR rating on ADB performance of satisfactory. D. Evaluation of Performance (program completion report assessment and validation) (i) (ii) Relevance. The PCR rated the program very relevant, as the aims and purposes were consistent with the government s development strategy and ADB s country assistance plan (PCR, para. 34). All components were appropriate, and program inputs were sufficient to generate the planned outputs. This validation agrees that the program was satisfactorily conceived and is reasonably comprehensive (section C[i]). In addition, to be effective, program-related achievements need to be sustainable. In this sense, the program remains important and very relevant to the country s road sector. The NHA is said to be committed to continuing improvements in institutional efficiency and reforms in the management of the national road network. The PCR rightfully points out that this policy program should have been implemented with TA, particularly for the transport policy and road safety components (section C[iv]). In particular, implementation of the road safety components did not meet expectations, while the transport policy theme reportedly benefited from a subsequent stand-alone transport policy TA project. This observed design weakness detracts from a full relevance rating at completion. Taking into consideration the overall program quality-at-entry at both approval and completion, this validation rates the program relevant. Effectiveness in achieving outcome. The PCR states that the lack of resources brings the continued implementation of all aspects of the policy reform into doubt. Therefore, while the initial efforts to set up new policies might have been largely successful, since they are largely on paper or the responsibility of newly established agencies, any possible positive effect might be lost as if nothing 8 These comprised a consultation mission (February 2002), review mission (November 2002), and a special program administration mission (November 2003). The November 2002 mission was a multiproject mission covering three other loans and a TA project.

8 6 ever happened without proper follow-up. The validation agrees with the PCR rating of effective inasmuch as the desired effect is still in progress. At completion, the result for each of the major themes is summarized in Table 2. Table 2: Assessing Effectiveness in Achieving Program Outcomes Policy Theme Rating Assessment Transport Policy Effective ADB intervention was successful and effective in developing a draft transport policy document for submission to the cabinet after interministerial discussion. While effectiveness is relatively positive up to program completion, the proof will come after the cabinet and further approvals and adequate funding. However, its approval is lingering for a long time due to evaluation required by many ministries and departments. a Road Asset Management Road Asset Preservation Institutional Efficiency Improvement Road Safety Effective Effective Effective Less Effective The road asset management system has been developed and helps ensure that the road sector agencies carry out asset management and preservation effectively. However, budget allocations need to be rationalized through revenue sources, so that an appropriate balance between revenue sources and appropriate allocation for the roads sector is maintained. The NHA, as one of the major road agencies, can now supplement government revenues through off-budget initiatives such as tolls on the national highway system. Thus, the NHA is more capable of undertaking both highway upgrading and maintenance. An axle load control program was introduced to protect and increase the service life of road assets. While legislation was enacted to restrict maximum axle loads on the national highway network, there is no such legislation on the provincial highways. The National Highway and Motorway Police (NH&MP) were responsible for identifying the operational parameters for enforcement. The policy reform is being monitored through the development of awareness programs and measuring axle load levels at regular intervals on the network. An enforcement procedure and institutional framework were established. Associated penalties for violation of load levels were set and are being levied. The NHA has moved toward becoming a commercial agency with a view to economy and efficiency in operations by operating on a fee-for-service basis. More recently, the NHA changed its organizational structure and is involved in institutional capacity building under follow-on ADB-financed programs. b Achieved outcomes here will probably be sustained considering follow-on assistance. Concerning the road safety theme, various measures were introduced to help reduce road accidents in Pakistan. The NHA and NH&MP are improving their knowledge, capability, and personnel training. Community awareness and stakeholder participation is being pursued, as well as professional-level accident recognition and implementation of remedial measures such as accident blackspot c identification and rectification. The PCR states several positive efforts in this section, but in section C(v) shortcomings are identified such as the implementation of some of the road safety policy components, which have not had the expected impact and sustainability. This was due to lack of funds, poor interministerial coordination, and lack of needed legislation. ADB = Asian Development Bank, NHA = National Highway Authority, PCR = program completion report. a A comprehensive National Transport Policy for Pakistan was completed on 30 May The final document was submitted to the executing agency for further processing and approval by the government. A departmental review was conducted until early The executing agency is still coordinating with the Planning Commission and the Cabinet for their final review and approval. Previously, several attempts were made by the government to articulate a comprehensive transport policy that encompasses all subsectors. The draft policy document was presented to the interministerial National Trade Corridor (NTC) Task Force. The NTC core group will take up the policy development process further and to formalize it as the government document. Implementation of the policy will be coordinated by the Planning Commission. b c ADB Report and Recommendation of the President to the Board of Directors: Proposed Multitranche Financing Facility and Loan to the Islamic Republic of Pakistan: National Highway Development Sector Investment Program. Manila (Loan 2210-PAK[SF], for $3 million, approved on 13 December and Loan 2231-PAK [Tranche 1], for $180 million, approved on 5 February 2006); and ADB Report and Recommendation of the President to the Board of Directors: Proposed Multitranche Financing Facility and Technical Assistance Grant to the Islamic Republic of Pakistan: National Trade Corridor Highway investment Program. Manila (Loan 2400-PAK, for $545 million, approved on 17 December and Loan 2401-PAK[SF], for $10 million, approved on 17 December). In terms of road safety management, this refers to a place where road accidents have historically been concentrated. Source: Independent Evaluation Department staff based on ADB Completion Report: Road Sector Development Program in the Islamic Republic of Pakistan. Manila.

9 7 (iii) Efficiency in achieving outcome and outputs. In terms of implementation efficiency, the policy loan cost was in line with approved loan amount while tranche releases were delayed. 9 The first tranche was delayed by only 4 months while the second tranche disbursements were late by 17 months because of delayed government action on release conditions. Despite these delays, the loan was closed as originally scheduled. The PCR rated the program efficient. Based on the PCR findings at completion, this validation agrees with that assessment. Table 3 briefly summarizes PCR assessments on program efficiency under the reform program. Table 3: Assessing Efficiency in Achieving Program Outcomes Policy Theme Rating Assessment Transport Policy Efficient The transport policy has gone through several drafts, the first to qualify for tranche release, and subsequent revisions have been made to reflect the views of the various agencies involved. MOC has finalized the national transport policy, but final approval has not been achieved. Road Asset Management Road Asset Preservation Institutional Efficiency Improvement Highly Efficient Efficient Less Efficient The National Highway Council has approved a medium-term road resource management plan. This initiative seems to be working efficiently as evidenced by the condition of the national highway and motorway network, which has improved steadily since Actions have been undertaken to improve road asset preservation. Weigh stations have been installed at strategic locations throughout the national highway system and on motorways and were integrated with the fine collection system. According to the PCR, enforcing road safety and axle load control on national highways is being extended regularly in phases. The PCR does not suggest exactly how far these phases have progressed. To achieve institutional efficiency improvement, the NHA has undertaken rightsizing exercises but with additions and subtractions, there has been little effect. NHA standard operating procedures have been revised and updated. The NHA has acquired financial management expertise, which should improve the efficiency of in-house financial management capability. Road Safety Inefficient The lack of achievements under the road safety theme is discussed in section C of the PCR. Lack of funds is cited as the main reason for inaction. This initiative was reportedly being pursued under the Northwest Frontier Province Road Development Sector Project, but was dropped by the government citing difficult conditions. a MOC = Ministry of Communications, NHA = National Highway Authority, PCR = program completion report. a Asian Development Bank (ADB) Report and Recommendation of the President to the Board of Directors: Proposed Loan to the Islamic Republic of Pakistan for the Northwest Frontier Province Road Development Sector and Subregional Connectivity Project. Manila (Loan 2103-PAK, for $296.2 million, approved on 18 November and Loan 2104-PAK[SF], for $5 million, approved on 28 November). Source: Independent Evaluation Department staff based on Asian Development Bank Completion Report: Road Sector Development Program in the Islamic Republic of Pakistan. Manila. (iv) Preliminary assessment of sustainability. While the PCR does not specifically rate sustainability in the sustainability section (PCR, paras ), the impacts section (PCR, para. 54) states that the policy program is expected to result in a sustainable impact over the long term (considering the effectiveness and efficiency of the outputs and outcomes). In broad terms, the PCR explains sustainability this way (para. 50): The government is trying its best to sustain the actions initiated under this Program. Because of the lack of resources and TA, some actions were not implemented and some objectives could not be achieved. Additional external assistance would be necessary for complete implementation of all policy actions and establishment of alternative funding source for their continuity. This would improve the Program s performance, sustainability, and intended benefits. The government committed itself to supporting this program before loan approval, but has apparently failed in meeting its obligations (section C[v]). Aside from external financial assistance, there seems to be little case for an expectation of high sustainability. The NHA and other concerned agencies will likely sustain actions where they see a clear benefit and will generally eschew actions deemed less beneficial or politically unacceptable. 9 Program loans do not fit the typical measure of efficiency as indicated by the economic internal rate of return of an infrastructure project.

10 8 Accordingly, this validation sees mixed results in the long-term sustainability of the program actions, i.e., depending on the interest of the donors in pursuing specific policy actions. In the likelihood that program sustainability is to become supply-driven or donor-dependent, this validation rates the sustainability less likely to be sustainable. (v) Impact (both intended and unintended). The PCR states many positive impacts such as an increased maintenance allocation now covering about 50% of the development budget, up from 38% (PCR, para. 51 and Appendix 4). The installation of weigh stations at strategic locations on national highways has been partly accomplished. However, as stated in the PCR, the main issue outstanding is full enforcement of legal loading, which is not fully operational. Maintenance planning is better executed and a maintenance manual developed. On the negative side, the government did not create the necessary organizations for the efficient treatment of road accident data. Regarding road infrastructure, stakeholder consultations are being carried out. The PCR concludes that the policy program s socioeconomic impact can be perceived as moderate. This validation agrees with this rating. E. Overall Assessment, Lessons, and Recommendations (validation of program completion report assessment) (i) (ii) (iii) Overall assessment. The government and ADB recognized for some time that a comprehensive program of policy development and reform was necessary within the transport sector, particularly within the highways and roads subsectors. Understanding the existing policies or lack thereof takes considerable time and interaction between ADB staff and concerned government officials. This sector development program was well conceived and at the beginning was readily agreed by the government. At program completion, the objectives of the policies were achieved with most agreed policy actions being carried out, except for road safety. Accordingly, the PCR rated the policy loan satisfactory. However, such a program is only as effective as the government wishes it to be. The PCR continually refers to policy objectives as having been generally achieved. The conditional generally is not defined in the PCR but may imply something less than full achievement (section C[iv]). 10 For the program to be fully effective, the PCR states that the government should ensure that these initiatives are continued in the long term to ensure sustainable operations. Likewise, continued donor engagement in pursuing program actions will be critical for proper follow-up. Based on program information at completion, the overall progress made under difficult conditions, and the level of donor interest, the validation rates the program at the low end of successful. 11 Lessons. There are several important lessons cited in the PCR. Considering their importance, they are highlighted below. (a) In the implementation of an interministerial policy, the capacity of the executing agency to coordinate with all the involved ministries is crucial. The implementation structure should reflect the organizational arrangement existing within the government. One of the basic problems in the sector is the lack of coordination, collaboration, and cooperation among the government agencies involved. Since this is an important requirement for success, the reasons for this failure must be understood and ways to ensure coordination developed. (b) Support for the implementation of the policy program could have been achieved with TA, particularly for transport policy and road safety. ADB is discussing ways to support various components. Recommendations. If the national transport policy document is endorsed by the government, it would require additional assistance in the development of implementation strategy and development of subsector specific guidelines and regulations. ADB s support in the endorsement process and implementation would go further in achieving the goals set out in this intervention. The PCR also made a number of sound recommendations, and these are presented and/or summarized below. 10 For example, at program completion, the national transport policy formulated by MOC with assistance from ADB is being submitted for cabinet approval at some future unspecified date. 11 The program is assessed to be relevant, effective, efficient, and less likely to be sustainable. Its equivalent overall weighted average of 1.8 falls within the range of 2.7<S 1.6 for scoring successful projects/programs.

11 9 (a) (b) (c) (d) (e) Three first tranche conditions require further actions to ensure full compliance. The government should accelerate efforts to establish the accident reporting center, driver instructor/training school, and accident evaluation laboratory (PCR, para. 60). There should be a set of mechanisms associated with covenants to guide and ensure their compliance. There is a need to coordinate, through relevant government organizations, implementation of the road safety program on the entire road network. To improve the program s performance and help make it sustainable, additional assistance is needed in the form of loans, grants, and TA. For such a complex and interactive initiative to be undertaken comprehensively, it is essential that the executing agency keep involved staff engaged through the duration of the program. Similarly, there should be continuity of ADB staff involvement both in review frequency and staff. F. Monitoring and Evaluation Design, Implementation, and Utilization (program completion report assessment and validation) Monitoring was left to a consultation mission in 2002, a review mission in 2002, and a special program administration mission in To maintain an active dialogue, this input on the part of ADB was inadequate. The two missions in 2002 focused on tranche approval and had one mission leader, while the one in 2003 was led by another ADB staff. No missions followed, as the two tranches had been released upon general compliance with the covenanted reforms. G. Other (e.g., safeguards, including governance and anticorruption; fiduciary aspects; government assessment of the program, as applicable) (program completion report assessment and validation) Specific safeguard measures were not needed as part of the program. Better coordination was required at the highest levels of government. Corruption was not reported to be an issue. The government did not provide the necessary operation and maintenance funds for the many activities required to sustain reform. Poor governance must be concluded. H. Ratings PCR IED Review Relevance: Very Relevant Relevant Effectiveness in Achieving Outcome: Efficiency in Achieving Outcome and Outputs: Preliminary Assessment of Sustainability: Borrower and Executing Agency: Reason for Disagreement/Comments A weakness in program design relates to use of attached TA projects to support the transport policy and road safety themes (sections C[iv] and D[i]). Effective Effective At program completion, initial efforts to set up the new policies appear successful (although largely on paper). Efficient Efficient While tranche disbursements were delayed, the loan was financially closed as originally scheduled. Not Rated Less Likely Given funding constraints, sustainability of policy actions will be mixed (i.e., actions will be where government see a clear benefit and will generally eschew actions deemed less beneficial or politically unacceptable). In the absence of external assistance, there seems to be little case for optimistic expectations on sustainability (section D[iv]). Satisfactory Satisfactory PCR rating at completion confirmed based on program achievements amidst a challenging reform process. To sustain this, longer term government performance must ensure the policy reforms are continued. Performance of ADB: Satisfactory Satisfactory The reform process would have also benefited further from closer field supervision by ADB staff (section C[vi]). Impact: Moderate Moderate Overall Assessment: Satisfactory Successful The policy loan is rated at the low end of successful. Quality of PCR: Satisfactory

12 10 I. Comments on PCR Quality The PCR appears thorough and comprehensive. It generally conformed to IED guidelines. To a certain extent, this validation appreciates the PCR s candid approach in pointing out some of the program s shortcomings during implementation. In particular, this validation also takes note of the good lessons identified by the PCR and some of the good recommendations made. However, the lack of easily understood criteria for meeting reform requirements as covenanted caused some confusion in the PCR, as general compliance rather than specific compliance was acceptable. Where assessment of the program and the PCR proved difficult when general compliance was not clear, some questions emerged on program effectiveness. J. Recommendation for Independent Evaluation Department Follow-Up IED may want to perform a detailed evaluation of the sector development program given the many difficulties the policy loan faced and the difficulty that sustainability seems to be facing. K. Data Sources for Validation The sources for this validation report included the PCR; report and recommendation of the President; minutes of the Management Review Meeting, Staff Review Committee, and discussions of ADB s Board of Directors; as well as back-to-office reports of supervisory missions.

13 REGIONAL DEPARTMENT S RESPONSE TO THE PROGRAM COMPLETION REPORT VALIDATION REPORT On 14 December 2009, Director, IED2, Independent Evaluation Department, received the following response from the Transport and Communications Division, Central and West Asia Department. We have no comment.