Validation Report. Cook Islands: Economic Recovery Support Program Subprograms 1 and 2. Independent Evaluation Department

Transcription

1 Validation Report Reference Number: PVR-460 Project Numbers: and Loan Numbers: 2565 and 2946 November 2016 Cook Islands: Economic Recovery Support Program Subprograms 1 and 2 Independent Evaluation Department

2 ABBREVIATIONS ADB Asian Development Bank DMF design monitoring framework ERSP economic recovery support program FY fiscal year GDP gross domestic product IGF Infrastructure Governance Framework IMP Infrastructure master plan MFEM Ministry of Finance and Economic Management NSDC National Sustainable Development Commission PCR program completion report RRP report and recommendation of the President TA technical assistance NOTE In this report, $ refers to US dollars. Key Words adb, asian development bank, cook islands, economic, fiscal management, fiscal stimulus, inclusive growth, infrastructure investment, public sector management, validation, vulnerable population The guidelines formally adopted by the Independent Evaluation Department (IED) on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of IED management, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. The final ratings are the ratings of IED and may or may not coincide with those originally proposed by the consultants engaged for this report. In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, IED does not intend to make any judgments as to the legal or other status of any territory or area.

3 PROGRAM BASIC DATA Project Numbers , PCR Circulation Date 15 Sep 2015 Loan Numbers 2565, 2946 PCR Validation Date Nov 2016 Project Name Economic Recovery Support Program Subprograms 1 and 2 Sector and subsector Loan 2565 Loan 2946 Multisector Public Sector Management Public expenditure and fiscal management Theme and subtheme Economic growth Promoting economic efficiency and enabling business environment, widening access to markets and economic opportunities Safeguard categories Loan 2565 Loan 2946 Environment C C Involuntary Resettlement C C Indigenous Peoples C C Country Cook Islands Approved ($ million) ADB Financing ADF: 0.00 ($ million) Loan 2565 Loan 2946 Signing Date Loan 2565 Loan 2946 Project Officer OCR: (Loan 2565) 6.00 (Loan 2946) Actual ($ million) Total Program Costs Loan Loan 2565 (Subprogram 1) Loan (Subprogram 2) Borrower Beneficiaries Others Cofinancier Total Cofinancing Approval Date Effectiveness Date 13 Oct 2009 Loan Jan Mar Nov 2012 Loan Dec Oct 2013 IED Review Director E. M. Veve 29 Oct Sep 2013 W. Kolkma, IED1 Closing Date Loan 2565 Loan 2946 Location SPSO 31 Mar Dec 2013 From Oct Mar Jan 2014 To Jan 2014 Team members B. Graham, Senior Evaluation Specialist, IED1 F. Ahmed, Lead Evaluation Specialist, IED1 L. Ocenar, Evaluation Officer, IED1 B. Prakash, Consultant E. Breckner, Consultant ADB = Asian Development Bank; ADF = Asian Development Fund; IED1 = Independent Evaluation Department, Division 1; OCR = ordinary capital resources; PCR = project completion report; SPSO = Pacific Subregional Office in Suva, Fiji.

4 2 I. PROGRAM DESCRIPTION A. Rationale 1. Following the Asian Development Bank (ADB) program cluster approach, the Economic Recovery Support Program (ERSP) was a policy-based loan consisting of two sequential subprograms. These were designed to assist the Government of Fiji to continue an infrastructure funding program during a period of low revenues after the global economic downturn in 2008 and 2009, and to support the country s ongoing reform agenda. 1 The crisis had led to a contraction in the economy, largely because of a drop in tourism. As a result, tax collections had fallen by 8.2% in real terms, and also gross domestic product (GDP) from 28.3% to 25.7% during fiscal year (FY) 2008 FY The rationale and design of the program were influenced by a 2009 ADB study that suggested a short-lived, value-for-money fiscal stimulus as a sensible way to counter the downturn, assuming that development partners and governments were able to expedite their programs to give the stimulus maximum and immediate effect. The study argued that the primary role of the stimulus would be to lift internal demand to help offset the decline in external demand during the crisis period, while a secondary role would be to protect the vulnerable groups The ERSP was to provide a broad range of benefits by helping to counter the negative impact of the global economic crisis, set a foundation for higher private sector-led economic growth, and support the vulnerable groups with better-targeted government programs. B. Expected Impact, Outcomes, and Outputs 4. The expected impact of the ERSP, based on the original design and monitoring framework (DMF), was higher and more inclusive economic growth. In subprogram 2, this changed to economic recovery combined with protection of the most vulnerable and the fiscal position which, in fact, was the original outcome of subprogram 1. Based on the DMFs, this impact was to be measured by a progressive increase in the real GDP growth rate from a baseline rate of 2.4% in 2009 to 3.5% by 2014 (for subprogram 1) and by 2016 (for subprogram 2). The impact was also to be measured by an improvement in the relative standard of living of the vulnerable groups of the community, based on household income and expenditure survey data. 5. Subprogram 1, approved in October 2009, was designed to (i) facilitate infrastructure investment in a period of constrained revenue, (ii) help prioritize infrastructure investments, (iii) raise the policy profile of vulnerability issues, and (iv) strengthen aspects of public financial 1 An infrastructure governance framework (IGF) had been developed earlier with support from ADB (ADB Report and Recommendation of the President to the Board of Directors: Proposed Program Cluster and Loan to Cook Islands for Subprogram 1: Economic Recovery Support Program. Manila), including the attached Corrigendum dated October ADB Taking the Helm: A Policy Brief on a Response to the Global Economic Crisis. Manila. The study noted that as of 2009, several Pacific economies, including the Cook Islands, had already taken an expansionary fiscal stance, not as a deliberate response to the economic crisis but as a result of allowing increases in budget deficits (thus, a fiscal stimulus was already in effect). The study noted that the magnitude of the Cook Islands stimulus was, in fact, already too high and that it had already relaxed fiscal policy too much.

5 3 management. 3 Subprogram 2 was expected to be considered by the Board of Directors about 24 months after the effectiveness of subprogram 1, or in late A total of 19 policy triggers were set for the release of subprogram 2, compared to 21 (policy actions) for subprogram 1. It was not approved until November 2012 and the loan agreement was not signed until September Subprogram 2 had the following objectives: (i) further support government budget priorities, and (ii) promote public sector reforms and measures to help protect the vulnerable and safeguard the fiscal position to promote higher and more inclusive growth. 6. The outcome for subprogram 1 was economic recovery combined with protection of the most vulnerable and the fiscal position. As indicated earlier, this was adopted as the impact statement in subprogram 2. Hence, the outcome for subprogram 2 was changed and became the government ensures sustainable budget allocations for needed capital investment and social protection. It is not clear why both the impact and the outcome statements were changed. 7. The outcome targets and indicators for subprogram 1 were (i) economic contraction avoided in FY2010 and FY2011, (ii) growth in construction value-added to exceed 5.5% in FY2012, (iii) progressive implementation in the Infrastructure Master Plan (IMP), (iv) Infrastructure Governance Framework (IGF) implemented on schedule, and (v) stability or upgrade in the government s sovereign credit rating (from BB in August 2009) to the end of FY2011. For subprogram 2, the targets and indicators were (i) growth in construction valueadded of 5.5% or greater in FY2012 and FY2013, and (ii) welfare payments above FY2011 baseline rate of NZ$12.3 million in FY2012 and FY2013. Table 1 shows the intended impact, outcomes, and outputs. Table 1: Expected Impact, Outcomes, and Outputs Subprogram 1 Subprogram 2 Impact Higher and more inclusive economic growth Economic recovery combined with protection of the most vulnerable and the fiscal position Outcomes Economic recovery, combined with protection of the vulnerable and the country s fiscal position 1. Short-term boost to economic activity from higher capital expenditure by the government 2. Structural and governance improvements 3. Support for the vulnerable through targeted actions 4. Maintenance of fiscal responsibility 5. Communication and ownership of economic and fiscal management initiatives Outputs MFEM = Ministry of Finance and Economic Management. Sources: Design and monitoring frameworks of subprograms 1 and 2. The government ensures sustainable budget allocations for needed capital investment and social protection 1. Higher capital expenditure by the government 2. Priority structural reforms and governance improvements implemented across the infrastructure sector 3. The most vulnerable members of the community receive targeted support from the government 4. MFEM maintains fiscal responsibility 5. MFEM communication of economic and fiscal management initiatives 3 ADB Completion Report: Economic Recovery Support Program Subprograms 1 and 2 in Cook Islands (Loans 2565-COO and 2946-COO). Manila.

6 4 8. The stated outputs were slightly revised from subprogram 1 to subprogram 2 but did not fundamentally change. The bulk of ERSP financing was to support Output 1 for both subprograms 1 and 2 (Table 1). The primary targets for these outputs were a budgeted capital expenditure to GDP ratio of more than 12.2% in FY2010 (for subprogram 1) and more than 12.0% in FY2012 and FY2013 (for subprogram 2). 9. The efficacy of the ESRP rested heavily on its ability to achieve the first and most critical output boost in capital expenditure. This was supposed to lift aggregate demand and economic activity during the global economic crisis, in turn, helping the economy avoid a contraction in FY2010 and FY2011. This output was, therefore, highly time-sensitive and required quick disbursement and expenditure of resources. As indicated by ADB in the lead up to the program, for the stimulus to be effective, projects must be implemented quickly The remaining outputs focused on structural and governance improvements, support for the vulnerable groups, fiscal responsibility, and communication-related initiatives. Appendix 1 provides the two subprograms impact, outcomes, and outputs and their performance targets and indicators. 5 C. Provision of Inputs 11. ADB provided two single tranche loans for each subprogram $10 million for subprogram 1 in 2009 and $6 million for subprogram 2 in These were to be disbursed following successful completion of the policy actions specified in the policy matrix. The policy matrix for subprogram 1 contained triggers for the release of funds under subprogram 2. Approval of the subprogram 2 tranche was delayed by 1 year due to difficulties in achieving several of the subprogram 2 triggers. Although specific technical assistance (TA) was not provided in support of the ERSP, a number of TA projects in other areas as well as regional TA projects had assisted in the preparation of the project. 6 D. Implementation and Monitoring Arrangements 12. The Ministry of Finance and Economic Management (MFEM) was the executing agency while the National Sustainable Development Committee (NSDC) served as the program coordinating committee. The NSDC was required to monitor progress, oversee implementation of the program, and provide guidance and direction to MFEM. 13. Within 3 months of loan effectiveness, the government was to establish and maintain a performance evaluation system acceptable to ADB, and MFEM was to report to ADB semiannually over the term of the program on the implementation of policy actions and their impact on budget outcomes and public enterprise performance (including financial and performance indicators). Program completion reports (PCRs) were to be prepared for each subprogram, with an overall report to be prepared within 1 year following the end of the program to evaluate its overall success. This comprehensive report was to assess the impacts of the 4 ADB. South Pacific Subregional Office Loan Fact-Finding Mission Economic Stimulus and Infrastructure Reform Program Loan and Country Programming Mission to Country Operations Business Plan ( ) for Cook Islands. Back-to-office-Report April (internal). 5 The two subprograms have their respective development policy letters and policy matrices. Specific outputs (i.e., policy, institutional, and capacity-building measures) are provided in each report and recommendation (RRP) of the President (RRP), and in the PCR. 6 See footnote 3, (footnote 24).

7 5 policy reforms. The government was to monitor and evaluate the program until the final PCR was prepared ADB, in collaboration with the NSDC and other development partners, was to review program performance 12 months after loan effectiveness to look into the outcome of subprogram 1 and begin preparing subprogram 2. The review was to (i) evaluate the implementation of the policy reform measures under subprogram 1; (ii) assess the impact on each sector; (iii) identify lessons; and (iv) outline reforms and assistance needed for sustained economic recovery, to enable the processing of subprogram 2. II. EVALUATION OF PERFORMANCE AND RATINGS A. Relevance of Design and Formulation 15. Following ADB guidelines for the evaluation of public sector operations, this validation assessed the alignment of the ERSP to country and ADB strategies and also the relevance of program design. 8 The assessment included the (i) underlying theory of change and results chain linking activities, outputs, outcome, and impact; (ii) associated indicators and targets; (iii) risks and assumptions; (iv) readiness and implementation capacity of the government; and (v) appropriateness of the lending modality. 16. The PCR rated the program relevant. It considered the program appropriate and relevant at appraisal, during implementation, and at completion. It argued that it was aligned with the government s National Sustainable Development Plan for subprogram 1, and the subsequent National Sustainable Development Plan for subprogram 2. The PCR also considered the program relevant to the ADB country partnership strategy for In assessing relevance, other factors must also be considered and it is here where some findings warrant attention, as they offer lessons for the future. While this validation found the ERSP relevant overall, the following aspects related to its rationale, ownership, and design raise some concern. 18. First, the rationale for the program was inconsistent with ADB s own policy guidance issued in the high-profile 2009 publication, Taking the Helm (footnote 2). In this report, ADB specifically cautioned against further fiscal stimulus in the Cook Islands, arguing that it was among the four Pacific economies that had already effectively adopted an expansionary fiscal stance heading into 2009, as measured by the magnitude of their fiscal deficits. It cautioned that the Cook Islands was among the four countries that had already relaxed fiscal policy too much. ADB also cautioned against financing new public works as a short-lived fiscal stimulus measure, unless projects are already very well advanced in terms of planning and readiness. While the impending economic downturn and its likely fiscal effects provided justification for ADB support to the Cook Islands, the fact remains that in 2009, ADB issued policy guidance cautioning against further fiscal stimulus in the country, yet in that same year it processed and approved a loan specifically to support fiscal stimulus. This contradiction should have been reconciled in the report and recommendation of the President (RRP) of subprogram 1. An 7 ADB Report and Recommendation of the President to the Board of Directors: Proposed Program Cluster and Loan to Cook Islands for Subprogram 1: Economic Recovery Support Program (Loan 2565-COO). Manila. 8 ADB Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations. Manila. The 2006 guidelines have since been replaced by the following: ADB Guidelines for the Evaluation of Public Sector Operations. Manila.

8 6 explanation would have helped to clarify why ADB s position on the matter had changed within the same year. 19. Second, the government s commitment to ensure the readiness of the program and its timely and effective implementation could have been stronger. In particular, the lack of implementable projects at the outset of the program, limited monitoring and reporting during implementation, significant delays and difficulty in meeting subprogram 2 policy triggers, and foregoing the development of a comprehensive final assessment of the program s impact, altogether challenged the implementation, monitoring, and evaluation of the ERSP. 20. Third, on the lending modality, the ERSP evolved from what was originally intended to be an infrastructure-focused loan to what essentially became a wider-scope, hybrid intervention using a program cluster modality to support infrastructure investment as a countercyclical stimulus and economic recovery measure, combined with elements of governance reform and support for the vulnerable groups. 9 The program cluster modality is designed for a phased implementation where deviations from the plans in the initial stage can cause significant subsequent deviations. The slippage in achieving the targeted infrastructure investments under the first phase delayed increasing the ratio of capital expenditure to GDP (a performance target) until FY2013. This had a knock-on effect of delaying the implementation of subprogram 2. This raises the question of whether a single-tranche program loan may have been more appropriate in providing countercyclical support while ensuring timely implementation of infrastructure investments. 21. Fourth, and related to this earlier point, the program design was overly complex. The tranche conditions were wide-ranging when, ideally, these should have been narrowed down to focus on issues related to infrastructure and quality of public expenditure, as suggested during the management review. 10 While the ERSP had fewer policy actions and conditions compared to the earlier Cook Islands program, the number of policy actions and triggers 21 for subprogram 1 and 19 for subprogram 2 (excluding sub-triggers) and the nature that spanned many different areas, still qualify this as a relatively wide-scoped and complex program. Ultimately, it is unclear how the design of the ERSP (i) responded to the initial concerns raised during the Management Review Meeting, or (ii) reflected ADB s past experiences with policybased lending, which suggest the use of a minimum number of strong and effective conditions rather than a large number of minor ones. 22. Fifth, the change in the impact and outcome statements from subprogram 1 to subprogram 2 with no explanation is a concern. The program cluster approach methodology recognizes imperfections in information and its availability, and allows for flexibility to cater to this in designing policy packages. It also allows for the reframing of solutions to policy issues in response to changes in the external environment. 11 Therefore, some design changes may be necessary and appropriate when transitioning from subprogram 1 to subprogram 2 of a program cluster. However, in this case, both the impact and outcome statements were revised from the first to the second subprogram with no clear explanation for the underlying reasons and 9 This was indicated in the 3 July 2009 Management Review Meeting. Management observed that the design had changed from an infrastructure program loan to being more strongly focused on budget support in response to the global financial and economic crisis (GFEC). In response, The project team said that the concept had evolved to include a much-needed response to the GFEC while still maintaining support for infrastructure development, which would provide a basis for longer-term economic growth a hybrid response. 10 This was indicated during the Management Review Meeting in July ADB Operations Manual Bank Policies (BP). D4/BP, issued on 7 April 2010, p.3, paras.13 and 14. See also subpara. (iii) in para. 15.

9 7 rationale for the transitions. In most program cluster approaches, the impact and outcome statements do not usually change significantly from subprogram 1 to subprogram 2, even if the program cluster guidelines allow for flexibility in the transition Moreover, the PCR only included the impact and outcome statements of subprogram 2, even though the RRP of subprogram 2 suggested that the overall ERSP impact and outcome were those specified for subprogram 1. The RRP indicated that The ERSP will achieve higher and more inclusive economic growth (ERSP impact), and will initiate economic recovery with protection of the vulnerable and the country s fiscal position (ERSP outcome). 13 However, both the documentation for the subprogram 2 loan and the PCR did not provide a clear explanation on how the first set of statements logically transitioned into the second set. Without a clear explanation and justification, these changes raise questions on the quality of design and ultimately made monitoring and assessment of the overall ERSP difficult. 24. Lastly, some of the key performance targets and indicators used in the DMFs were problematic. One of the performance targets for subprogram 2 was a reduction in subsidies for airline services by FY However, during implementation, and upon further examination, these subsidies were actually found to provide a net benefit to the country. 15 The PCR also pointed to the lack of household survey data to show an increase in the relative standard of living of the vulnerable (an impact target), lack of sex-disaggregated data for welfare payments (an outcome target), and unreliable GDP estimates (against which several DMF targets and indicators should have been assessed). 25. These issues suggest that more time and effort should have been invested in preparing the program and conducting a thorough political economy analysis necessary for a program of such complexity, even with the urgency of the situation and the need to get the program designed, agreed, and approved within a relatively short period. 16 Nonetheless, the program was a timely and appropriate effort by ADB to help a country with a good track record of reform prepare for a major economic downturn. Moreover, the areas of policy action selected largely built on existing areas of ADB engagement and supported the reform priorities of the government. More important, since the Cook Islands is not a member of the International Monetary Fund or the World Bank Group, it had limited access to external financial support in the face of the looming crisis and this provided strong impetus for the program and quickresponse support from ADB. Furthermore, the targeted assistance to vulnerable groups in the program design was important from an inclusion perspective, even if this added complexity to the overall program. Even with the six major concerns expressed above, altogether, the ERSP is assessed relevant. 12 This was the case in two contemporary program clusters in the Pacific the RMI Public Sector Program and the Samoa ERSP. Neither of these programs changed the impact or outcome statements or even the outputs, for that matter from subprogram 1 to subprogram 2. This is also the case with most other program clusters that IED has reviewed in other regions. 13 ADB Report and Recommendation of the President to the Board of Directors: Proposed Program Cluster and Loan to Cook Islands for Subprogram 2: Economic Recovery Support Program (Loan 2946-COO). Manila. 14 This is indicated in the DMF, under output An independent review of the subsidy found that the economic benefits of underwriting the route exceeded the economic cost to government. The service was put out to competitive tender and Air New Zealand was re-awarded the contract for another 4 years. 16 According to the Basic Data section of the PCR, the appraisal mission (August 2009) consisted of two people and lasted 5 days. A prior fact-finding mission (April 2009) consisted of three people and lasted 7 days.

10 8 B. Effectiveness in Achieving Program Outcome and Outputs 26. The PCR rated the program effective. This was based on its assessment that the ERSP enabled the government to retain levels of recurrent expenditure in FY2010 and FY2011.The expected outcome for the program Economic recovery, combined with protection of the vulnerable and the country s fiscal position was changed to The government ensures sustainable budget allocations for needed capital investment and social protection. Fiscal and economic data presented in the PCR, in addition to information from other sources for the FY2010 through FY2014 period, showed a mixed picture of the program s overall effectiveness particularly in meeting specific outcome and output targets. 27. The main thrust of the ERSP was to boost capital expenditure as a short-term fiscal stimulus to help shore up the economy during the anticipated downturn during FY2010 FY2013. However, the approval of subprogram 1 in late 2009 did not result in a timely boost in capital expenditure because it was found after Board approval that no investable infrastructure projects were actually ready for implementation. As the PCR put it, At appraisal it was intended that ADB and other development partner support would enable priority investments in airport expansion and road and water supply rehabilitation to be brought forward into FY2010. This did not happen as the necessary project design and other preparatory work (including agreeing project funding with development partners) were not sufficiently advanced for implementation (PCR, para. 5).This was despite assurances given just prior to approval that the Cook Islands had a number of much-needed implementation-ready capital works projects and that the program loan would enable government to select which of these is the highest immediate priority Nonetheless, while the transfer of funds did not result in an immediate boost in capital expenditure as intended, some fiscal effects were seen on the recurrent side, with recurrent spending growing by 5% in FY2010 and 30% in FY2011. Part of this went into expanding welfare payments, which grew from NZ$12.3 million in FY2012 to over NZ$13.0 million in FY2013 and FY2014, as intended under the ERSP. An eventual and significant 59% boost in capital spending was also seen in FY2013, with recent analysis on the economy suggesting that the government has maintained an expansionary fiscal stance to support growth through investments in infrastructure. 18 Therefore, the ERSP was able to support a somewhat blunt and delayed fiscal stimulus, starting on the recurrent side and eventually manifesting on the capital side. 29. On specific targets and indicators, the ERSP had the following outcome metrics: Subprogram 1: Economic contraction avoided in FY2010 and FY2011, growth in construction value-added to exceed 5.5% in FY2012, progressive implementation in the IMP, IGF implemented on schedule, and stability or upgrade in the government s sovereign credit rating (from BB in August 2009) to the end of FY2011. Subprogram 2: Growth in construction value-added of 5.5% or greater in FY2012 and FY2013, and welfare payments above the FY2011 baseline rate of NZ$12.3 million in FY2012 and FY ADB. Pacific Department COO: Economic Recovery Support Program Loan Appraisal Mission and S-PATA 7149 COO: Public Sector Review & Improvement Review Mission. Back-to-office-Report August (internal). 18 ADB Technical Assistance Consultant s Report: Cook Islands Macroeconomic Assessment. (Financed by the Technical Assistance Special Fund under the Pacific Economic Management Phase 2, RETA 8565). Manila.

11 9 30. Along with data reported in the PCR, additional information gathered during the validation show mixed achievement against the targets and indicators. Although the GDP estimates were acknowledged as unreliable, the PCR indicated that GDP growth peaked at 4.1% in FY2011 but is projected to be negative until FY2018, except for FY2015 and FY2016 when 1.4% real GDP growth is projected. ADB Key Indicators for 2015 reported that GDP contracted in FY2010 by 3% but then rebounded and grew at 1% in FY2011 and 4% in FY Another set of estimates suggest that GDP grew by 4.1% in FY2011 but turned negative during FY2012 FY2014 (footnote 18).While these estimates vary slightly, they suggest that the country was unable to avoid an economic contraction in the immediate FY2010 FY2011 period and has since seen volatile growth. Yet, it is reasonable to assume that growth rates during the crisis period would have been lower in the absence of the blunt stimulus provided under the ERSP. 31. Meanwhile, the target growth of construction value-added by 5.5% of GDP in FY2012 and FY2013 was partially achieved. The PCR indicated that construction value-added grew 30.0% in FY2012, but then contracted by 47.0% in FY At a higher (impact) level, one of the program targets was an increase in the expenditure share of the lowest 20% of households. However, without recent data on household income and expenditure, it cannot be determined whether this objective was achieved; but in the absence of a clear economic recovery and the ongoing low-growth situation, its achievement is unlikely. 33. The PCR failed to provide information on the Cook Islands sovereign credit rating and whether it remained stable or was upgraded (from BB in August 2009) as intended. However, by the end of FY2011, the country s credit rating was downgraded by Standard & Poor s. 20 Credit rating information from the MFEM indicates that from 2011 through 2015 the rating remained unchanged at B+/Stable/B By all indications, the objectives relating to the IMP and IGF were generally met, with ADB indicating that the main principles of the IGF continue to guide decision making and resource allocation on infrastructure investments. Moreover, data provided in the PCR suggest that welfare payments in FY2012 and FY2013 exceeded the FY2011 baseline amount of NZ$12.3 million, although sex-disaggregated data were not provided by the government. 35. The majority of targets and indicators at the output level were achieved. While under Output 1 (para. 8) the target of boosting capital expenditure to more than 12% of GDP in FY2010 was not achieved, this was eventually achieved in FY2013. The PCR stated that the capital expenditure ratio also exceeded 12% by FY2014 and FY2015. For Output 2, the IGF objectives on the review of economic regulation, and competitive tendering were generally met. However, the Output 2 target of reducing subsidies to air services by FY2013 was not achieved but was rendered irrelevant after a review of the subsidy found that its benefits exceeded the costs. 36. The main milestones for Output 3 were met, although some detailed data were not provided and the PCR rightly raised a question on the sustainability of some of these activities. These included (i) maintaining the real per head value of welfare payments to the vulnerable groups at FY2009 levels, (ii) piloting performance monitoring of services used by the vulnerable groups, (iii) reviewing social welfare benefits, and (iv) budgeting allocations for new targeted 19 ADB Key Indicators for Asia and the Pacific Manila. 20 Cook Islands News reported the downgrade in a 16 December 2011 article, Cooks Credit Rating Takes a Hit See (accessed 15 March 2016).

12 10 social welfare services. Output 4 targets and indicators were generally met. These supported the objective of maintaining fiscal responsibility, with milestones relating to budgeting, meeting key financial ratios, and performance reviews. Output 5 targets of supporting public communication and ownership were largely achieved, although a specific action to undertake a stakeholder survey in 2011 does not appear to have been achieved. 37. In conclusion, while the fiscal stimulus was not as sharp or timely as intended and the economy has yet to revert to stable growth, and some outcome and output targets were not achieved, on balance, the ERSP was still helpful in supporting the country s fiscal position, protecting the vulnerable groups, and getting the country through a difficult period. Altogether, these factors suggest the ERSP was borderline effective. C. Efficiency of Resource Use in Achieving Outcomes and Outputs 38. The PCR considered the program efficient in meeting its outcomes and outputs. It stated that several capital works were brought forward under the ERSP but no specific instances or any evidence was provided. The PCR further stated that fiscal targets set out in the program were achieved and the government met the conditions for the release of funds. Given the single-tranche nature of both subprograms, and front-loading of conditions, it is understandable that the benefits resulting from the transfer of funds could have occurred whether in the form of enhanced private sector demand or augmented demand from consumers. This is simply a result of any budgetary transfer. 39. A better test of efficiency would be in terms of the opportunity cost of resources, but this cannot be done with just a little information. Measuring the full benefits of the $16 million intervention is also difficult because the comprehensive assessment of the impacts of the ERSP policy reform, which was supposed to be undertaken at the program s completion, was never undertaken. 40. On process efficiency, both subprograms 1 and 2 were delayed. DMF indicators that were mostly complied with under the ERSP took place late in the subprogram cycles. Subprogram 2 was not approved until November 2012 (1 year later than originally planned) and the loan agreement was not signed until September 2013 nearly 4 years after subprogram 1 s effectiveness date. As described in the PCR, the delay in expenditure of funds under subprogram 1 meant that subprogram 2 triggers were achieved later than originally anticipated and consequently, the disbursement of funds under subprogram 2 was also delayed. While all funds were eventually disbursed, these delays dampened the stimulus to the economy during the most critical crisis period. 41. In the limited sense of budgetary resource transfers, the program can be assessed to have been efficient as it succeeded in transferring the full amount of funds to the country, however, it was not efficient from a process efficiency standpoint. Taking the two together, this validation assesses the program less than efficient. D. Preliminary Assessment of Sustainability 42. The PCR considered the program likely sustainable given the commitment of the government to ongoing improvements in capital asset management (including improving the timeliness and levels of operations and maintenance expenditure), and ongoing public financial management improvements. As discussed earlier (para. 34), by all indications, the main principles of the IGF continue to guide decision making and resource allocation on infrastructure

13 11 investment. Recent analyses indicate sustained fiscal expansion via infrastructure investment (para. 28 and footnote 18) through FY2015 and a possible return to stable GDP growth by FY2016, although GDP estimates remain unreliable. 22 Recent estimates also indicate sustained levels of social and welfare expenditure through FY2015, although the PCR casts some doubt on the sustainability of the welfare pilot programs with program counterpart funds (PCR, para. 36).With a positive but cautious view, this validation assesses the ERSP likely sustainable. E. Institutional Development 43. The PCR provided no rating or assessment of institutional development but asserted that the program reinforced improvements in infrastructure governance, public financial management, delivery of public welfare services to vulnerable communities, and public communication. The PCR also stated that the program provided increased opportunities for the country s private sector although this validation finds no particular evidence given for this. Based on the actions listed in the PCR policy matrix, this validation cannot pinpoint any major institutional initiative bearing fruit in the absence of stable economic growth, although there are some modest ones. Post-PCR information provided by ADB indicated that the secretariat of the Infrastructure Committee was established and the functions of the Ministry of Outer Islands and Infrastructure and Planning were split and relocated within the Office of the Prime Minister and Infrastructure Cook Islands to reduce discrepancies in budgeting and expenditure. Altogether these improvements provide sufficient evidence to rate the program s institutional development impact significant. F. Impact 44. The PCR did not provide a rating for the program s impact and did not discuss it in any depth. The expected impact of the ERSP was higher and more inclusive economic growth. As earlier noted, this was changed in subprogram 2 to economic recovery combined with protection of the most vulnerable and the fiscal position. Impact was to be measured by (i) a progressive increase in the real GDP growth rate to 3.5%; and (ii) an improvement in the relative standard of living of the vulnerable groups of the community, as measured by a 20% increase in the expenditure share of the lowest 20% of households. Recent forecasts suggest that a progressive increase in GDP may be possible, but the data still point to volatility in growth rates. 23 Since no recent household survey data is available, it is not possible to assess if living standards for the lowest 20% of households have improved, and any improvement may be limited given the continued economic volatility. Moreover, a comprehensive report on the overall impact of the program was never developed. The budget transfers of $10 million and $6 million under subprograms 1 and 2 did have some stimulus effects and the sustained welfare expenditures could have helped support the vulnerable groups during the period and in recent years, but on the whole, the evidence suggests that the ERSP impacts were less than satisfactory This is based on the 20 July 2016 estimate of the ADB Economics and Regional Cooperation Department. 23 See footnote Beginning May 2016, IED adopted the ratings terminology of the April 2016 Guidelines for the Evaluation of Public Sector Operations on development impacts. In this terminology, a less than satisfactory rating coincides with the moderate rating that was used before this date.

14 12 III. OTHER PERFORMANCE ASSESSMENTS A. Performance of the Borrower and Executing Agency 45. The PCR rated the performance of the borrower and executing agency satisfactory. It cited the good performance during program design and eventually during implementation. It also credited the government for undertaking a number of infrastructure sector reforms, improving governance, continuing to improve financial management and reporting, and taking some steps toward better targeting of assistance to vulnerable members of the community. 46. At the same time, the PCR documented a range of shortcomings, beginning with the initial delays in utilizing subprogram 1 funds due to insufficient planning and other preparatory work for infrastructure projects. This was clearly a setback to the ERSP and its ability to deliver a quick-response stimulus during the immediate crisis period. The PCR also indicated that the government provided limited monitoring of program performance, provision of relevant data (including sex-disaggregated data for welfare payments), and reporting in written form. The PCR did not indicate whether the proposed periodic meetings with key line ministries and consultative meetings with civil society and development partners to solicit feedback on the effectiveness and emerging needs and other impacts of the program were undertaken as expected. The preparation of a comprehensive report on the overall impact of the reforms set out in the policy letter and matrix is a critical loan covenant that was not complied with. Moreover, no separate PCRs for each subprogram appears not to have been developed. 47. These shortcomings in monitoring and reporting can be attributed, in part, to changes at the minister and secretary levels in MFEM in 2009 and 2010, as well as changes in senior management and decision-making appointments in other agencies. Moreover, the elections in late 2010 will have had some impact on ongoing monitoring, reporting, and overall implementation. It is clear, therefore, that implementing the program and reporting on its progress was a major challenge during this period. Nevertheless, the government took on an ambitious agenda and was eventually able to see it through, albeit with difficulty and delay. On balance, the experience suggests that its performance was satisfactory. B. Performance of the Asian Development Bank 48. The PCR rated the performance of ADB satisfactory, but noted that it could have assisted the program s implementation more effectively. The program had some design deficiencies and, during implementation, a more rigorous effort to help the government identify investable infrastructure projects should have been made, given the time-sensitive nature of the program. As also noted in the PCR, ADB could have insisted the borrower in submitting proper reports, especially before releasing the second tranche. 49. A closer look at the DMF and policy matrices in the PCR indicates that some of the conditions were considered fulfilled but with minimum compliance. For example, one of the policy conditions was the preparation and Cabinet approval of policy statements for each infrastructure sector (roads, airports, ports and harbors, government buildings, energy, water supply, sanitation and waste management) including provisions for user charges, cost recovery and community service obligations (PCR, Appendix 2). This was one of the triggers that caused the delay in the release of subprogram 2. Eventually, compliance with this ended up with just the release of the National Sustainable Development Plan, which set out infrastructure objectives and strategies. The PCR indicated that ADB could have assisted more by insisting on the government to better document proposed and actual expenditure financed under the

15 13 program (PCR, para. 32).With these shortcomings, the performance of ADB is assessed less than satisfactory. IV. OVERALL ASSESSMENT, LESSONS, AND RECOMMENDATIONS A. Overall Assessment and Ratings 50. The PCR rated the project successful with subcriterion ratings of relevant, effective, efficient, and likely sustainable. This validation found the evidence and arguments in the PCR and in post-pcr consultations with ADB staff just sufficient (borderline) for an overall successful rating. The program suffered from some design deficiencies but was overall relevant. It provided a blunt and delayed fiscal stimulus and saw mixed results in its outcome and output targets, thus warranting a cautious borderline effective rating. Its implementation was problematic, thus, it is rated less than efficient. Finally, this validation found sufficient evidence that the ERSP achievements are likely sustainable, even with concerns over the sustainability of expanded welfare programs. Table 2: Overall Ratings Criteria PCR IED Review Reason for Disagreement and/or Comments Relevance Relevant Relevant While rated relevant overall, the program had clear design weaknesses (paras ). Effectiveness Effective Effective The program is borderline effective. The ERSP-supported fiscal stimulus was blunt and delayed and it did not achieve all of its outcome and output targets, but was generally helpful (paras ). Efficiency Efficient Less than efficient The program experienced significant delays and process efficiency was weak (paras ). Preliminary assessment of sustainability Overall assessment Likely sustainable Likely sustainable The ERSP s achievements are likely sustainable given continued government commitment to improving fiscal management, maintaining healthy levels of capital and welfare expenditures, and ongoing reforms and improvements (para. 42). Successful Successful This validation found the evidence and arguments in the PCR and in post-pcr consultations with ADB staff just sufficient (borderline) for an overall successful rating. Institutional development Impact Not rated Less than satisfactory Performance of borrower and executing agency Not rated Significant Some institutional improvements were realized (para. 43). The program was to contribute to an economic recovery and improve the standards of living for the most vulnerable the evidence for this is limited and growth remains volatile (para. 44). Satisfactory Satisfactory The limited monitoring and reporting on the program was a major omission from an evaluation perspective (paras ).

16 14 Criteria PCR IED Review Reason for Disagreement and/or Comments Performance of Satisfactory Less than The program design had weaknesses and ADB ADB satisfactory could have assisted the program s implementation more effectively (paras ). The PCR had several weaknesses (paras ). Quality of PCR Less than satisfactory ADB = Asian Development Bank, ERSP = Economic Recovery Support Program, IED = Independent Evaluation Department, PCR = project completion report. Note: This report uses the ratings terminology of the April 2016 Guidelines for the Evaluation of Public Sector Operations. Source: ADB Independent Evaluation Department. B. Lessons 51. The ERSP provides some important lessons. The first is on the use of infrastructure financing as a fiscal stimulus measure. The PCR correctly noted that the provision of finance for infrastructure funding will only have an immediate effect on the overall level of economic activity if a pipeline of projects is in place this was not the case for the ERSP. The ADB publication, Taking the Helm (footnote 2), which influenced the design of the ERSP, cautioned that support for public works requires long lead times and that, New public works, unless already very well advanced in planning, can probably be ruled out as a short-lived fiscal stimulus measure, so that the emphasis should be on accelerating existing works programs. The second and related lesson is that ADB should have invested more effort to better understand the institutional capacity and overall readiness of the Cook Islands for the program. 52. The third lesson is on the scope and design of crisis response interventions. When designing a program that is meant to respond rapidly to an impending crisis, it is not a good practice to adopt a wide-scoped, overly complex program with numerous policy actions and conditions especially in the Pacific. This concern was raised at the design stage during the Management Review for the ERSP, but was unfortunately not reflected in the program s final design. The fourth lesson is on program cluster approaches. While some flexibility in program design is necessary, including allowance for some mid-course adjustment from one subprogram to the next, it is uncommon and problematic to significantly revise both the impact and outcome statements, especially with no explanation for the underlying reason and rationale for the transition. Finally, as the PCR indicated, ADB should have pressed harder for better documentation of proposed and actual expenditures under the program, and for more consistent reporting on the program s implementation and progress along the way. As noted in the PCR, ADB could have insisted that the borrower submitted proper reports, especially before releasing the second tranche. C. Recommendations for Follow-Up 53. This validation concurs with the need, as outlined in the PCR, to provide a TA project to assist the government in handling its debt and potential negative contingencies, including the impact of repayment of large international loans. The ADB South Pacific Subregional Office staff confirmed that ongoing TA projects for these and several other priorities are being provided. 54. ADB should consider organizing a consultation meeting with the Government of the Cook Islands to discuss the experiences and lessons of the ERSP and to ensure that future interventions of this nature are more effectively designed and executed.

17 15 V. OTHER CONSIDERATIONS AND FOLLOW-UP A. Monitoring and Evaluation Design, Implementation, and Utilization 55. Despite covenants requiring a monitoring and reporting system for the program and a completion report for each subprogram, the PCR reported that there appeared to have been limited reporting in written form, at least from the government to ADB. B. Comments on Program Completion Report Quality 56. The PCR had several weaknesses. Paragraph 10 of the RRP for subprogram 2 indicated that the expected overall impact and outcome of the program are those outlined in the RRP of subprogram On this basis, the impact and outcome of subprogram 1 should have been used in the overall program analysis by the PCR. The PCR, like the RRP for subprogram 2, did not provide a clear explanation on the change in impact and outcome from subprogram 1 to subprogram 2 and made no attempt to rationalize these for presentation and assessment. There should have been more efforts to assess the achievement of impacts, outcomes, and outputs. 57. The assessments of relevance, effectiveness, efficiency, and sustainability aimed to rationalize the positive ratings given to these criteria, even though the evidence for these positive assessments was thin and required additional consultations and research during this validation. This made the initial basis for the overall successful rating not compelling. Also, no ratings were given for institutional development or impact even if these were required for policybased lending. As a result, this validation has taken much longer time than is normally required. The PCR quality is assessed less than satisfactory. C. Data Sources for Validation 58. Data sources for this validation included minutes of key internal meetings (Country Portfolio Review Mission, Management Review Meeting), program RRPs, loan review mission reports, the PCR, and government publications such as the Budget Consultation Document. For FY2014 and FY2015 Additional updated fiscal and economic data were obtained from ADB and other sources. This validation also involved two rounds of consultations between the Independent Evaluation Department (IED) and the Pacific Department, both of ADB. D. Recommendation for Independent Evaluation Department Follow-Up 59. It is strongly recommended that this validation be immediately followed by a full program evaluation. This will enable IED to consult with government officials and other key stakeholders involved in the ERSP and allow for a more complete assessment of the program s performance, impact, and sustainability. This full evaluation would also serve as an important case study to the 2017 IED evaluation on ADB Support for Policy-Based Lending. 25 The RRP stated that The ERSP will achieve higher and more inclusive economic growth (ERSP impact), and will initiate economic recovery with protection of the vulnerable and the country s fiscal position (ERSP outcome).