Evaluation Approach Validation of Indonesia Country Partnership Strategy Final Review November 2014

Transcription

1 6 ADB Avenue, Mandaluyong City, 1550 Metro Manila, Philippines Tel ; Fax evaluation@adb.org; Evaluation Approach Validation of Indonesia Country Partnership Strategy Final Review November 2014 Team Leader: Hyun H. Son, Principal Evaluation Specialist ( hhson@adb.org) Contact: evaluation@adb.org I. Introduction A. Final Review Validation Objectives and Procedures 1. Indonesia is a founding member of the Asian Development Bank (ADB). ADB is currently preparing a new Country Partnership Strategy (CPS) for the country for ADB s Southeast Asia Regional Department prepared a Final Review of CPS in October, the findings of which are to inform the next CPS. The Independent Evaluation Department (IED) received the Final Review on 5 November 2014 for validation. 2. The Final Review assessed ADB CPS as successful, based on positive ratings given to the relevance, effectiveness, sustainability and impact of the program and partnership although it gave a less than efficient rating to the program. Despite the Final Review noting the delays in CPS finalization and the CPS covering a large number of sectors and thematic areas, the CPS was assessed satisfactory for strategic positioning given the program s responsiveness to changes in the development setting, ADB and the government s priorities, among others. ADB s program was assessed to be relevant as projects and programs were well designed and responsive to stakeholder inputs and to government s priorities albeit project monitoring could have been better formalized and recorded. ADB support was assessed to be partly efficient given the high transaction costs of doing conventional project operations due to complex country systems for the use of external assistance, and reluctance of executing and implementing agencies to apply ADB project policies and procedures when these differ from country systems. 3. The CPS was rated as effective. The majority of the outcomes intended in the CPS results framework and in the core sectors and thematic areas of support have been achieved. ADB s program was rated likely sustainable given improving financing and cost-recovery arrangements in the key sectors of ADB support, community involvement in the design and delivery of much of the rural infrastructure, and strong ownership of ADB-supported policy reforms. The Final Review assessed the development impact of ADB support satisfactory noting that ADB did make a significant contribution to inclusive economic growth and sustainable development. 4. The Final Review recommended (i) sharpening the sector focus to a small number of sector operations where ADB is likely to have an influence at the sector level; (ii) according greater emphasis on policy-based lending including the use of results-based lending that is more suitable for middle-income countries; (iii) overcoming delays by avoiding overly complex 1

2 project designs and adopting a wider array of country systems; (iv) improving knowledge management by enhancing the dissemination of lessons learned in ADB s operations in the field, among others; (v) exploring options for ADB lending directly to state-owned enterprises (SOEs); (vi) leveraging both financial and policy resources such as carbon funds, private sector support and bilateral co-financing; and (vii) improving results-tracking by according greater emphasis on outcome, sustainability, readiness and sector performance monitoring during project supervisions and in the annual portfolio review. Key lessons included the need for appropriate adjustments in the assistance program given changes in the country s development setting and for the government to take ownership of reform programs, which take time. 5. IED will prepare a report to (i) validate the findings and assessments of the Final Review including the relevance, efficiency, effectiveness, sustainability, and development impacts of the country portfolio and its targets; and (ii) identify lessons and recommendations for ADB s future activities in the upcoming CPS period. The validation report will be prepared in accordance with the Guidelines on Preparation of the CPS Final Review Validation Reports, issued by IED in August II. Background A. Analysis of Socio-economic Constraints in Indonesia 6. Indonesia is one of the leading economies in Asia and the Pacific. Now a (lower) middleincome country, Indonesia has the 16th largest economy in the world and the biggest economy in the Association of Southeast Asian Nations (ASEAN) The Indonesian economy has shown resilience and vibrancy. It grew at an average of 8% in before significantly decelerating in due to the Asian financial crisis. The country s economy subsequently rebounded, growing at an average of 5.4% in accounting for a minor dip in 2009 following the global financial crisis. Growth is primarily driven by the services sector, which grew at an average 7.1% in compared to 4.4% in industry and 3.4% in agriculture. 3 Indonesia s growth has been accompanied by significant poverty reduction. The proportion of people living under $1.25-a-day declined from 54.3% in 1990 to 18.1% in 2010 (footnote 3). 8. Despite these achievements, Indonesia is finding it difficult to move from middle income to high income status. Indonesia attained middle-income status as early as 1993 but slipped back following the Asian financial crisis in It bounced back to the middle-income group in To avoid falling in the middle-income trap, Indonesia s per capita gross domestic product will need to expand at an average of 14.8% annually in This is highly unlikely as growth averaged around 5% in the last decade or so. The Organization of Economic Co-operation and Development estimates that Indonesia could reach high-income status by 2042, but this is only a best case scenario. 6 1 ADB Guidelines for the Preparation of Country Partnership Strategy Final Review Validation Report. Draft. 2 ADB Indonesia: Fact Sheet. Manila. 3 ADB Key Indicators for Asia and the Pacific. Manila. 4 Rhee, C Indonesia Risks Falling into the Middle Income-Trap. Published 27 March 2012 in Jakarta Globe. Retrieved 17 February 2014 in 5 Felipe, J Tracking the Middle-Income Trap: What is It, Who is It, and Why? Part 1. ADB Economics Working Paper Series No Manila OECD. Economic Outlook for Southeast Asia, China and India 2014: Beyond the Middle-Income Trap. Paris. 2

3 9. The government s draft National Medium-Term Development Plan (abbreviated RPJMN) identifies the middle income trap as one of the crucial challenges Indonesia is facing. The draft plan envisions that Indonesia will move out of the middle income status by 2030 by achieving 6 8% growth per year. It identifies Indonesia s priority policy directions in the medium term including (i) human capital and welfare development; (ii) inclusive and sustainable growth; (iii) good governance for robust development; (iv) equitable growth across regions; (v) accelerated infrastructure development for growth and equity; (vi) natural resource management; and (vii) disaster and climate change risk mitigation To overcome the middle-income trap, Indonesia must deepen a range of reforms and address daunting challenges, some of which are covered in this validation. 11. Infrastructure gaps. Adequate infrastructure plays an important role in promoting growth and reducing poverty in Indonesia. A 1% increase in infrastructure investment contributes an estimated 0.3% to the country s gross domestic product (GDP). 8 Indonesia s infrastructure investments however have yet to recover since the Asian financial crisis. Between 1995 and 2011, the country s total infrastructure investment declined from an average of 7% in to around 3 4% of GDP in recent years except in when it reached an average of 5.7% of GDP due to the fast track program of investment in coal power generation by the state-owned electricity utility PLN. Indonesia s average infrastructure investment in real terms in was about two thirds of what it was in Indonesia also lags behind its neighbouring Asian economies in terms of infrastructure investments. The country s average infrastructure investment level of 3% of GDP is well below China, Thailand and Vietnam s investment levels of above 7% of GDP Decentralization has induced changes in the composition of Indonesia s infrastructure investments, in terms of source and sector share, in the last decade or so. Subnational governments are now the largest source of infrastructure spending in Indonesia. The share of subnational governments in total infrastructure spending increased from 7% in to 39% in In contrast, the central government s share declined from 26% to 21%, as did private infrastructure investments from 31% to 10%, and SOEs from 37% to 30% over the same period. Due to decentralization, expenditure on management and investment of local roads and water and sanitation was transferred from central to sub-national governments, primarily to district/city governments. 13. The private sector s involvement in infrastructure projects has waned in the past decade primarily due to issues relating to land acquisition, and inter-government and inter-agency coordination. Bankable public-private partnerships (PPP) projects are also hard to come by in the country. In 2012, only three PPP projects were deemed ready for offer out of a total of 58 PPP projects in 11 different areas including toll roads, maritime transportation and water resources, valued at a total of $51.2 billion in the period Across sectors, transport investments share in total infrastructure increased from 22% in to 52% in , while energy s share decreased from 46% to 26% over the 7 The final version of the National Medium-Term Development Plan is scheduled to be released in January A copy of its draft version can be accessed at 8 Mustajab, M Infrastructure Investment in Indonesia: Process and Impact. Accessed 29 September 2014 at 9 World Bank Indonesia Economic Quarterly Update: March Jakarta. 10 World Bank Indonesia: Avoiding the Trap. Indonesia Development Policy Review Jakarta. 3

4 same period. The increase in the share of the transport sector is mainly due to subnational road investments. Between 2001 and 2009, the length of district road increased from 287,577 km to 384,810 km. This increase has been marred however by inadequate funds for operation and maintenance, substandard design and road quality, and new roads that do not fit in an integrated transport network Given the important role of infrastructure in development and significant gaps in infrastructure investments in Indonesia, the validation will include a sector analysis of infrastructure particularly transport and energy investments. This analysis will also take into account ADB s support for investments in these areas. 16. Fiscal Management. Indonesia is expected to have limited fiscal space in the coming years. Assuming a GDP growth of % per year, with no fiscal shocks or reforms, fiscal deficit is projected at 2.1% of GDP in 2015, gradually decreasing to 1.7% of GDP. Considering the fiscal rule of a maximum deficit of 2.5% of GDP, Indonesia s fiscal space for spending priorities is deemed too limited, particularly if shocks arise such as oil price increase and rupiah depreciation on subsidy spending. Investors and economists have long called for a significant reduction, if not the complete phase out, of Indonesia s fuel subsidies. Its fuel subsidy program is estimated to cost 2.6% of GDP in 2014 and exerts tremendous pressure on the government s limited fiscal space Newly-elected President Joko Widodo is reportedly seeking to gradually reduce fuel subsidies in the next few years. The Indonesian government increased fuel prices by more than 30% in November 2014 to save more than $8 billion in Indonesia s fuel prices, one of the cheapest in the world, were increased by more than Rp 2,000 ($0.16) Fiscal risks limit the spending priorities of the Indonesian government and the provision of basic services such as education and health. To this end, the validation will include a sector analysis on public sector management. The analysis will also take into account ADB s support for public sector management in the country. 19. Rising inequality. Indonesia has experienced rising income inequality over the last decade. In fact, it has one of the fastest rising rates of income inequality in the region in the last decade. 14 Indonesia s Gini coefficient increased from 30% in 2000 to 41% in 2012, compared to Malaysia s, which increased from 38% in 2004 to 46% in 2009, and the Philippines, which instead decreased from 46% in 2000 to 43% in 2009 (footnote 10). Several factors contribute to Indonesia s rising income inequality. 20. Progress in improving access to basic services has been uneven across geographic and income groups. In education, primary net enrolment rate (NER) at the national level increased from 93% in 2003 to 96% in However, NER in poor provinces still lags somewhat behind the national average: for instance, NER in East Nusa Tenggara (NTT), which is one of the poorest provinces in the country, increased from 88% in 2003 to 94% in Improving access 11 World Bank Investing in Indonesia s Roads: Improving Efficiency and Closing the Financing Gap. Jakarta. 12 World Bank Indonesia Economic Quarterly Update. May Reuters Indonesia Hikes Fuel Prices, Saving Government $8Bn Next Year. Accessed 19 November 2014 at 14 Comparing Gini coefficients across countries have three important caveats: (i) countries can use either income- or consumption-based welfare measures; (ii) household surveys adopt varying methodologies in computing the consumption aggregate and are collected in varying years across countries; and (iii) countries adjust differently for the spatial cost of living. 4

5 to secondary education poses a greater challenge than in the primary level. Secondary NER increased from 63% to 74% in 2003 and 2013, respectively; this is still low and indicative of the poor preparation of many young Indonesians to face the labour market. 15 The quality of education is also problematic. In 2012, Indonesia ranked 57 th in reading, math and scientific literacy out of 65 countries in an international test administered to 470, year old students (footnote 10). Poor access to and quality of education has also translated into labor market challenges. Demand for high skilled labor has been increasing: despite this, more than two of every three workers have not completed high school education. Moreover, as the returns to education have been consistently increasing since the 1997 Asian financial crisis, the gap between wages for workers with primary education and those with university education has been widening. 16 In health, Indonesia is expected to miss the 2015 deadline of achieving the Millennium Development Goal (MDG) on maternal health Secondly, fiscal policy particularly relating to fuel subsidies may have exacerbated income inequality in Indonesia. As discussed earlier, a large portion of Indonesia s budget accrue to fuel subsidies. However, the rich benefit more than the poor from the fuel subsidies. The richest 10% of households receive 40% of the benefits of fuel subsidies: in contrast, the poorest 10% only receive less than 1% of the benefits (footnote 10). According to the National Socioeconomic Survey 2010 data the bottom 5% of households consumed 1.7 liters of subsidized gasoline per month compared to the top 5% who consumed 82 liters of subsidized gasoline per month. 18 The high spending on fuel subsidies also crowds out investments in basic services such as education, health and infrastructure. The poor are likely to be affected more by under-investments in basic services than the rich. 22. In view of the above observations, the validation will examine the implications of inequality to development priorities in Indonesia and the impact of fuel subsidies on the welfare of the poor. 23. Decentralization. Decentralization has been one of the major governance reforms in Indonesia over the last decade. Since 2001, Indonesia has embarked on fiscal and administrative decentralization. Thirteen years after its roll out, Indonesia has yet to maximize the benefits of decentralization in improving access to basic services. While decentralization has improved access to information and increased engagement in local political processes, it has also imposed specific constraints that hamper effective service delivery. 24. One, coordination across ministries and levels of government has impeded infrastructure development and effective delivery of basic services. Before Indonesia switched to a decentralized system in 2001, infrastructure planning and implementation was handled by the central government. Now, under a decentralized system, subnational governments have assumed greater responsibilities in managing infrastructure. Provincial and district roads now account for more than 80% of Indonesia s total road network. Subnational governments also spend about 1.5% of GDP on infrastructure, which is more than the central government s infrastructure expenditures, estimated at 1% of GDP. Given the increased role of subnational 15 The estimates are from the Indonesian Statistics Office and can be accessed at 16 Asian Development Bank, International Labor Organization, and Islamic Development Bank Indonesia: Critical Development Constraints. Mandaluyong City. 17 ADB Key Indicators in Asia and the Pacific. Manila. 18 Pradiptyo R, and G. A. Sahadewo A Growing Pain: An Experimental Approach to Discover the Most Acceptable Strategy for Lifting Fuel Subsidy Scheme in Indonesia, mimeo. Faculty of Economics and Business, Universitas Gadjah Mada, Yogyakarta. 5

6 governments in infrastructure development, infrastructure projects with impacts that spill across districts require extensive consultation and discussion between central and local governments. Moreover, infrastructure development plans at the central, provincial and district levels are often not mutually consistent (footnote 10). 25. Two, subnational governments have inadequate capacity to raise revenues and a huge portion of their limited fiscal space is allocated to spending on personnel. About 90% of the budgets of subnational governments are financed by fiscal transfers from the central government. Moreover, more than 40% of subnational government spending is allotted to personnel. Excessive spending on personnel may be due to the composition of central government transfers. Almost 60% of central government transfers to subnational governments in 2012 were composed of the General Allocation Fund (Dana Alokasi Umum or DAU), which are untied block grants that encourage disproportionate personnel spending. In contrast, the Special Allocation Fund (Dana Alokasi Khusus or DAK), which are similar to output-based subsidies aimed at funding special activities of the region in accordance with national priorities, compose 6% of these central government transfers and are thinly spread across too many sectors (footnote 10). 26. Three, variations exist in legal and regulatory frameworks on doing business across provinces, districts and cities or municipalities. For instance, it takes about 44 days to obtain a construction permit in the city of Bandung in West Java compared to 158 days in Jakarta. Similarly, official procedures for starting a business take about 27 days in the city of Palangka Raya in Central Kalimantan compared to 45 days in Jakarta. These variations can influence the attractiveness of regions to investment. The silver lining is that regions with streamlined processes can provide good practices that can be disseminated through supportive legal or regulatory reforms at the national level (footnote 10). 27. The validation will consider the implications of decentralization and other related governance issues in the CPS design and implementation, and well as for ADB s future operations. ADB has provided support for Indonesia s decentralization efforts in past years. B. ADB Country Strategies 28. ADB s country strategies for Indonesia prior to include two country strategy programs (CSPs) for and , an interim operational strategy for and country operational strategies (COS) for 2001 and CSP updates were also released in , , and The 1994 COS emphasized the need for expediting Indonesia s transformation from a predominantly agriculture-based to an industrialized economy through human resource and infrastructure development, and sustainable utilization of natural resources. The programs outlined in the 1994 COS were overtaken by the 1997 Asian financial crisis. During the crisis, ADB adopted an interim operational strategy that was in place from 1998 to 2000 to mitigate the impacts and help address the structural causes of the crisis. Under the interim operational strategy, ADB targeted financial sector restructuring to support economic recovery, and social safety nets through health, nutrition, and education support to address the surge in poverty. 30. The interim operational strategy was replaced by the 2001 COS, which emphasized poverty reduction through a decentralized pro-poor and "bottom up" approach to directly and indirectly target the poor. The 2001 COS supported five priority areas: creating basic institutions 6

7 and improving governance, encouraging pro-poor and pro-jobs sustainable recovery, balanced regional development for regional equity, social development, and mainstreaming environment management and providing for sustainable use of natural resources. To this end, the planned program of assistance during was outlined in the country assistance plan. 31. CSP was closely patterned after the 2001 COS. To improve the effectiveness of ADB assistance, the strategy adopted a geographical focus instead of a sector-specific approach - by targeting poor provinces with low financial resource base or low human capital base such as West Nusa Tenggara, East Nusa Tenggara and Papua. The strategy supported five focal areas: improving governance through anticorruption initiatives, and legal and judicial reform; supporting decentralization to meet rural and urban development needs; promoting human development through improved access to education and health services; mainstreaming environment management and sustainable use of natural resources; and promoting sustained growth through investments in infrastructure, strengthening financial and corporate governance, and encouraging private sector development. However, the implementation of the strategy was marred by delays due to the impact of decentralization, lack of coordination between agencies, and weak institutional and human capacities. In particular, ADB lacked adequate resources for project supervision and monitoring for decentralized projects, which involved some 200 districts. 32. CSP was designed around two pillars: promoting pro-poor, sustainable economic growth through macroeconomic, infrastructure, and financial sector reforms; and social development through policy and institutional reform, selective expenditure programs and investments, resource management, and capacity development. Five operational areas were identified under the strategy including: infrastructure, finance, decentralization, MDG achievement (particularly in water and sanitation, health, and education), and environment. 33. Inclusive growth and environmental sustainability were the two strategic pillars of CPS To help promote inclusive growth, ADB sought to support government efforts on connecting poor people and regions to markets by upgrading infrastructure, improving logistics, and enhancing the skills base, and by strengthening local governments capacity for public service delivery. To support environmental sustainability, ADB targeted efforts to improve land, coastal, and water resource management. CPS differed from the previous CPS in the following aspects. It included climate change considerations as a strategic thrust, supported the gradual rebalancing of ADB s portfolio toward project investments, and focused on a reduced set of specific activities within six sectors: agriculture and natural resources, education, energy, finance, transport, and water supply and sanitation. C. ADB Program 34. Indonesia has received more than $12 billion in sovereign and nonsovereign financing from ADB in (Figure 1). The largest share of ADB s financing, at 41% (or equivalent to $5.0 billion), was allocated to public sector management, followed by finance at 16% (or equivalent to $2.0 billion), multisector projects at 15% (or equivalent to $1.8 billion), energy at 10% (or equivalent to $1.2 billion), and transport and ICT at 8% (or equivalent to $1.0 billion). 7

8 Figure 1: ADB Loans and Grants to Indonesia, Water Supply and other Municipal Infrastructure Services 3% Transport and ICT 8% Agriculture and Natural Resources 3% Education 2% Energy 10% Finance 16% Public Sector Management 41% Multisector 15% Health and Social Protection 1% Industry and Trade 1% Note: ICT=information and communication technology Sources: Loan, TA, Grant, and Equity Approvals and Loan and Grant Financial Information Services, accessed on 10 November D. Lessons and Issues Identified from Previous Country Evaluations 35. A Country Assistance Program Evaluation (CAPE) was prepared by the Operations Evaluation Department in CAPE 2005 identifies lessons for ADB s operations and policy dialogue. Lessons for operations stress that: (i) ADB should continue to target support in sectors where it had comparative advantage including transport and communications, social infrastructure, and finance; (ii) ADB s infrastructure assistance can indirectly impact poverty, particularly for infrastructure located in areas with a high relative incidence of poverty, without making project design overly complex by including subcomponents that aim to reduce poverty directly, which can increase the transaction costs of the main project; (iii) reduce the geographic coverage of ADB s assistance to reduce the transaction costs and complexity of implementation and management of operations and improve the sustainability of outcomes by providing essential physical and social infrastructure in the same local governments where support has been provided for strategic priorities such as capacity building for governance; (iv) governance should be mainstreamed in ADB operations as legal, regulatory, civil service, and anticorruption reforms are needed to attract private sector investments; and (v) ADB should work with the government to reduce the payment of excessive commitment fees by developing a set of project readiness filters and allowing sufficient time for project processing to ensure that loans are not submitted for Board consideration prematurely. 36. Lessons for policy dialogue note that ADB should (i) help strengthen financial flows to local governments; (ii) support greater roles for provincial governments particularly in coordinating and prioritizing projects between districts in their provinces, and monitoring 8

9 development effectiveness, governance, and the delivery of services at the local level; (iii) strengthen capacity of local officials on environment and natural resources management (ENRM) and strengthen voluntary compliance with ENRM policies; (iv) together with the government, define an agenda for good governance and anticorruption; and (v) use technical assistance effectively. III. Evaluation Methodology A. Scope and Coverage 37. The validation period will cover newly approved, active and completed loans, grants and technical assistance from 2012 to Indonesia received a total of $5.4 billion in loans and grants from ADB, with the largest share of 24% (or equivalent to $1.3 billion) accruing to energy projects followed by finance at 20% (or equivalent to $1.1 billion), public sector management at 18% (or equivalent to $1.0 billion), and transport and ICT at 18% (or equivalent to $964.2 million). Figure 2 presents newly-approved, active, and completed loans and grants in Figure 2: ADB Loans and Grants to Indonesia, Water Supply and other Municipal Infrastructure Services 7% Transport and ICT 18% Public Sector Management 18% Agriculture and Natural Resources 6% Finance 20% Education 4% Energy 24% Health and Social Protection 3% Note: The estimates include new approvals, completed, and active loan and grants in Completed loans and grants with closing dates falling within the time frame are included as part of the current portfolio in Many multisector programs/projects have been reclassified according to the new Project Classification System including Agriculture and Natural Resources, Transport and ICT, and Water Supply and other Municipal Infrastructure and Services. ICT=information and communication technology Sources: Loan, TA, Grant, and Equity Approvals database; and LFIS Mainframe, accessed on 10 November ADB has provided $186.7 million in technical assistance, including project preparatory technical assistance, to Indonesia in Education, agriculture and natural resources, and public sector management received the bulk of the funding for technical assistance at 57% ($105.9 million) and 14% ($25.3 million), respectively. Figure 3 presents newly-approved, active, and completed technical assistance in

10 Figure 3: ADB Technical Assistance to Indonesia, Public Sector Management 6% Multisector 2% Health and Social Protection 6% Transport and ICT 3% Water Supply and other Municipal Infrastructure Services 5% Agriculture and Natural Resources 14% Finance 4% Education 57% Energy 5% Note: The estimates include new approvals, completed, and active technical assistance, including project preparatory technical assistance, in Completed technical assistances with closing dates falling within the time frame are included as part of the current portfolio in Many multisector programs/projects have been reclassified according to the new Project Classification System including Agriculture and Natural Resources, Transport and ICT, and Water Supply and other Municipal Infrastructure and Services. ICT=information and communication technology Sources: Loan, TA, Grant, and Equity Approvals database; and TAIS Mainframe, accessed on 10 November B. Final Review Validation Approach and Methodology 39. The validation will assess the performance of ADB s county assistance strategy and program in Indonesia using the following criteria: (i) strategic positioning, (ii) program relevance, (iii) efficiency, (iv) effectiveness, (v) sustainability, and (vi) development impacts. 40. The validation methodology will include (i) sector analyses of infrastructure (transport and energy), public sector management and education; (ii) a desk review of relevant documents and reports; and (iii) consultation with government agencies, development partners, Indonesia Resident Mission, and staff at ADB headquarters. As discussed, gaps in infrastructure, public sector management, and human capital development are posing challenges to Indonesia s growth prospects particularly as it seeks to move toward a high income trajectory. Thus, these key sectors warrant more detailed analysis in the validation, the findings of which will be useful in ADB s operations in the country. These sectors, particularly infrastructure and public sector management, have been allocated a large portion of the CPS resources envelope in

11 2014. Moreover, ADB has supported important interventions in these areas, particularly in education, that can provide important lessons for the formulation of the next CPS The validation will be reviewed by two peer reviewers: Tomoo Ueda, Principal Evaluation Specialist, and Emile Gozali, Principal Evaluation Specialist. It will also be examined by two external reviewers, who have yet to be confirmed. These external reviewers will have extensive experience on evaluation and research in this field. C. Key Issues to be Addressed by the Final Review Validation 42. Linkage between CPS and diagnosis of binding constraints to Indonesia s development. The Final Review noted that in the diagnosis of the binding constraints to Indonesia s development (which reinforced the strategic positioning of ADB support in CPS ), the need for the country to escape the middle-income trap by shifting from resource-driven growth dependent on cheap labor and capital to a growth process that is based on high productivity and innovation was underscored. However, the danger and solutions to the middle-income trap are not clearly reflected in CPS The validation report will examine how well diagnostic findings of country constraints are reflected in the CPS. It will also explore the factors that can impel Indonesia toward a trajectory that can take it to a high income country. 43. Sector choice. The Final Review mentioned that the CPS was not geographically focused and covered a large number of sectors, sub-sectors and thematic areas. CAPE 2005, and the final review of CPS and its validation also recommended reducing the geographic coverage of ADB support in Indonesia. However, a number of ADB interventions in the CPS period targeted policy reforms that spread across various sectors and thematic areas, and the impact of which are observed across many provinces and regions. Attention will be also given to the circumstances and response concerning the agriculture and natural resources (ANR) and public sector management (PSM) sectors; the former saw implementation fall drastically from the CPS planned activities, while PSM went from a crosssectoral approach to become the lead sector, capturing 38% of actual expenditures. 44. Project design and Loan Modality. The Final Review noted that overly complex project designs contributed to project delays. Another important aspect that should be discussed is how project design accommodated the CPS s focus on inclusive growth. The validation will examine the specific adjustments made on project design (e.g. targeting of recipients) to help promote inclusive growth. It will assess whether the reference to inclusive growth in the CPS and its implementation recognized the three pillars that ADB professes characterizes this concept (high sustained growth, access to opportunities, and safety nets). The Final Review also noted that the CPS focused considerably more on policy-based lending than investment operations. The validation will examine the relevance of using policy-based loans to Indonesia s development challenges and opportunities. 45. Results monitoring. A quarter of the outcome indicators included in the sector and country results framework were neither formally monitored or reported on regularly by government agencies or by the Central Bureau of Statistics, according to the Final Review. Similarly, project monitoring emphasizes financial performance than progress toward 19 The study team selected infrastructure, public sector management and education for sector analyses following initial consultation with the Indonesia Resident Mission (IRM). IRM has supported the selection of these sectors, noting their importance in ADB s operations in the country. 11

12 development results and there is a lack of baseline surveys, the Final Review added. The validation will examine ADB and the government s efforts, and factors that affect their capacity and coordination, to regularly monitor country, sector and project level outcomes. 46. Progress on policy reform. The Final Review noted that policy reforms supported by ADB have important contributions to macro-stability, connectivity, financial depth, and the enabling environment for the private sector. The validation will examine the government s capacity to sustain these policy reforms given changes in the political and economic landscapes. It will also examine other areas of policy reform particularly fuel subsidy reduction, which can have potential significant impacts in Indonesia s economy. 47. Government capacity and coordination. The Final Review noted that the weak capacity and in some cases reluctance of executing agencies and implementing agencies in applying ADB project and program procedures, as well as complex government aid approval and delivery systems have contributed to project delays. The validation will examine government coordination in managing ADB projects and programs, particularly those supporting policy reforms. It will also examine the government s capacity and revenues, with an emphasis on local governments given Indonesia s decentralized system, to sustain achievements made under ADB interventions. D. Proposed Final Review Validation Schedule and Cost Estimates 48. A two-week mission to Indonesia will be fielded on November The mission will verify the key findings of the final review and gather data for the sector analyses. The mission will undertake interviews with representatives of government agencies, executing and implementing agencies, and select development partners. The mission team will include the team leader and one international consultant. 49. The following team members will be responsible for designated tasks and sectors (Table 1) for the preparation of the report. The terms of reference for the consultants are in Appendix 1. Table 1: Independent Evaluation Team Department Assigned Task or Sector Responsible ADB Staff or Consultant Team Leader, education sector analysis, Hyun H. Son, Principal Evaluation Specialist economic analysis, and preparation of the final report Review and technical support Farzana Ahmed, Lead Results Management Specialist Analytical support and data processing Lucille Ocenar, Evaluation Officer Administrative support Valerie Anne Melo-Cabuang, Senior Evaluation Assistant Public sector management sector analysis One international consultant Infrastructure (energy and transport) sector One international consultant analysis Analytical support and research assistance One national consultant 12

13 50. The validation will be prepared between November 2014 and March The following schedule is proposed for the preparation of the validation report. Milestone Schedule Approval of concept note IV September 2014 Recruitment of consultants IV September II November 2014 Submission of EAP to ILT and peer review III November 2014 Interdepartmental comments for EAP and finalization III November I December 2014 Field Mission III-IV November 2014 Peer review IV December 2014 One-stop review I-II January 2014 Interdepartmental review Incorporation of interdepartmental comments Government commenting II-III January 2015 III-IV January 2015 IV January-I February 2015 Report revision and circulation for DG Level II-III February 2015 OSEC editing III-IV February 2015 Incorporate comments from editors IV February 2015 Circulation prior to Informal Board Seminar Informal Board Seminar I-II March 2015 III-IV March 2015 Appendix: Terms of Reference for Consultants 13

14 Appendix: Terms of Reference For the validation of the Country Partnership Strategy Final Review (CPSFRV) for Indonesia, two international consultants and a national consultant will be engaged to assist the IED team leader in the preparation of CPSFR VR documents. 1. Country Program Evaluation Specialist The Country Program Evaluation Specialist will be recruited for 35 working days on an intermittent basis. The consultant will provide advice to the team in providing for the tone and direction of the CPSFR VR. He/she will review and analyze country assistance program context, evaluate and discuss ADB's performance in relation to objectives and targets indicated in the ADB s strategies for Indonesia. He/she will conduct a sector analysis on public sector management and should have extensive knowledge on decentralization and other governance issues in Indonesia. He/she will review the details of the evaluation issues, framework and questions presented in the evaluation approach paper. Inputs will be provided from the desk review of the related documents and reports. The specific tasks are given below: (i) conduct desk review of relevant background materials, in particular, but not limited to the CPSFR, CPS, country operational business plans, and project documents; (ii) assess the CPSFR, against corresponding CPS with regard to the evaluation criteria; assess also the quality of the final review report; (iii) assess the appropriateness of ADB program and strategic choices in light of the country s development context and government priorities; (iv) prepare a contextual review of the implementation of the CPS, including (a) country context and development goals, (b) country strategy, (c) sector-level outputs and outcomes, (d) CPS implementation, and (e) ADB s institutional performance; (v) prepare a sector assessment on public sector management focusing on decentralization and other governance issues that have implications on ADB s operations, analyzing the key milestones, and priority challenges and opportunities in the sector; (vi) review Report and Recommendation of the President (RRPs) and project completion reports (PCRs), plus ADTA papers and TA completion reports (TCRs), in the public management sector to gauge the strategic positioning, relevance, effectiveness, efficiency, sustainability, and impact of sector operations; (vii) prepare a draft of the validation report, using IED s Validation Report Guidelines and template documents; (viii) participate in the independent evaluation mission (IEM) including field visits, together with the evaluation team; (ix) conduct consultations with relevant stakeholders in Indonesia (government officials, incountry development partners and civil society organizations) and personnel based in ADB headquarters and the Indonesia Resident Mission; (x) assist the team leader to present the findings of the IEM; (xi) revise draft of validation report, incorporating findings from the mission, comments/feedback from various stages-i.e. from peer reviewers, IED management, inter-department and government reviews; and (xii) perform other tasks that may be assigned by the team leader in the context of the CPS FR validation. 14

15 2. Infrastructure Evaluation Specialist (Transport and Energy) An evaluation specialist, with in depth knowledge on the operations and evaluation of infrastructure projects, will be recruited for the CPSFRV for 15 working days on an intermittent basis. He/she should have experience in undertaking strategic planning at the country level, conduct of economic analysis and should be familiar with the transport and energy sectors of Indonesia. He/she will review the details of the evaluation issues, framework and questions given in the evaluation approach paper and will assess the outcomes of the ADB program in the areas mentioned above against the targets indicated in the results framework presented in ADB s country strategies. The specific tasks are given below: (i) review the key development milestones of Indonesia in the transport and energy sectors with focus, particularly in relation to the economic and political environment. Review the government's transport and energy sector plans and determine if it is focused on priority development needs; (ii) review ADB's program strategies and portfolio in the transport and energy sectors to determine if they are (a) consistent with the government's plan; (b) consistent with ADB's Strategy 2020; (c) recognize capacity constraints; (d) suitably coordinated with other development partners; (e) focus on ADB's comparative advantage in the transport and energy sectors, as identified by ADB's experience in the country and sector; (f) had a suitable rationale for guiding the types of loans (project loans vis-a-vis sector loans, among others); (iii) review updated sector results frameworks and Design and Monitoring Frameworks (DMF) of projects and advisory TAs (ADTAs) in the transport and energy sectors to determine if indicators were relevant and appropriate, baselines were established, targets were specific and time-bound, and gauge ADB's impacts and achievement of results; (iv) review RRPs and PCRs, plus ADTA papers and TCRs, in the transport and energy sectors to gauge the strategic positioning, relevance, effectiveness, efficiency, sustainability, and impact of sector operations; (v) analyze the key issues, challenges, opportunities, constraints and needed reforms in the transport and energy sectors and crosscutting themes as they relate to the sectors; (vi) identify key lessons and provide forward-looking recommendations for consideration by ADB in its future country strategy in the transport and energy sectors of Indonesia; (vii) assess commitment to and progress on institutional reforms and organizational changes relevant to the project; (viii) review project economic internal rates of return (EIRR), update data and EIRRs as needed; (ix) interview project directors of transport and energy sector projects on any problems with project implementation; how ADB has responded; changes in sector policy or conditions that affected the project; government's commitment to the project and availability of financing and other resources; experience with the resident mission; and risks facing the project; and (x) perform other tasks that may be assigned by the CPSFR VR team leader in the transport and energy sectors in the context of the Indonesia CPS. 3. Research Associate A Research Associate will be engaged for 66 working days on an intermittent basis. He/she will assist the IED team in the preparation of CPSFR VR through data gathering, conduct of portfolio 15

16 review and analysis, synthesis of important lessons and findings. The detailed responsibilities are as follows: (i) Collect and assist in the review of relevant ADB documents including current procedures, strategies, policies, guidelines, and economic and sector work related to Indonesia; (ii) Undertake portfolio review and prepare detailed graphs, charts, figures, and tables; (iii) Contribute to the distillation and identification of key lessons, taking into account past and current issues and future challenges; (iv) In the context of the IED guidelines, synthesize relevant information that can be used to substantiate relevance, effectiveness, efficiency, sustainability, and contribution to impacts of ADB-financed technical assistance, loans and grants; (v) Assist in additional research, as needed, in responding to comments received during interdepartmental and management reviews; and (vi) Perform other tasks as required by the team leader and the assisting national officer in the context of the report/study. 16