METHODOLOGY FOR EVALUATIONS OF BUDGET SUPPORT OPERATIONS AT COUNTRY LEVEL. Methodological Details

Transcription

1 METHODOLOGY FOR EVALUATIONS OF BUDGET SUPPORT OPERATIONS AT COUNTRY LEVEL Methodological Details April 2009

2 The methodological advices expressed in this paper are based on the document called "Issue Paper" (IP). The paper gives the evaluation team some ideas on how to perform evaluations of budget support operations. 1. Introduction The methodology for the evaluation of budget support operations at country level takes explicitly into account the fact that budget support is a contribution to the implementation of the policy and public spending actions of a partner government. This implies that the final outcomes and impact of GBS/SBS depend predominantly on the quality and effectiveness of the strategies and actions of the government. The proposed evaluation methodology consists of a five-level analytical framework (inputs, direct outputs, government strategy (induced) outputs, outcomes and impacts see IP -fig 4 p. 14), which is applied in a sequence of three steps/components (see IP section 4). Step/Component One consists of an assessment of the inputs, direct outputs and induced outputs clearly attributable to budget support. Step/Component Two focuses on those outcomes and impacts of the government strategy that GBS/SBS intends to support. Step/Component Three deals with an exploration of the linkages between the GBS/SBS processes and the outcomes and impact of the government strategy. Evaluations of budget support should be comprehensive, which means covering all budget support operations in a particular country over a certain period. This is not only because it is conceptually very difficult, if not impossible, to attribute the individual effects of budget support operations to any single development agency but also because it is now established practice that budget support should be provided through a harmonised, unified framework, the combined effects of which should strictly be evaluated together. Furthermore, budget support evaluations ought to be conducted on a joint basis between the recipient governments and those donors providing budget support. The final outcomes and impacts of budget support reflect not only the particular design choices made by development agencies but also the nature of the joint government-development partner management arrangements, policy dialogue, and the effectiveness of the government strategies supported by GBS or SBS. Nevertheless, evaluations can and should examine some of the specific effects on incentives and strategy choices which individual budget support operations might have. 2. The Evaluation Questions As a practical way of applying this methodology, the evaluation team should focus on addressing a number of key Evaluation Questions (EQs) as proposed in the ToR. For each evaluation question to be analysed and answered, clear judgement criteria and indicators have to be formulated. A judgement criterion: (i) specifies and disaggregates the focus of the question; (ii) gives ex-ante the judgement on which the answers to the EQ will be given. An indicator is an aspect (characteristic or variable) which has to be measured in order to answer the evaluation question Evaluation Questions both for GBS and SBS The ToR include a table with a list of compulsory EQs. This paper gives again the table with a tentative breakdown of judgement criteria (JC). The JC are not compulsory but they intend 2

3 to cover much of the possible concerns and focuses of a GBS/SBS evaluation, to offer an exhaustive example for the evaluators. In the actual evaluations, the JC must be adapted to (i) the policy content and purpose of the budget support provided to a specific country and (ii) to the political, economic and social context of that particular country. The JC must be reduced, merged or split, and reformulated according to the priorities shown by the specific framework. Other JC can be asked. This should be done by the evaluation team in consultation with the Evaluation Management Group and the Evaluation Steering Committee during the inception phase of the evaluation (see chapter of the ToR). This implies a careful reconsideration and selection of the JC to reflect the actual focus of the evaluation. The JC included in the table hereunder are of a standard nature and mainly aim at clarifying the content of the question. 3

4 Table 1 List of evaluation questions and possible judgement criteria EF Level Level 1: Inputs (Step/Component One) Key evaluation questions 1.1. Which inputs have been provided and to what extent do they correspond to the envisaged GBS/SBS inputs? 1.2. To what extent are the budget support operations put in place consistent with the country s strategic and policy framework and with the overall DPs development strategies? Possible Judgement Criteria i. Funds, as committed by DPs, are disbursed and transferred to the central bank as planned. ii. Performance assessment frameworks (PAFs) aligned with government systems are agreed upon and can be monitored on a yearly basis. iii. DPs have put in place appropriate capacities and have agreed with government adequate mechanisms to ensure a coordinated policy dialogue process consistent with the key country development priorities. iv. DPs have put in place appropriate means and mechanisms to ensure progressive harmonisation of budget support operations. v. Technical assistance and capacity building initiatives have been planned in a coordinated manner and mechanisms for coordinated management among DPs and with government exist. i. The objectives and approach of GBS/SBS are consistent with - and relevant to - the country s strategic and policy framework (economic and social policies). ii. The objectives and approach of GBS/SBS are relevant to and have adjusted to respond to changes in the overall DPs development strategies. iii. The objectives and approach of GBS/SBS are consistent with the Paris Declaration objectives as regards ownership, harmonisation and alignment How well was the design (including the mix of GBS/SBS inputs) adapted to the specific political, economic and institutional context? i. Favourable entry conditions for budget support operations were in place at the time of design of GBS/SBS operations, in particular political commitment and positive dynamics in PFM. ii. The design of GBS/SBS as a whole (i.e. the quantitative and qualitative allocation of inputs, their expected interactions, their explicit and implicit prioritisation) responds to the strengths and weaknesses of the government with regard to establishing sound PFM and macro / sector policies. iii. The design of GBS/SBS operations is flexible and can be adapted in response to changes that may occur as suggested by embedded feed-back and learning mechanisms. iv. GBS/SBS design includes specific features tailored on the priorities of the main GBS/SBS providers (e.g. focus of policy dialogue and specific conditionalities). 4

5 Level 2: Direct outputs (Step/Component One) 2.1. To what extent has budget support contributed over time to an increased size and share of external funding subject to government s budgetary process and improved predictability of aid flows overall? 2.2. To what extent has budget support contributed to the establishment of a framework of policy dialogue, focused on key government strategies and priorities? 2.3. To what extent has budget support contributed to the provision of non-financial inputs, such as technical assistance and capacity building which are strategic and focused on government priorities? 2.4. To what extent has budget support contributed to harmonisation and alignment of external assistance, and reduced transaction costs over time? i. GBS/SBS financial inputs have been efficiently delivered. ii. The size and share of external funding transferred to the National Treasury of the partner country has increased. iii. The size and share of external funding subject to Government's budget process and PFM system has increased. iv. The predictability of disbursements of budget support and of global aid flows has increased (annual deviation of actual budget support from the forecast provided by donor agencies; in-year timeliness of donor disbursements / compliance with aggregate quarterly estimates 1,.). i. There has been a shift in focus of policy dialogue, toward more strategic issues, including government strategy, aid management and performance assessment. ii. The Performance Assessment Framework, supported by adequate monitoring systems, is the basis for mutual accountability and partnership. iii. There is an increased degree of involvement of relevant ministries both at central and sector level, and of decentralised institutions. i. GBS/SBS non-financial inputs have been efficiently delivered. ii. TA and capacity building initiatives are increasingly coordinated and focused on key government strategic priorities. i. The overall level of DPs harmonisation has increased. ii. The overall alignment of external aid with the government budget cycle (financial management and procurement systems) has increased. iii. There is evidence of a reduction in transaction costs of aid delivery. 2 1 See also Public Expenditure and Financial Accountability Performance Measurement Framework (June 2005), PEFA secretariat, World Bank, Wash. DC. Indicator D- 1 Predictability of Direct Budget Support. 2 Evidence of reduced transaction costs should be assessed with reference to the alignment of procedures at the various stages of negotiation, disbursement, utilisation / implementation and monitoring of aid (e.g. non-standard procurement systems, brain-drain effects of parallel project management structures, etc.). See also areas indicated by the OECD/DAC Guiding Principles and Good Practices for Budget Support under heading 3 Transaction costs incurred by budget support should be minimised. OECD/DAC (2006). Harmonising Donor Practices for Effective Aid Delivery, Volume 2. In addition, there may also be conversion costs in moving from one instrument to another, and different elements of risk for different types of transaction. 5

6 Level 3: Induced Outputs (Step/Component One) 3.1. To what extent have there been improvements in fiscal discipline and macroeconomic management and how far has budget support contributed? 3.2. To what extent did budget management and overall PFM improve and how far has budget support contributed to those improvements? 3.3. To what extent have there been improvements in the quality of policy processes and policy implementation (including ownership and transparency), and how far has budget support contributed to those improvements? 3.4. How has the level and composition of public spending changed and with which main consequences for the production of public goods and services? i. There have been improvements in fiscal and monetary policies and the related policy targets have been achieved (standard IMF requirements). ii. Budget financing has improved (reduced arrears, reduced Central Bank overdrafts, reduced domestic borrowing, increased debt sustainability). iii. Policies directly related to the balance of payments and exchange rates have been improved ensuring stability and competitiveness. i. The quality of PFM system and outcomes has improved, according to PEFA and/ or similar indicators. 3 ii. Relevant PFM reforms have been/ are being successfully implemented to improve PFM systems and outcomes. 4 i. Government ownership and initiative in the definition of new / revised policies and strategies has increased. ii. The quality of policy processes in key areas of concern for GBS/SBS operations has improved (for example, social sector reforms, business competition framework, policies ). iii. The quality of consultations, the related information and the transparency of policy processes have improved. iv. Government capacity to implement policies has increased, including legislation, institutional and operational arrangements. v. Monitoring and evaluation systems to measure output and outcome indicators have improved. i. The internal composition of public expenditure has improved, including: increased level and shares of government expenditure on national development policy priorities (including all crosscutting issues and key areas of concern for GBS/SBS programmes); improved balance between recurrent and capital expenditures; and - within the recurrent expenditures - improved balance between salary and non-salary spending. ii. Improved trends of output indicators in key sectors (e.g. increased number of classrooms, teachers trained, etc). 3 Refer to the PFM performance indicators that cover the six dimensions of an open and orderly PFM system. 1. Credibility of the budget The budget is realistic and is implemented as intended. 2. Comprehensiveness and transparency The budget and the fiscal risk oversight are comprehensive, and fiscal and budget information is accessible to the public. 3. Policy-based budgeting The budget is prepared with due regard to government policy. 4. Predictability and control in budget execution The budget is implemented in an orderly and predictable manner and there are arrangements for the exercise of control and stewardship in the use of public funds. 5. Accounting, recording and reporting Adequate records and information are produced, maintained and disseminated to meet decision-making, control, management and reporting purposes. 6. External scrutiny and audit Arrangements for scrutiny of public finances and follow-up by the executive are operating. Public Expenditure and Financial Accountability Performance Measurement Framework (June 2005), PEFA secretariat, World Bank, Wash. DC. 4 In evaluations specifically focused on EC operations, the degree of compliance with PFM requirements set for the release of EC fixed and variable tranches should be considered here (e.g. adherence with fundamental PFM requirements for the release of the EC s fixed tranche and actual performance against specific indicators set for the release of the EC s variable tranche). 6

7 3.5. To what extent have there been improvements in governance and democratic accountability, particularly regarding the relative roles of Parliament and Civil Society in relation to the Budget? To what extent did the rule of law improve in the country? i. The roles of Parliament, Prosecutor general, Court of Auditors and Civil Society, with regards to the budgeting process, are better defined and respond to the need of an increased accountability (reports, judicial reviews, follow-up actions, exhaustive and transparent information). ii. Specific anticorruption measures in the public administration are effectively implemented and corruption is prosecuted (as shown also by the trends in the international anticorruption indexes). iii. Improved data produced by monitoring & evaluation systems informs the domestic policy debate including parliament and civil society. i. The level of institutional independence of the judges has actually increased (as shown by the effective implementation of key institutional changes and by the perception of the civil society). ii. The number, diversification, efficiency and accessibility of courts has increased (as shown by the actual institutional changes and the decreasing trends in: the duration and the number of prosecutions and trials, in specific sectors; the cases dropped due to inability to afford the costs; the cases handled related to disputes about property rights and payments for goods and services provided, etc.). 5 It is postulated that both political and administrative accountability can be reinforced through the dissemination of information, and by the advocacy and mobilisation of civil society groups. Within this framework, some DPs, and especially DPs engaged in GBS/SBS operations, are pursuing anticorruption strategies and actions aimed at increasing political and administrative accountability by other, complementary, means, including the provision of TA to support accountability institutions (audit agencies, parliaments etc), and support to civil society organisations engaged in such issues. 7

8 Levels 5 + 4: Impacts and Outcomes (Step/Component Two) 4.1. How did the economy, in particular the enterprises, perform and interact with the economic and institutional environment? For SBS, this question will highlight the relationships between the overall economic context and the sector performance How did the overall livelihoods (impact), including citizens security and access to services (outcome) of the target groups change over time (for example, with respect to the key social sectors, in particular health and education and/or justice for the entrepreneurs, all citizens and especially the low income population)? For SBS, this question will focus on the specific sectors To what extent have there been changes in the income of the citizens and the income distribution (e.g. Gini Index, income growth per income quintile) with a special focus on the poorest part of the population? 4.4. To what extent can significant changes be identified in some key cross cutting themes in the society, such as environment protection, gender equality and youth participation? i. Trends of basic economic indicators have improved (stability, growth, investment, trade, sector shares in GDP, GDP per capita ). ii. Performance of key sectors has improved ensuring structural and sustainable change. iii. Economic infrastructure and services, including finance, have become more competitive. iv. The small holders sectors, including agriculture and SMEs in the marginal areas, have positively responded to the changes and have enhanced participation of the poor to economic growth (income poverty). i. Trends of selected outcome indicators indicate clear improvements in access and coverage of basic social services (e.g. relevant MDGs indicators, increased enrolment rates, improved enrolment by social group and gender; increased access to health services, by area and group, ). ii. Trends related to impact indicators (e.g. relevant MDGs indicators, morbidity and mortality rates; literacy rates; etc.) when available indicate improvements in non-income poverty dimensions. iii. The civil society organisations show an increased confidence in the fairness of the political debate and their capacity to influence government policy. i. Trends of selected outcome indicators indicate clear improvements of the income of the citizens (disaggregated data by region, rural/urban, socio-professional category, gender ) and income distribution. ii. Trends related to impact indicators (e.g. housing, consumption, equipment ). i. The major environmental challenges due to the ongoing demographic and economic processes are addressed with positive records (as shown by the existing monitoring indicators regarding the main environmental threats MDGs indicators under goal 7). ii. Gender equality in the key sectors has changed, as shown by specific health, education and economic (access to market) indicators (e.g. relevant MDGs indicators). iii. Youth employability and employment have changed, as shown by the specific education and employment indicators (e.g. relevant MDGs indicators). 8

9 Level 4 + 5: the determining factors of change (Step/Component Two) 5.1. To what extent can changes in the performance of the economy in particular in the performance of enterprises be related to changes in macroeconomic and fiscal management and/or changes in other government policies or policy processes and/or to other external or internal factors? 5.2. To what extent can changes in the overall livelihoods and income, including citizen's security and access to services be related to changes in government policies or policy processes, and/or to other external or internal factors? 5.3. To what extent can changes in the income of citizens and in the income distribution be related to changes in government policies or policy processes, and/or to other external/ internal factors? 5.4. To what extent can changes in some key cross-cutting themes be related to changes in government policies or policy processes and/or to other external or internal factors? i. Existence and strength of qualitative and quantitative causality links between improved macroeconomic and fiscal management and improved trends of economic indicators. ii. Existence and strength of qualitative and quantitative causality links between the implementation of new / improved economic and sector policies and improved trends of economic indicators. iii. Existence and strength of external factors - changes in the regional and international economic and political environment (e.g. major changes in international prices, regional economic and political destabilisation) - that justify part of the change registered in the economic indicators. iv. Existence and strength of other internal factors (e.g. political change) that justify part of the change registered in the economic indicators. i. Existence and strength of qualitative and quantitative causality links between improved access to services by and improved livelihoods and standards of living for - the poor and changes in public expenditure. ii. Existence and strength of qualitative and quantitative causality links between improved access to services by and improved livelihoods for - the poor and improvements in government policy processes. iii. Existence and strength of income related increases that may justify increased access to services by - and improved livelihoods for - the poor. iv. Existence and strength of other factors that may justify increased access to services by - and improved livelihoods for - the poor (e.g. specific non government related interventions, health improvements ). i. Existence and strength of qualitative and quantitative causality links between improved transparency and changes in government policies. ii. Existence and strength of qualitative and quantitative causality links between improved access of the poor and entrepreneurs to justice and changes in government policies. iii. Existence and strength of external factors - changes in the regional and international political / security environment - that justify part of the change registered in the governance / transparency indicators. iv. Existence and strength of other internal factors (e.g. political change) that justify part of the change registered in the governance / transparency indicators. i. Existence and strength of qualitative and quantitative causality links between improved outcomes in key cross-cutting themes and changes in public expenditure. ii. Existence and strength of qualitative and quantitative causality links between improved outcomes in key cross-cutting themes and changes in government policies. iii. Existence and strength of external factors (e.g. environmental changes due to climatic factors,. ) that justify part of the change registered in the selected outcomes. 9

10 Step/Component Three: identification of significant causal links Step Three will be addressed upon completion of the first two steps/components, and will not imply the formulation of completely new questions. It should aim at highlighting the consistencies and/or discrepancies between the answers provided in Steps/Components One and Two, to identify any significant causal link (contribution) between GBS/SBS and the actual outcomes/ impacts identified. The rationale for the formulation of the EQs may be defined as follows: Step/Component One s general question asks: to what extent did GBS/SBS contribute to the achievement of the government outputs (including allocative and operational efficiency) in Level 3 of the IL? Step/Component Two s general question asks: to what extent the achievement of the development outcomes that GBS/SBS aimed to support has been favoured, hampered, or basically not influenced by the government outputs (including those related to changes in allocative and operational efficiency of public expenditure) in level 3 of the IL? Step/Component Three s general question asks: (i) to what extent were the GBS/SBS contributions to government outputs consistent with those outputs contribution to the intended development outcomes? and (ii) which is the possible explanation, in particular, of any inconsistency? 6 In practical terms, it is suggested that the evaluators start from the establishment of a comparison table as outlined in the footnote above, between the results of Step/Component One and Step/Component Two. Then they will decide whether some specific questions should be formulated to explain specific consistencies and discrepancies shown in the comparison table. In most cases, the answers provided in Step/Component One and Two will be sufficient to explain the results of the comparison table and no further EQs will be needed. Appropriate comments to the comparison table will be sufficient to complete the evaluation. The presentation of such comments may be included in just one question (the general one), or split in more questions according to the different subjects addressed. The choice will depend upon the complexity of the results of the comparison table as well as on the complexity of the evaluation Specificities of Sector Budget Supports 6 In practice the evaluators may fill up a table as in the example below. The table will help compare the results of Step/Component One and Step Two. If such results are consistent, then the conclusion is relatively unambiguous. If the results are not consistent, then the evaluators should find a possible explanation. For instance: if a positive outcome was mainly determined by a government output, to which GBS/ SBS did not contribute, then it is possible that GBS/ SBS should be reviewed, to better support such an output in the future. If a positive outcome was mainly determined by non-government factors, while GBS/ SBS contributed significantly to those government outputs that potentially could have affected such an outcome, then the Step One assessment may be reviewed, and if confirmed various explanations may be found: e.g. the effects of the government output will be visible on the longer term, or the contribution of the government output will be more visible when the positive external factors will disappear. The table will help build the conclusions of the evaluation and find (and possibly explain) the main causality links between GBS/ SBS and the outcomes/ impacts of the government strategy. EQ 5.1 Type of link Output Outcome (Step/Component 2) Main determinant, or Co-determinant, or Weak contribution, or Type of link GBS/ SBS Output (Step/Component 1) Main determinant, or Co-determinant, or Weak contribution, or Explanation of the consistency/ inconsistency.... No contribution No contribution

11 Besides the need to apply most standard EQs at both aggregate and sector level, some aspects of SBS demand specific consideration and emphasis. Key issues specifically related to SBS include for example the sector policy focus, sector level induced outputs targeted in the SBS arrangement, performance measurement at sector level, adequate funding of the sector budgets, government DPs relations at sector level, alignment (e.g. SBS versus basket funds, procurement issues), etc. 3. The Comprehensive Evaluation Framework The evaluation framework to be applied is presented in summary form in the IP p. 14 (fig 4). It is based on a logical framework which sets out a hypothesised sequence of effects of budget support which needs to be systematically tested Content of the framework The evaluation framework is divided into five levels as follows: Level 1: GBS/SBS inputs Level 2: Direct outputs Level 3: Induced outputs Level 4: Outcomes Level 5: Impacts Level One: the GBS or SBS Inputs The first level of the Framework concerns the GBS/SBS inputs. Although there is an element of negotiation between the government and the budget support providers, the level and nature of inputs are mostly decided by the providers of budget support. The evaluation framework considers GBS and SBS as a package of inputs provided by the donors consisting of: a transfer of funds to the Consolidated Fund of the National Treasury, a national or sector policy dialogue (including disbursement modalities and related conditionalities), technical assistance or capacity building linked to the budget support, and efforts to align budget support with national policies and systems or to harmonise GBS/SBS management systems across the budget support providers. Level one will include an inventory of inputs planned and actually provided 7, and an assessment of whether such inputs have taken into account the initial conditions (relevance of GBS/SBS with respect to the choice of the instrument and its specific design). Other inputs provided by the Government should also be listed. At this level, it is possible to highlight the specific contribution of the main GBS/SBS 7 Information to be indicated in the inventory should include: i) programme intent, e.g. stated objectives; ii)content of policy dialogue and procedures for dialogue; iii) amounts transferred and disbursement criteria and procedures, including progress in aligning GBS/SBS with national systems; iv) conditionality/performance indicators; v) content of technical assistance/capacity building accompanying measures. 11

12 providers, including particular conditionalities or disbursement modalities (e.g. EC variable tranches, other partners conditions, etc.), so as to assess the rationale of such specificities and make possible an assessment of their particular effects (if any) in the next evaluation steps Level Two: Direct Outputs Level two is concerned with the direct outputs 8 of the budget support inputs. So long as they are of a sufficient scale and are not offset by countervailing actions or events 9, budget support inputs will have a direct effect on the relationship between external assistance and national or sector budget and policy processes. Specifically, the Framework postulates that budget support will produce the following direct outputs: the size and share of external funding made available through the national budget has increased; the size and share of budget available for discretionary spending is increased; the predictability of external funding flows is increased; policy dialogue and conditionalities become more coordinated, consistent with government priorities and more conducive to effective implementation of government strategy; TA and capacity building become more coordinated, more consistent with government priorities and more conducive to effective implementation of government strategy; external assistance related to GBS/SBS becomes more harmonised and aligned, with consequent transaction cost reductions 10, and has a positive influence on harmonisation and alignment of the whole package of external assistance Level Three: Induced Outputs At Level three, the direct outputs of GBS/SBS combine with the effects of government and other inputs into the budget and policy processes. The framework postulates that, when GBS/SBS and government strategy work in the same direction and there are no significant adverse and unforeseen external factors, direct outputs of budget support will have positive effects upon the financing and institutional framework for public spending and public policy, and consequently on public policy management and service delivery. Financing effects are likely to manifest themselves relatively quickly, whereas the policy and institutional effects 8 The definition of direct outputs is based on the particular nature of the evaluation framework. This is a nested framework in which the effects of GBS/SBS are interlocked with those of the government strategy which GBS/SBS is supporting. In this framework some outputs depend directly and almost exclusively on GBS/SBS implementation. Other outputs depend on implementation of government strategy, albeit strictly interrelated with the GBS/SBS. In the present methodology, to establish a clear distinction between them, the latter are defined as induced outputs, the former as direct outputs. Other definitions could have been adopted: for example, in similar cases, a distinction between immediate effects and output, or output 1 and output 2 has been adopted. 9 For example disbursement delays, transfer bottlenecks between the Central Bank and the Treasury, various factors hampering mobilisation of inputs, etc. All such factors when identified should be assessed and included in the evaluation to draw possible conclusions and recommendations. 10 A key point to note is that, while the initial level of Harmonisation and Alignment (H&A) associated with budget support can be considered a GBS/SBS input, the GBS/SBS process also aims to deepen harmonisation and alignment of development assistance as a whole. As such H&A of the whole assistance, i.e. a stronger partnership framework between government and DPs, can be considered a direct output of GBS/SBS. 12

13 are likely to take longer. It is suggested that there are six particular ways in which these outputs are likely to manifest themselves, at either national or sector levels : improved aggregate fiscal discipline and macroeconomic management; strengthened PFM and procurement systems; improved quality of policy processes and policy implementation; increased funding for discretionary public spending with effects on the quantity and quality of goods and services provided by the public sector; enhanced governance and democratic accountability; improved rule of law in the country Level Four: Outcomes As one proceeds up the framework towards impact level, the hypotheses being tested are less directly related to the provision of budget support per se and more to the entire set of policies and strategies of the government and the reactions of the relevant stakeholders to changes in the public policy-making and resource allocation processes. This reflects the fact that budget support is conceived as a contribution to a government process of public spending and policymaking. The outcomes postulated in the framework relate to the positive effects on the wider economy and the poor of the enhancement of government capacity to influence in a positive way the proximate determinants of economic growth and poverty reduction at national and sector levels. They derive specifically from the interaction of a more efficient, effective and accountable public sector with the wider economy and society. Thus, in the medium to long term, one expects to see outcomes related to: increased use and appreciation by the beneficiaries of the goods and services provided by the public sector; a positive response by the general economy to the improvements in government initiatives targeted by budget support arrangements; increased business confidence deriving from more effective macroeconomic and regulatory policies; improved general confidence of the population and enhanced democratic accountability, particularly in relation to the budget process Level Five: Impacts Impacts may be expected to materialise in the long term as long as the expected outcomes are produced and key assumptions hold true with regard to the growth and development process. However, at this level the changes taking place are most likely not influenced by budget support alone, but are the combined effects of underlying processes of change and trends in the domestic policy making process, and within society and the economy, sustained by GBS/SBS. The desired final impact of GBS and SBS, when combined with appropriate public policy and spending actions is that: there is enhanced and sustainable economic growth; 13

14 there are reductions in income and non-income poverty; there is empowerment and social inclusion of poor people and disadvantaged groups (including women); there may well be other impact areas to consider, according to the specific partnership framework and the related priorities established by the government and the main development partners such as significant improvements in democracy, human rights, and environment protection What is driving change within the Framework? The evaluation framework identifies two main categories of effects of GBS or SBS: flow-of-funds effects deriving directly from the budget support funds themselves; and policy or institutional effects resulting from the interplay of budget support funds and the non-financial GBS/SBS inputs with domestic processes of policy-making, budget formulation and budget execution Diversity and specificity of the GBS effects The flow of funds effects derive from the additional funds which are made available to the national budget through GBS or SBS. Depending on the relative significance of these flows and on the extent to which they are targeted, these may permit either additional levels of public spending or reduced levels of borrowing. In so far as GBS or SBS resources are fungible, they may also reduce the costs of financing the budget, for example by reducing the need for within-year treasury operations to meet cash flow requirements (hence reducing the need for Central Bank overdrafts or for Treasury Bill issues, or for accumulation of payment arrears.) They may also have macroeconomic effects, in particular on interest and exchange rates: a reduction of government borrowing may help control the interest and the inflation rates; an increase of the local currency demand on the currency market may contribute to an appreciation of the exchange rate. Policy and institutional effects are the effects on policies and institutional processes which budget support might generate. Some endogenous policy and institutional effects arise merely from the increased use of the national Public Finance Management (PFM) system, as opposed to the use of aid modalities which are less aligned with the PFM system. Depending on the relative scale of budget support provided (and the extent to which its effects are reinforced by restricting growth of less aligned aid modalities), it may have positive institutional effects by restoring importance to the national budget and to the related processes of accountability. As matching policies and budgets becomes more important, a greater attention to policy priorities and related resource allocation within the government, in the Parliament, among Civil Society Organisations (CSOs) and the public at large should be expected. The Ministry of Finance and the National Audit Office should strengthen their coordination and control roles, while the line ministries should develop their policy-making and consensus building capacities to ensure adequate resources in their sectors. As a result, policy and spending decisions should be scrutinised more carefully and changes may be demanded; there may also be domestic pressure on public servants to improve the quality of budget formulation and execution processes. The relative significance of these effects will be determined by the quality, transparency and comprehensiveness of the budget process at the outset, by the extent to which there is a genuine shift from off-budget to on-budget decision-making and by the extent to which the Executive is compelled to respond to democratic pressures. 14

15 Other exogenous policy and institutional effects will result from the new external influence exerted through the GBS or SBS arrangements. These result directly from the processes of policy dialogue with Development Partners (DPs), from the budget support disbursement conditionalities and from the external capacity-building support which might be provided. For example the attention given by DPs to the importance of improving PFM systems may promote a more intense implementation of PFM reforms than might otherwise have been the case. The introduction of GBS or SBS disbursements linked to performance indicators 11, as applied by the European Commission for variable tranches in its budget support programmes, may act as an incentive to promote faster improvements in effective service delivery Systemic effects and causality chains An attempt should be made - although it will not be easy - to distinguish the flow of funds and policy and institutional effects. In practice, these effects are often intertwined or there is interplay between them, but it is important for an understanding of the evaluation framework to recognise that it is this range of effects and their interaction with the external factors which drives the systemic changes at different levels of the Framework 12. During such an assessment, specific sequences of links between individual factors across the framework may be identified. When sequences of links, or causality chains, appear particularly significant, they need to be highlighted and analysed in depth to draw out possible lessons, although in most circumstances it is not advisable to formulate these causality chains ex ante as hypotheses to be tested. The experience has shown that structuring the evaluation along such hypothesised causality chains may jeopardise an understanding of the main systemic changes and generate arbitrary interpretation. According to the proposed methodology, it is not possible to build linear causal relationships from GBS/SBS inputs to government strategy impacts, as the changes at each level of the framework are determined by a system of causes combining both GBS/SBS and other factors, where the latter assume a growing role during the advance through the various stages of the framework. As an example, it would be difficult to conclude as suggested by the Joint Evaluation of General Budget Support (JEGBS) causality map that the input policy dialogue has a causality relationship only with the immediate effects of increased predictability of funds and more strategic focus of policy dialogue and conditionality ; or that alignment and harmonisation do not have a causal relationship with increased 11 The indicators used to determine the amount of the variable tranche to be disbursed need to be measured annually and capable of reasonable annual variation - probably in response to government action. Thus the kinds of results in question are not medium-term or long-term impacts such as literacy rates or life expectancy, but indicators, linked to the Millennium Development Goals or powerful proxies for them, at the level of annual service delivery outcomes [ ] A typical example would be primary school enrolment. This is classified as outcome in the EC input-output-outcome-impact typology: whereas the amount spent on primary education would be classified as an input, the number of primary teachers trained would be an output, and literacy rates would be an impact. European Commission Budget Support: An Innovative Approach to Conditionality DG DEV, European Commission, in Budget support as more effective aid? Recent Experiences and Emerging Lessons, (eds) Koeberle S., Stavreski Z., and Walliser J., The World Bank (2006). 12 For example, better access of the poor to education (at outcome level) may be ensured thanks to better public service delivery, as the consequence of a number of GBS/SBS related factors: stronger political consistency and accountability (possible GBS/SBS policy and institutional endogenous effects); better policy formulation (possible GBS/SBS policy and institutional exogenous effect); and increased funds allocation (possible GBS/SBS flow of funds effects). Each of such factors may in turn be positively influenced by the others and by non-gbs/sbs factors. On the other side a better access of the poor to education (at outcome level), although facilitated by the better services considered above, might be determined by an increase in household incomes or an improvement in transport infrastructure not directly related to GBS/SBS. 15

16 proportion of funds subject to the national budget. On the other hand, the need to represent the complexity of the relationships has pushed the JEGBS to extend the use of the multiple causality links, thus missing the original purpose of the ex ante causality map. 4. How to deal with the three steps/components of the methodology The methodology explicitly recognises that budget support is a contribution to a government strategy (to a set of domestically determined public policy and spending actions). By implication, the evaluation of GBS/SBS must involve, on the one hand (Step/Component One), an assessment of the logic of budget support, and on the second hand (Step/Component Two), an assessment of those outcomes and impacts of the government strategy that GBS/SBS intends to support. The Step/Component Three is the exploration of the linkages between the GBS/SBS processes and the outcomes and impact of the government strategy. This approach aims to achieve a balance in the evaluation between the assessment of the proper effects of budget support (inputs, direct outputs and induced outputs) and the assessment of the influence that GBS/SBS is able to produce on the government strategy to achieve its outcomes and impacts. At the same time it recognises the difficulty of assessing the influence of GBS/SBS at outcome and impact levels without a broader understanding of global and sector social and economic performance at country level Overview of the methodology The effects of budget support per se may be seen to be significant up to the level of the induced outputs. To identify the influence of GBS/SBS at higher levels of the logframe, the evaluators must establish a clear picture of those outcomes and impacts of the government strategy that GBS/SBS aimed to support. The factors that contributed to successes or failures, either internal or external, must also be identified. Finally, an assessment should be undertaken of how, through which links, and to what extent, GBS/SBS may have contributed. The required approach combines the following: Step/Component One: a thorough evaluation of the budget support at levels 1, 2 and 3 of the Framework. This is focused on the relevance of the inputs provided, the direct outputs of these inputs, and the quality and adequacy of the changes supported in the government systems (induced outputs). Step/Component Two: a broad assessment of the GBS/SBS related outcomes and impacts of the government strategies. Step/Component Two, unless explicitly requested by the government and its partners, does not consist of an evaluation of the government strategy, but provides exhaustive information on the level of achievement of the development results that GBS/SBS aimed to promote and on the related determining factors. In particular, it should allow the evaluators to understand: - whether the achievements in terms of economic growth, poverty reduction and other development objectives correspond to the GBS/SBS supported targets of the government strategy; and - what were the determining factors of such positive/negative achievements, including the specific role of the outputs of the government strategy. Step/Component Three: an assessment of the contribution of GBS/SBS to the factors that have contributed to the success or failure of the government strategy. It is carried out combining and comparing the results of Components One and Two. 16

17 Step/Component One covers the first three levels of the evaluation framework from GBS/SBS inputs (level 1) through direct outputs (level 2) to induced outputs (level 3). It should be realised that the induced output at level 3 is a combined effect of GBS/SBS direct outputs and government actions. Step/Component Two - the government strategy assessment goes from the impact (Level 5) and outcome (Level 4) of the strategy back to its determining factors, including the outputs of the government strategies in general and in particular those induced by GBS/SBS. The accomplishment of these two steps/components is a pre-condition for allowing a final assessment of the GBS/SBS contribution to the success or failure of the government strategy, which is the third component in the methodology. This is in fact a contribution analysis, which is a very complex process 13. Therefore it is recommended that the GBS/SBS contribution to the actual success or failure of the government strategy be addressed in a qualitative manner, in which probable linkages are identified without attempting to quantify them Step/Component One: Analysis of Inputs, Direct Outputs and Induced Outputs (Levels 1 to 3) In Step/Component One, the evaluators will be examining the logic of GBS/SBS, in particular the context within which GBS/SBS have been provided, the nature of the inputs actually supplied, their direct outputs, the combined (induced) outputs of these interventions and the related government actions (Levels 1-3). At these three levels, evaluators need to explicitly examine what type of influence GBS/SBS may be exerting through flow of funds effects and policy and institutional effects, as described in section 3.2, as well as other possible systemic effects. The evaluators need to consider both the intended effects and the unintended, - and potentially negative - effects. Step/Component One comprises the following four sub-stages: An overview of the evolution of the context within which GBS/SBS have to be evaluated. This comprises (i) the overall country context in which the government strategy was conceived and is implemented, (ii) the entry conditions under which provision of budget support has started, and (iii) the pre-existing framework of external aid in the country, which in most cases GBS/SBS would be expected to affect directly. An assessment of the GBS/SBS inputs actually provided, vis à vis the context and the scope of the GBS/SBS, taking account of the actual volume of funds transferred, the content of dialogue and conditionality, any technical assistance linked to the GBS/SBS, and the level of harmonisation and alignment of GBS/SBS arrangements. Such assessment will address in particular the relevance of GBS/SBS as an aid instrument, and the relevance of its design. An assessment of the direct outputs of GBS/SBS, in terms of modified relationships between external assistance and the national budget and policy processes. Here, evaluators should focus on the effects of GBS/SBS on the levels of Official Development Assistance (ODA) actually included in the budget process, on the composition of those funds and their predictability, as well as on the content of the policy dialogue, on the nature of technical assistance provision and on the composition and nature of development 13 Mayne, J. Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly. Discussion Paper. Ottawa: Officer of the Auditor General of Canada,