Institut für Strukturforschung und Planung GmbH

Transcription

1 Institut für Strukturforschung und Planung GmbH GOPA Gesellschaft für Organisation, Planung und Ausbildung mbh WORKING PAPER NO. 2 EVALUATING PROGRAMME-BASED APPROACHES: GENERAL BUDGET SUPPORT Evaluating Programme-based Approaches: General Budget Support December 2008

2 Authors: Barbara Dreis-Lampen Martin Güldner Hans-Rimbert Hemmer Felipe Isidor-Serrano Contact: Prof. Dr. Hans-Rimbert Hemmer Institut für Strukturforschung und Planung GmbH Hindenburgring Bad Homburg Tel +49 (6172) Fax + 49 (6172) hans-rimbert.hemmer@gopa.de

3 Contents Contents Forword Introduction Basics of evaluation methodology Logical framework analysis as basis Determining the actual results Evaluation in the form of a goal-oriented gap analysis What is programme-oriented joint financing? Programme orientation Joint financing mechanisms Synthesis Problems of budget management and risks inherent in general budget support Budget management is of central importance for functional budget support Major risks with general budget support - an overview Fiduciary risks Corruption a special form of fiduciary risk Macroeconomic risks and political risks Risk of programmes being rejected Risks associated with PBA-specific benefits Risks caused by the cooperation of other donors Conclusion Specific challenges for the evaluation of PBA in the form of general budget support Preliminary remarks Orientation of the logframe Measuring of actual results and making a gap analysis The evaluation Relevance i

4 Contents Effectiveness Efficiency Impact Sustainability Coherence, complementarity and coordination Assessing checklists Questions used in connection with the causality map Rating/Assessing the success of budget support programmes Literature ii

5 ABBREVIATIONS Abbreviations AfDB BMZ CEB DAC DANIDA DC DED DEZA DFID EU FC FDI GOPA GTZ IDD IEG InWEnt ISP IMF KfW African Development Bank Bundesministerium für wirtschaftliche Zusammenarbeit und Entwicklung Council of Europe Development Bank Development Assistance Committee Danish International Development Agency Development Cooperation Deutscher Entwicklungsdienst Direktion für Entwicklung und Zusammenarbeit Department for International Development European Union Financial Cooperation Foreign Direct Investment Gesellschaft für Organisation, Planung und Ausbildung Deutsche Gesellschaft für Technische Zusammenarbeit International Development Department Independent Evaluation Group Internationale Weiterbildung und Entwicklung Institut für Strukturforschung und Planung International Monetary Fund Kreditanstalt für Wiederaufbau iii

6 Abbreviations LDCs LENPA LFA MDBS MDGs NGO OECD PBA PBA-2 PEFA PRSC PRSP PSIA RBM SWAp TC WB Least Developed Countries Learning Network on Programme-Based Approaches Logical Framework Analysis Multi-Donor Budget Support Programmes Millennium Development Goals Non-Governmental Organisation Organisation for Economic Cooperation and Development Programme-based Approach Jointly Financed Programme-based Approach Public Expenditure Financial Accountability Poverty Reduction Strategy Credit Poverty Reduction Strategy Paper Poverty and Social Impact Analysis Results-based Management Sector-wide Approach Technical Cooperation World Bank iv

7 FORWORD Forword This is the second working paper by the Institute for Structural Research and Planning, an independent think tank belonging to the GOPA Group. It builds on working paper No. 1 which examined issues related to the evaluation of Programme-based Approaches (PBA), using sector programmes as an example. This paper starts with a brief introduction explaining why the issue of evaluating PBAs, especially jointly financed PBAs (in the terminology of this working paper: PBA-2), needs to be addressed. Chapter 2 then describes the basic principles of evaluation methodology, using the Development Assistance Committee (DAC) evaluation system as a basis. Chapter 3 defines the content of PBA-2, in particular that of budget support, and distinguishes it from other forms of programme aid. These three chapters are a summary of the corresponding chapters in working paper No. 1, and will serve to introduce those not yet familiar with the subject to the evaluation problems inherent in budget support approaches. For more detailed information, readers should consult working paper No. 1. Otherwise, readers can go straight to chapter 4 which examines major problems in budget management and the ensuing risks for general budget support. Chapter 5 then sets out the main theme of working paper No. 2: the specific challenges involved in evaluating PBA-2 in the form of general budget support. It focuses on the practical aspects of implementing budget support evaluations, and provides detailed checklists for individual evaluation criteria. This working paper aims to support the development of practical methodologies, i.e. methods that can be applied without undue effort. This sets it apart from the numerous publications and deliberations on PBA-2 evaluation published by national and international development cooperation institutions. An analysis of the growing number of contributions to the debate on PBA-2 and budget funding reveals that most offer little in the way of concrete information about the effectiveness and efficiency of this development instrument, or about how these may be determined. Learning from evaluation is therefore crucial if we wish to free PBA-2 and its future role from dogmatic explanations and developmental dreams. This is easier said than done, for budget support evaluation is a complex field which involves adapting traditional evaluation frameworks to suit the questions at hand. This working paper therefore includes a large number of new and indispensable checklists for use when conducting specific budget support evaluations. While the checklists provided are comprehensive, they do not claim to be exhaustive. But why would a consulting company think tank concern itself with the evaluation of PBA-2 in the first place? Here are three answers: 1

8 FORWORD Because PBA-2 evaluations involve such a wide range of highly complex tasks, it makes sense to entrust them to larger teams of specialists not to individual evaluators, as is common practice when evaluating individual projects. In addition to sector-specific and method-specific knowledge at microeconomic, mesoeconomic and macroeconomic levels, these evaluation teams also require fiscal expertise, specifically in the field of public financial management. Usually, building and steering teams of this calibre is a task for the consulting industry. The consulting industry s main concern is always practical relevance, i.e. the question of how/whether the insights generated and the solutions proposed by academic research can be implemented in the field. As a result, this working paper aims to be a practical manual for all those concerned with evaluation, not a theoretical discourse on PBA-2 evaluation. As such, it claims to be neither tendentious, nor guided by vested interest. Ultimately, to evaluate general budget support is to review political processes, not individual project results. This means we must also ask how ownership is to be reviewed something that requires us to first consider and answer questions such as Who is evaluating, and on whose behalf? In future, evaluation services must (like other development cooperation services) be driven more by local partner structures, and used less as a way of providing donorcontrolled proof as to how means are being allocated. In its recent report, the Independent Evaluation Group (IEG) of the World Bank described its experience in Chile and Colombia as follows: some governments, such as those in Chile or Colombia, outsource all the evaluations conducted as part of the government M+E system to academia or consulting firms This includes donor-funded projects. The document continues: One reason for outsourcing these evaluations, rather than having them conducted by government officials, is to afford them higher levels of independence, objectivity, and credibility - and also to avoid any (potential or perceived) conflict of interests that can arise from self-evaluation. One frequently made point of criticism cannot be dismissed namely, that development institutions/ organisations tend to award themselves (unduly) high marks because their self-evaluations are driven by vested interests. The economist William Easterly, who spent sixteen years working for the World Bank, has compared this practice to allowing university students to fill in their own degree certificates. This is another reason why (private) local and international consultants should be asked to apply their international expertise to local structures. The KfW Development Bank (KfW) has recently pointed out that as a development instrument, PBA-2 makes sense only in countries that are willing and able to carry out reforms, and which adhere to democratic principles and the rule of law. It is therefore necessary to firmly establish and institutionalise know-how as to which reforms are most pressing and how they can be implemented in both research and the private 2

9 FORWORD sector. Evaluations involving the private consulting industry can greatly contribute towards achieving this. 3

10

11 Introduction CHAPTER 1 CHAPTER 1 Introduction (1) With the Millennium Declaration of 2000, and the Millennium Development Goals (MDGs) derived from it, the 189 member states of the United Nations set themselves concrete goals in the fight against global poverty, and initiated a process whereby the community of states would use the development effects of international development cooperation (DC) as a guiding principle. The Paris Declaration on Aid Effectiveness in March 2005 emphasised that achieving the MDGs will require not just an increase in the volume of DC, but a significant increase in aid effectiveness too (OECD 2005). As DC focuses on the aid effectiveness of its policies, the need arises for a more intensive examination of the methods used to analyse that (aid) effectiveness. 1 As a rule, evaluations are the method used to monitor the effectiveness of measures. Past mistakes must be identified in order to make cooperation with developing countries more effective, and more successful, in the future. Indeed, few political areas are as hotly contested in public debate as DC, despite the fact that virtually no other political sector conducts more evaluations, or has institutionalised evaluations so extensively. The demand for analysing aid effectiveness Evaluations can be undertaken before a DC intervention starts (ex ante evaluation), while it is being implemented (ongoing evaluation) or after its completion (ex post evaluation). (2) In recent years, complex DC programmes (complementary to isolated, stand-alone projects which had been dominant up to then) have been constantly growing in importance. The reason for this development is that compared with isolated projects, they are the better form of administering DC in terms of development policy. Within these programme-based approaches, jointly financed programme-based approaches (in our terminology PBA-2) are constantly growing in importance. So far, however, (existing) empirical knowledge has not been systematically looked at. This means there is no comprehensive evidence to prove that the postulated advantages of PBA-2 over individual projects broader effectiveness with regard to growth and to poverty reduction, more sustainability, stronger participation and ownership, to name but a few actually come to bear. The international trend towards PBA-2 (and thus the move away from a projectbased approach) and the orientation towards the impact of DC activities (i.e. determining the actual impact and benefits for the partners) mean that in order to prove the claims made ex ante for its developmental superiority, all DC stakeholders must not only familiarise themselves with this new form of aid, but have to specifically prove the superior developmental effectiveness of this type of DC over individual projects. (3) Moreover, correctly performed evaluations generally yield knowledge that can be used to improve the quality of on-going and future projects. This aspect of institutional learning is precisely what stands out so clearly in the Paris Declaration mentioned ear- The growing importance of jointly financed PBA Evaluations and institutional learning 1 For more details see ISP (2007), chapters

12 CHAPTER 1 Introduction lier. This said, it is presupposed that the development of evaluation procedures can, in a compact and reasonable way, yield the desired insights into completed DC measures without contravening the dictates of efficiency and effectiveness of the evaluation as such. Chapter 2 explains the exact meaning of these terms in detail. Accountability towards the general public Aims of the GOPA working papers on PBA- 2 evaluation (4) The importance of results and lessons learnt from evaluations is growing, particularly in the light of existing accountability to the general public, and specifically taxpayers, as to how the entrusted means are used. The fact that in jointly financed programmes, certain effects can no longer be attributed to individual programmes, makes it all the more important to evaluate the developmental impact of DC measures. (5) GOPA Consultants intends to conduct an in-depth examination of the various types of DC programmes and their evaluation. This is to be achieved by the Institut für Strukturforschung und Planung (Institute for Structural Research and Planning), the GOPA Group s independent think tank. The most important results of these efforts will be published in a series of working papers. This also enables GOPA to respond to the demand of some donors for identifying consulting agencies with sufficient expertise and/or experience in the different types of PBA-2 evaluations. 6

13 Basics of evaluation methodology CHAPTER 2 CHAPTER 2 Basics of evaluation methodology 2.1 Logical framework analysis as basis (1) The purpose of DC is to achieve specific development goals. To make sure this planning instrument for development reforms is used appropriately, clear and consistent goals must be named for each DC intervention (which can be either an individual project or an entire programme). Since many of the desired results (e.g. poverty reduction) cannot be achieved directly, objectives (i.e. interim targets) are also set. These objectives are more closely related to the measures taken (and can thus be verified more quickly), and are logically related to the achievement of the desired end result. They are usually set within the framework of a logical framework analysis (logframe; LFA). First introduced in the 1970s, LFA is now well established in the international donor community. In a structured and systematic way, logframes list the expectations that particular development projects must meet. This involves: LFA as a planning approach for DC interventions a comprehensive description of the project s key strategic elements (resources, activities, outputs, objectives and goals); an explicit list of the assumptions that underpin these strategic elements (and hence of the risks that implementing them would entail); setting indicators to examine the extent to which objectives and goals are being achieved. (2) Indicators are quantitative or qualitative variables that allow the changes caused by projects or programmes to be measured in a simple and reliable way. This means that indicators can also reflect the changes induced by DC interventions. In this case, they are obtained by simplifying complex issues appropriately and reducing them to an observable dimension (DEZA 1999, p. 21). Indicators can be derived from various sources such as previously available statistics, documents prepared by institutions involved in the project or programme (e.g. the sponsor), or datasets collected specifically for the project or programme in question. Some indicators for concrete measures will need to be calculated from the start an aspect that is not discussed in this paper. (3) The indicators required to establish the extent to which objectives and goals are achieved comprise two components. On the one hand, their task is to determine, on respective levels, changes that can be considered as being representative of the entire spectrum of desired effects. On the other hand, it involves linking these indicators to targets (= aspiration levels) in order to ascertain the extent to which the respective objectives and goals have been achieved. What are indicators? Requirements for the indicators used during the planning process In practice, however, a conflict often occurs: indicators that are easily identified cover only a part of the parameter being measured, whereas indicators of wide-reaching 7

14 CHAPTER 2 Basics of evaluation methodology significance either cannot be calculated at all, or require a disproportionate amount of time and effort to calculate. This means that in the project or programme being examined, a compromise must be made between accuracy and ascertainability, or several indicators must be created when determining a particular issue. LFA key terms (4) The key LFA terms are inputs, activities, outputs, purpose and goals. These are defined as follows: Term Inputs Activities Outputs Purpose Goals Definition The financial, human, and material resources to be used by the development intervention All actions to be taken or work to be performed through which inputs, such as funds, technical assistance and other types of resources are mobilised to produce specific outputs The products, capital goods and services which should result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of the outcomes The publicly stated objectives of the development programme or project Note: Project or programme objectives are the intended physical, financial, institutional, social, environmental, or other development results to which a project or programme is expected to contribute. The higher-order objectives to which a development intervention is intended to contribute Source: OECD (2002). 8

15 Basics of evaluation methodology CHAPTER 2 (5) The method used to distinguish between objectives and goals is important. Evaluation theorists have proposed at least three different ways in which this can be done: Differentiating between objectives and goals Objectives Short-term and medium-term effects Direct or immediate effects Effects on target group Goals Long-term effects Indirect or collateral effects Overarching effects All three suggestions are valid and can be justified in terms of evaluation theory. Which one is ultimately used will depend on the following criteria: a) the specific viewpoint of the evaluator, or of the evaluation client; b) the nature of the specific intervention to be evaluated; c) the difficulties that arise when differentiating between these individual levels in specific cases, and identifying the effects/impact observed on the respective levels; d) the question of whether representative, empirically verifiable and resilient indicators can be found for measuring the extent to which the goals and objectives concerned are being achieved. (6) The DAC uses the time factor as its basis for differentiating between objectives and goals. Thus, objectives are results desired in the short to medium term, while goals are results desired in the long term. The explanation by DEZA (1999, page 22) supports this approach by defining a goal as a longer-term development goal to which a project/ programme contributes by means of its results, but whose achievement also lies beyond the responsibility of that project/programme, and which is also impacted by other influences. In this interpretation, target group-related aspects no longer play a crucial role when classifying the target level. (7) Since the process of transforming inputs into outputs takes place within DC interventions, the achieved outputs can usually be controlled directly within the DC intervention. By contrast, the achieved effects cannot be controlled directly. While it is true that the process of transforming delivered outputs into effects is indeed planned (using endogenous and exogenous assumptions 2 ), most exogenous assumptions lie beyond the control of the domestic and foreign partners who cooperate in the DC intervention, so that transformation cannot be controlled directly within the DC intervention. This means the logframe can only describe risks and formulate the expected likelihood of purposes and goals being achieved to a defined extent. This likelihood diminishes along the results chain from resources (highest) to achievement of the higher-level goal (lowest). How the DAC differentiates objectives and goals How to measure risks 2 If the assumptions required to achieve an effect can be partially influenced by the DC intervention, then we are dealing with endogenous assumptions. If the required assumptions are outside the control of the donors and/or partners/responsible organisation(s), then they are exogenous assumptions. 9

16 CHAPTER 2 Basics of evaluation methodology Setting aspiration levels (8) To make valid statements about the effects and benefits that developmental measures actually achieve, we must first determine the extent (aspiration level) to which the individual objectives are to be reached. These aspiration levels must relate to the development potential of the country concerned. 2.2 Determining the actual results Basic approach Key terms (1) To assess DC measures in developmental terms, the assumptions or causal relationships made ex ante in the logframe must be reviewed ex post to see whether they actually occurred in the field. (2) The distinctions between inputs, outputs, outcome and impact are now commonly used in both literature and practice to identify actual effects that occur at different times, and taken to compare these with the planning parameters in the logframe [Prennushi et al. (2000), p. 108]. This enables a gap analysis to be made. Content-wise, these terms are defined as follows: Term Inputs Activities Outputs Outcome Impact Results Results Chain Definition The financial, human, and material resources used by the development intervention Actions taken or work performed through which inputs, such as funds, technical assistance and other types of resources are mobilised to produce specific outputs The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcome The likely or achieved short-term and medium-term effects of an intervention s outputs, positive and negative, intended and unintended, which result directly from the usage of the outputs of the DC intervention and can be directly attributed to it in terms of cause/causally and/or quantitatively Positive and negative, often long-term effects, indirectly produced by a development intervention, i.e. those that can be attributed to it The entirety of positive or negative outputs and effects (intended or unintended) of a development intervention The causal sequence of a development intervention - necessary to achieve the desired objectives beginning with inputs, moving through activities and outputs, and culminating in outcome and impact Source: OECD (2002). 10

17 Basics of evaluation methodology CHAPTER 2 Indirect changes actually effected at goal level by a DC intervention are checked at impact level, whereas all effects that were a direct result of a DC intervention s outputs being used (and which can be attributed to it causally or qualitatively) are recorded at objective level, using the outcome of all effects that occurred. On the level of actually delivered outputs, all goods, services and potentials are identified that resulted from the intervention. So while these outputs create the relationship with planned outputs in the logframe, inputs compare the actual resources used with those that were planned. The planned and unplanned, positive and negative outputs and effects of a DC intervention i.e. the actually identifiable results are thus measured at all levels. Measured results as a summary of outputs and effects Measuring is done with the help of indicators the same indicators defined in the logframe ex ante to later on check the determined assumptions and objectives. (3) By comparing the measured results established ex post with the planning figures in the logframe, a gap analysis is obtained. For this gap analysis to provide systematic quality control of the intervention, it must compare the underlying aspiration levels of objectives and goals with the actually identifiable direct effects and higher-level impact. Any differences between planned and factual degrees of achievement are explained by scrutinising the assumptions that underpinned them, since the greatest risk of unsatisfactory achievement lies in their having been compromised. This means the gap analysis should not be restricted to the goal level, but should consider the underlying assumptions too a process that takes place during evaluation. (4) The comparison of situations with and without intervention has become known as the with and without principle. This is now an integral part of any good ex post evaluation, because observable changes can be attributed to the intervention made. 3 Identifying an appropriate without situation ( counterfactual ) as a benchmark poses a difficult challenge. The problems connected with this issue will be discussed in a separate working paper. Gap analysis The quest for the without scenario 2.3 Evaluation in the form of a goal-oriented gap analysis (1) Developmental assessment of DC interventions takes place during the evaluation process. The test criteria of the OECD Development Assistance Committee (DAC) are now widely accepted in developmental policy circles, and are used internationally as a basis for evaluating development projects. [IDD (2005), p. 18]. Developmental assessment always includes the following criteria: relevance, effectiveness, efficiency, impact and sustainability. Their content can be described as follows: DAC evaluation criteria 3 Another principle is the before-after principle which compares conditions at two points in time. Any changes that occur are then attributed to the development intervention. The danger with this approach is that changes that were not caused by the intervention may still be attributed to it. 11

18 CHAPTER 2 Basics of evaluation methodology Criterion Content Relevance Effectiveness Efficiency Impact Sustainability Are we doing the right things? The extent to which the objectives of a development intervention are consistent with target groups and beneficiaries requirements, country needs, global priorities, partners and donors policies as well as the basic political orientation of the donor s government Are we achieving the objectives of this developmental measure? (at the level of project/programme objectives) The extent to which the development interventions contribute to the achievement of the (direct) objectives of the intervention (gap analysis), as well as the achievement of other objectives which have not been determined previously but are considered to be important Are the objectives of the developmental measure being reached in an economical way? This criterion relates to the relationship between resources and results (inputs : outputs = production efficiency, inputs : outcomes = allocation efficiency). Does the developmental measure contribute to achieving (higher) developmental goals? (level of goals and beyond) In the context of this criterion it needs to be checked if and how the development intervention contributes to achieving the desired development goals. Furthermore, it will be checked whether other positive and negative higher developmental effects (along the entire chain of effects) have been produced. Will the positive effects/benefits last? The continuation of benefits from a development intervention after major development assistance has been completed; the probability of continued long-term benefits Source: OECD (2002). (2) Within the context of German DC a sixth criterion (coherence, complementarity & coordination) has been defined: Coherence, complementarity & coordination Have the developmental activities of individual donors been harmonised with each other? This criterion assesses the extent to which the developmental claims made on the partner country by individual donors are contradictory to or compatible with each other. The coordination of these political claims on the part of the donors with those of their partners is not taken into account here, since it is already incorporated in other evaluation criteria, such as relevance. Source: BMZ (2006). 12

19 Basics of evaluation methodology CHAPTER 2 This criterion is particularly important for programme and country evaluations, especially of jointly financed programmes. It can usually be ignored for project evaluations. (3) German DC assesses interventions in terms of relevance, effectiveness, efficiency and impact on a six-stage scale to obtain an overall assessment of their developmental effectiveness. The six stages are explained below. The six-stage rating system of German DC Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 Stage 6 Very good result; clearly exceeds expectations Good result; fully meets expectations; no major shortcomings Satisfactory result; falls short of expectations; positive results predominate Non-satisfactory result; falls well short of expectations; some recognisable positive results, but negative results predominate Clearly unsatisfactory result; some positive partial results, but negative results clearly predominate The intervention is useless, or has made the situation worse. Source: Own translation of BMZ (2006). 13

20 CHAPTER 2 Basics of evaluation methodology Stages 1-3 denote a positive or successful assessment. Stages 4-6 denote a non-positive or non-successful assessment. Evaluating sustainability (4) The following four-stage scale is used when evaluating sustainability: Stage 1 Very good sustainability: High probability that the project s developmental effects which have been positive so far, will continue unchanged, or even increase Stage 2 Stage 3 Stage 4 Good sustainability: High probability that the project s developmental effects which have been positive so far, will only decline slightly and will remain distinctly positive (this is normally what can be expected) Satisfactory sustainability: High probability that the project s developmental effects which have been positive so far, will decrease considerably, yet still remain positive 4 Insufficient sustainability: High probability that the project s developmental effects which were not sufficient at the time of evaluation, will not improve to the extent that positive developmental effectiveness can be achieved 5 Source: Own translation of BMZ (2006). The overall assessment Other Rating Systems (5) The overall assessment on the six-stage scale is obtained by weighting the five individual criteria in a ratio that must be calibrated individually for each intervention. Stages 1-3 denote a successful project, whereas stages 4-6 denote a non-successful project. In this context, it must be remembered that for a project to be deemed successful in developmental terms, it must normally score satisfactory (stage 3) or higher in terms of its effectiveness (the extent to which it achieved its objectives), its effects at goal level (higher-ranking developmental impact) and its sustainability. (6) Other donors have opted for a four-stage rating system instead. Stages 1-3 describe successful measures and correspond to the German system. Stage 4 covers all projects that were rated non-successful, and corresponds to stages 4-6 in the German system. In the authors view, the rating system used is of secondary importance. 4 This stage also applies when a project s sustainability up to the time of evaluation is deemed insufficient, but will in all probability improve, so that the project will achieve positive developmental effectiveness. 5 This stage also applies when developmental effectiveness has so far been positive, but will in all probability seriously decrease, and hence no longer meet the demands for stage 3. 14

21 What is programme-oriented joint financing? CHAPTER 3 3 What is programme-oriented joint financing? CHAPTER To better understand the term PBA which is still defined in different ways in international literature, we will start by briefly discussing the meaning of programme orientation and joint financing (both of which are part of PBA) before examining the structure and the various forms of PBA. 3.1 Programme orientation (1) There is no consistent definition of the term programme orientation either in literature or in the field. While the DC debate on programmes and their design often covers the strategic relationship between the multiple projects of a single donor, including those carried out independently (with a view to clustering them by content), this terminology cannot be linked directly to programme-based aid. The policy document/position paper of the Federal Ministry for Economic Cooperation and Development/BMZ (2001) points out that project clustering can indeed be a form of programme-based aid but only if the clusters are consistently oriented to the programmes of the partners. Furthermore, the Ministry claims that programme orientation must also involve acting jointly with other donors to create a consistent overall programme, in conjunction with the partner s general and sectoral strategies and implementation mechanisms. This involves the use of existing partner structures in order to improve ownership, and hence to increase the developmental effectiveness of DC contributions [Klingebiel (2003), p. 1]. The potential forms of manifestation therefore depend on the specific programme objectives, and on conditions in the cooperation countries. (2) The extent to which programme-based aid also presupposes joint action within the donor community is a subject of international controversy. One frequently-voiced view is that all programme-based aid must involve joint funding. Strictly speaking, however, separate funding by different owners can constitute programme-based aid too, provided funding is based on joint policies, and provided their content and concepts harmonise while keeping the partner s strategies in mind [Schäfer (2004a), p. 4]. (3) Due to the growing importance of programme-based aid in international development policy, and the different views expressed in literature as to what should be subsumed under this term, a consistent definition of programme-based aid is necessary. In the authors view, it seems wise to use the terminology coined by the Learning Network on Programme-Based Approaches (LENPA) 6 which was founded in Canada. The LENPA definition describes four key elements: What is programme orientation? Joint action The LENPA key elements 6 LENPA is an informal network of international analysts, specialists and development institutes set up for the exchange of information and analyses on programme-oriented 15

22 CHAPTER 3 What is programme-oriented joint financing? Leadership by the host country or organisation A single programme and budget framework Donor coordination and harmonisation of procedures Efforts to increase the use of local procedures over time with regard to programme design and implementation, financial management, and monitoring and evaluation [Lavergne and Alba (2003), p. 2]. The term programmebased approaches (PBA) The need for joint dialogue The (co)responsibility of the partners (4) The advantage of this LENPA programme terminology is that, in addition to covering sector programmes, it also includes poverty reduction strategies, cross-section programmes, regional and local programmes, and programmes by NGOs [Schäfer (2004a), p. 3]. In this context, programmes can also be cross-project programmatic approaches with a clearly recognisable common overarching concept [Klingebiel (2003), p. 5]. LENPA therefore uses the term programme-based approach (PBA) which has become fairly common among development professionals. (5) The starting point for the programme approach is concerted dialogue between donors and partners, in which the two parties agree on the (programme) objectives and the results that are to be achieved. These should be oriented to the development strategies of the partner countries, e.g. the Poverty Reduction Strategy Papers (PRSPs) or sector strategies. As part of these agreements, and before any measures are taken, efforts should be made to better harmonise the procedures used by donors and recipients, and to better coordinate the approaches taken by different donors, building on the partner s processes and structures. Essentially, this involves the areas of contract award, expenditure planning, reporting, capacity development as well as monitoring and evaluation. (6) In keeping with the intentions of the programme-oriented approach, the responsibility for the activities required to achieve the desired objectives and goals rests with the partners. The idea is to use local structures to strengthen institutional capacities sustainably. The donor s active support is required only if national procedures are not sufficiently transparent or efficient, or if the necessary reforms are too great for the partners to absorb. (7) The main forms of programme-based DC are sector-wide approaches (SWAps), structural adaptation measures in the form of poverty reduction strategy credits (PRSCs), and budget support [BMZ (2001), p. 4]. DC aimed at optimising DC efforts in general. International conferences are organised annually and serve as an exchange forum [Lavergne (2005), p. 1]. 16

23 What is programme-oriented joint financing? CHAPTER Joint financing mechanisms (1) While different forms of PBA show similarities in terms of how DC is being utilised, joint financing is primarily a specific form of financing. Joint financing implies that the use of the allocated funds to implement several coherent measures can flow into PBA, as joint financing would not make sense if individual donors did not agree on how funds were to be used. Hence, a consistent disbursement process should be in place. The means by which resources are allocated can be controlled in different ways. Joint control by the donors and the partner is the rule, although control can also be transferred to the partner, or can remain completely with the donors [Klingebiel (2003), p. 8]. (2) According to the German Federal Ministry for Economic Cooperation and Development, joint financing of a specific project or programme relies on an ex ante agreement between several donors and the (host) partner with regard to the objectives, important measures and expected results. The planned interventions are directed by the partner. The stakeholders agree on the content and on uniform procedures, as well as on auditing criteria, and accounting and monitoring procedures [BMZ (2001), p. 4]. In this sense, joint financing includes central elements of PBA. As a result, it is often regarded in the relevant literature as being one of several types of PBA a type in which specific measures cannot be directly attributed to individual donor contributions. (3) The authors believe it would be more appropriate and understandable to view joint financing as a separate form of support that is independent of PBA even though in reality, joint financing and PBA usually do go hand-in-hand. The reasons are that joint financing incorporates specific financing aspects that are not covered by PBA, and that stand-alone projects or programme-independent projects can also be jointly financed. We therefore recommend the following definition: What is joint financing? The definition of the German Federal Ministry for Economic Cooperation and Development The authors definition Joint financing is a specific form of support that comprises central elements of PBA, but also includes the provision, allocation and disbursement of financial resources as well as stand-alone projects. In joint financing, specific measures can no longer be directly attributed to individual donor contributions. (4) Normally, joint financial support from several donors that is granted and used by means of a consistent procedure takes one of two forms: budget funding or basket funding. a) In budget funding, donor funds are channelled into the national budget of the partner countries. There are two types of budget funding sectoral, and general. In sectoral budget funding, the use of financial resources is earmarked for a certain sector; general budget support is not tied to ex ante conditions in terms of its allocation, but is usually linked to indicators that can be assigned to the social sector or elsewhere to ensure that at least some of the resources are indeed used in that sector. Budget funding 17

24 CHAPTER 3 What is programme-oriented joint financing? Basket funding b) In basket funding, funds provided by the donor community are paid into a common basket outside of the national budget [Schäfer (2004b), p. 46]. This ensures that they will indeed be used to implement specific measures, and helps the partners control more effectively how resources are deployed. 3.3 Synthesis PBA-2: Simply combining PBAs and joint financing is not enough Simultaneous recording of expenditures and receipts The two main directions of PBA-2: Macroeconomic programmes (1) The use of the term PBA-2 presupposes a combination of programme orientation (in the PBA sense) and joint financing modalities (basket or budget funding). Yet PBA-2 can be more than this: in addition to consulting services for formulating and implementing the policies involved, it also includes technical support in the form of training and advanced training. The intense political dialogue at the heart of PBA-2 also envisions reforms and measures designed to overcome structural obstacles in the developing countries [Djafari (2005), p. 18]. (2) Furthermore, PBA-2 sets the focus on expenditure due to the need for partners to spend financial resources responsibly. This can lead to the income aspect of the budget in other words, the mobilisation of internal resources being neglected. Naturally, there is more to development policy than spending policy alone, so income-related aspects (and other relevant questions) must also be included in the political dialogue [Schäfer (2004b), p. 48], something that is happening increasingly in the context of PBA-2. (3) By summing up these building blocks, the following forms of PBA-2 can be obtained [see also BMZ (2006a), appendix 1]: a) Macroeconomic programmes: Macroeconomic programmes are designed to support cross-sector economic and institutional reforms, and also reforms that concern local regulatory policies. Frequently, these programmes also introduce selected sector policy issues into the political discussion. Conditions often serve as an instrument for political dialogue, for coordinating donors, and as prerequisites for disbursement. Donors and the host partner country collectively determine the objectives, measures, monitoring indicators and prerequisites for disbursement (triggers). Donor resources are transferred to the general budget. This type of funding is carried out in the form of general budget support by means of: Poverty Reduction Support Credits (PRSCs) granted by the World Bank 7 which are often co-financed by other donors. These programmes are undertaken to support (politico)-economic, social and institutional reforms; Multi-Donor Budget Support programmes (MDBS) by other donors [bilateral programmes, European Commission, regional development banks (e.g. AfDB) 8 ]. In terms of content, these macro(economic) programmes mainly, though not exclu- 7 PRSCs are one of the World Bank s development policy lending instruments. When used as an instrument for general budget support, they accompany the implementation of PRSPs. 8 Recently, the World Bank has been open for participation in MDBS too. 18

25 What is programme-oriented joint financing? CHAPTER 3 sively, address the social sector (education, health water, etc.), as well as governance issues (public budget management, etc). b) Sector programmes Sector programmes Sector programmes (Sector Wide Approaches SWAps) support reforms and finance expenditure within a single sector. While they can be carried out in any DC focus area, they have traditionally and particularly been important in the sectors of education, water/effluents, health, transportation and agriculture (which are frequently the focus of PRSPs). They can also include overarching themes such as decentralisation/communal support, and can include several selected sectors. As a rule, sector programmes are funded via one of the following: sector-related budget support: donor resources flow into the general budget and are earmarked for the sector concerned. 9 No individual measures are defined in this context; basket funding: donors jointly finance an outgoing payments budget for the implementation of a basket of measures that normally results from the sector strategy. This basket is earmarked for the measures agreed. It is not possible to trace how the resources of individual donors were used. (4) Among the different forms of PBA, particularly budget support has been growing in importance for some years now. Budget support can occur both in the form of general budget support (macroeconomic programme) and sector-related budget support (sector programme). From the current developmental point of view, however, general budget support seems to be the more interesting alternative, the more so as there are many overlaps between sector-related budget support and basket funding. Therefore, the following remarks on PBA(-2) budget support confine themselves to (the topic of) general budget support. Amongst international donors there is an increasing number in favour of providing DC primarily (or even exclusively) in the form of general budget support, with the consequence that funds are transferred directly into the (national) budget of developing countries without any further conditionalities. This trend is enforced by the World Bank and the International Monetary Fund (IMF) and by several bilateral donors with their respective development agencies [AGEZ et al. (2005), p. 3]. 10 The concrete expectations that such general budget support measures are supposed to meet are not only that they contribute directly to increasing growth and reducing poverty [Amtsblatt (2005), p. 31], but also that this form of DC achieves more leverage, by strengthening and stabilising the responsibility of the partners (ownership) as well as local institutional structures (i.e. first and foremost the quality of public administration and public household systems). Furthermore, regular political dialogue is to engender ongoing/continuing reforms and adjustments. PBA-2 budget support chances and risks 9 This means that expenditures in a particular sector must reach a defined, agreed (high) level (basic concept of additionality of donor resources for a specific sector). 10 Most importantly, the British Department for International Development (DFID), the European Commission (EU) and the state development agencies of the Nordic Plus Group (Denmark, Finland, Great Britain, Ireland, the Netherlands, Norway and Sweden). 19

26 CHAPTER 3 What is programme-oriented joint financing? Budget management problems and risks (5) General budget support differs from other currently used forms of DC support in particular in terms of funding and complexity. The pooling of donor contributions and their direct transfer into the budgets of developing countries leads to a situation where clearly higher demands are made on the budget management of the partners as well as on their willingness and capacity for cooperation (or their absorptive capacity). The higher demands go hand in hand with greater risks as regards the fulfilment of expectations in connection with budget support. It is for this reason that in discussing budget management issues the fourth chapter is going to deal closely with the risks resulting from severe shortcomings in budget management. These are often subsumed under the term of fiduciary risks. On the other hand, since an analysis of the risks entailed in general budget support is not meant to focus exclusively on the weaknesses and shortcomings of the partners, chapter four also discusses risks not related to the partners which can adversely affect the success of this type of support to a considerable degree. The fifth chapter will then deal with the consequences that these risks have for the evaluation of general budget support programmes. 20

27 Problems of budget management and risks inherent in general budget support CHAPTER 4 CHAPTER 4 Problems of budget management and risks inherent in general budget support 4.1 Budget management is of central importance for functional budget support (1) The international donor community has generally agreed that for general budget support to take hold, partner countries public administration institutions must have basic levels of functional governance and administrative skills, transparency and efficiency. Since budget support funds are transferred directly to the budgets of developing countries, donors pay particular attention to national budget management which needs to harmonise several goals (namely, that of the donors and that of the partner) at different levels. Whereas at the macroeconomic level, the fiscal discipline of the overall budget must be ensured, at the strategic level, budget management should ensure that resources are allocated according to the politically determined and jointly agreed priorities. The efficiency and effectiveness of budget implementation is then ensured at the operative level [Schiavo-Campo und Tommasi (1999), chapter 1, p. 3f.]. (2) Clearly, functional budget management is a basic prerequisite for the success of general budget support. Consequently, the diagnosis and reform of public budget management play an important role in the cooperation between donors and partner governments in the area of PBAs. As a rule, general budget support is granted only after a comprehensive and systematic ex ante analysis of public budget management has been carried out. 11 This analysis serves to identify concrete reform needs, e.g. determination of the most appropriate form of assistance: general or sector-related budget support, sector programme(s), isolated stand-alone projects as well as the appropriate mix of FC-TC instruments for the respective developing countries. The analysis can thus serve as a kind of mechanism for finding the right projects or programmes for donor and partner institutions within the scope of programme-oriented approaches [cf. Leiderer (2004), p. 24]. (3) Public Expenditure Financial Accountability (PEFA) is the most important initiative of this type of ex ante assessment of budget management to date. It was jointly launched by the World Bank, the European Commission, DFID and other donors and has had an office at the World Bank since The instruments for analysis now available are relatively comprehensive, but need to be harmonised more as far as content and method are concerned [cf. Leiderer (2004), p. 64]. However, neither the strengths or weaknesses nor the risks entailed in PEFA 12 will be discussed in greater detail in this working paper; Goals of budget management Ex ante analysis in order to assess the quality of budget management Public Expenditure Financial Accountability (PEFA) as an example of an ex ante assessment of budget management 11 This differs from ex ante analyses which focus on an analysis of the entire country context and are discussed later on. 12 More information in Leiderer (2004). 21

28 CHAPTER 4 Problems of budget management and risks inherent in general budget support rather, the reader s attention is to be called to the fact that PEFA constitutes a necessary ex ante set of instruments for assessing budget management: on the one hand, the allocation of funds in the form of general budget support depends largely on the results of PEFA analyses; on the other hand, these results are ultimately reflected in the assumptions and risks used to specify the likelihood of the defined outputs and the objectives being achieved. Budget management systems still have considerable deficits Capacity development and ownership as prerequisites for successful general budget support (4) One reason why ex ante analyses of budget management systems are so important is because the quality of public budget management systems directly affects the extent to which the external financing sources required for national development and poverty reduction strategies are made available. Insufficient planning and implementation capacities (and also insufficient internal and external control) increase the risks associated with budget support. Most developing countries still have considerable deficits in these areas. In particular, serious weaknesses may still be found in the completeness and realism of national budgets, and in terms of transparency, responsibility and accountability with regard to their implementation. These include a lack of staff and technical capacities, inappropriate budgeting systems, corruption, and the lack of political will to provide transparent, poverty-oriented budget management. Isolated TC projects (still the dominant form) also place an extra burden on partner countries which must meet a wide range of donor demands concerning the administration of the means allocated, with all the reporting activities this involves. This ties up a considerable amount of the partner countries internal resources, thereby hindering the efforts of their governments to develop policies according to their political priorities, and to earmark budgets for these activities in order to deliver public services efficiently and effectively. [Leiderer (2005), p. 2]. (5) Another important aspect of budget management is the fact that the absorptive capacity of recipient countries is often very limited as regards financial contributions which are often pooled and directly transferred to national budgets. This raises the question of how much effort donors should make to enable their partners to induce sustainable progress in their reforms with these financial transfers. In the context of general budget support, measures designed to either lay the foundations for, or encourage responsible policies in partner countries are crucial. The key measures here are those which strengthen capacity development 13 ) and increase the ownership of the partner country for the PBA budget support s underlying development programme. This means that as part of general budget support, a degree of expertise and a certain mentality must be transferred to partner countries to ensure that the funds placed unreservedly at their disposal will be directed to the predetermined core areas responsibly, and will thus 13 On the basis of the DAC, the term capacity building will be replaced by the term capacity development. The reason is that certain capacities are normally available in the partner countries, even though they may not yet be developed sufficiently. The development of these capacities makes better allowance for the orientation to the partners priorities. That is one of the main characteristics of programme-based approaches (OECD-DAC 2006a). 22

29 Problems of budget management and risks inherent in general budget support CHAPTER 4 help to initiate or further a sustainable development process. In other words, for general budget support to be successful, a continuous learning process must be induced to create public awareness in developing countries. This will enable them to independently identify limiting factors and to select appropriate measures by which to overcome them. However, this type of learning process also poses risks since budget support initially involves placing blind faith in those who receive it as funds must initially be transferred to budgets unconditionally in order to foster the desired improvements in ownership. These and other risks will be examined in greater depth in the following. 4.2 Major risks with general budget support - an overview (1) Due to its special characteristics (alignment of DC with partners programmes and structures, promotion of independence and of autonomous action and ownership), PBA budget support is generally seen as an instrument that is fraught with risk [Bernasconi (2006, p. 49]. Since the degree and controllability of risks are of particular significance when making evaluations, the problem of the risks inherent in general budget support must be examined in more detail. Risks with general budget support In the broadest sense, risk may be defined as any factor that jeopardises the achievement of development goals. Therefore, an evaluation of such risks should always take the goals which underpin those risks into consideration. These goals will, in turn, depend on the respective form of DC, as well as on country-specific parameters. Due to the variability of goals, the risks inherent in general budget support must be identified separately for each country. (2) The main risk groups are: Main types of risk fiduciary risks; risk of corruption as a special case of fiduciary risk; macroeconomic and political risks; risks of the programme being rejected; risks when implementing PBA-specific benefits; risks caused by the collaboration of other donors. There are no clear barriers between the various types of risk; individual risks can converge and amplify each other Fiduciary risks (1) When donor funds are paid directly into the budgets of partner countries, the main problems are the fiduciary risks. These mainly relate to poor economic and political governance and administration caused by weaknesses in public finance administration. As a rule, budget subsidies are used in accordance with the partner countries national budgetary procedures; once transferred, they are almost entirely beyond the donors control. Any shortcomings in public financial systems will pose a high risk of funds being misappropriated. Also, promises of general budget support constitute a long-term The basic problem of donor funds directly transferred to the budgets of partner countries 23

30 CHAPTER 4 Problems of budget management and risks inherent in general budget support commitment: donors pay funds into developing systems over a period of years, without receiving sufficient guarantees in terms of transparency, reliability and efficiency [Amtsblatt (2005), p. 6]. (2) There are four main fiduciary risks: The main fiduciary risks What determines the levels of these fiduciary risks? Transfer risk: means could be diverted before they reach the target country s ministry of finance. Diversion from intended use: means might not be used for the purpose intended; budget implementation might not adhere to the budget passed by Parliament. This can, for instance, lead to conditionalities not being observed. Non-registration: financial flows might defy external controls, due to inadequate accounting. Also, if controls are not aligned with quality needs or deadlines, they might distort perceptions of the actual financial flows. Sub-optimal use of resources: resources might not be used efficiently and effectively [Bernasconi (2006), p. 49]. (3) In general budget support, donors are no longer responsible for the way that means are allocated. As a result, the levels of fiduciary risk depend mainly on how well budgets are formulated, how they are handled, and how independent their supervision is. 14 Budget formulation: Budget formulation Although budget formulation can foresee clear rules for aspects such as steering the budgetary process, the way in which specific areas of responsibility are allocated may be unclear on the partners part. This could increase many of the risks mentioned above. Although the political priorities may be apparent, the way in which political interests, planning and the budgetary process interact can still be ambiguous, as can the relationship between resources and political goals. This can lead to deficits in the efficiency and/or effectiveness with which resources are used. Since budgets are a yardstick for current and potential expenses, any underestimates made when planning the budget income can lead to disproportionate expenditure. Budget management: Budget management If funds are distributed, but not used, they become immobilised cash assets which benefit banks at the expense of national governments. This situation is caused by inefficient resource management, and can lead to a reduction in services on the part of recipient countries. If fund allocators cannot be sure when they will receive the money earmarked for them from the Treasury (and in what amount), this can undermine the budget s credibility and reduce the availability or quality of public services. 14 For more detailed examples, see DFID (2002), p.18ff. 24

31 Problems of budget management and risks inherent in general budget support CHAPTER 4 Both unauthorised reporting standards and faulty and late financial statements increase the risk that funds will not be registered, and hence that resources might be mismanaged in future. Ultimately, this can lead to sub-optimal use of resources. If fiscal data and/or bank documents [i.e. written evidence of a business or accounting transaction] are flawed, the result can be disproportionate expenditure. Independent supervision: Independent, regular audits of public accounts can fail if the General Accounting Office respectly the Court of Auditors is either understaffed, or is not authorised to carry out precise audits. This increases the risk of maladministration and fraud. Although Parliament is supposed to disclose incompetence and corruption by strictly auditing public spending, there is a danger that o budgets might be represented in a roundabout way, and would thus lack transparency; o legislature may be insufficiently trained for this type of auditing. Independent supervision As a result, implementation and follow-up recommendations are often ineffective, and do not necessarily improve the quality of services rendered by the State Corruption a special form of fiduciary risk (1) One risk often associated with general budget support is the risk of corruption. However, an accurate measurement of the impact of corruption upon procurement is virtually impossible, even though perceptions of the incidence of corruption can have a profound effect upon the decision-making of government, donors and suppliers alike [Finland (2007), p. 13]. Furthermore, it is often impossible to make precise distinctions between unlawful payments, divergent documentation schedules, and divergent updating criteria. (2) On the other hand, practice has shown that sensitive issues like the extent of corruption and the amount or structure of military expenditure 15, can be dealt with more easily in the context of general budget support, since the possibility of active participation in political dialogue and the close monitoring/observance of reforms this involves tend to create more leverage in the fight against corruption than an isolated (project) approach can achieve [cf. Adler (2004), p. 50]. (3) Some development experts even assume that general budget support is no more prone to corruption than project-based support. They cite scientific evidence showing that corrupt individuals prefer to engage in projects which require high monitoring effort, or comprehensive and separate control of funds. However, in the political dialogue on the overall budget, this type of control mechanism for budget resources is rarely an option. It is also argued that projects often bypass tax authorities, and are hence more The difficulty of measuring corruption The opportunities of political dialogue Conclusion: in terms of corruption risk, budget support is no worse than traditional projectbased support 15 One example is Uganda, where military expenditure was first increased, then significantly reduced following intervention by donors as part of budget support activities [cf. Adler (2004), p. 50]. 25

32 CHAPTER 4 Problems of budget management and risks inherent in general budget support susceptible to corruption [Bernasconi (2006), p. 50]. On the whole, the risk of corruption is unlikely to be any higher for general budget support than it is for traditional projectbased support Macroeconomic risks and political risks Threat of Dutch Disease The impact of exchange rates Neglecting the income aspect Political risks Poverty Social Impact Analysis (PSIA) (1) When donor funds are pooled in general budget support, influxes of funds into partner countries are considerable. While empirical evidence is still lacking, the risk of budget support triggering Dutch disease effects cannot be ruled out. Increased supplies of money can trigger a sudden increase in demand for domestic goods and services, and may lead to national currencies being revalued. This can put domestic manufacturers at a disadvantage in international markets [Bernasconi (2006), p. 50]. (2) Taking this further, domestic price rises normally increase the demand for foreign goods and services. Particularly when exchange rates are fixed, the result can be that the currency holdings earmarked for financing the necessary imports become depleted (due to higher imports and lower exports). There is also the risk that stable exchange rates (e.g. in the context of a stable anchor currency) a key factor for national stability might come under attack from speculators and eventually have to be abandoned. 16 These issues highlight the tremendous demands that macroeconomic management entails. (3) Due to the influx of unbound resources in general budget support, the focus often rests on partners expenditure policies, so there is the risk that the income aspect of the budget (i.e. the mobilisation of internal resources) could be neglected. However, development policy is more than a mere expenditure policy [Schäfer (2004b), p. 48], and establishing performant, sustainable income systems in partner countries is an important task in long-term development policy. (4) Budget-financed approaches are often linked to existing political risks which are to be overcome by means of relevant measures in the area of good governance. Political risks are generally taken to mean precarious political events, such as wars, revolutions or civil unrest. If a diversion effect s mode of impact causes funds that were allocated under budget financing to be used for other areas which have no significance for the development process, this can have serious consequences for the development process as such. (5) Macroeconomic risks create special challenges since installing preventive management measures is a very complicated matter. Risks should therefore be addressed and action taken before general budget support is granted rather than to wait for them to occur. When conducting an ex ante assessment to determine the consequences of general budget support for a particular country, the usual procedure is to perform a Poverty and Social Impact Analysis or PSIA. PSIA is an instrument that the World Bank is obliged to use, due to an initiative by its former President Wolfensohn [Asche (2005), p. 17]. PSIA is used as a basis for weighing up the different possibilities of political action [Djafari 16 Other potential macroeconomic effects of price hikes (inflation) or abandoning fixed exchange rates (e.g., changes to interest rates), will not be discussed here. 26

33 Problems of budget management and risks inherent in general budget support CHAPTER 4 (2005a), p. 19]. PSIA focuses on analysing the distribution effect of possible reforms ex ante, especially with regard to the poor. Basically, these are impact analyses; unlike impact evaluations, however, they are not carried out ex post, but ex ante, and do not rely on models and/or econometric methods. Apart from analysing potential economic effects, PSIA is also used to analyse the different groups which are affected by the reform and which can, in turn, influence it. PSIA also involves analysing the institutional situation of the partners [World Bank (2003a), p. 1ff.]. PSIA has now become an integral part of PBA [Djafari (2005a), p. 20]. Given the complexity und breadth of effectiveness of PBA, this kind of ex ante assessment is an important form of evaluation for the systematic anticipation of effects. However, it should be pointed out that within PSIA there have been efforts recently to examine the implementation and the effects of reforms ex post [Thilo-Körner (2004), p. 23f.] Risk of programmes being rejected (1) In PBA budget support, it is essential to coordinate the different donors. This means additional effort. Previous experience has shown that due to the learning and coordination needs, transaction costs in general budget support may even surpass those incurred in traditional project-based support. Particularly comprehensive programmes can cause high transaction costs due to the efforts required to coordinate the participation of different donors. As a result, either donors or recipients may reject these programmes, even though they would be of vital importance for the development process [Ashoff (2004), p. 3]. (2) Coordination and transaction costs should therefore not be regarded separately, but always be weighed up against the potential impact/effectiveness of DC. Proponents of budget-financed approaches view donor coordination as a way to improve the effectiveness of DC, since it leads to a reduction of transaction costs in the long term. The top priorities, apart from reducing the number of donors, are to reduce the number of DC measures, and to harmonise the various procedures [Ashoff (2004), p. 1]. Generally speaking, basically sensible forms of PBA and those with a broad effectiveness (in particular general budget support) should not be dispensed with in favour of other less costly DC projects, as this could lead to significant losses in terms of impact. The fear of high coordination and transaction costs The need to consider the potential impact/ effectiveness of DC Risks associated with PBA-specific benefits (1) In theory, there are many arguments in favour of general budget support, assuming one accepts that it generally increases the leverage of DC for example by generating ownership, allowing for major structural reforms, etc. In practice, however, these anticipated benefits occur only in the medium or long term. Experience to date also shows that they do not come about automatically, but require constant attention, technical support and political dialogue. It is questionable whether these can be guaranteed continually for long periods (approx. 12 to 15 years per budget support cycle). Many benefits of budget financing only materialise in the long term 27

34 CHAPTER 4 Problems of budget management and risks inherent in general budget support If technical support is poor or if partners cannot realistically assess of the chances and risks of general budget support, donors might forego taking measures that would be indispensable for the success of the programme they have been/are financing, because they lack trust in their partner. This highlights the importance of political dialogue which should include discussing the potential risks. The issue of the ownership required from partner countries The need for predictable financial flows into partner countries (2) General budget support both enforces and requires increased ownership on the part of partners. However, permanent coordination with donors can lead to problems as far as the development of ownership is concerned, since it notably affects sovereignty in budgetary matters, and hence reduces options for an independent formulation of policies in a particular country [AGEZ et al. (2005), p. 7]. Also, placing the focus on spending policy increases the risk that the power of the executive might be strengthened unilaterally at the expense of the legislature and other stakeholders [cf. Schäfer (2004a), p. 6]. (3) Regular, predictable financial flows are intended to help partner countries achieve stability, and development goals, by enabling them to plan reliably. Yet evaluations (e.g. DFID) have shown that general budget support is easy to stop, and technical or political reasons often lead to its being suspended [AGEZ et al. (2005), p. 10]. One possible cause could be (the) conditionalities. If disbursement is denied because conditionalities have not been met, general budget support can lead to considerable financial bottlenecks in partner countries, and thus to development losses particularly when community programmes funded by several donors involve substantial sums of money that are disbursed according to uniform procedures. In situations of this kind, the dilemma faced by donors is whether to accept non-compliance or suspend payments Risks caused by the cooperation of other donors Coordination issues New donors (1) Losses in efficiency and effectiveness are also the result of coordination problems both within the donor community, and/or between donors and their partner. As experience has shown, it is not just donor-recipient coordination that causes difficulties: often, donors cannot agree on uniform procedures, conditionalities, et cetera among themselves. (2) Coordination issues are further aggravated when as now happens increasingly often countries which used to receive development aid themselves become active as donors. Examples include China, India, Brazil, Mexico, South Africa and Thailand, all of which now assist poorer developing nations. In some cases, their activities are overshadowed by their own foreign economic and political interests to a far greater extent than is the case with OECD-DAC donors. At first sight, this new situation may well seem positive. However, an examination of the challenges involved in achieving a defined impact in developing countries by means of various interventions is enough to reveal that the involvement of these new donors entails risks too. Bearing in mind that coordination difficulties occur even within the donor community of OECD-DAC members, it is hardly likely that coordination with new donors will improve the overall situation. Differing goals and standards, particularly in 28

35 Problems of budget management and risks inherent in general budget support CHAPTER 4 terms of reporting, accounting or monitoring, can hamper the desired effectiveness of specific interventions. This means it is important to maintain an ongoing dialogue with new donors, and to weigh up carefully whether cooperation is desirable and possible. 4.3 Conclusion (1) Since the risks associated with PBA budget support largely depend on the general conditions in respective partner countries, they must be analysed on a case-by-case basis; blanket judgments are inappropriate. Thus, in some countries fiduciary risks might play a minor part, whereas macroeconomic risks must be classified as high. In other countries, the reverse could be the case. (2) On the other hand, differing procedural parameters used by donors could further increase individual risks as these procedures have not yet been sufficiently harmonised within the donor community. For example, different donors could have established different standards and methods for accounting, preparing monitoring reports, evaluation methods, as well as different conditionalities. This can cause considerable problems for recipients, and thus increase the risks involved in general budget support. These include inefficient and ineffective accounting, and also the excessive demands placed on partners as a result. (3) The willingness of donors to support a partner country (via budget support) in implementing its development strategy will largely depend on how donors assess the risks described in this chapter. Their risk assessments are also very important when evaluating budget support: for one thing, they provide ex ante information on the likelihood of achieving goals; for another, they can be used to explain the non-achievement of goals at a later stage. As a result, the risks listed here, and their evaluation, should be taken into account at all times when making concrete evaluations. The significance of general conditions in partner countries Risks caused by differing procedural parameters Risk assessment is also crucial for the evaluation process 29

36 CHAPTER 4 Problems of budget management and risks inherent in general budget support 30

37 Specific challenges for the evaluation of PBA in the form of general budget support CHAPTER 5 5 Specific challenges for the evaluation of PBA in the form of general budget support CHAPTER 5.1 Preliminary remarks (1) The risks discussed in chapter 4.2. come into play when capturing the inputs, outputs, outcomes and impact behind the evaluation criteria. In reality, however, these can never be captured in full due to the complexity of budget support programmes. Instead, spot checks are performed, using parameters that are considered most representative, and which can be determined with reasonable effort. The same risks also have a considerable influence on the sustainability of the programmes concerned, especially since evaluations are usually ongoing, not ex post. The evaluation checklists used for PBA budget support programmes should take these considerations into account. The key components of such questions will be dealt with in section 5.3. (2) Since evaluation requirements in general budget support are very complex, only the most important aspects will be examined in detail here. As a basic rule, the approach described in chapter 2 applies to general budget support too. This approach involves using four steps (setting up a logframe determining actual results gap analysis evaluation) to capture and evaluate the development effects induced by the DC intervention in this case, budget support. The processes involved in setting up the logframe and determining its effects both pose considerable challenges. Firstly, the risks described must be carefully incorporated into the corresponding network of a programme s effects (and be taken into account); secondly, a large number of changes must be analysed, due to the high complexity and desired broader effect of general budget support. (3) Country-specific context and budget management quality are determined using specific ex ante analyses. These can be either an integral part of the planned budget support, or can be performed as part of other concurrent development interventions. A third option is to fall back on analyses from previous DC interventions. The choice of analysis will affect the evaluation results greatly, as the relationship between analyses and budget support is particularly important for the criterion of efficiency. Where necessary, interdependencies with other interventions must be identified even before the logframe is set up, and the usefulness of their results must be checked. Consequences for budget support evaluations Same methodology, higher complexity Ex ante analyses and their impact on evaluation results 31

38 CHAPTER 5 Specific challenges for the evaluation of PBA in the form of general budget support Specific characteristics of general budget support and their consequences for evaluations Capacity development and ownership as an evaluation approach General budget support targets causes, not symptoms The need to consider DAC evaluation principles (4) The specific characteristics of general budget support pose special challenges. Whereas funds in sector-related budget support are harnessed to individual sectors (which largely determines the programmes objective) 17, general budget support is a socalled full-coverage intervention. This can affect any area of a country, and unlike sector programmes, there is normally no prescribed area of intervention. As a result depending on the country-specific context the first step is to identify which sectors should be supported in order to accelerate the development process (leading sectors). These sectors can represent the core of the entire evaluation process. (5) Difficulties also arise when determining the evaluation approach in the first place. The first concerns of general budget support are to strengthen staff capacities and increase ownership. Only then is it possible to assess the interventions aimed to improve the identified areas, together with the degree to which their targets have been achieved, by using the evaluation criteria as a reference. In other words, the basic difficulty when evaluating general budget support is to determine whether only the implementation of interventions and the degree to which their goals have been achieved; or whether the definition of interventions, as an indicator for the responsible use of the funds provided, should serve as the starting point of the evaluation. (6) In terms of a sustainable development process, and as a response to the previous success of DC, the authors of this working paper view general budget support less as a method designed merely to cure symptoms, but more as an attempt to get to the roots of the fundamental problems of partner countries. Specifically, this involves generating ownership, which means that the changed attitude of partner countries and their efforts to overcome poverty should undoubtedly be an important subject for evaluation. This also allows for the verification of the basic assumption that PBA makes a vital contribution towards inducing structural changes in partner countries. The increased focus on the impact of DC interventions should not, however, lead to a situation in which evaluations of the extent to which interventions achieve predetermined targets are completely neglected. Instead, evaluations should adopt a two-fold strategy, and should account for both aspects of reform progress, and should weight them according to the country-specific context. (7) Like all DC evaluations, budget support evaluations should follow DAC evaluation principles (such as independence, transparency, expediency, pertinence, participation, reporting, etc.) 18 consistently. This is of particular importance as general budget support evaluations should, as far as possible, be conducted by teams of evaluators from both donor and partner countries. To enable them to evaluate correctly, all team members must be sufficiently independent of the organisations involved in providing support. 17 Cf. Working Paper No. 1 issued by our Institute (ISP 2007). 18 For details see OECD-DAC (1994), p , and OECD (2006). 32

39 Specific challenges for the evaluation of PBA in the form of general budget support CHAPTER Orientation of the logframe (1) As a basic rule, general budget support evaluations should always be preceded by a logframe analysis, on which they can build. If not, evaluators should provide a logframe ex post. It is particularly important to determine purposes/objectives and higher-level goals, their indicators, and the corresponding targets (aspiration levels) of objectives and goal achievement. Key bottleneck factors must also be defined, as must the high potential sectors that can contribute significantly to reducing poverty. Since the focus in general budget support no longer rests on individual projects or sectors, and since ultimately, any sector in a country can be the subject of budget support, the relevant logframe categories inputs, activities, outputs, purpose and goals should be defined fairly broadly. Because it is impossible to record all these categories in full, representative values which can be obtained using reasonable effort must be used for the logframe instead, also along with their indicators. In conducting evaluations, it has become common practice to use the same terms for assessing actual achievements: outcome instead of purpose, and impact instead of goals. We will follow this practice in the following discussion. (2) Partners who wish to receive budget support must always develop a Poverty Reduction Strategy Paper (PRSP). This paper or at the very least, the submission of a poverty reduction strategy is currently seen as an indispensable prerequisite for the granting of budget support. This puts poverty reduction and the partner s willingness to reform (as identified in subjective assessments) at the centre of intended macroeconomic reforms [cf. Schmidt (2005), p. 27]. To accommodate country-specific objectives and general framework parameters, sectors considered as important in PRSPs should be dealt with in the logframe. On the basis of these analyses, the purposes/objectives, higherlevel goals and assumptions and risks should then be substantiated in the logframe. This makes sense, because a PRSP is in itself a kind of written ex ante deliberation about how to reform the country concerned. However, since most PRSPs cover all social and economic sectors in a country, they can hardly be used to derive recommendations for action. Instead, PBA budget support logframes should try to operationalise the objectives and goals defined in PRSPs, and break them down into smaller units. This will establish a logical relationship between all the levels in the logframe, and the objectives and goals of general budget support without the risk that the general budget support objectives and goals determined ex ante could run counter to the goals in the PRSP. The difficulty of determining outcomes, impacts, indicators and targets ex ante PRSP as a prerequisite Only in exceptional cases can the requirement for a PRSP be dispensed with. In such cases, the partner government must, as a minimum requirement, declare that it accepts the goals which form the normative core of budget support. (3) As a rule, the goals determined in a PRSP are incorporated in the objectives and goals system of the general budget support logframe. When operationalising individual objectives and goals, highly aggregated macroeconomic indicators are often used as a tool. The downside of such overall indicators is that frequently, they do not reveal the significant differences that exist in individual sectors or social groups. When designing general Breaking down overall indicators to facilitate institutional learning 33

40 CHAPTER 5 Specific challenges for the evaluation of PBA in the form of general budget support budget support, it therefore makes more sense to disaggregate indicators (especially macroeconomic indicators) into individual indicators, then assign these to the different logframe levels. This can add visibility to the development effects that are considered important and which are often found at local or mesoeconomic level. Also, most overall indicators used in PRSPs cannot isolate effects that occur at different times the very factor that is so important for institutional learning from evaluations (otherwise, we could merely speak of trends in development projects). Because of this, the decision to which extent indicators should be disaggregated is just as important as their selection. Overall indicators should be disaggregated jointly by donors and partner countries, since functional disaggregation will depend on the type and scope of available data, objectives and (higher-level) goals, as well as on the strategies proposed. For instance, overall indicators could be divided according to geographic, administrative or gender-specific aspects [cf. Prennushi et al. (2000), p. 111f.]. Logframe and impact chains (4) To display realistic impact chains of planned interventions in the logframe, the relevant means of transmission for inputs up to the impact level must first be identified. The causality map in Figure 5.1 below meets the general logframe requirements as set out in this paper. Owing also to the lack of experience with PBA budget support, this causality map is the most frequently method in literature for evaluating general budget support. In line with the logframe concept described at the beginning of this paper, and the conceptual modification discussed in item one (1) of this chapter, it distinguishes between the levels of inputs, outputs, outcomes and impact. The authors of the causality map have also introduced an extra level called immediate effects. The connections between these levels are illustrated in figure 5.1 [cf. Lanser et al. (2006), p. 2, as well as Booth and Lawson (2004), p. 11f.]. The six different types of input on the first level appear to be significant for all forms of PBA, and reflect the specific characteristics of PBA well. The second level examines the immediate effects of the inputs; in the opinion of the authors of this causality map, these should focus on changes in the relationship between aid, the national budget and national policy processes. The authors believe that the extra level would augment planning security, since the likelihood of desired results occurring at subsequent levels would be higher if the immediate effects listed for the second level actually materialised. Here too, it must be verified whether these immediate changes can be attributed to the conditionalities set by donors, or are the result of a collaborative policy dialogue in which both parties are on an equal footing. This seems sensible, as it is advisable or even necessary to identify the driving force behind the changes. Thus, statements are possible about progress achieved in generating ownership even at an early stage. 34

41 Specific challenges for the evaluation of PBA in the form of general budget support CHAPTER 5 Figure 5.1: Causality map for a general budget support programme (Diagram 1) Source: Booth and Lawson (2004), p. 12, and p

42 CHAPTER 5 Specific challenges for the evaluation of PBA in the form of general budget support The significance of auxiliary conditions (assumptions and risks) (5) In the view of the authors of this working paper it makes more sense to anchor the relevant analyses and effects in the auxiliary conditions, and to integrate them in assumptions and risks in order to at least reduce the complexity in presenting the logframe. At the very least this will simplify the logframes appearance. The additional level also makes it harder to distinguish clearly between immediate effects and outputs 19. In the medium term, positive effects on previous levels can lead to positive changes in general financial and institutional parameters at output level which are shown on level three in figure 5.1. The outcomes on level four reflect the expectation that strengthening capacities at government level will enhance the contiguous determinants of poverty. Level five shows the desired final effects of PBA budget support a reduction in poverty, and the social integration of all individuals into society [cf. Booth und Lawson (2004), p. 11ff.]. The basic structure is not enough (6) The approach presented here conforms to that of the DAC. This causality map also reflects the core areas that must be considered in PBA budget support. Thus, figure 5.1 can be used to identify the goals and expectations connected with the various levels. However, the precise content can be designed only by using a corresponding country analysis. It must also be remembered that the logframe shown above is merely the basic structure for evaluations of PBA budget support, and must still be extended, given the complexity of this form of aid. For instance, figure 5.1 does not include the interdependencies or feedback loops that exist on the same level, and which can lead to substantial changes. [cf. IDD (2005), p. 24f.]. The logframe could also include the prevailing conditions before interventions were implemented, or before budget support began. An assessment of trends that would arise without the support of the donors could also be made in this context. Generally speaking, general budget support evaluations require a more complex logframe diagram, as shown in the causality map below. This version is currently the one cited most frequently, notably because it has already been used in seven budget support evaluations in different countries. 20 The question of whether this structure should be adopted for all budget support evaluations (and thus be set in stone ), and to what extent the simplified causality map should be expanded, will be discussed on the basis of figure The criticism of the additional level raised here does not mean that the issues it implies should not be taken into account. One the one hand, the intention is to conform to DAC norms (or to provide seamless evaluations, in the case of German projects); on the other, to reduce complexity. This will become clearer in the figure below. As long as we are dealing with the level of immediate effects and the determination of activities belonging to the programme, there are no objections to their being shown on a separate level. However, difficulties do arise when the effects that result from inputs which are basically assumptions for the achievement of the outputs at output level, are listed separately. 20 These are the budget support evaluations in Burkina Faso, Malawi, Mozambique, Nicaragua, Rwanda, Uganda and Vietnam which were carried out on behalf of DFID by the International Development Department of the University of Birmingham (United Kingdom). Cf. IDD (2005). 36

43 Specific challenges for the evaluation of PBA in the form of general budget support CHAPTER 5 (7) The causality map in figure 5.2 shows both the vertical and horizontal impact interrelations that are expected from budget support. It correlates the different desired results with other results on the same level, and also with results that occurred previously. Although it is difficult to show the actual network of a programme's effects, such complex figures are necessary in order to capture (and above all, explain) the most important changes not least because different effects can reinforce each other, or even cancel each other out. This is the only way to assess the intensity of the effects and to identify reasons why a (non) intended change leads to desired or unexpected results (either on one level or across levels). Where necessary, this type of analysis also includes an analysis of repercussions at subordinate levels that can be induced via effects at impact level. (8) The areas listed in the causality map in figure 5.2 (cf. the various boxes) are indeed core areas in the evaluation process in general budget support. However, they hardly meet the demands (made in chapter 4) for capturing simultaneously both increased accountability (in the sense of interventions identified and carried out specifically for this purpose), and the changes induced by these interventions. Even when viewed separately, the attention paid to these sectors is far from sufficient. As can be seen in figure 5.2 overleaf, the first three levels (from entry conditions to outputs) are mostly based on the donors efforts to lay the foundations for a responsible policy. This is particularly true of the boxes level 1 and level 2. Only at output level (level 3) do changes and advancements that directly affect the partners outputs begin to be considered. Logframe should reflect the complexity of budget support Criticism levelled at the often-cited causality map Even at this level, however, the efforts made by the partners themselves are not scrutinised explicitly. The description does not reveal which contribution (e.g. toward achieving higher efficiency) the partners have made. Donors contributions, on the other hand, are defined by the inputs and immediate effects shown in the causality map. Only at the level of purposes/objectives does the focus rest exclusively on the efforts of the partners, or on issues that take ownership into account. This is essential because after all, the aim of general budget support is to enable partners to conduct their own reforms long term, and thus to achieve a manageable and sustainable level of development. 37

44 CHAPTER 5 Specific challenges for the evaluation of PBA in the form of general budget support Figure 5.2: Causality map for a general budget support programme (Diagram 2) Source: Booth and Lawson (2004) 38