STUDY ON EVALUATING SWISS COOPERATION STRATEGIES IN HIGHLY HARMONISED SETTINGS

Transcription

1 STUDY ON EVALUATING SWISS COOPERATION STRATEGIES IN HIGHLY HARMONISED SETTINGS

2

3 Study on Evaluating Swiss Cooperation Strategies in Highly Harmonised Settings Commissioned by the Swiss Agency for Development and Cooperation (SDC, Controlling Section and Focal Point Aid Effectiveness), Bern Final Report December 9, 2009 Rolf Kappel, Walter Egli, Marie-Laure Müller, Jutta Werner NADEL, ETH Zürich, 8092 Zürich;

4 Table of Content Page List of Acronyms...3 Executive Summary Introduction Fundamentals of Cooperation Strategy Evaluations Evaluating Portfolios of Aid Interventions Evaluation of Inputs, Outputs and Effects Process Evaluations Joint evaluations Evaluations of Cooperation Strategies: Current Practices and Future Perspectives Why Cooperation Strategy Evaluations? Current Practices Contributions to Mutual Accountability Outlook Evaluations of Programme Aid: Current Practices and Future Perspectives Current Practices Outlook Are Total Aid Evaluations Feasible? Recommendations for SDC...26 References...29 List of Interview Partners...33 Terms of Reference

5 List of Acronyms CGE Model DAC DANIDA DFID GBS IA IOB JAS M&E NORAD ODA OECD PGBS PRS PRSP SBS SDC SECO SIDA SWAP Computable General Equilibrium Model Development Assistance Committee Danish Ministry for Development Cooperation, Department of Foreign Affairs UK Department for International Development General Budget Support Irish Aid, Department of Foreign Affairs Policy and Operations Evaluation Department, Netherlands Ministry of Foreign Affairs Joint Assistance Strategy Monitoring and Evaluation Norwegian Agency for Development Cooperation Official Development Aid Organisation for Economic Development and Cooperation Partnership General Budget Support Poverty Reduction Strategy Poverty Reduction Strategy Paper Sector Budget Support Swiss Agency for Development and Cooperation Swiss State Secretariat for Economic Affairs Swedish International Development Cooperation Agency Sector Wide Approach 3

6 Executive Summary Cooperation strategies are important steering instruments for aid at the country level. Evaluations of these strategies satisfy accountability needs of stakeholders of aid and serve as instruments for organisational learning to increase aid effectiveness. Cooperation strategy evaluations must necessarily build on results of project and programme aid evaluations in a country portfolio. The higher the quality of these existing evaluations is, the higher is the quality of evaluations at the country level. To date many project and programme aid evaluations do not apply rigorous, quantitative methods to separate aid effects from other influences on target variables. In the case of programme aid (budget support) it is not possible to rigorously separate aid effects of all or individual donors from overall programme effects. Qualitative methods are frequently used to construct counterfactuals and to identify plausible aid effects. Investing more into high quality, rigorous evaluations of project and programme aid has also positive consequences for cooperation strategy evaluations that build on these results. Document analyses and interviews with evaluation experts indicate that donors diagnose weaknesses of current cooperation strategy evaluations and want to improve the benefit cost ratios. Current practices put a high emphasis on the evaluation of inputs and outputs of aid as well as the quality of the overall aid process. Much less emphasis is put on the evaluation of the overall or combined effect of all aid activities at the country level, a methodologically very difficult and challenging task. The efforts to collect new data and information, which complement existing evaluation results of an aid portfolio, vary widely between donors and between country cases. Many cooperation strategy evaluations rely heavily on qualitative approaches, while the use of quantitative methods is relatively rare. Donors frequently carry out these evaluations unilaterally, because governments of recipient countries have limited capacities to participate in such exercises, and because the evaluation results for an individual donor are often not sufficiently interesting for others. Exceptions are country cases with Joint Assistance Strategies or, more generally, country cases where the shares of budget support in total aid are high. If the harmonisation of aid increases in the future, the potential and relevance for joint cooperation strategy evaluation will also increase. To improve the benefit-cost ratio of cooperation strategy evaluations we recommend SDC to consequently use existing evaluation and monitoring results of project and programme aid effects for meta-evaluations of the whole aid portfolio at the country level. These metaevaluations should be complemented by a process evaluation of the whole cooperation strategy. To get the best results, we also recommend SDC to heavily invest in rigorous, quantitative evaluations of project and programme aid effects. These investments will contribute to improve aid effectiveness at the project, programme and country level. In addition we recommend that programme aid evaluations should under all circumstances be carried out as joint evaluations with the recipient government and all donors participating in the programme aid. With increasing shares of budget support in total aid, i.e. increasing harmonisation, these joint evaluations will become more and more relevant. In countries with 4

7 Joint Assistance Strategies and / or high shares of budget support joint evaluations may approximate evaluations of total aid in the foreseeable future. We recommend SDC to undertake an information campaign that explains to Swiss stakeholders of international cooperation the rationale of programme aid and SDC s commitment to the Paris Declaration and the Accra Agenda of Action. The information campaign should also help stakeholders to better understand why in the case of budget support the aid effects of a single donor cannot be separated from overall programme effects and overall aid effects. Moreover, stakeholders should be informed that with SDC s continued efforts to improve evaluations at the project, programme, and country level the quality and visibility of Swiss aid will be strengthened. 5

8 1. Introduction Following SDC s terminology we use the term cooperation strategy as a synonym for terms like country assistance strategy, country programme, etc. A cooperation strategy consists of a portfolio of aid activities, which can be divided into the three categories given below. (a) Project aid describes aid activities implemented by a donor or a contracted implementation agency, with financial flows and steering procedures outside the government budget and government entities of the recipient country. (b) Programme aid of budget support describes financial assistance to the budget of a recipient country. Programme aid includes general budget support (GBS) and sector budget support (SBS). SBS is often described with the term sector wide approach (SWAP). Basket funding is a special type of SBS. Several donors pool their funds and unify procedures, but funds and procedures are outside of those of the government. 1 We use the terms programme aid and budget support interchangeably. We distinguish between the evaluation of outputs and aid effects and process evaluation. Outputs are immediate results of aid activities (e.g. the number of schools built); aid effects comprise outcomes (e.g. the additional number of students being educated) and impacts of aid activities (e.g. higher incomes of more young people due to higher education rates). Process evaluations are concerned with the design and implementation of the cooperation process, including issues of relevance, alignment, harmonisation, and consistency of aid activities as well as the quality of the policy dialogue. We use the term results for the combination of these characteristics and the outputs and effects of aid. The term rigorous evaluation is used as a synonym for quantitative evaluation, and implies that intervention effects can be separated in quantitative terms from other influences (the counterfactual), either with a control group approach (ideally with randomised treatment and control groups), with a cost-benefit analysis, or with a set of control variables in a regression analysis. We do not define precisely the term highly harmonised, i.e. we do not give a cut-off point for programme aid as a share of total ODA (Official Development Assistance) after which aid is called highly harmonised. But for the subsequent discussion it is important to note that the higher this share the more important the role of joint programme evaluations and the higher the chances for total aid evaluations. The whole study rests on the basic assumption that SDC will continue to use cooperation strategies as a steering instrument for cooperation with priority countries (or regions). Given SDC s efforts to focus on fewer priority countries, given the trend to focus on fewer domains and sectors of intervention, and given the commitment to follow the principles of alignment and harmonisation, it seems imperative that SDC indeed uses cooperation strategies as a 1 Some donors and authors use the term basket funding for sector budget support with funds and procedures either within or outside government systems. 6

9 steering instrument and the evaluation of cooperation strategies as a unit of account(ability). Our literature survey and interviews with evaluation experts in several donor organisations strongly indicate that virtually all donors share this position. Various studies on approaches to cooperation strategy evaluations focus mainly on the guiding questions and the structures and processes of such evaluations, and put hardly any emphasis on methodological question in the strict sense of the word. For instance Morazán et al. state that evaluations at the country level are open for virtually all methods of the social sciences that can be applied for such tasks (2007, p. 5). That conclusion is not very helpful, because evaluations at the country level inevitably face certain time limits and resource constraints, which also constrain methodological choices. In other words, there is no way around the request to find methodological approaches with acceptable benefit-cost ratios. It is against this background that we focus in the study at hand more on methodological choices than many previous studies, and attempt to propose a methodological approach that promises indeed good benefit-cost ratios for evaluations at the country level. 7

10 2. Fundamentals of Cooperation Strategy Evaluations 2.1 Evaluating Portfolios of Aid Interventions As noted before, the cooperation strategy of a donor consists of a portfolio of project aid and programme aid interventions. The results-oriented evaluation of such a portfolio therefore requires the following three steps: (a) Project aid interventions must be evaluated with respect to outputs, effects (outcomes and impacts), and the quality of aid processes. (b) Programme aid interventions must be evaluated with respect to outputs, effects, and the quality of aid processes. (c) Somehow these evaluation results must be integrated into an overall evaluation of the whole portfolio, which, as it is often said, is supposed to be more than the sum of its individual elements. It goes without saying that evaluations of cooperation strategies cannot and do not fully and in all details accomplish steps (a) and (b). Given limited time and resource constraints for evaluations at the country level, experts involved in evaluations have two choices to get results for (a) and (b): either they make use of already existing results of project and programme aid evaluations, or they evaluate the effects and results of project and programme aid interventions not completely and to some extent rather superficially. As we will explain in chapter 3, in the real world evaluation experts often combine these two choices and then try to put the whole puzzle together. If one accepts the proposition that the evaluation of a cooperation strategy must inter alia be based on the evaluation of the project and programme aid included in the country portfolio, two conclusions follow. First, the best results at the country level can be achieved if as many detailed evaluation results as possible for project and programme aid are already available. Second, the quality of cooperation strategy evaluations increases with the quality of these evaluations results. Based on these conclusions and the analysis of current practices we suggest at the end of this study a concept for cooperation strategy evaluations comprising three recommendations. First, we recommend SDC to carry out high-quality project and programme aid evaluations. This is a worthwhile investment in the search of aid effectiveness at the project and programme level, but also at the country level. Second, we recommend SDC metaevaluations of these results as the main instrument to evaluate cooperation strategies. Third, we recommend process evaluations at the country level, which focus on assessing the quality of the overall aid process, and complement the meta-evaluations. Before elaborating on the analysis of current practices and the details of the rationale of this concept we must explain the methods to achieve high quality project and programme aid evaluations. 8

11 2.2 Evaluation of Inputs, Outputs and Effects Evaluating inputs and outputs of aid is usually the easiest part of an evaluation. Inputs delivered and outputs achieved can often simply be measured or described, in quantitative and / or narrative terms, and then compared against the target values that were set in the planning documents of projects and programmes. Evaluations of aid effects, i.e. outcomes and impacts, are much more difficult. Although evaluations of project aid and programme aid have certain things in common, they are also characterised by considerable differences. Project Aid Effects Using a stylised functional presentation for simplicity and clarity, the major task of evaluating the output or effect of a project intervention can be explained as follows: (1) Y = f ( I i, X j ) with Y being an observed change of the target variable, I i being a set of project interventions, and X j being a set of control variables influencing the target variable. The fundamental evaluation problem is to separate the effect of the project interventions from influences of other variables on the target variable (including perhaps domestic public actions that support or hamper project interventions). In other words, the evaluation must implicitly answer the counterfactual question: what would have happened if the project interventions had not taken place? Depending on the nature of a project several methods are available to rigorously evaluate project effects in quantitative terms: randomised field experiments with control groups, non-randomised experiments with control groups, regression analyses, and cost benefit analyses (see e. g. the contributions of various authors in Banerjee 2007). It is well known that many project evaluations do not succeed in clearly identifying and quantifying effects that can be attributed to project interventions. Programme Aid Effects In the case of programme aid evaluations this fundamental attribution problem exists as well and can be solved in similar ways as for project aid. The solution of the problem can be illustrated with the following equation: (2) Y = f ( P i, X j ) Y is again the change of the target variable, but P i describes now a set of programme interventions carried out by the recipient government and supported by several donors. X j denotes again a vector of covariates. It must be noted that P i measures programme interventions, not programme aid interventions. In other words, equation (2) explains that programme effects can rigorously be separated from the effects of covariates, but that a further rigorous attribution of effects to all (or individual) donors supporting the programme is usually not feasible. As all donors and the recipient government contribute funds and other inputs to the same programme it can only be stated that donors contribute to the overall programme effect. Of course, on the input side the financial contributions of all donors or a single donor can be put in relation to the total funds of the programme, but such proportions cannot be transferred to the result side. 9

12 Programme effects can be evaluated in rigorous, quantitative terms either with regression analyses (e. g. Elbers, Gunning, de Hoop 2009; De Kemp 2008) or with computable general equilibrium (CGE-) models (e.g Bourguignon and da Silva 2003, 2008). Based on rigorous evaluations of programme effects some authors go even one step further and try to separate the effects of aid interventions by using (non-rigorous) scenario analyses. For instance White and Dijkstra (2003) construct some sort of soft counterfactuals by comparing macroeconomic effects of programmes, e.g. on the external account or on the fiscal position, with and without aid. Such scenario analyses can also be carried out with CGE-models. Hence, given a rigorous, quantitative evaluation framework that identifies overall programme effects a combination with qualitative elements (scenario assumptions) can help to separate at least some presumed effects of programme aid. As mentioned before, most project evaluations do not use rigorous evaluation methods to measure effects of aid interventions. Project evaluations are in many cases restricted to before-after comparisons sometimes even without baseline data, which then requires evaluators to (re-) construct estimates of conditions before the intervention. At best such evaluations consider some qualitative assumptions about the influence of external variables, which then may help to capture the presumed, plausible project effects. The same holds for many programme aid evaluations. Particularly evaluations of comprehensive policy programmes such as poverty reduction strategies (PRS), for which governments receive general budget support, are often restricted to testing chains of causality in qualitative terms. Many of these evaluations are also limited to before-after-comparisons either without counterfactuals or, at best, with qualitative attempts to construct counterfactuals. A good example for such a soft approach is the work of IDD and Associates (2006) on the joint evaluation of budget support within the Partnership General Budget Support (PGBS) initiative in seven countries. Taken together, it should be clear that high quality evaluations of aid effects are characterised by their degree of reliability to separate project and programme aid interventions from other influences on target variables. This attribution problem cannot always be solved in quantitative terms, but it is broadly agreed that a rigorous approach can and should be followed more often. It is undeniable that a quantitative evaluation offers the highest level of rigour or reliability, and hence quality. And the higher the quality of project and programme evaluations the higher is also the quality of a cooperation strategy evaluation based on (inter alia) these results. In other words: reliable project and programme evaluations are the backbone of reliable cooperation strategy evaluations. Therefore we recommend investing in rigorous evaluations of project and programme aid effects. 10

13 2.3 Process Evaluations Process evaluations focus on dimensions of aid quality that are part and parcel of aid effectiveness but must be considered in their own right. Process evaluations attempt to answer questions of the following sort: (a) To what extent are aid interventions of a donor relevant for the development of the recipient country? (b) To what extent are aid interventions of a donor aligned with the declared goals of development and with the implemented development strategy of the recipient country? (c) To what extent are aid interventions of a donor harmonised with interventions of other donors and policy programmes of recipients, i.e. integrated into the recipient country s budget and procedures or pooled in a common basket of donor activities? (d) How well do aid interventions of a donor complement and reinforce each other within the portfolio of a cooperation strategy? (e) How well do aid interventions of a donor complement and reinforce aid interventions of other donors and public action of the recipient country s government? (f) Does the donor help to avoid the crowding of aid in some sectors / domains ( aid darlings ) and the neglect of other sectors / domains ( aid orphans )? (g) To what extent has the policy dialogue between a donor and the government and other donors contributed to strengthen the cooperation process and aid effectiveness? Process evaluations are also carried out at both the project and programme level, and it is by all means helpful to synthesise and critically discuss these results when conducting a cooperation strategy evaluation. In addition to using existing process evaluation results of projects and programmes a process evaluation focusing on the whole project and programme aid portfolio of a cooperation strategy should be carried out. 2.4 Joint evaluations The request of joint evaluations must distinguish between different aid instruments. In the case of project evaluations the term refers to an endeavour jointly carried out between a donor and the recipient government. Depending on the project, government representatives involved in the joint evaluation may come from the national, provincial, district, or even community level. As governments usually face a large number of projects they will often not be able to fully participate in project evaluations due to capacity constraints for such tasks. In principle this reasoning also holds for cooperation strategy evaluations of individual donors. These evaluations satisfy primarily learning and accountability needs on the donor side, and although they provide also value added for recipient governments the recipient side may decide not to fully participate because of limited personnel and financial resources. It is therefore realistic to assume that evaluations of projects and cooperation strategies will in many cases not only be donor-driven, but also carried out unilaterally, with limited participation of recipient governments. That is a second-best solution, but unavoidable as 11

14 long as project aid makes up substantial shares of total aid, and as long as individual cooperation strategies are not substituted by Joint Assistance Strategies (JAS), which are also largely implemented through budget support. More important than full participation of the recipient government is the independence of evaluators and the quality of their work. In the case of programme aid (budget support) joint evaluations, in which donors and the recipient government fully participate, are a must. As in a rigorous manner only the overall programme effect can be evaluated with soft assumptions perhaps the effect of all donors, but certainly not the effect of an individual donor it makes absolutely no sense that donors tackle the issue on a bilateral basis or even unilaterally. It is generally assumed that programme aid and joint evaluations reduce the transaction costs for recipient governments; therefore both approaches rank high on the agenda of the Paris Declaration and the Accra Agenda for Action of course only for recipient countries where the quality of policies and governance justify programme aid. There seems to be a broad agreement that joint evaluations incur relatively high transaction costs for donors, but that the total benefits (including lower transaction costs for recipient governments) outweigh these costs. Apart from the learning effect joint evaluations are also an important contribution to mutual accountability as donors and recipients work together in critically assessing the results of their joint development efforts. Hence, the higher the share of programme aid in to total ODA in a given country, i.e. the higher the degree of harmonisation, the more important are joint evaluations. 12

15 3. Evaluations of Cooperation Strategies: Current Practices and Future Perspectives 3.1 Why Cooperation Strategy Evaluations? As mentioned in the introduction, Switzerland and other donor countries tend to reduce the number of priority countries where they provide aid, and the number of sectors or domains in which they intervene. This process goes along with a trend of aid effectiveness at the country level getting more important for two reasons. First, stakeholders in donor countries, like the parliament and civil society at large, have a legitimate and increasing interest to be informed about the successes, failures and challenges of bilateral aid at the country level. Second, donor agencies are interested in learning from evaluations of their cooperation strategies about aid effectiveness at the country level. Hence, cooperation strategy evaluations exhibit a clear value added on the donor side, as units of account and as instruments of learning to increase aid effectiveness. Cooperation strategy evaluations contain also relevant information for recipient governments. First, they provide a synthesis of the overall achievements, failures and challenges of an individual donor at the country level. Second, such evaluations are a reliable source of information about the extent to which an individual donor fulfils the goals of the Paris Declaration as well as other commitments laid down in the cooperation strategy. Both sorts of information are relevant and of interest for recipient governments. Despite the usefulness of such information recipient governments have often only limited interest in aid results of individual donors. Given that many recipient governments cooperate with a large number of bilateral and multilateral donors (as well as non-government organisations), the transaction costs of participating in numerous evaluations and digesting them may be considered too high in relation to the value added of the findings. Therefore governments are more interested in programme aid and joint evaluations of aid, where as many donors as possible participate: more programme aid and fewer joint evaluations lead to lower transaction costs than lots of project aid and many bilateral evaluations. Joint cooperation strategy evaluations of several donors are also not the rule, because evaluation results for one donor are hardly of special interest for other donors. Exceptions are country cases where donors follow a single Joint Assistance Strategy (JAS) and / or country cases where donors provide an overwhelming share of total aid in the same sectors and as budget support. Under such conditions joint cooperation strategy evaluations are highly beneficial. If shares of budget support in total aid are sufficiently high, such evaluations can approximate total aid evaluations. 13

16 3.2 Current Practices The most ambitious goal of a cooperation strategy evaluation is to look at the combined effect of the portfolio of a donor s project and programme aid interventions. If one focuses on rigorous evaluations this ambitious goal poses two methodological problems that are insurmountable. First, we have no rigorous method to aggregate project aid effects into an overall aid effect at the country level. Second, as explained before, in the case of programme aid it is impossible to rigorously separate the aid effects of a single donor from the total programme and total aid effects. Donor organisations and evaluators usually deal with this problem by examining mainly the overall cooperation process, and by assessing the effects of individual components of the cooperation strategy separately and individually. The subsequent attempt to take stock of current practices of cooperation strategy evaluations is based on the analysis of selected evaluation documents, of selected methodological studies and the information from a few interviews with evaluation experts of several donors. We want to emphasise that this exercise must be interpreted with great care above all the stylised classification of some donor practices summarised in Table 1. Approaches to cooperation strategy evaluations can differ considerably among donors. Moreover, the approaches of individual donors can differ quite considerably case-by-case, dependent upon the content of the cooperation programme itself, the availability of project and programme evaluation reports to build on, the rationale and lead questions that trigger and guide the evaluation, the time horizon covered by the evaluation, the budget ceiling for the task, and (last but not least) methodological preferences and experiences of the experts involved in the evaluation. In other words, apart from some very general principles that donors usually apply it is virtually impossible to identify the approach preferred and followed by one or the other donor. With this big caveat, Table 1 attempts to illustrate some common and some differing practices of cooperation strategy evaluations of a sample of six donors: DANIDA, DFID, IA, NORAD, OBI, and SIDA. We discuss some insights from Swiss practices of SDC and SECO separately. As we make at the end of this study recommendations for SDC we do not discuss expected changes in the future as for the other donors. Again, it should be noted that the entries high, medium, and low in Table 1 as well as the arrows signalling expected increasing, stationary, and declining trends in the future are our interpretations from reading selected documents and conducting a few interviews. The differences between the ratings of the donors in our sample are more important and telling than the ratings themselves. 14

17 Table 1: Stylised Classification of Practices to Evaluate Cooperation Strategies Donor 1 Relevance 2 Emphasis on inputs & outputs 3 Emphasis on overall aid process 4 Emphasis on overall aid effects 5 Emphasis on new data collection 6 Emphasis on field visits and interviews 7 Emphasis on a qualitative approach 8 Emphasis on quantitative elements 9 Unilateral evaluation 10 Evaluation jointly with government DANIDA Past High High High Low Low Low High Low Medium Low High Denmark Future 11 Evaluation jointly with donors DFID Past High High High Low Low Medium High Medium Medium Medium Medium England Future IA Past High High High Low Low Medium High Low Medium Medium Medium Ireland Future NORAD Past Medium High High Low Low Low High Low Medium Medium Medium Norway Future IOB Past Medium High High Low Low Medium High Medium Medium Medium Medium Netherlands Future SIDA Past Medium High High Low Low Medium High Low Medium Low High Sweden Future SDC Past High High High Low Low Medium High Low Medium Medium Medium Switzerland Future SECO Past Medium High High Low Low Medium High Low Medium Medium Medium Switzerland Future 15

18 In general, all donors of our sample classify cooperation strategy evaluations as relevant (column 1). In the case of NORAD, IOB and SIDA we got the impression that these organisations expect the relevance of evaluations at the country level to increase in the future. Particularly IOB expects to be confronted with more frequent demands from parliament to report about successes and challenges of cooperation strategies. DANIDA, on the other hand, has recently concluded three country programme evaluations that span rather long time periods between 14 and 16 years (DANIDA 2008) time frames that do not suggest very frequent evaluations at the country level. Of course, all donors require that cooperation strategy evaluations include descriptions of inputs and outputs of aid interventions in the country portfolio (column 2). Inputs delivered and outputs achieved are compared with the target values in planning documents and provide a first overview of targets that have been achieved and that have been missed. This first step is usually followed by an analysis of the causes for achieving and missing targets. In close relation with these analyses all donors put a strong emphasis on evaluating the overall aid process at the country level (column 3), whereas the emphasis on the evaluation of the overall or combined aid effect of all interventions is significantly lower (column 4). Given the methodological difficulties mentioned above, this is not surprising. Therefore the effects of aid interventions in the country portfolio are usually assessed individually and separately from the overall process evaluation. Most donors put relatively low emphasis on the collection of new data on aid effects during the cooperation strategy evaluation (column 5). Statistical surveys with beneficiaries are typical instruments applied for this task, but with the limited time frames and budgets for evaluations at the country level they are not in all cases and intensively applied. The most important sources of data and information are existing evaluation and monitoring reports on aid effects. From our document analysis we conclude that DANIDA and NORAD expect to invest more in data collection in the future, which would be consistent with the abovementioned expectations that the relevance of evaluations at the country level may increase. On the other hand, DFID and IA are quite outspoken followers of what the organisations call a light-touch approach to cooperation strategy evaluations; they may continue with relatively small efforts to collect data for this task. More important than data collection in the strict sense of the term are field visits to get more qualitative information about effects and processes of project and programme aid, although these visits are usually selective and cover by no means all interventions (column 6). Interviews and focus group discussions with beneficiaries are the main instruments of information collection to complement existing evaluation and monitoring results. In addition interviews with aid experts and government representatives involved in project and programme aid play an important role in evaluating the overall country portfolio of aid activities. These practices also contribute to the high emphasis on qualitative approaches that characterises all cooperation strategy evaluations (column 7). The use of quantitative evaluation results of project and programme aid to evaluate the overall portfolio plays at the time being a relatively minor role, simply because rigorous project and programme aid evaluations are still rather scarce (column 8). From our sources we conclude that DFID and 16

19 OBI are the most advanced organisations in that area. But there seems to be a general willingness of donors to put more emphasis on rigorous project and programme evaluations in the future, which then will also be used as inputs for cooperation strategy evaluations. Cooperation strategy evaluations are quite frequently carried out unilaterally (column 9), but virtually all donor organisations in our sample feel being caught in a dilemma. On the one hand, as mentioned before, the unilateral practice often makes sense, because the evaluation results for the strategy of an individual donor are of relatively low interest for the recipient government and other donors. On the other hand all development partners have committed themselves in the Paris Declaration and the subsequent Accra Agenda for Action to carry out joint evaluations whenever possible. In principle all donors are willing and interested to fulfil that commitment, but some of them are outspokenly realistic and accept that in the case of cooperation strategy evaluations unilateral exercises may be unavoidable (columns 10 an 11). In our sample of donors IOB and IA made very clear that they expect to carry out cooperation strategy evaluations in most cases unilaterally, because the benefits of joint evaluations are for the recipient government and other donors too low in relation to the overall effort. There is of course the exception of country cases where donors have subscribed to Joint Assistance Strategies (JAS). Under these conditions joint evaluations at the country level are highly beneficial for all partners involved. The evaluation department of DANIDA for instance explores with other donors the possibility of joint evaluations in Tanzania, Uganda and Zambia (DANIDA 2008). DANIDA (2009) also reports about a joint cooperation strategy evaluation with Belgium, the European Commission, France and Luxembourg in Niger. The five donors hold the opinion that they have a sufficiently critical mass in total aid and such a strong overlap in their sector and domain support that a joint exercise is beneficial for all. This example indicates that whenever the share of programme aid in total ODA for a given recipient country is high the chances are high that joint evaluations pay off. DANIDA, NORAD, and SIDA seem to have more experience from joint cooperation strategy evaluations than others in our sample, and expect to continue or even strengthen this practice. The practices of SDC and SECO are not that different from what we summarised for the other donors of our sample. Both organisations consider cooperation strategies as relevant, but for SECO this instrument seems to be of less importance than for SDC. The methodological approaches are very similar to those followed by other donors. Cooperation strategy evaluations focus strongly on the assessment of aid inputs and outputs and on the quality of the overall aid process. There is a rather strong emphasis on qualitative methods and on field visits and interviews to complement the document analysis of existing evaluation and monitoring results. The emphasis on collecting new data, on making use of quantitative 17

20 evaluation results, and on evaluating overall aid effects is relatively low. 2 SECO concludes that with the current emphasis of cooperation strategy evaluations on processes of aid the main beneficiaries of these efforts are the operational staff involved in development cooperation, not so much headquarters and stakeholders who tend to be more interested in overall aid effects. Apart from cooperation strategy evaluations both SDC and SECO emphasise the relevance of good monitoring systems as steering instruments at the country level. Particularly SDC s proposal to focus the annual reports of Swiss cooperation offices more on results achieved than (primarily) on plans for the coming year is a bold step in this direction. Both SDC and SECO are also interested and willing to carry out joint cooperation strategy evaluations with recipient governments and other donors, but they face of course the same constraints as the other donors of our sample. Sometimes the timing of a planned joint evaluation, which serves the needs and preferences of some donors, may be a major impediment for others. SECO, which to date is responsible for most of Swiss budget support, succeeded so far to carry out all its programme aid evaluations jointly with recipient governments and other donors. And both SDC and SECO pay always attention to have representatives from recipient countries in the evaluation teams they put together. To what extent this involvement in the case of cooperation strategy evaluations can be interpreted as an active involvement of recipient governments is beyond our knowledge. Particularly SDC expects that the relevance of evaluations at the country level may increase in the future, as both an instrument of learning and an instrument of accountability. Both SDC and SECO feel that also evaluations at the country level should strengthen the domestic and international visibility of Swiss aid contributions and effects. However, neither the representatives of SDC nor SECO that we interviewed would argue in favour of mandatory and frequent cooperation strategy evaluations in each and every priority country. They would rather advocate a case-by-case approach, where the decision for and the timing of such evaluations would be determined by specific questions that should be answered and by good opportunities to carry out such an exercise with high benefits. 3.3 Contributions to Mutual Accountability It is worth mentioning that the Paris Declaration suffers from a notable asymmetry in its request for mutual accountability. While the accountability of donors to their citizens and parliaments is recognised in the declaration s introductory Statement of Resolve, it is completely missing in the subsequent requests and recommendations on mutual 2 A good example of the frequently modest attempts to evaluate overall aid effects at the country level is contained in SDC s evaluation of the cooperation strategy for NEPAL ( ), where the evaluators write: Table 3 provides our assessment again highly subjective of the impact that development assistance (the wider donor effort within Nepal of which SDC is part) has had over the review period on key outcomes aligned to HMG development strategy. We also make an assessment of the significance of the SDC contribution to that impact achievement (SDC 2005, p. 38). The statement is followed by a table with four entry lines for four strategic objectives. Impact assessment grades comprise low, medium, and high ; SDC contribution is graded with similar codes. That s about all on overall aid effects; the overwhelming part of this evaluation at the country level is on the aid process. 18

21 accountability. It seems that this deficiency has been addressed in the Accra High Level Forum meeting of 2008, reminding recipient governments not to neglect or underestimate accountability obligations of donor governments in their home countries. Accepting accountability of donors in their home countries as part and parcel of mutual accountability is of course a strong argument in favour of cooperation strategy evaluations. It goes without saying that mutual accountability among donors and between donors and recipient governments must be practiced and enforced in the day-to-day work and the regular monitoring and evaluation of the cooperation process. But cooperation strategy evaluations are the most comprehensive and precise retrospective pieces of information about the extent to which single donors actually fulfil the aid commitments laid down in their cooperation strategies. Therefore evaluation reports of cooperation strategies can be used as valuable inputs to strengthen the process of mutual accountability between donors and recipient governments, and among donors themselves. Mutual accountability is a soft concept in the sense that donors and recipients are barely able to enforce compliance with hard sanctions. In principle, donors have the strongest mechanism to sanction non-compliance of recipient governments by reducing their support. However, this seemingly strong option is weaker than often presumed. First, there is clear evidence that donor conditionality and sanctions cannot enforce compliance if a recipient government is not willing and capable to fulfil conditions. Second, sanctions in the form of reduced aid are at conflict with the overall goal to provide aid, and with the goal to reduce aid volatility and increase aid predictability. Mutual accountability is therefore largely practiced through peer pressure and the desire to protect reputation. Essentially the success or failure of mutual accountability seems to rest on three pillars. First, a shared agenda with clearly spelled out goals and reciprocal commitments; second, jointly agreed and executed procedures of monitoring and evaluation; and third, functioning forums for informal debate, policy dialogue, and negotiations. All efforts towards strengthening these three pillars will also strengthen mutual accountability. As mentioned before, evaluations of cooperation strategies can make a modest, but perhaps notable contribution to achieve this goal. The fact that in many recipient countries the government cooperates with a large number of bilateral and multilateral donors creates a problem of collective action. With so many actors, particularly in the field of programme aid, there is the risk that no one feels responsible for the outcome of the cooperation process. The problem can only be solved if the recipient government takes ownership and leadership in the development and cooperation process. If that is the case, mutual accountability can indeed contribute to strengthen development and aid effectiveness. If not, mutual accountability will not succeed in solving the collective action problem. 19

22 3.4 Outlook As a result of our analysis we see four potentially important trends for the future development of cooperation strategy evaluations, which, in our opinion, should be followed. The first is the general interest of donors to strengthen the quality of evaluations at the country level with regard to aid effects. We expect that more rigorous project and programme aid evaluations will (and should) contribute to this trend. Second, in recipient countries where harmonisation increases, i.e. where budget support covers an increasing share of total aid, programme aid evaluations become more and more important. While programme effects can be evaluated rigorously, programme aid effects cannot be separated with rigorous counterfactuals. Donors will (and should) continue to further develop soft approaches to separate the contribution of total programme aid effects. Third, in countries with high degrees of harmonisation and JAS, or a comparable set of policy programmes strongly supported by donors, joint evaluations of programme aid can approximate evaluations of total aid. Donors should promote such an attempt in order to gain experience about the feasibility of this goal. Fourth, although donors have committed themselves to joint evaluations whenever possible, they will (and should) carry out cooperation strategy evaluations unilaterally if need be. Legitimate accountability needs in donor countries justify such practices. As increasing programme aid and harmonisation, together with their implications for high quality evaluations, play such an important role in the future, we will elaborate on current practices and recommendable developments of these issues in the next chapter. 20

23 4. Evaluations of Programme Aid: Current Practices and Future Perspectives 4.1 Current Practices Donors provide programme aid in two forms: in the form of general budget support (GBS) and in the form of sector budget support (SBS). We will discuss the evaluations of the two types of programme aid separately. Nonetheless, it must be noted that GBS evaluations inevitably cover also important aspects of sector programme evaluations. This stems from the fact that GBS is in the end allocated to budget lines for public action in specific sectors. Therefore the effects of GBS can be divided into two categories depicted in Figure 2: the overall (sector-independent) effects of macro policies 3 and the effects of sector policies. Figure 2: Two-pronged evaluation of GBS General budget support Macro policies Sector policies Macro policy effects Sector policy effects Macro effects evaluation Sector effects evaluation The two types of effects have important implications for the evaluation of GBS. On the one hand the effects of macro policies must be evaluated with suitable tools for an overall macro evaluation, on the other hand sector effects must be evaluated with suitable tools for sector evaluations (see also Compernolle and de Kemp 2009). GBS is always connected to a national development strategy, in most cases a poverty reduction strategy (PRS). Recipient governments have the lead in the monitoring and evaluation of PRS processes and effects, but receive for this task support from donors. Important monitoring products on a country basis are the regular PRS progress reports. These documents are units of account of governments to their domestic stakeholders and to donors. In essence, PRS progress reports describe the attainment of policy and development goals as they are laid down in the PRSP. To some extent the documents also include (self-) evaluations of the results of the PRS, i.e. the reporting governments sometimes try to separate the effects of their policies from other influence on the PRS-target variables. PRS progress reports inevitably cover a broad spectrum of policy areas and effects. Therefore, and because they must record progress in the short term, they rely on 3 For the sake of simplicity we subsume under the term macro policies also structural or micro policies, which also affect the whole economy. 21

24 narrative and descriptive before-after comparisons. Despite these limitations the reports can and should serve as inputs to programme aid evaluations. Useful as PRS progress reports are, they result from monitoring activities and not from evaluations in a strict sense. Moreover, they say nothing about the contribution of GBS or other aid interventions to the attainment of development goals. As explained in Chapter 2, most GBS outcome and impact evaluations to date are not based on rigorous methods but on qualitative approaches. This also holds for the attempt to separate aid effects from overall programme effects. As mentioned before, the most prominent example of such an approach is the joint evaluation of GBS in seven countries within the Partnership General Budget Support (PGBS) initiative, in which 24 bilateral and multilateral aid agencies participated (IDD 2006). The evaluation methodology was the same in all countries and focused (inter alia) on the following topics: relevance of the PGBS; effects on harmonisation and alignment; effects on public expenditure; effects on planning and budgetary system; effects on policies and policy processes; effects on macroeconomic performance; effects on service delivery (this was the sector-specific part of the evaluation as indicated in Figure 1); effects on poverty reduction; and sustainability of PGBS. For each of these policy areas the evaluation described the supposed chain of causality and attempted to explain the counterfactual, i. e. the supposed policy effects without the PGBS. The evaluation method, although not rigorous in quantitative terms, succeeded in making transparent all underlying hypotheses on causal linkages and counterfactuals. That is an important achievement, as it allows for critically scrutinising the plausibility of hypotheses, the plausibility of empirical evidence, and the plausibility of interpretations and judgments of evaluators. In addition to these effects-oriented topics the evaluation also dealt with more process-oriented issues, such as strategic biases of the PGBS, the impact on capacity development, the role of conditionality in the context of partnership, the relation to other aid modalities, and the management of risk involved in GBS. To date, also most evaluations of sector budget support (SBS) are based on qualitative approaches, i.e. before-after comparisons of programme effects without rigorous separation of control groups or control variables. However, several sector programmes have undergone quantitative evaluations, and there seems to be a clear tendency to apply rigorous methods more often in the future. Examples of quantitatively analysed sector programmes include the evaluation of the Mexican programme PROGRESA (Skofias 2005); an evaluation of land reform in India (Besley and Burgess 2000); and World Bank evaluations of educational reforms in Ghana (OED 2004) and child health in Bangladesh (OED 2005). Other examples are the Dutch evaluations of primary education programmes in Uganda and Zambia (summarised by de Kemp 2008), which illustrate particularly well the potential of quantitative sector programme evaluations. The evaluation results show that the education programmes indeed succeeded in both countries in enhancing access to primary education through the expansion of education capacities and the abolition of school fees. The growth of enrolments notwithstanding, pupil-teacher ratios, pupil-classroom ratios and pupil-book ratios could decline in Uganda, and are expected to decline in Zambia in the coming years as a result of increasing budgets for education. The impact analyses, which focus on examination results 22