EVROPSKI PARLAMENT 2014-2019 Odbor za proračunski nadzor 30.3.2015 DELOVNI DOKUMENT o posebnem poročilu Evropskega računskega sodišča št. 18/2014 (razrešnica za leto 2014): Sistem vrednotenja in sistem v rezultate usmerjenega spremljanja, ki ju uporablja EuropeAid Odbor za proračunski nadzor Poročevalec: Bart Staes DT\1055894.doc PE544.401v02-00 Združena v raznolikosti
Introduction In the current economic and political context, there is an increased need for the Commission to ensure and demonstrate that EU funds are used in an economic, efficient and effective manner. The European Parliament and the Council have indicated a need for a clearer view of the results achieved with regard to the EU s main objectives. Information on results achieved can only be obtained with a results accountability framework comprising three elements: monitoring, evaluation and reporting. Within the Commission s decentralised organisational framework, the Directorate General for Development and Cooperation - EuropeAid (EuropeAid) has set up its own results accountability framework which comprises the monitoring, evaluation and reporting of its activities. This report examines two of the key elements of this framework, namely evaluations and results oriented monitoring (ROM). Evaluation is the systematic and objective assessment of an ongoing or completed programme or policy, its design, implementation and results. Evaluation is essential for planning, designing and implementing EU policies and interventions, and enhancing transparency and democratic accountability. EuropeAid manages two types of evaluations, which are carried out by external contractors: (a) strategic evaluations, which comprise geographic evaluations (covering a country or region) and thematic evaluations, which cover a sector (e.g. education, food security) or a method of aid delivery (e.g. budget support, cooperation with development banks) and (b) programme evaluations, that can be carried out during the implementation (mid-term evaluation), on completion (final evaluation) or after completion (ex post evaluation) of an intervention. These evaluations assess performance against five criteria developed by the Development Assistance Committee of the Organisation for Economic Cooperation and Development (OECD-DAC), along with the international donor community relevance, effectiveness, efficiency, sustainability and impact. In addition, evaluations have to assess the EU s comparative advantage and the coherence with other EU policies and programmes, other donors actions and partner countries policies. ROM is a standardised external review, specific to external aid, designed to look at programmes performance through short on-site visits to programmes. The main purposes of these tools of the accountability framework are to improve the implementation of ongoing programmes and the design of future programmes and policies through feedback and lessons learned, and to provide a basis for accountability. Audit Scope and Objectives The Court examined whether EuropeAid has reliable evaluation and ROM systems. The audit focused on three main questions: - Are EuropeAid s evaluation and ROM functions appropriately organised and resourced? - Do EuropeAid s evaluation and ROM systems ensure the provision of relevant and robust findings? - Do the evaluation and ROM systems ensure the maximum use of the findings made? The audit covered the evaluation and ROM systems in place in the period from 2007 to 2013. PE544.401v02-00 2/6 DT\1055894.doc
Where adequate information was available and could be checked by the Court, it took account of changes adopted or planned by EuropeAid for the period from 2014 to 2020. The audit was carried out between November 2013 and March 2014 and included (i) a review of EuropeAid s evaluation and ROM guidelines and of the main documents and processes in the management of evaluations, (ii) interviews with officials from eight units at EuropeAid, (iii) visits to the EU delegations in Benin and India, involving interviews with the delegations staff and a review of relevant documentation, (iv) a review of programming documents, programme financing agreements, strategic and programme evaluations, and ROM reports; and (v) a survey sent to 50 EU delegations to which 41 answers were received. Court's Findings and Observations Are EuropeAid s evaluation and ROM functions appropriately organised and resourced? The Court found that overall, EuropeAid s evaluation and ROM functions were well organised with a sound distribution of responsibilities between the Evaluation unit, the Quality and Results unit in charge of the ROM and the ROM coordinators but there was a lack of overall supervision by senior management of programme evaluation activities which are decentralised to EuropeAid operational units and EU delegations. Also, insufficient attention was paid to the efficient use of evaluation and ROM resources. EuropeAid has insufficient information on resources used by its evaluation and ROM systems. The total annual costs of evaluations and ROM would amount to 34-38 million euro but EuropeAid has no reliable information on the number of programme evaluations and since their respective costs cannot be easily identified in the IT management information system. More EuropeAid had no management information on staff resources involved in ROM and programme evaluations; the Court estimated it to be likely above 80. Due to this lack of information, EuropeAid cannot ensure that staff and financial resources are appropriate and proportionately allocated to the various evaluation and ROM activities. Finally the Court observed that there was an inadequate coordination between programme evaluations and ROMs, which was likely to lead to inefficiencies. Programme evaluations and ROMs follow completely different planning approaches and criteria with the risk that certain programmes were over and under-assessed and sometimes not assessed at all. Do EuropeAid s evaluation and ROM systems ensure the provision of relevant and robust findings? The Court audit work revealed that the evaluation and ROM systems did not sufficiently ensure that relevant and robust findings are produced. If multiannual plans for strategic evaluations were based on appropriate criteria (achieving full geographic coverage, selection on the basis of the materiality funds allocated and increased number of evaluations of aid delivery methods or selection of thematic evaluations according to the priority themes of the Agenda for Change), the criteria for programme evaluation plans were based on insufficiently clear prioritisation criteria and significant differences in approach. The Court found that there was no monitoring system to identify and address frequent deviations from evaluation plans and to address the systemic causes of delay leading to a situation where only 70% of the selected evaluations were completed as planned. DT\1055894.doc 3/6 PE544.401v02-00
With regard to quality control procedures, the Court estimated that they were not implemented consistently for ROM and programme evaluations as reference groups (set up for each evaluation to advice all along the evaluation process) were not implemented for programme evaluations (only 39% of EU delegations set up a reference group always or most of the time and it was rarely the case by EuropeAid's operational units and EU delegations visited by the Court). More, the assessment of the final evaluation reports was only carried out by 71% of the EU delegations with only 37% of them filling assessment grids. With regard to the assessment of the quality of ROM reports, compulsory response sheets were only done for less than 60% of ROM reports from 2011 to 2013. Furthermore, the evaluation and ROM systems do not provide adequate information on results achieved due to insufficiently well-defined objectives and indicators, the latter being prepared in a political context and not sufficiently translated into operational objectives. The Court's sampling done for thematic, regional and country level programming documents and individual programmes also showed that objectives and indicators were unclear, lacking or without baselines or targets against which to assess performance. The absence of clearly defined objectives and indicators made difficult for an evaluation to establish evidence-based performance assessments. It also appeared from the Court audit work that evaluations and ROMs focused more on the implementation of programmes than on results achieved. There was a limited proportion of ex post evaluations and ex post ROMs for assessing impact and inherent limitations in the evaluation methodology for budget support i.e. difficulty to quantify the budget support contribution in a qualitative manner. Likewise, it should also be noted that the reporting from EU delegations did not provide much evidence on policy achievements but more on activities implemented and the EuropeAid's contribution to the Commission's Article 318 evaluation reports, despite positive developments, did not provide sufficient, relevant and reliable evidence on EU's policies achievement. The aforementioned factors limit considerably EuropeAid s capacity to account for the actual results achieved. Do the evaluation and ROM systems ensure the maximum use of the findings made? The Court found that the evaluation and ROM systems did not ensure that maximum use was made of findings because proper mechanisms were not in place to monitor their follow-up and dissemination. The Court identified weaknesses in the follow-up of strategic evaluations findings and recommendations issued, on their acceptance and use for new policies and programmes. Detailed information on the planned actions, their timetable and the allocation of responsibilities were also insufficient and the follow-up of the action plan by the Evaluation unit was limited to one year after the publication of the evaluation report. The follow-up for programme evaluations and ROM findings was not enough documented. The Court considered that in the absence of an appropriate follow-up system, EuropeAid could not ensure that findings and recommendations were used effectively. With regard to dissemination, it was found that evaluation and ROM findings were well disseminated to all EuropeAid s units, EU delegations, external stakeholders and other EU institutions and Member States. A synthesis of strategic evaluation results and key lessons learned and the aggregated ROM performance data were included annual activity report and in the preparation of the Commission s Article 318 evaluation report. However for PE544.401v02-00 4/6 DT\1055894.doc
programme evaluations, the Court found that the dissemination was inadequate as the EAMRs reviewed by the Court contained no or little information in this respect. Moreover, programme evaluations report cannot be systematically used as there was no central database to manage, store and analyse ROM and programme evaluations. Priporočila poročevalca za morebitno vključitev v letno poročilo o razrešnici Komisiji 1. Pozdravlja posebno poročilo o sistemu vrednotenja in sistemu v rezultate usmerjenega spremljanja (ROM), ki ju uporablja EuropeAid, ter v nadaljevanju navaja svoje ugotovitve in priporočila; Splošne pripombe 2. je resno zaskrbljen zaradi nezadostne zanesljivosti sistema vrednotenja in sistema ROM, ki ju uporablja EuropeAid, neustrezne stopnje nadzora in spremljanja vrednotenja programa, in tudi dejstva, da EuropeAid ne more zagotoviti ustreznosti osebja in finančnih sredstev in da so učinkovito dodeljeni za različne dejavnosti vrednotenja; 3. poudarja, da je nujno potrebno, da se Parlamentu in organu za proračunski nadzor zagotovi jasen pregled, v kateri meri so bili dejansko doseženi glavni cilji Unije; 4. opozarja, da bi bilo treba zagotoviti zunanje, objektivne in nepristranske povratne informacije o uspešnosti projektov in programov pomoči Evropske komisije kot del prizadevanj Komisije za zagotavljanje kakovosti; 5. meni, da so rezultati vrednotenja ključni elementi za vključevanje v politiko in postopek pregleda politik, da bi prilagodili strateške politične cilje in povečali splošno skladnost z drugimi politikami EU; 6. meni, da vlaganje v analizo in združevanje dokaznih rezultatov iz različnih vrednotenj zagotavlja ne le splošno sliko trendov, ampak tudi omogoča pridobivanje izkušenj za večjo končno učinkovitost postopkov vrednotenja, obenem pa nudi boljše dokaze, ki potrjujejo da smo na pravi poti pri sprejemanju odločitev in boljšemu oblikovanju politik za vsak posamezni instrument pomoči; 7. meni, da je izmenjava znanja z vsemi sredstvi in orodji ključnega pomena ne le za razvoj kulture vrednotenja, ampak zlasti za učinkovito kulturo uspešnosti; 8. podpira priporočila Sodišča v zvezi s sistemom vrednotenja in sistemom v rezultate usmerjenega spremljanja, ki ju uporablja EuropeAid; Priporočila Računskega sodišča Sodišče podaja priporočila v zvezi z učinkovito rabo virov za vrednotenje in ROM, določanjem prednostnih nalog in spremljanjem ocen, izvajanjem postopkov nadzora kakovosti, prikazom doseženih rezultatov in spremljanjem ter razširjanjem ugotovitev na podlagi vrednotenja in ROM. DT\1055894.doc 5/6 PE544.401v02-00
9. EuropeAid bi moral vzdrževati ustrezne informacije o upravljanju in ocene potreb opravljati redno, da bi zagotovili osveščeno dodeljevanje finančnih in kadrovskih virov med vrednotenja programa in ROM; 10. Da se zagotovi, da opravljena vrednotenja odražajo prednostne naloge organizacije, bi EuropeAid moral: določiti jasna izbirna merila za prednostno obravnavo vrednotenj programa in dokumentirati, kako so bila uporabljena pri pripravi načrtov vrednotenja, ob upoštevanju dopolnjevanja z ROM; znatno izboljšati svoj sistem za spremljanje in poročanje o izvajanju načrtov vrednotenja, vključno z analizo razlogov za zamude in opisom ukrepov za njihovo odpravo; okrepiti splošen nadzor dejavnosti vrednotenja programa, ki jih izvaja EuropeAid: 11. Da bi zagotovili kakovost vrednotenj programa in ROM, bi EuropeAid moral: vztrajati, da operativne enote in delegacije uporabljajo zahteve za kontrolo kakovosti, vključno z uporabo referenčne skupine in dokumentiranjem opravljenih kontrol kakovosti pri vrednotenjih programa; redno preverjati izvajanje teh kontrol; 12. Da bi okrepili zmogljivost sistema vrednotenja za zagotavljanje ustreznih informacij o doseženih rezultatih, bi EuropeAid moral: strožje uporabljati regulativne določbe, ki zahtevajo uporabo ciljev in preverljivih kazalnikov SMART; spremeniti sistem spremljanja, da bi še naprej zagotavljal podatke o programih, vsaj tri leta po njihovem zaključku; ter občutno povečati delež naknadnih vrednotenj programa. PE544.401v02-00 6/6 DT\1055894.doc