Evaluation and Monitoring of European Research Framework Programmes

Transcription

1 1 Evaluation and Monitoring of European Research Framework Programmes Tokyo, July 2008 Dr. Peter Fisch European Commission Directorate General Research A.3

2 2 Roadmap The European Research Framework Programmes (FP) Evaluation and Monitoring of FP6 and FP7 Open Issues for organising evaluations

3 EU Research Most important multinational research programme world wide Complementary to national research activities and programmes 27 Member States and Associated Countries contributing 100+ countries involved Annual budget 6bn, increasing research contracts per year, involving partners 3

4 4 EU Research History 1952: European Coal and Steel Treaty; first projects started : EURATOM Treaty; Joint Research Centre set up 1984: First Framework Programme ( ) 1987: European Single Act Treaty; Research becomes a EU policy Second Framework Programme ( ) 1990: Third Framework Programme ( ) 1994: Fourth Framework Programme ( ) 1998: Fifth Framework Programme ( ) 2000: European Research Area 2002: Sixth Framework Programme ( ) 2006: Seventh Framework Programme ( )

5 5 EU Research Challenges EU-25 US Japan R&D intensity (% of GDP) (2004) Share of R&D financed by industry (%) (1) Researchers (FTE) per thousand labour force (2) Share of world scientific publications (%) (2003) Scientific publications per million population (2003) Share of world triadic patents (%) (2000) Triadic patents per million population (2000) High-tech exports as a share of total manufacturing exports (%) (2003) Share of world high-tech exports (%) (2003) Notes: (1) EU-25: 2003; US, JP: (2) EU-25: 2004; US: 2002; JP: 2003.

6 6 EU Research Lisbon Strategy 2000 Lisbon agenda : making the European Union the most competitive economy in the world and achieving full employment by Growth and jobs strategy Research as key player in the Knowledge triangle

7 EU Research European Research Area Increased cooperation and integration of research activities at European/national/regional level Launched in 2000 as background to FP6 relaunched in 2007 with Green Paper underlining potential Internal market for research Free circulation of researchers, technology and knowledge Effective European-level coordination of national and regional research activities, programmes and policies Focusing on seven axes researchers; infrastructure; institutions; knowledge sharing; coordinated research programmes; world access 7

8 8 billion 12 EU Research Framework Programmes Annual Budgets between 1984 and NB: budgets in current prices. Source: Annual Report 2003, plus FP7 revised proposal

9 9 FP Cooperation Collaborative research Structure Multi-partner projects, top-down agenda Ideas Frontier Research Individual projects, bottom-up agenda Capacities Research Capacity Sharing of infrastructures People Marie Curie Actions Mobility programmes for post-docs, Training courses, + Other activities (EURATOM, JRC, )

10 10 FP Budgetary Breakdown ( million)

11 11 FP Thematic Areas ( Cooperation ) I. Cooperation * Not including non-nuclear activities of the Joint Research Centre: million Budget ( million, current prices) 1. Health Food, agriculture and fisheries, and biotechnology Information and communication technologies Nanotechnologies, materials and production Energy Environment Transport Socioeconomic research Space Security Total

12 Actors FP Evaluation System (1) Political level (Member States) European Commission level Directorate General level (Ministries) Member States evaluation Rules Framework Programme Decisions European Commission Financial Regulations and related rules European Commission Communications on Evaluation Other types of internal European Commission Regulations 12

13 13 FP Evaluation System (2) FP6 Annual Monitoring by independent experts Five Year Assessment by highlevel independent experts Impact surveys at FP level Evaluation studies at operational level National impact studies Ad-hoc research-related activities FP7 Internal monitoring of implementation -Indicators to track progress FP7 interim evaluation Ex post assessment of each FP, 2 years after its completion by high-level independent experts Strengthened programme of coordinated strategic-level evaluations Evaluation studies at operational level (portfolio, programme) Coordinated national impact studies FP research on evaluation tools and approaches

14 14 FP7 Monitoring Implementation Move from external monitoring (FP6) towards internal monitoring (FP7) Move from an ad-hoc approach towards a continuous and systematic collection of information and indicators Monitoring primarily aimed to support management in implementing FP7 Annual report to be presented to the Programme Committee and to be published on internet Possible information source for future FP7 evaluations Flexible system to develop as FP7 will become more mature

15 15 FP7 Monitoring Issues at Stake Application numbers Proposal Evaluation Time to contract Success rates User Feedback

16 FP6 Ex-post Evaluation Actors: Expert group; Commission services Legal base: FP Decision Scope: All Community research activities Organisation: Coordinated approach between Commission services for FP activities Input: Reports from INFSO and JRC; evidence base of 30+ studies; Member States studies; expert analyses; self assessments, Output: Expert group report (findings and recommendations); Commission response Timing: Panel report by end 2008 Dissemination: Policy makers and FP management 16

17 17 FP6 Ex-post Evaluation Issues at Stake Longer term impact and consequences of FP activities? Changes in behaviour of participants? Networking patterns (geographical / institutional)? Networks of Excellence? Integration of new Member States?

18 Evaluation of FP7 Roadmap Progress Report Interim Evaluation Ex-ante Evaluation in time for new Commission proposal on FP8 Ex-post Evaluation 2015 FP7 interim evaluation Independence Comprehensive evidence base Build on the ex-post evaluation of FP6 18

19 FP Evaluation Perspectives Growing EU Research budget also means: Increased need for accountability Efficiency of the European RTD system under scrutiny Timing of forthcoming evaluations in line with need to have an informed debate on future EU RTD policy Need to focus more on the fundamental aspects and less on minor implementation issues Need to develop evaluation capacities in Europe as part of the European Research Area 19

20 FP Evaluation Achievements and Challenges Individually robust evaluations Independence Assessment of impact on scientific knowledge Justification of interventions European network Some evolution of tools Coordination Assessment of longer-term impacts Assessment of socio-economic impact Use of results in implementation 20

21 Open Issues Intervention Logic Specify a clear intervention logic in the basic legal acts Develop a hierarchy of (measurable) objectives throughout the different levels (programmes, projects) FP evaluation: Difficult to achieve in a complex environment like the European Union Need to develop new types of indicators in order not to be blocked by just aiming at what you can measure 21

22 Open Issues Overall Evaluation Strategy Clear evaluation strategy from the outset Complete coverage of all activities Right timing in view of revisions and development of new actions FP evaluation: Concise long term-time planning Need to improve on overall coherence of the evaluation activities carried out in different fields of the FP 22

23 Open Issues Diversity and Coordination Calls for strong coordination and harmonisation clash with the need to use a wide spectrum of different evaluation approaches FP evaluation: Diversity is one of the big assets of Europe Different traditions and schools across Member States Unique opportunity to bring together wide range of evaluation approaches Challenge to turn constructive chaos into operational structures 23

24 Open Issues Control or Understanding Different views on what drives evaluation Need to control activities and actors Wish to understand what is going on Major implications on the organisation and focus of evaluations FP evaluation: Focus on evaluation as a management task to gain a better understanding of the activities undertaken 24

25 Open Issues Longer Term Impact Full impact of research activities can only be assessed after a long period of time, as full implementation can take several years FP evaluation: In order to be on time (politically), evaluations are carried out very early Test-run planned for a study looking back at projects which were finished some 8 to 12 years ago Easier said than done 25

26 Open Issues Concise Messages A too much detailed level of analysis prevents many evaluation reports from having a real impact in the wider political and societal context FP evaluation: Meta-Evaluation through expert panels as a tool to condense otherwise too dispersed evaluation findings Further efforts are needed to focus on key messages for communication with the target audiences 26

27 Open Issues Organise Mutual Learning There is no ideal evaluation All actors are trying out different approaches Need to organise more mutual learning FP evaluation: European RTD Evaluation Network, involving experts from 30+ countries Interest in more active exchange at global level 27

28 28 Contact Dr. Peter Fisch Head of Unit European Commission DG RTD A.3 SDME 2/ Bruxelles Belgium