Policy evaluation in the EU. Linking ex ante and ex post evaluation. Workshop, Friday 20 June 2014 Centre for European Law and Governance Cardiff Jean Monnet Centre of Excellence Cardiff University, UK The Centre for European Law and Governance (CELAG) is an interdisciplinary centre for the study of EU law and European governance. It is a Jean Monnet Centre of Excellence, recognised by the European Commission for its excellence in research and teaching on the European Union. Centre for European Law and Governance www.cardiff.ac.uk/celag
Page 1 Policy evaluation in the EU Workshop, Friday 20 June 2014 Centre for European Law and Governance, Cardiff Jean Monnet Centre of Excellence Cardiff University The concept of policy evaluation is traditionally used for the ex post assessment of policy programmes, particularly regarding public intervention relying on the public purse. A scientific community (with journals as Evaluation, or Evidence & Policy), as well as a community of consultants and policy experts, have developed around this, with extensive debate on what would be the best methodology for such evaluation. At EU level, such ex post evaluations have traditionally been performed in sectors such as cohesion and research policy. More recently, in the context of the Smart Regulation agenda, the European Commission has proposed a broader conceptualisation of policy evaluation. A more general evaluation culture is proposed, applicable to all types of policymaking, including regulatory intervention, as well as to different stages of policy-making. More particularly, it is argued that ex post evaluation and ex ante evaluation should be linked. Ex ante evaluation in the EU has been boosted by the European Commission s systematic use of the Integrated Impact Assessment procedure since 2003. The recent Smart Regulation emphasis that ex post evaluations of the current regulatory framework should more regularly be used as starting point and source of information for ex ante assessments of the impacts of new regulatory proposals seems a logic argument. However, there are many challenges to the idea of a broader concept of policy evaluation throughout the entire policy cycle, and linking ex ante and ex post evaluation. Ex ante and ex post evaluation have traditionally been used in different policy sectors and for different types of policy interventions. Moreover while they may both fit ideas of better regulation they may answer to different objectives. They raise similar questions in terms of principal agent relationships, accountability, participation, issues of coordination, appropriate methodology etc. but the answers may well be different. The aim of this workshop is to reflect on a broad conceptualisation of policy evaluation in EU policymaking, and on the potential and challenges it brings. The challenge is also an academic one. The topic of ex post evaluation, and of ex ante impact assessments is largely dealt with by two separate scientific communities, with different approaches, and often different disciplinary backgrounds. While there is a well established scientific community studying policy evaluation, few of them are EU specialists. The topic of European impact assessments instead has attracted considerable scientific attention by scholars in EU and/or regulatory studies; few of whom whoever deal with ex post policy evaluation. The aim of this workshop is to bridge this gap and to provide for an interdisciplinary exchange. Moreover, the workshop aims at an interaction between academic scholars and practitioners dealing with ex ante and ex post evaluation at EU level.
Page 2 Programme 9.45. Arrival. Coffee 10.00 Prof. Stijn Smismans (Cardiff University) and Dr Emanuala Bozzini (University of Trento): Towards a policy evaluation culture in the EU: objectives, instruments and actors 10.30 Dr Claire A. Dunlop (University of Exeter) (paper together with Prof. Claudio M. Radaelli (University of Exeter)): Policy Appraisal in the European Union: Challenging the Conventional Wisdom? 11.00 Steven Hojlund (Copenhagen Business School): The European Commission and evaluation: How and why do evaluation systems develop? 11.00-11.15 Coffee break 11.15 Prof. Anne Meuwese (Tilburg University): A Framework for Regulatory Policy Evaluation: OECD and member state perspectives 11.45 Dr Melanie Smith (Cardiff University): The policy cycle, evaluation and the salience of infringement data 12.15 Dr Paul Stephenson (Sciences-Po Paris): Reconciling audit and evaluation: the shift to performance and effectiveness at the European Court of Auditors 12.45-14.00 Lunch 14.00 Dr Lorna Schrefler (Centre for European Policy Studies, Brussels) and Dr Giacomo Luchetta (Centre for European Policy Studies, Brussels): Cumulative cost assessments 14.30 Dr Rachel Minto (Cardiff University) (paper together with Dr Lut Mergaert (Yellow Window)): Ex ante and ex post evaluations: two sides of the same coin? The case of gender mainstreaming in EU Research Policy 15.00 Dr. Emanuela Bozzini (University of Trento) and Dr Jo Hunt (Cardiff University): Policy evaluation and non-learning in the CAP 16.30-16.00 Coffee break 16.00-17.30 Practitioners debate Bettina Rafaelsen; Chief Market Manager, COWI, Consulting Group
Page 3 Sara Piller; Unit C1, Evaluation, Regulatory Fitness and Performance, Secretariat General Commission. Dr. Frank Siebern Thomas; Deputy Head Unit, European Commission, DG MARKT, Economic Analysis, Impact Assessments, Evaluation. 20.00 Dinner at Laguna Restaurant Other participants Cliff Wirajendi, Executive Manager European Journal of Risk Regulation Andy Fraser, Head of the Natural Resources Management Programme, Welsh Government. Lori Frater, Environment Team, Legal Service Department, Welsh Government. Joanna Legg, EU Research Analyst, Europe Directorate, Foreign and Commonwealth Office. Sara Drake, Senior Lecturer Cardiff Law School. Annette Morris, Lecturer, Cardiff Law School. Ivan Mugabi, PhD researcher, Cardiff Law School. Other contributors to special issue: Prof. Alberto Alemanno (HEC Paris): Policy evaluation and the courts
Page 4 Abstracts Prof. Stijn Smismans (Cardiff University) and Dr Emanuala Bozzini (University of Trento): Towards a policy evaluation culture in the EU: objectives, instruments and actors The paper provides an overview of the challenges related to creating a culture of policy evaluation in EU policy-making, and in particular the European Commission s attempt to ensure policy evaluation as a reflexive process throughout the policy cycle. The Commission s intentions are placed in the context of the fragmented literature on the topics of both ex ante and ex post evaluation, not only identifying the two policy communities dealing (rather separately) with ex ante impact assessments and ex post evaluation but referring to broader theoretical debates about policy learning and reflexive governance. The different objectives, normative expectations and methodologies of both ex ant and ex post evaluation are identified. The analysis will then focus on an empirical assessment of the different actors involved in both ex ante and ex post evaluation, drawing both normative and practical conclusions from current practices. Dr Claire A. Dunlop (University of Exeter) and Prof. Claudio M. Radaelli (University of Exeter): Policy Appraisal in the European Union: Challenging the Conventional Wisdom? In this paper we reflect on the extent and ways in which empirical research on policy appraisal in the European Union challenges the conventional wisdom about appraisal instruments, especially but not exclusively ex-ante impact assessment. At the outset, we define the conventional wisdom, and explain its components. We then draw on the findings of our four-year project (funded by the European Research Council) to distil lessons and explore its implications for EU-level policy appraisal. Specifically, we tackle conventional propositions on the role of appraisal in decision-making, the balance between institutions and mechanisms, issues control and policy learning, the feasibility of drawing lessons for the diffusion of appraisal across Europe, and equifinality. We conclude that EU appraisal has distinctive features that question the conventional wisdom, and therefore researchers should not assume that we can use models and explanations developed in the US uncritically. Steven Hojlund (Copenhagen Business School): The European Commission and evaluation: How and why do evaluation systems develop? This article accounts for the historical development of the European Commission's evaluation system from 1980 and onwards. Based on more than 58 interviews, the article explains how the Commission mainly adapted to internal and external pressures in a multi-level governance framework in order to secure both accountability and organisational efficiency. The analysis relies on organisational institutional theory to understand different types of adaptation that unfold simultaneously and over time in the Commission. The article further expands the view of the Commission by using the concept of institutional entrepreneur to understand the latest developments of the Commission s evaluation system and the Commission s role in shaping it. Dr Lorna Schefler (Centre for European Policy Studies, Brussels) and Dr Giacomo Luchetta (Centre for European Policy Studies, Brussels): Cumulative cost assessments Cumulative cost assessment are a new entry in the EU REFIT agenda. The paper covers three aspects: methodological considerations (and the link between ex post and ex ante assessment),
Page 5 governance implications as regards the connection between this additional tool and others in the smart regulation toolkit, and finally include an empirical perspective based on our recent work. Dr Melanie Smith (Cardiff University): The policy cycle, evaluation and the salience of infringement data The infringement process contained in Article 258 TFEU and the related financial sanction in Article 260 TFEU require the Commission, as guardian of the Treaties, to monitor the performance of member states in relation to the implementation and application of EU norms. As the Commission carries out this duty it amasses enormous amounts of data in relation to the realization of policy objectives through its discovery of the (non) implementation of EU norms. This paper explores the Commission s duty as guardian of the Treaties through the lens of better regulation and examines the potential for the infringement data to play a major role in the latter stages of implementation and evaluation in the policy cycle. The paper will focus on the concept of evaluation and explore the extent to which the Commission has included the infringement data as part of its learning cycle. In other words, to what extent is this important data being used to inform the better regulation agenda and subsequent policy design choices? The paper will look to the institutional and practical barriers to making an effective use of infringement data as part of the policy evaluation and will conclude with some observations on how evaluation might be integrated more fully into the policy process. Dr Paul Stephenson (Marie Curie Fellow, Sciences-Po Paris): Reconciling audit and evaluation: the shift to performance and effectiveness at the European Court of Auditors The audit function for the European Communities auditors started out in the 1950s by focusing on complicance (legality and regularity). This approach to auditing policy expenditure financed by the EU budget has dominated for much of the last 36 years at the European Court of Auditors. Auditing has been seen as a purely technical function, tied up with accounting and technocracy. Decisionmakers and politicians have concerned themselves largely with error and fraud, particularly in light of the introduction of the Declaration of Assurance (DAS) following Maastricht. However, in recent years, and in the climate of credit crisis, member state resistance to increases in the EU budget (with calls for renationalisation of programme expenditure), and citizen dissatisfaction with integration, the EU institutions have been forced to make attempts to better justify the value-added of policies pursued using EU budgetary expenditure. In the last decade, the ECA has published over 300 special reports, as it has shifted its activity more towards performance audit, questioning policy effectiveness, efficiency and impact. This has given the Court a new role in policy evaluation, bringing an increased political dimension to its work, but also creating tensions in terms of interinstitutional relations. This paper examines what a modern day, post-enlargement Court of Auditors should be doing - how it should interpret its mandate - and which institutional and systemic factors might account for the ongoing shift in the nature of its activity. What are its own interests and what is at stake in the reform process? Prof. Anne Meuwese (Tilburg University): A Framework for Regulatory Policy Evaluation: OECD and member state perspectives Prof. Alberto Alemanno (HEC Paris): Policy evaluation and the courts
Page 6 Dr Lut Mergaert (Yellow Window) and Dr Rachel Minto (Cardiff University): Ex ante and ex post evaluations: two sides of the same coin? The case of gender mainstreaming in EU Research Policy This paper investigates the role of evaluations in the development of European Union (EU) research policy, with particular attention to their function in the promotion of gender equality. It engages with two themes of contemporary EU governance: firstly, the role of evaluations within an effective and efficient policy-making process; secondly, the EU s constitutionalised commitment to promoting gender equality in all its activities (gender mainstreaming, Article 8 TFEU). This paper focuses on the interface between ex ante and ex post evaluation, exploring how these two sides of evaluation are linked in the promotion of gender equality. A case study approach was followed, with EU research policy as the object of analysis. This is a policy area in which gender mainstreaming has known some success (albeit qualified) and in which several ex ante and retrospective evaluations have been undertaken as part of a cyclical process of policy development. Analysis charts and critiques policy learning over the multiple iterations of the EU s multiannual framework programmes (from FP5 to Horizon 2020), permitting an assessment of the strengths and weaknesses of evaluation in terms of the EU s constitutionalised commitment to gender mainstreaming. For more information about events and activities at the Centre for European Law and Governance, please see the website: www.cf.ac.uk/celag