MINISTRY OF REGIONAL DEVELOPMENT National Coordination Authority The Czech approach to evaluating communication activities Radek Kobza Head of EU Publicity Department 6 December 2018, INFORM, Brussels
Why to take interest in evaluation? In last few years, evaluations have helped us: 1) to improve some of our communication activities 2) to stop doing some of others 3) to negotiate our annual communication activities plans with senior management of the ministry (ministries, deputy ministries ) Last but not least: 4) to get the EFFIE award for best media campaign in public sector in the Czech Republic in 2016
What this presentation will be about? 3 examples of tools and how we have been using them to evaluate: media campaigns for general public (includes TV spots + print + on-line) publications and promo materials / give-aways events for general public (Open Days)
Coordination of ESI funds communication in CZ EU Publicity Department Evaluation Unit Working Group for Publicity
Coordination between MA and NCA Common Communication Strategy for ESIF 2014 2020 sets common objectives defined by primary measurable result indicators and three secondary result indicators specifies main topics and the content of communication defines target groups define rules for communication of the EU Funds, roles of individual institutions in the implementation structure set fundamental rules for the evaluation of the communication activities performed (each OP has to evaluate communication activities) Available at: https://dotaceeu.cz/en/evropske-fondy-v-cr/narodni-organ-pro-koordinaci/publicita (both CZ and ENG)
Communication objectives Take away no.1: Know your target (s) / goals before you start.
Indicators We have high awareness of EU Funds ( 80 %), but we are much more interested in positive perception through increasing a knowledge of projects in our neighborhood. Q: How do you see benefits of EU funded projects for Your country: 65,8 % (definitely or rather beneficial) Your region: 64,8 % You, personally / close family: 55,6 % Data from survey in 2017, CAWI, n=1500 respondents We focus on everyday aspects; we all use benefits of more than 100 000 of EU funded project in creative briefs
Evaluation of media campaign Target group: general public (there was more detailed segmentation) Goal: raise positive awareness and beneficial results of EU funded projects in our neighborhood Media split and tools: TV (1 spot) + digital (banners) + print (ad in newspapers in different size, supplements) + landing page www.kazdydenpomahame.eu We have focused on Creativity (visual and copy) + Media planning
Evaluation of media campaign II Creativity (TV spot) We do PRE-TEST focused on: Attractiveness (How the spot is made, is it interesting for you?) Credibility (Would You believe the information / message in the spot? Are the actors trustable for You believable?) Did You get the main information / message from the spot? Understandability (Was copy clear for You? Which part was hard to understand?) Methodology: CAWI (on-line panel), 800 respondents, surveys takes up to 10 minutes (10 14 questions + sociodemo questions) This is an integrated part of our cooperation with creative agency. We do benchmark results with standards and our previous campaigns. In post-production we can still change some aspects: copy / texts, colors, sound before campaign starts.
Evaluation of media campaign III Creativity (TV spot) Up to two months after the campaign we do POST-TEST focused on: General and specific indicators (general awareness, positive perception, knowledge of projects (see Common Communication Strategy ) Awareness and effectiveness of our campaign (Have you seen this and where? Whas it attractive?) Methodology: CAWI (on-line panel), 800 respondents, surveys takes up to 15 minutes (up to 20 questions + sociodemo questions) This is an separate order / cooperation with one of survey contractors (due to impartiality). And we benchmark results again and compare them with pre-test results. POST-BUY delivered by media agency (there is lots of metrics in both off- and on-line media) + POST-TEST helps us to better media planning for next campaigns (= better knowledge of media types and formats for defined target groups)
Evaluation of publications We produce a wide range of publications for different target groups: General public Professional public Applicants Evaluation tool: FOCUS GROUPs Done by external contractor. We have realized that prepare a good focus group is not so easy. It is important to know why and to whom you produced every single publication, what you wanted to achieve =) contractor can prepare valid questions for focus group to evaluate how we have fulfilled our goals.
Evaluation of promo materials / giveaways We buy a wide range of give-away as a supporting communication tool for events and competitions. Evaluation tool: PANEL OF EXPERTS Done by external contractor. Main outcomes: Useful comments on the quality of products Price benchmark Tips for better alternatives / products
Evaluation of events / Open Days on Projects Photo from ELI Beamlines + HILASE (laser centers), 3000 participants, 1 000 of them took guided tour inside the research centers + activities for whole family. Q: Was this event successful?
Evaluation of events / Open Days on Projects II From 2018, we describe success of events in the beginning of preparation (when we start budget calculation). Events are run as projects and this success descriptions is also an set up expectation for event managers. We try to put on the table: Hypothesis - Do participants assess EU funds more positively after visiting specific projects when they could see and touch benefits? Goal description in numbers More then 500 participants / 1 event; more than 50 % of them would appreciate the event, more than 50 % would recognize the influence of EU funding, more than 50 % would know more then 3 EU funded projects in that region
How we measure it? On the spot: Hostess is asking three questions to people leaving the event using tablet application: 1. Have you known any project cofunded from the EU Funds before you attended this event? 2. Did this event help you to get a better idea of where the EU Funds help in the Czech Republic? 3. Do you think it is useful to be informed by public institutions about projects supported from the EU Funds, e. g. by similar events?
How we measure it? II 5 days after event: on-line survey for participants whose took part in the voting competition about best project in the region during the event (we have their email address and presented them more EU funded projects. In the email there is also a small gift: link to the photo album from the event Questions with options: How did you learn about the event? With whom did you attend the event? How do you see the number of projects presented? Which projects do you remember from the event? Would you recommend attending a similar event to your friends? Is it useful to be informed by public institutions (such as the Ministry) at similar events about which projects are supported from the EU Funds? Would you like to get more information about the ESI Funds? Do you know any other project co-funded from the ESI Funds, besides those presented at the event?
How we measure it? III Open answer questions: What did interest you the most at the event? What did you miss and what would you like to see/experience next time? Which of the presented projects interested you the most? What would interest you? (information about the EU Funds) Which ones? (projects cofunded from the EU Funds)
Plan for 2019: Evaluation of Eurofon and Eurocentres Eurofon is a call centre for EU a EU Funds related questions. Operates free of charge phone line +420 800 200 200, responds on chats on website (www.dotaceeu.cz - umbrella web portal for ESIF in CZ) and general emails. Suggested evaluation tools: mystery calling focus on the quality of operator s interpersonal skills and quality of information. Eurocentre is a network of 13 regional information offices responsible for presentations in schools, organizing debates for public and realizing seminars and consultation for applicants and beneficiaries. Suggested evaluation tools: mystery shopping focus on the quality of officer s interpersonal skills and quality of service.
Thank you for your attention. Radek Kobza radek.kobza@mmr.cz