The Case for IVR. Presented to

Size: px

Start display at page:

Download "The Case for IVR. Presented to"

Maria Underwood
5 years ago
Views:

1 The Case for IVR Presented to February 26, 2015

2 No Regular Political Polling (2010) In 2010 there was no regular political polling in Canada This lack of regular polling made it difficult to track political and social changes Forum was interested in reintroducing regular political polling to the Canadian context Created The Forum Poll 1

3 Making the Right Choice The Forum Poll Highlights Needed to evaluate various methodologies and approaches Methodologies considered: CATI Online Panel IVR 30+ Years of polling experience 6 Years at the helm of the Gallup Poll Backed by a team of dedicated analysts 2

unlisted numbers and numbers on the Do Not Call list Results are statistically accurate

4 Choosing IVR The Forum Poll proceeded with an Interactive Voice Response System (IVR) Telephone sample selection is random and representative, includes landlines, cell phones, unlisted numbers and numbers on the Do Not Call list Results are statistically accurate within a defined range of probability Quick (rapidly changing opinions) Cost effective (large samples) 3

Advantages of IVR Random sample selection allows a

live interviewer calls Allows large sample sizes

5 Advantages of IVR Random sample selection allows a stated Margin of Error (MoE) Online methodologies are not random and have no MoE (MRIA) IVR eliminates Social Desirability Bias which affects live interviewer calls Allows large sample sizes and quick turnaround Response rates equalto live operator 4

6 Progress Made The Forum Poll is the flagship product of Forum Research, one of Canada s leading public opinion research firms The Forum Poll regularly examines important political, social and business issues across Canada The Forum Poll has appeared in well over 100 publications in the last 12 months The Forum Poll has conducted close to 200 polls over the last 12 months with approximately 200,000 Canadians surveyed 5

7 Forum Poll Accuracy The Forum Ontario Poll has focussed on accuracy: Random selection Quick turnaround Large samples Proprietary solutions to potential IVR methodology issues Proprietary weighting Corporatepolicy policy ofundertaking final election polls (9todate) 6

8 ON Accuracy 2011 Election Ontario (October 3 rd to 4 th, 2011 Election) For the 2011 Ontario election, the Forum Poll was accurate within 0.4% overall. Pollster Popular Vote Deviation % Lib Con NDP Green Other Deviation Method Actual Results Forum Poll Results Actual Results % 3% Forum Research (n=1034) IVR 23% 35% 38% 23% 36% 37% Abacus (n=926) Nanos (n=900) Angus Reid (n=1003) EKOS (n=2165) IPSOS Reid (n=1020) Online CATI Online IVR CATI Seat Distribution Results % Lib Con NDP Green Other Deviation Actual Results Forum Research Inc

9 SK Accuracy 2011 Election Saskatchewan (November 7 th, 2011 Election) In Saskatchewan, the Forum Poll predicted the election results with a total deviation of less than 1%. Actual Results Forum Poll Results Pollster Popular Vote Deviation Sask Deviat % NDP Green Liberal Other Method Party ion Actual Results % 3% 1% 64% 34% 3% 1% 62% Saskatchewan Party NDP Green Forum Research (n=1687) Insightrix (n=1000) Praxis (n=1041) IVR n/a CATI Liberal Seat Distribution Results Sask % Party NDP Deviation Actual Results 49 9 Forum Research Inc

10 AB Accuracy 2012 Election Alberta (April 23 rd, 2012 Election) Pollster Popular Vote Deviation In Alberta, the Forum Poll not only achieved a prediction with a 2.7% total deviation, i but also managed to detect a last minute voter swing. Actual Results Forum Poll Results 1% 10% 1% 10% 34% 44% 10% 12% 3% 1% PC 38% 36% Wildrose Liberal NDP Alberta Party Other % Cons WAP Lib NDP AP Other Deviation Method Actual Results Forum Research (n=1950) Leger (n=1200) Return on Insight (n=800) ThinkHQ (n=1425) Campaign Research (n=1545) Angus Reid (n=800) Abacus Data (n=1076) Seat Distribution Results % Cons WAP Lib NDP AP Other Deviation Actual Results Forum Research Inc IVR CATI CATI n/a IVR Online IVR 9

11 QC Accuracy 2012 Election Quebec (Sept. 3 rd, 2012 Election) For the 2012 election, the Forum Poll was accurate within 1.5% overall%. Pollster Popular Vote Deviation Deviati Meth % PQ Lib CAQ QS Green ON Other on od Actual Results Forum Poll Results Actual Results % 6% 31% 32% 6% 29% Leger (n= 1,856) Forum Research (n= 2,703) CROP 25% (n= ~1,000) EKOS (n= 1,764) 36% Online IVR CATI IVR Liberal PQ CAQ QS ON Green Other Seat Distribution Results % PQ Lib CAQ QS ON Deviat ion Actual Results Forum Research Inc

BC Accuracy 2013 Election British Columbia (May 14 th, 2013 Election) In British Columbia, the Forum poll was the most accurate of nine firms in predicting the election with a deviation of 1.

12 BC Accuracy 2013 Election British Columbia (May 14 th, 2013 Election) In British Columbia, the Forum poll was the most accurate of nine firms in predicting the election with a deviation of 1.6 across all parties. Actual Results 2% 8% 6% 43% 41% Forum Poll Results 8% 5%3% Lib 40% 44% Seat Distribution Results % Lib NDP Other Deviation Actual Results Forum Research Inc NDP Green Cons Other Pollster Popular Vote Deviation % Lib NDP Green Cons Other Deviati on Actual Results Forum Research (n=1147) Ipsos Reid (n=800) Angus Reid (n=803) Hill & Knowlton (n=804) EKOS (n=861) Oraclepoll (n=1000) Abacus Data (n=1042) Insights West (n=855) Justason (n=700) Method IVR Online Online Online IVR CATI Online Online Online 11

13 NS Accuracy 2013 Election Nova Scotia (October 8 th, 2013 Election) In Nova Scotia, the three firms arrived at very similar and very accurate predictions. In this case, the Forum Poll predicted the results with a total deviation of 1.6%. Actual Results 1% Forum Poll Results 26% 23% 27% 26% 3% Cons Lib NDP Pollster Popular Vote Deviation % Cons Lib NDP Green Other Deviati on Actual Results Abacus (n=600) Forum Research (n=922) Method CATI IVR CRA (n=270) CATI 46% 48% Green Other Seat Distribution Results % Cons Lib NDP Deviation Actual Results Forum Research Inc

14 QC Accuracy 2014 Election Quebec (April 7 th, 2014 Election) In Quebec, the Forum Poll was the most accurate firm in predicting the 2014 election with and average deviation of 1.00%. Pollster Popular Vote Deviation % Lib PQ CAQ QS ON Green Other Deviation Method Actual Results Actual Results Forum Poll Results Forum Research (n= 1536) 8% Angus Reid 6% (n=1410) EKOS (n=1422) 42% Leger 23% 23% 44% (n=1220) Ipsos Reid 25% 24% (n=1012) CROP (n=1400) IVR Online IVR Online Online Online Liberal PQ CAQ QS ON Green Other Seat Distribution Results % Lib PQ CAQ QS Deviation Actual Results Forum ou Research Inc

15 ON Accuracy 2014 Election Ontario (June 12 th, 2014 Election) For the 2014 Ontario election, the Forum Poll was accurate within 2.6% overall. Pollster Popular Vote Deviation % Lib Con NDP Green Other Deviation Method Actual Results % Actual Results 5% 39% 31% Forum Poll Results 3% 20% 41% 35% Abacus (n = 1882) Angus Reid (n = 1866) Oracle (n = 1000) Forum Research (n= 1054) Leger (n = 1050) EKOS (n = 1332) Ipsos Reid (n = 1991) Online Online CATI IVR Online IVR Online Seat Distribution Results % Lib Con NDP Green Other Deviation Actual Results Forum Research Inc

16 NB Accuracy 2014 Election New Brunswick (Sept. 22 nd, 2014 Election) For the 2014 New Brunswick election, the Forum Poll was accurate within 2.1% overall. Pollster Popular Vote Deviation % Lib Con NDP Green Other Deviation Method Actual Results Forum Poll Results Actual Results % 7% 43% 6% 12% 40% CRA (n=333) Forum Research (n= 1326) CATI IVR 35% 40% Seat Distribution Results % Lib Con NDP Green Other Deviation Actual Results Forum Research Inc

17 Provincial Election Accuracy Research Firm Ranking of Most Accurate Firms by Deviation Most Accurate Second Most Accurate Third Most Accurate Elections Predicted Forum Research Inc Abacus Data Leger CRA Angus Reid Insightrix 1 1 IPSOS Reid 1 4 CROP 1 2 Nanos 1 1 Praxis 1 1 Return on Insight 1 1 EKOS 1 5 Oraclepoll 1 2 Campaign Research 1 Hill & Knowlton 1 Insights West 1 Justason 1 ThinkHQ 1 16

18 Provincial Election Accuracy Overall Average Deviation Research Firm Average Deviation Number of Elections Polled Nanos Forum Research Inc Insightrix CRA Leger Angus Abacus Praxis Ekos Crop Oracle Return Ipsos Think Campaign Hill & Know Insight Justason

19 Trends First Identified by Forum Rob Ford to win Toronto Mayoralty in 2010 NDP surge in 2011 federal election NDP surge after Mulcair won federal party leadership Liberal surge before / after Trudeau won the leadership PQ in majority territory in Quebec in early 2012 First Wildrose Party surge in 2012 Alberta provincial election Last minute voter shift from Wildrose Party to Conservative in 2012 Alberta provincial election Last minute voter shift from NDP to Liberals in BC in 2013 provincial election Only pollster to correctly predict order of finishing i parties in Quebec election of

20 Key Federal Issues Polled & Published Key Federal Issues Polled Sample of Outlets Covering Issues Assisted Suicide Banking Services Bilingualism Catholic School Funding in Ontario City Service Issues Climate Change Customer Satisfaction Death Penalty Fighting in Hockey Marijuana Legalization Monarchy Mortgage Rates National Horserace Political Orientation Senate Reform Separatism Tuition Costs 19

21 Conclusions The Role of IVR in Survey Research CATI surveys have long been the gold standard for survey research IVR methodology, may not be a comprehensive replacement, but it can play a vital and economical role when conducting survey research today IVR methodology is fast, accurate and cost effective IVR can be used to screen for low incidence populations (i.e. MDs) IVR can be combined with other methodologies, such as IVR screener then transfer to a live agent survey IVR recruit to online survey 20

22 Conclusions Summary Limited use in Canada compared to the United States Methodology concerns are a key reason for the lack of acceptance Overall, more accurate than Computer Assisted Telephone Interviews (CATI) and online panel surveys in predicting the past 9 provincial elections 21

23 Conclusions Questions? 22

Canada 2015 Tracking Voting Intentions in Canada. For Publication Friday, September 4, 2015

Canada 2015 Tracking Voting Intentions in Canada For Publication Friday, September 4, 2015 Methodology Online survey conducted with Canadian citizens across all Canadian regions. This study was conducted