What does the failure of the polls tell us about the future of survey research? Professor Patrick Sturgis, University of Southampton

Transcription

1 What does the failure of the polls tell us about the future of survey research? Professor Patrick Sturgis, University of Southampton

2 Cathie Marsh

3 Polls and Surveys Polls are what pollsters do; surveys are what academics/official statisticians do Polls are low-quality, non-random; surveys are high-quality, random (face-to-face) Polls are about politics/democracy, surveys are about explanation/understanding

4 The 2015 election polls

5

6 Election Result v Average of final Polls (GB) V o t e 25 S h a r e % Tory Labour UKIP Lib Dem Green Other Party

7 seats Seat shares Tory Labour SNP Lib Dem DUP Other

8

9 Had the forecasts been different, then the nightly news bulletins would surely have concentrated rather more on the vast spending cuts to come, and rather less on the potential role of Scottish nationalists in a hung parliament. That might have influenced the result. Editorial 14/5/15

10

11 How bad was the 2015 miss?

12 Error on Conservative vote share Chart by Will Jennings

13 Error on Labour vote share Chart by Will Jennings

14 Mean absolute error on Con/Lab lead Chart by Will Jennings

15 Consensus of the final polls Fieldwork Sample Con Lab Lib Ukip Green Other Populus 5 6 May Lord Ashcroft 5 6 May Ipsos MORI 5 6 May YouGov 4 6 May ComRes 5 6 May Survation 4 6 May ICM 3 6 May Panelbase 1 6 May Opinium 4 5 May BMG 3 5 May TNS 30-4 May Result

16 Whodunnit?

17 The suspects Late swing

18 The suspects Late swing Sampling/weighting

19 The suspects Late swing Sampling/weighting Turnout misreporting

20 The suspects Late swing Sampling/weighting Turnout misreporting Don t Know/refusal

21 The suspects Late swing Sampling/weighting Turnout misreporting Don t Know/refusal Question wording

22 The suspects Late swing Sampling/weighting Turnout misreporting Don t Know/refusal Question wording Registration/postal

23 Consensus of the final polls Fieldwork Sample Con Lab Lib Ukip Green Other Populus 5 6 May Lord Ashcroft 5 6 May Ipsos MORI 5 6 May YouGov 4 6 May ComRes 5 6 May Survation 4 6 May ICM 3 6 May Panelbase 1 6 May Opinium 4 5 May BMG 3 5 May TNS 30-4 May Result

24 Herding Hey! I was just about to say that Moo

25 Implications

26 Polls are the public face of survey research Never so much public and media interest in survey methodology! Damaging for trust in official statistics? ( it is because the same instruments of data collection and assembly are used in the both surveys and polls that many have become as suspicious of the former as they are of the latter, Marsh 1982) Negative effect on response rates?

27

28

29

30 The effect of the 2015 polling miss on surveys will be, at most, minimal

31 Election results are very difficult to estimate In surveys we (should) ask carefully drawn samples from well defined populations. About well defined behaviours They have already done During a well-defined period Election polls ask about future behaviour Which is subject to social desirability bias Of a poorly defined population (likely voters)

32 And random surveys did ok The exit poll got it (about) right The British Election Study post-election survey got it right (though not subject to late swing)

33 British Election Study face-to-face post-election estimates Conservative Labour Liberal Democrat SNP Plaid Cymru UKIP Green Party

34 And random surveys did ok Exit poll got it (about) right British Election Study got it (about) right 2015 British Social Attitudes.?

35 Design Differences All pre-election polls were either phone or online Non-random sampling methods Opt-in internet panels starting to dominate: 0% in 2001; 37% in 2005; 61% in 2010; 80% in 2015

36 Opt-in internet panels Enormous growth in market share

37 Global spend on online market research Chart: Mario Callegaro, source Inside Research

38 Opt-in internet panels Enormous growth in market share Low cost, ~ 5 per complete interview? Rapid data turnaround Multimedia & mobile functionality Increase access to survey data, build capacity Prediction success (until recently )

39 Poll overload? During the short campaign (March 30 th - May 7 th ) 91 polls for GB 13 polls for Scotland 4 polls for Wales 61 polls for constituencies (Ashcroft). During the long campaign (May ) 1,942 polls for GB Compared to approx. 3,500 polls for GB between 1945 and 2010

40 Opt-in accuracy

41 Yeager et al 2011

42 Erens et al (2014) Men: (all: n=35; behavior CAPI: n=9; behavior CASI: n=18; opinion: n=8) in 4 Web surveys compared to National Survey of Sexual Attitudes and Lifestyles (Natsal-3)

43 So all is well with random surveys?

44 Response Rate (%) British Social Attitudes Survey Response Rate x Survey Year

45 Low and declining response rates RDD even worse, in the US routinely < 10% (increasing mobile-only+do not call legislation) Survey sponsors ask what are we getting for our money? Is a low response rate survey better than a well designed quota?

46 Increasing costs Per achieved interview costs are high and increasing Simon Jackman estimates $2000 per complete interview in 2012 American National Election Study My estimate= ~ 250 per achieved for PAF sample, 45 min CAPI, n=~1500, RR=~50% Compare ~ 5 for opt-in panels

47 Cost drivers Average number of calls increasing More refusal conversion

48 US Survey of Consumer Attitudes (Curtin et al 2000) Response Rate = 70% (1979) -> 68% (1996) Mean contact attempts % refusal conversions

49 Cost drivers Average number of calls increasing More refusal conversion More incentives (UKHLS, 30) 30%-40% of fieldwork costs can be deployed on the 20% hardest to get respondents

50 Externalities of survey pressure Poor data quality of hard to get respondents Fabrication pressure on respondents (community life survey) Fabrication pressure on interviewers (PISA) Ethical research practice?

51 Is all this effort (and cost) worth it?

52 r(response rate, nonresponse bias) Groves (2006)

53 Correlation response rate & bias Edelman et al (2000) US Presidential exit poll Keeter et al (2000) compare standard and rigorous fieldwork procedures In general, the field finds weak response propensity models

54 What if we didn t bother with the hard to get?

55 Response rate RESPONSE RATE PER CALL NUMBER 60% of costs 40% of costs 100% 90% 80% 70% 65% 68% 70% 72% 73% 76% 60% 54% 60% British Crime Survey 50% 46% Taking Part 40% 30% 33% British Election Study Skills For Life 20% 15% 10% 0% All After x visits

56 Williams, Sturgis, Brunton-Smith (2015) Take estimate for a variable after just one call E.g. % smokers = 24% Compare to same estimate after all calls Now % smokers = 18% Absolute % difference = 6% Relative absolute difference = 6/18 = 33% Do this for lots (>600) of variables over multiple surveys

57 Average absolute % difference after 1 call 1.8

58 Average absolute % difference after 1 call (post-stratified) 1.37

59 Average absolute % difference after 3 calls 0.75

60 Average absolute % difference after 3 calls (post-stratified) 0.6

61 What is to be done?

62 Tolerance for low response rate surveys We know that RRs shouldn t matter as much as they do but we don t seem to be able to move on from maximising response rate (at great expense) Partly to do with how contracts are tendered? There is a lot we know about how to keep response rates up, survey agencies tied to this

63 Greater diversity of cost/quality options At present, choice options are: very expensive, or very cheap Need to develop more mid-range options, e.g. postal invite to complete online Develop better understanding of what different cost options offer in terms of data quality

64 We should be wary of opt-in panels No satisfactory underlying theory of inference Such designs work well until they don t; there is little theory undergirding their key features, Groves (2011, p867) Robust evidence of substantial bias in estimates, plus variability across providers Double selection mechanism (into panel, into survey)

65 probability web panels The future for random surveys? Draw random sample into panel for periodic online surveys, provide access for off-line population High fixed costs, very low marginal costs per interview Tablet/phone compatibility as standard

66

67 The Sailing Ship Effect Advent of the steam ship in the 19 th Century seemed to herald the demise of the sailing ship Yet, we still have very nice sailing ships today acceleration of innovation in the old technology in response to the threat from the new (Foster 1988) A sailing ship effect for large, random (in-home) surveys? As online surveys become more common, in-home contact will be an increasingly distinctive experience

68 Reports of the death of random surveys greatly exaggerated

69

70

71 Thank You