Citizenship Survey Incentive experiment report

Transcription

1 Citizenship Survey Incentive experiment report

2 Queen s Printer and Controller of Her Majesty s Stationery Office, 2011 Copyright in the typographical arrangement rests with the Crown. You may re-use this information (not including logos) free of charge in any format or medium, under the terms of the Open Government Licence. To view this licence, visit or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or psi@nationalarchives.gsi.gov.uk. This document/publication is also available on our website at If you require this publication in an alternative format please alternativeformats@communities.gsi.gov.uk Any enquiries regarding this document/publication should be sent to us at: Department for Communities and Local Government Eland House Bressenden Place London SW1E 5DU Telephone: April 2011 ISBN:

3 Citizenship Survey Incentive experiment report Ipsos MORI April 2011 Department for Communities and Local Government

4 Contents 1 Introduction Background Research aims Structure of this report 3 2 Methodology Selecting the sample Selecting the incentive Administering the incentive Fieldwork dates Data processing and analysis 5 3 Results Impact measurements Impact of incentives Impact of incentives on response rates by sub-groups Impact of incentives on respondents profile Impact of incentives from the interviewers perspective 13 4 Conclusions 14 5 Appendix: Advance letters 16 Letter to incentivised group 16 Letter to control group 17 1

5 1 Introduction 1.1 Background The Citizenship Study is a continuous survey 1 among people aged 16 or over living in England and Wales using a random preselected sampling approach. It is conducted on behalf of the Department for Communities and Local Government. The study consists of three components: A core sample, which is a nationally representative sample of approximately 10,000 adults living in England and Wales A ethnic minority boost sample that aims to achieve approximately 5,000 adults interviews; and A Muslim boost sample that aims to achieve approximately 1,200 adult interviews. In the 2010/11 survey year, an incentive experiment was introduced to the core sample of the survey in order to examine the impact of incentives (in the form of a book of stamps) on response rates. The experiment was conducted by Ipsos MORI and TNS-BMRB. It was introduced in the second quarter of the study, which runs from July-September 2010 and continued into the third quarter of the study (October to December 2010). The experiment was cost neutral to the Department for Communities and Local Government. This report focuses upon the results from the quarter 2 incentive experiment. 1.2 Research aims The purpose of the incentive experiment is to test the impact of incentives on response rates. In analysing the results, this report will examine: 1. Whether using an incentive had a positive impact on overall response rates We look separately at the impact on contact rates and the co-operation rates 2 2. Whether incentives had an effect on relative re-issue rates We compare the two samples (incentivised and control sample) but also looking at re-issue rates for Quarter 2 in 2009/10 1 Following a public consultation on the future of the Citizenship Survey, further Citizenship Surveys will be discontinued in order to make substantial savings. 2 See section 3.1 for definitions. 2

6 3. Whether the impact of incentives varies by different sub-groups We examine whether the impact of incentives on response rates differed significantly by sub-group including Government Office Region, Acorn Group 3 and Ethnic Minority Quintiles 4 4. Whether the profile of respondents differed across the two samples (incentivised and control sample) 5. Whether using an incentive had a positive impact in field from the interviewers perspective. 1.3 Structure of this report The report is structured as follows: Methodology This chapter presents key aspects of the incentive design, including how it was administered. Results This chapter examines the results of the experiment in terms of contact, cooperation and response rates from Quarter 2. It also looks at the impact of incentives on non-response. Conclusions This final chapter revisits the aims of the incentive experiment and in doing so draws conclusions on the value of using incentives (in the form of a book of stamps) for the Citizenship Survey. 3 Acorn is a geodemographic segmentation of the UK s population which segments small neighbourhoods, postcodes, or consumer households into five categories, 17 groups and 56 types. 4 Ethnic Minority Quintiles are calculated based on the proportion of the population within each ward classified as Black, Mixed, Asian and Other. Wards across England and Wales were classified into five quintile groups based on these proportions. 3

7 2 Methodology 2.1 Selecting the sample The decision to introduce incentives for the core sample only was made for two main reasons: 1. The core sample is the only sample type where we had not yet achieved the target response rate (the achieved response rate for Quarter 1 in Year 2010/11 was 57 per cent compared to a target of 60 per cent). 2. The core sample is the only sample type within the Citizenship Survey where we inform potential respondents in advance about the survey, via a letter sent to the address before the interviewer calls. Advance incentives are more problematic to administer (and less effective) if people cannot be informed in advance about them. It was important that there was a control group to test the impact of incentives against and it was equally important that there was no bias in terms of which addresses were selected to receive the incentive. Allocation was done within each primary sampling unit 5, so that exactly half of the addresses were randomly allocated to the incentive sample and half to the control sample. Every alternate address in the primary sample unit was assigned as an incentive address. 2.2 Selecting the incentive The Department for Communities and Local Government indicated that any incentive to potential participants or to the public should be cost-neutral. This instruction, coupled with a view that core addresses should not be treated vastly differently from other sample types, led to a decision that a monetary incentive (e.g. a 10 voucher) would be inappropriate. It was agreed, therefore, that the incentive would be a book of six first class stamps sent with an advance letter to the selected core addresses. 2.3 Administering the incentive The stamps were sent to the randomly selected core addresses with their advance letter and were unconditional: every address assigned to the 5 Further information on the overall sampling methodology used for the Citizenship Study can be found in the Citizenship Study Technical Report. 4

8 incentive sample received the stamps irrespective of them taking part in the Survey. In the advance letter, one additional sentence was added to those who received the incentive: As a token of our thanks, please find enclosed a book of stamps for your use. Apart from this sentence, the advance letter sent to addresses in the incentive and control groups was the same. The letters are provided as Appendix 1. Interviewers were made aware of which addresses had received stamps by the appearance of the letter S on the contact sheets. For those in the control group, the letters NS were on the contact sheets. Interviewers were advised to refer to the stamps on the doorstep when trying to encourage participation in the survey. As standard, all new interviewers to the Citizenship Survey receive a full day face-to-face briefing. The aims of the incentives experiment and how to refer to the incentives at the point of contact were presented and discussed during these briefings. For all existing interviewers, an incentives briefing note was circulated. 2.4 Fieldwork dates The incentive experiment was introduced at the start of Quarter 2 in July 2010 and continued in Quarter 3. This report examines the results for Quarter 2, which had a fieldwork period from 1 July 2010 to 30 September Data processing and analysis A variable to record whether the sample was an incentive or control sample address was added to the outcome code reports and incorporated into the SPSS dataset used for non-response weighting. 5

9 3 Results 3.1 Impact measurements In analysing the results it is important to look not only at the overall response rates achieved in the two samples but also at the contact rate and the cooperation rate. The contact rate measures the proportion of all eligible cases where the interviewer made contact with a household member, irrespective of their participation in the Survey or ability to provide further information about the household 6 The co-operation rate measures the number of achieved interviews as a proportion of those contacted during the fieldwork period 7 The response rate indicates the interviews achieved as a proportion of those eligible for the survey Impact of incentives Table 1 below presents the contact, co-operation and response rates of the incentive and control groups based on the outcome of the first issue. 9 First issue outcomes The results showed that while the incentive made no difference to the contact rate, respondents who received the incentive were more likely to co-operate than those who did not (65.02% vs %, p<0.01). This impact is also reflected in the overall response rate (54.5% vs %, p<0.01). 6 Peter Lynn, Roeland Beerten, Johanna Laiho and Jean Martin: Recommended Standard Final Outcomes Categories and Standard Definitions of Response Rate for Social Surveys, ISER Working Papers Number , p.27 7 Ibid, p.27 8 Ibid, p.26 9 All sampled addresses are issued and worked by interviewers. Where no interview has been achieved at a given sampled address, this address will be re-issued and re-allocated to another interviewer so that we maximise the chances of achieving an interview. There are a few cases when the address would not be re-issued, including the address being classified as deadwood, the selected respondent being physically or mentally unable to complete the interview, or having given a firm refusal to complete the interview. Addresses can be issued up to three times. 6

10 Table 1: Summary of first issue outcome measures by sample group Control group: No stamps Incentive group: All sample Stamps % % % Contact rate Co-operation rate ** Response rate ** Base (n) 2,240 2,257 4,470 ** significance at 1 per cent level based on a 1 tailed test Reissues Due to the higher response rate after the addresses were issued for the first time there was, not surprisingly, significantly fewer re-issues required in the incentivised group (28.93% vs %, p<0.01). The results also showed that there was a significantly lower proportion of addresses which were reissued this year compared to the corresponding quarter last year (Q2 2009/10) where no incentives were used (30.33 per cent of addresses reissued this year compared to per cent last year). The principle is that by reducing the number of re-issues, important costs savings can be achieved which makes the use of incentives a cost neutral exercise. Final issue outcomes Table 2 below shows that the initial differences between the experiment and control group in terms of initial co-operation and response rates are broadly maintained with the final outcomes. This is important because it suggests that the benefits of incentives persist and are not eliminated as a result of reissuing (despite less reissuing being carried out among the experiment group) 7

11 Table 2: Summary of final outcome measures by sample group Control group: Incentive group: All No stamps Stamps sample % % % Contact rate Co-operation rate * Response rate ** Base (n) 2,230 2,240 4,470 * significance at 5 per cent level, **significance at 1 per cent level, based on 1 tailed test 3.3 Impact of incentives on response rates by sub-groups Further analysis was conducted to identify if there was any evidence that the impact on response rates by incentives differed significantly by Government Office Region, Acorn Group and Ethnic Minority Quintiles 10. No evidence was found to suggest this was the case. However there are some differences that, whilst not statistically significant, are none the less worth reporting. Charts 1-3 below show the differences in response rates across the various sub-groups for the incentive and control groups. Incentives do not appear to show a positive effect in London, the Eastern region, the North East and Wales, as shown in Chart 1. In the South West and Yorkshire and Humberside, however, the effect of incentives on response rates appears more pronounced. 10 Ethnic Minority Quintiles are calculated based on the proportion of the population within each ward classified as Black, Mixed, Asian and other. Wards across England and Wales were classified into 5 quintile groups based on these proportions. 8

12 Chart 1: Impact of incentives on response rates by Government Office Region No stamps Stamps 70% 65% 60% 55% 50% 45% 40% East Midlands Eastern London North East North West South East South West Wales West Midlands Yorkshire & Humber Chart 2 shows the effect of incentives on response rates by quintiles of residents from an ethnic minority background living in the local area. It is the areas with the highest proportion of residents from an ethnic minority background that go against the trend and show no evidence of incentives improving response rates. This finding reflects the above response by Government Office Region many of the areas with a higher proportion of ethnic minority respondents are in London and the South East. 9

13 Chart 2: Impact of incentives on response rates by Ethnic Minority Quintile No stamps Stamps 70% 65% 60% 55% 50% 45% 40% 1st quintile % Ethnic Minority 2nd quintile % Ethnic Minority 3rd quintile % Ethnic Minority 4th quintile % Ethnic Minority 5th quintile % Ethnic Minority Finally, looking at the impact of incentives on response rates by Acorn Group (shown in Chart 3), whilst not clear cut, there does appear to be a trend towards a higher impact in those areas classified as containing families of one kind or another e.g. flourishing families, secure families and struggling families. 10

14 Chart 3: Impact of incentives on response rates by Acorn Group No stamps Stamps 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% Wealthy Executives Affluent Greys Flourishing Families Prosperous Professionals Educated Urbanites Aspiring Singles Starting Out Secure Families Settled Suburbia Prudent Pensioners Asian Communities Post Industrial Families Blue Collar Routes Struggling Families Burdened Singles High-Rise Hardship Inner City Adversity 3.4 Impact of incentives on respondents profile There were no statistically significant differences in the profiling characteristics of respondents when comparing the incentive group against the control group. This suggests that the use of incentives did not have a major effect on the profile of the achieved sample. 11

15 Table 3: Summary of respondent profile measures by sample group Incentive group: Control group: ONS Mid- Stamps No stamps Year Estimates 2009 % % % Sex Male Female Age Ethnicity White * Mixed * Asian * Black * Other * NS-SEC Higher and lower management N/A Intermediate, small N/A employers, lower supervisory Semi-routine and N/A routine Other (never worked/longterm unemployed/ student) N/A Base (n) * Source: 2001 Census 12

16 3.5 Impact of incentives from the interviewers perspective Informal feedback collected from interviewers suggested that the stamps helped in a number of different ways. Whilst a small number of interviewers felt the stamps made little difference, overall, the feedback was positive. Feedback suggests that the stamps helped interviewers to set a positive tone to the conversation on the door step by giving them something different and specific to say and generally starting the conversation in a good way. People remembered the stamps, and the whole conversation was much more positive compared to previous rounds. [The stamps] made people remember the letter better, made it harder for them to refuse to help and generally got things off on a good footing. Some interviewers also reported that mentioning stamps made it easier for respondents to remember that they had received an advance letter, with some people remembering the stamps rather than the content of the letter. I feel the stamps had an impact. The people said: I have not received the letter. And I said: Do you remember the book of stamps with the letter. They then said: oh yes. Finally, interviewers also mentioned that stamps may also have encouraged participation. Several interviewers mentioned that respondents who had received stamps expressed positive feelings about them although some respondents mentioned receiving the stamps but still refused to take part in the survey. 13

17 4 Conclusions In drawing conclusions on the impact and value of the use of incentives, attention returns to the study aims: 1. Whether using an incentive had a positive impact on overall response rates The experiment showed that while the incentive made no difference to the contact rate, respondents who received the incentive were more likely to cooperate than those who did not. This was also reflected in the overall response rate, which were higher among those who received an incentive (by three percentage points). Importantly, the differences between the incentive and control group in terms of initial co-operation and response rates were maintained with the final outcomes showing that the benefits of incentives persisted and were not eliminated as a result of reissuing. It is worth noting that these results are consistent with the results of a previous experiment conducted by TNS-BMRB on the 2004/05 British Crime Survey using the same methodology to assess the impact of stamps on response rates. 2. Whether incentives had an effect on relative re-issue rates The results show that due to the higher overall response rate after the addresses were issued for the first time there was, not surprisingly, significantly fewer re-issues required in the incentivised group. The results also show that there was a significantly lower proportion of addresses which were reissued this year compared to the corresponding quarter last year (Q2 2009/10) where no incentives were used (30.33 per cent of addresses reissued this year compared to per cent last year). This is important in terms of the costs savings implied by the lower level of re-issues, making the use of incentives a cost neutral exercise. 3. Whether the impact of incentives varied by different sub-groups. There is no evidence of a significant impact of incentive on response rates by Government Office Region, Acorn Group or Ethnic Minority Quintiles. There were indications of differences but these were not statistically significant. 4. Whether the profile of respondents differed across the two samples (incentivised and control sample) Comparison of profiling characteristics for the achieved sample by those who received stamps to those who did not receive stamps showed no significant differences in the profile of respondents, suggesting that the use of incentives did not have a statistically significant effect on the profile of the achieved sample. 14

18 5. Whether using an incentive had a positive impact in field from the interviewers perspective Informal feedback collected from interviewers suggests that the stamps helped interviewers to set a positive tone to the conversation on the door step, and made it easier for respondents to recall the advance letter. Interviewers also mentioned that, overall, many respondents expressed positive feelings about receiving the book of stamps. Positive impacts were experienced on co-operation and response rates whilst being cost-neutral to the Department for Communities and Local Government. Based on these results, it was recommended that incentives were rolled out to all of the core sample. Subsequently, stamps were sent with advance letters to all the core addresses in Quarter 4 of the 2010/11 Citizenship Study. 15

19 5 Appendix: Advance letters Letter to incentivised group Reference: Merge serial number The Occupier Merge address 1 Merge address 2 Merge address 3 Merge address 4 Merge address 5 Merge Postcode Dear Sir / Madam, Communities Study: Your Chance to be Heard We would like you to take part in the Communities Study; an important study about your views on a range of important issues including: your neighbourhood immigration rights and responsibilities influencing political decisions economic downturn Your views will help the government and local councils understand and improve decisions made on these important issues. The results are also used by charities, voluntary organisations and researchers. The leaflet included with this letter tells you more about the study. The Communities Study is being carried out by the independent research organisations Ipsos MORI and TNS-BMRB on behalf of the government department Communities and Local Government. Your address has been selected at random and an interviewer will visit you in the next few weeks to explain the study in more detail. All Ipsos MORI and TNS-BMRB interviewers carry identification badges with their photo. Please only allow people who carry this official identification into your home. Your address is one of just 20 selected to represent your local area and cannot be replaced. Your answers will be treated as confidential. It will not be possible to identify any individual from the survey findings, and the answers you give will be used for research purposes only. No identifiable information about you will be passed to government departments, local authorities or any other bodies without your consent. If you would like to talk to someone about the study, please call Elizabeth Lane from the study team at Ipsos MORI on freephone or communitiesstudy@ipsos.com Thank you in advance for your help. As a token of our thanks, please find enclosed a book of stamps for your use. Yours faithfully, SIGNATURE 16

20 Letter to control group Reference: Merge serial number The Occupier Merge address 1 Merge address 2 Merge address 3 Merge address 4 Merge address 5 Merge Postcode Dear Sir / Madam, Communities Study: Your Chance to be Heard We would like you to take part in the Communities Study; an important study about your views on a range of important issues including: your neighbourhood immigration rights and responsibilities influencing political decisions economic downturn Your views will help the government and local councils understand and improve decisions made on these important issues. The results are also used by charities, voluntary organisations and researchers. The leaflet included with this letter tells you more about the study. The Communities Study is being carried out by the independent research organisations Ipsos MORI and TNS-BMRB on behalf of the government department Communities and Local Government. Your address has been selected at random and an interviewer will visit you in the next few weeks to explain the study in more detail. All Ipsos MORI and TNS-BMRB interviewers carry identification badges with their photo. Please only allow people who carry this official identification into your home. Your address is one of just 20 selected to represent your local area and cannot be replaced. Your answers will be treated as confidential. It will not be possible to identify any individual from the survey findings, and the answers you give will be used for research purposes only. No identifiable information about you will be passed to government departments, local authorities or any other bodies without your consent. If you would like to talk to someone about the study, please call Elizabeth Lane from the study team at Ipsos MORI on freephone or communitiesstudy@ipsos.com. Thank you in advance for your help. Yours faithfully, SIGNATURE 17