The Impact of Survey Nonresponse on Survey Accuracy

Transcription

1 The Impact of Survey Nonresponse on Survey Accuracy Scott Keeter Handout prepared for a conference on The Future of Survey Research: Challenges and Opportunities Sponsored by the National Science Foundation October 3-4, 2012 Arlington, Virginia Nonresponse in the National Household Education Survey (Brick and Williams 2009)

2 2 Nonresponse in the National Immunization Survey (Brick and Williams 2009) Trends in Contact, Cooperation and Response Rates at the Pew Research Center % % % % % % Contact rate (percent of households in which an adult was reached) Cooperation rate (percent of households contacted that yielded an interview) Response rate (percent of households sampled that yielded an interview) PEW RESEARCH CENTER 2012 Methodology Study. Rates computed according to American Association for Public Opinion Research (AAPOR) standard definitions for CON2, COOP3 and RR3. Rates are typical for surveys conducted in each year.

3 3 Percentage Absolute Relative Nonresponse Bias of 959 Respondent Means by Nonresponse Rate of the 59 Surveys in Which They Were Estimated. Groves, Robert M., and Emilia Peytcheva The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis. Public Opinion Quarterly 72(2): Pew Research Center 2012 Study of the Impact of Survey Nonresponse Key Comparisons Standard Pew Research survey High-effort Pew Research survey Survey estimates from the standard survey are compared with estimates on 40 identical questions from a high-effort survey that achieved a response rate of 22%. U.S. government surveys Survey estimates compared with benchmarks from high-response rate surveys on demographic characteristics, receipt of government benefits, voter registration health status, community and political engagement, and others. Voter and Consumer databases Survey respondents and non-respondents in the landline samples are compared on a range of household characteristics, including past voting behavior, partisan affiliation, financial status and personal interests using two national databases with data on nearly every U.S. household. PEW RESEARCH CENTER 2012 Methodology Study.

4 4 High-Effort Surveys Increase Contact and Cooperation Levels Contact rate (% of households in which an adult was reached) Landline Landline Landline Cell Total % % % % % Standard survey High-effort survey Cooperation rate (% of households contacted that yielded an interview) Standard survey High-effort survey Response rate (% of households sampled that yielded an interview) Standard survey High-effort survey PEW RESEARCH CENTER 2012 Methodology Study. Rates computed according to American Association for Public Opinion Research (AAPOR) standard definitions for CON2, COOP3 and RR3. Modest Differences on Many Measures but Large Gaps on Civic and Political Engagement Pew Research standard survey US government survey % % U.S. Citizen Homeowner Lived at current address 5 or more years Married Children in household Internet user Current smoker In prior year, received Unemployment benefits Social Security payments Food stamps or nutrition assistance Registered to vote* Contacted a public official in past year Volunteered for an organization in past year Talked with neighbors in past week PEW RESEARCH CENTER 2012 Methodology Study. All government figures are from the Current Population Survey except marital status, smoking and home ownership which are from the National Health Interview Survey. *Based on citizens.

5 5 Voter Database Comparisons Financial Situation and Lifestyle Comparisons Database identifies household as Characteristics of landline households who Responded Did not respond Estimated net worth % % $500, Under $25, Home value $500, Under $100, Overall financial status Top 20% Bottom 20% Min. N for financial measures 695 6,179 Estimated party affiliation of first person listed in household Republican Democrat N for party measure 399 3,287 Interests Reading Community & charities Politics/current affairs Religious/inspirational Hunting and shooting Camping and hiking NASCAR N for interest measures 718 6,418 PEW RESEARCH CENTER 2012 Methodology Study. Household database information comes from a marketing services and information company that has household data on more than 124 million U.S. households. Comparisons are between landline households in the standard survey who Characteristics of landline households who Did not Database identifies Responded respond household as % % Registered to vote Voted in Party registration Republican Democrat Other 6 9 No record of party/not a party registration state N 593 4,972 PEW RESEARCH CENTER 2012 Methodology Study. Voter database information is based on public voter registration and turnout data. Comparisons are between landline households in the standard survey who responded and those who did not respond. Voter Database Comparisons Characteristics of landline households who Survey respondent Refusal/ Breakoff Other/ Unknown All sampled households % % % % Registered voter Voted in Party registration Republican Democrat Other No record of party/ not a party registration state N PEW RESEARCH CENTER 2012 Methodology Study. Voter database information is based on public voter registration and turnout data. Database comparisons are for landline samples only. Comparisons are between all responding households, refusing households and other or unknown households in the landline sample for the standard survey.

6 6 responded and those who did not respond.

7 7 Katherine G. Abraham, Sara Helms and Stanley Presser. How social processes