Technical Report for the 2011 Minnesota Health Access Survey: Survey Methodology, Weighting and Data Editing

Transcription

1 Technical Report for the 2011 Minnesota Health Access Survey: Survey Methodology, Weighting and Data Editing SHADAC, January This report provides information concerning the data collection process and methodology behind the Minnesota Health Access Survey (MNHA), emphasizing the most recent administration of the MNHA which occurred in Section 1 provides a brief overview of the survey s purpose and history since its inception Section 2 describes the sampling strategy Section 3 explains the survey administration process Section 4 describes the content of the MNHA survey Section 5 provides information about response and cooperation rates Section 6 explains how the data are weighted to represent the Minnesota population Section 7 describes key data edits and variable construction which precedes the analysis, and Section 8 describes the analysis strategy 1. Overview of the Survey The Minnesota Health Access Survey (MNHA) is a periodic survey of non-institutionalized Minnesota residents. The survey collects detailed information on health insurance coverage options, access to coverage and health care services, and basic demographic data. The goal of the survey is to document trends in health insurance coverage, and access to insurance and health care at the state and regional level, as well as for select subpopulations (e.g., rural, low-income and populations of color). The MNHA data play an important role in monitoring policy-relevant trends in health insurance coverage and informing health policy development in Minnesota on topics such as affordability of coverage, redesign of public program coverage, and evidence of discrimination faced by enrollees in state public programs. In contrast to national surveys that to some extent allow for the development of Minnesota-specific estimates, the MNHA provides more precise and timely estimates, includes a range of coverage and 1 Initial Submission August

2 access relevant questions, can be easily modified to be responsive to developing state health policy issues, and ensures the availability of micro-data for time sensitive research and policy analysis. The MNHA has been conducted a number of times over the years; in 1990, 1995, 1999, 2001, 2004, 2007, 2009 and This technical report focuses on the MNHA years that are most comparable to one another based on sample and survey design; specifically, from 2001 forward. 2 Until recently, the timeline for data collection has been a function of the availability of funding. 3 Although the Minnesota Department of Health has played a role in the MNHA from the beginning, the MNHA surveys from 2001 forward represents a partnership between the Minnesota Department of Health s Health Economics Program and the University of Minnesota s State Health Access Data Assistance Center (SHADAC). 2. Sampling Methodology The design of a sample is important to ensure that estimates derived from a survey are representative of the overall population and inferences are largely unbiased. A key objective of the MNHA surveys is to fill gaps in knowledge about Minnesota s uninsured population and to monitor key trends relevant to understanding factors that influence access to health insurance and health care services in Minnesota across populations and over time. The sampling strategy is designed specifically to generate reliable health insurance coverage estimates for the state overall, several geographic regions, and populations of color in Minnesota. The MNHA surveys are telephone surveys. Although telephone surveys have lower response rates than in-person interviews, they are more economical. Mail surveys are more economical than both telephone and in-person interviews, but they typically have the lowest response rates of the three survey modes. Furthermore, the complexity of the MNHA (where people with different insurance types are asked different sets of questions) is not conducive to self-administered/mail format. Web surveys hold promise for creating a self-administered version of the MNHA in the future. However, no webbased sample frames exist that can ensure appropriate representation of the Minnesota population. Moving to an address based sample frame would allow for the adoption of a mixed mode survey format. Specifically, the sample would be mailed an advance letter describing the study and offering the options of calling the survey center or going to a secure website to complete the survey. Sample elements with listed phone numbers can be telephoned if there was no response to the mail survey. 2 Details from earlier versions of the Minnesota Health Access Survey are not presented in this technical report. For information regarding earlier reports, please contact Kathleen Thiede Call at callx001@umn.edu or Stefan Gildemeister at Stefan.Gildemeister@state.mn.us. 3 The funding source has varied over the years, beginning in 1990 with funding from the Minnesota legislature; continued in 1995 and 1999 with support from the Blue Cross Blue Shield Foundation of Minnesota (BCBS-MN); in 2001 with funding by a grant from the federal Health Resources and Services Administration (HRSA); in 2004 with support from BCBS-MN, HRSA, the Minnesota Department of Human Services (DHS), and Hennepin County. Beginning with 2007, the survey is funded by legislative appropriation to the Minnesota Department of Health as a biennial study. The 2007, 2009 and 2011 surveys have received support from DHS. 2

3 As with all telephone-based surveys, although the goal is to learn about people, the sampling units in the study are telephone numbers. From 2001 forward the MNHA surveys were based on stratified random digit dial (RDD) landline telephone samples that oversampled in rural Minnesota and areas with greater probabilities of reaching populations of color. Thus, the sample for the MNHA surveys consists of telephone numbers grouped into geographically contiguous areas (strata). The strata were created to resemble as closely as possible state and sub-state geography in the areas sampled. For example, each year data are presented for the 13 economic development regions and are also available for some of the more populous counties (e.g., Hennepin and Ramsey). Within each geographic stratum, each telephone number had an equal probability of selection for the survey. As a way of obtaining sufficient sample sizes among populations of color, telephone exchanges in certain counties that U.S. Census Bureau and other outside data indicate have higher rates of key populations were oversampled. Over time, the following data have been used in constructing the sample for the MNHA: U.S. Census Bureau estimates of population by race and ethnicity; Projections of population groups by zip code developed by Claritas, Inc.; Population estimates from Minnesota s State Demographic Office; School enrollment data by race and ethnicity; and Telephone exchange estimates from GENESYS Sampling Systems, Marketing Systems Group (MSG). The 2009 MNHA marked the first year in which a dual frame design was employed that included both landline and cell phone RDD frames. The primary reasons for this change were growing concerns about sample coverage attributed to the steady increase of people living in cell phone-only households and desire to include all households with at least one type of phone in the survey. The range of telephone usage households includes landline only, cell phone-only and those with both landline and cell phone service (the latter are referred to as dual usage). According to Blumberg and Luke, 34.0% of US households were cell phone-only in the last half of In 2011, based on data from the National Health Interview Survey (NHIS), the percent of adults over 18 who lived in cell phone-only households in Minnesota was 32.3%, and for children under 18 it was 31.6%. 5 The decision to include a cell phone RDD frame was also based on the knowledge that the cell phone-only population is systematically different from those who can be reached by landline telephone. As compared to people in landline households, those in cell phone-only households are more likely to be low-income, young adults, males, black, Hispanic and 4 Blumberg SJ, Luke JV. Wireless substitution: Early release of estimates from the National Health Interview Survey, July-January National Center for Health Statistics. June Available from: Accessed June 28, State-specific estimates of telephone usage were not available from the NHIS or other sources. 3

4 renters. 6 While those in cell phone-only households report better health status and more active life styles, they are also more likely to be uninsured, report barriers to health care, and have higher rates of drinking and smoking. 7 Further, it appears that post-stratification adjustments, which have been shown to be effective mechanisms for reducing bias in overall population estimates, may not do enough to reduce bias in estimates within subpopulations (e.g., populations of color and young adults). 8 The targeted number and proportion of cell phone interviews was increased in 2011 as compared to 2009 because of evidence that the cell phone-only population is growing over time. From 2009 to 2011 alone, the proportion of US households that were cell phone-only increased from 24.5% to 34.0%. Cell phone sampling is new enough in the field of survey research that definitive guidelines about which cell phone households to include, based on telephone usage (i.e., cell phone-only, cell phone-mostly, or all cell phone users) do not exist. While cell phone-only cases (those who have a cell phone but no landline telephone) are typically included in a sample to reach those who otherwise would be excluded from landline telephone studies, there are a number of arguments for including persons in the study who can be potentially reached through both phone types or sample frames. Research indicates distinct differences between landline and cell phone-only households and there is evidence that those who live in dual telephone usage households and use their cell phone for all or most of their calls (termed cell mostly ) may be different than landline-only and cell phone-only populations. 9 Completing interviews in the 2009 MNHA with all cell phone sample cases, regardless of telephone use patterns, allowed for an evaluation of the efficiency of the various approaches, both from statistical and cost perspectives. 10 This evaluation helped inform design of the 2011 sample frame which includes all households that have at least one type of phone, and to implement screening procedures to increase the proportion of cell phone interviews that were from cell phone-only households. Specifically, a screening question was added to the 2011 cell phone sample frame in order to increase the likelihood of interviewing cell phone-only households, and half of otherwise eligible dual landline and cell phone households were screened out. While only 40% of all interviews in the cell phone frame and 7.4% of all 6 Blumberg SJ, Luke JV. Wireless substitution: Early release of estimates from the National Health Interview Survey, July-January National Center for Health Statistics. June 2012; Brick JM, Edwards WS, Lee S. Sampling telephone numbers and adults, interview length, and weighting in the California Health Interview Survey cell phone pilot study. Public Opin Q 2007; 71(5): Blumberg SJ, Luke JV. Wireless substitution: Early release of estimates from the National Health Interview Survey, July-January National Center for Health Statistics. June 2012; Lee S, Brick JM, Brown ER, Grant, D. Growing Cell-Phone Population and Noncoverage Bias in Traditional Random Digit Dial Telephone Surveys. Health Serv Res. 2010; 45(4): Call KT, Davern M, Boudreaux M, Johnson PJ, Nelson J. Can post-stratification adjustments do enough to reduce bias in telephone surveys that do not sample cell phones? It depends. Medical Care, 2011; 49(4): Lee S, Brick JM, Brown ER, Grant, D. Growing cell-phone population and noncoverage bias in traditional random digit dial telephone surveys. Health Serv Res. 2010; 45(4): Cell phone surveys are more expensive for several reasons: (1) federal law prohibits the use of predictive dialing, (2) respondents are often provided an compensation to reimburse their phone charge for taking the call, and (3) many calls are screened out as ineligible based on age (minors) and residence (have a Minnesota exchange but live outside of the state). Cell phone frames typically have lower response and cooperation rates than landline frames. 4

5 interviews (both cell and landline frames) were with cell phone-only households in 2009, 59% of cell phone interviews and 23% of all interviews were conducted with respondents with no landline in As stated earlier, respondents reached through cell phone frames are younger and more racially/ethnically diverse than those captured in landline frames. Nonetheless, we wanted to replicate the sample design across both sample frames to improve the odds of obtaining a representative sample. Therefore we stratified the cell phone sample using switch points throughout the state to mimic the design of the landline frame. The results indicate that this worked quite well. The concordance between respondent-reported residential zip code and the switch point location (within sample strata) was 66% overall (lower in the Northwest and Central regions of the state) compared to 90% concordance between zip code and exchange in the landline frame. 11 Details about the landline and cell phone samples used for the 2011 MNHA are included in Appendix A. The decision to conduct interviews with all cell phone respondents adds to the complexity of weighting the data. Because base weights to adjust for the probability of selection are calculated separately for each survey frame, individuals with landlines and cell phones, the sizable overlap population, would be doublecounted in the construction of weights. Adjustments to correct for this factor are discussed in Section 6, Weighting Survey Responses. Consistent with 2009, a screening question was also included in the landline sample frame in 2011 to help with identifying and excluding a portion of households without members under the age of 65. Half of all landline households comprised of only adults age 65 or older were screened out for the first six weeks of the survey which was increased to 75% for the remainder of the data collection period. This was necessary given the disproportionately high participation among those 65 years and older in past administrations of the MNHA, landline samples, and surveys more generally. Additionally, when selecting the target from the household, children under age 18 within the household were given a 50% higher probability of selection than adults in the household. 3. Survey Administration Prior to 2009, the data were collected by the Survey Research Center at the University of Minnesota, School of Public Health, Division of Health Policy and Management. Following the closure of that center and a competitive bidding process, data collection in 2009 and 2011 was conducted by Social Science Research Solutions (SSRS), an independent research company based in Pennsylvania. The study received IRB approval from the MN Department of Health and the University of Minnesota. As part of the survey protocol, respondents receive a Tennessen warning at the time of the interview, as well as telephone numbers for the University of Minnesota Human Subjects office should they have concerns about the 11 Dutwin D. Stratification of cell phones: Implications for research. Presented at 66 th Annual AAPOR Conference. Phoenix, AZ. May

6 interview, and Dr. Call, should they have additional question or concerns about the goals of the study a use of the data. Each year the data were collected through Computer Assisted Telephone Interviews (CATI) following a brief field test. The CATI system was programmed (Computers for Marketing Corporation (CfMC ), version 8.1.) and thoroughly reviewed by all partners (SSRS, MDH, and SHADAC) prior to pretesting the instrument. The review consisted of a multiple iterations of analyzing the accuracy of the skip pattern logic and interviewer directions for this complex instrument. Interviewer training was conducted prior to the pretest and just before the study officially entered the field. Call center supervisors and interviewers were walked through each question and provided a Q by Q manual that explains the motivation behind each question and provides responses to common or potential inquiries from respondents. Interviewers were also given general training to help them maximize response rates and data quality. Interviewers were instructed to emphasize the social and policy relevance of the study and to reassure respondents that the information they provided was confidential. A total of 40 pretests were conducted (30 landline; 10 cell phone), followed by careful review of the recorded interviews and data. The pretest resulted in minor changes to the CATI program. Calls were monitored over the course of the study (live by SSRS supervisors, and via de-identified audio recording by MDH and SHADAC staff), with intermittent interviewer training provided as needed. The 2001 survey was conducted from November 2000 to May 2001; the 2004 and 2007 surveys were conducted from July to December of their respective year; the 2009 survey was conducted from August to November 2009; and the 2011 survey was conducted from September to December 2011, benefiting from more robust call center capacity. 12 A detailed data collection timeline is provided in Appendix B. The survey fielding enacted the following best practice procedures that partly targeted achieving high survey response rates: The targeted number for total call attempts was set at 12 and 10 maximum call attempts for landline and cell frames respectively. The time of day and the day of the week when call-backs were placed was varied using a programmed differential call rule. Respondents were permitted to set the schedule for a call-back and were offered the opportunity to initiate the phone-back on a toll free number. Any record dispositioned to have a privacy manager (a call screening service works with caller ID to stop unidentified callers before the phone rings) were immediately called back manually on an open line that relays caller ID information and therefore is not blocked. 12 Detailed data collection results for the 2011 MNHA authored by SSRS are available by request from Kathleen Call at callx001@umn.edu or Stefan Gildemeister at Stefan.Gildemeister@state.mn.us. 6

7 Initial refused interviews were rested for a period of two weeks, after which a refusal conversion attempt took place. Second refusals were put to bed for an additional four weeks, when a second conversion was attempted. At beginning of the field period, a limit of six total calls was placed on all pieces of sample. The pieces of sample that were still categorized as No Answer, Busy, or Answering Machine after six calls (and therefore still had a potential to be converted into a completed interview) were rested for one to two weeks. At the end of the data collection period, most unresolved calls had reached the ten or twelve call limit after multiple rounds of rest periods. Pieces of sample that were initially categorized as refusals were called back at least twice for an attempt to convert the refusal to a completed interview. Across both landline and cell phone sample frames, a total of 2,230 refusals were converted to completed interviews by specially trained refusal converters. Each year the MNHA has been administered, at a minimum, in English and Spanish. 13 The average length of time it takes to complete the MNHA interview has been maintained over time at around 15 minutes. The actual time it takes to conduct an interview, however, varies by household size, insurance status, telephone status, and survey language. Surveys completed in English took 15 minutes on average in 2009; it took an average of 24 minutes to complete the Spanish version and 18 minutes to conduct interviews with cell phone respondents. Cell phone interviews required additional time due to extra questions needed to establish eligibility, safety, and to gather contact information at the end of the interview to allow for mailing the compensation. a. Compensation and Voic Experiment In 2011, an experiment was implemented to understand the effect of different compensations schemes and voic messages on participation rates within the cell phone sample. The objectives were to evaluate how cell phone users respond to voic and compensation conditions independently and combined. The experiment tested six conditions: half (50%) of the cell phone sample was assigned a no compensation group and half (50%) was assigned to receive either a $5 (25%) or $10 (25%) compensation (See Figure 1). Sample that was not resolved within the first call attempt was included in the voic component of the experiment The 2001 and 2004 the survey was also translated into Hmong; respectively 32 and 85 surveys were conducted in Hmong. This in contrast to 185 and 296 Spanish surveys completed in 2001 and The 2004 survey was also translated into Somali, with a total of 38 surveys conducted in Somali. Because of the low number of surveys conducted in Hmong and Somali and the relatively high cost of these translations, the survey was only translated into Spanish in subsequent waves of the MNHA. A total of 187 and 140 interviews were conducted in Spanish in 2007 and 2009; 198 were conducted in Answering machine messages were left on calls 1 and 4 for the cell phone sample and on calls 3, 6 and 9 for the landline. 7

8 The no compensation group had two voic groups: one that received the standard voic informing them about the survey and providing the call-in number and another that did not receive a voic . The no voic (control) group was limited to a small sample size that was expected to yield 100 completes in order to limit any potentially negative effects of not leaving a voice mail. The compensation groups were split into two groups with approximately half receiving a voic that informed respondents about the compensation and provided the call-in number (compensation voic ) and the other half receiving the standard voic that did not mention the compensation. Respondents in the no compensation condition were provided compensation if they proactively requested it and provided the contact information necessary to receive the check. Figure 1. Design of Compensation and Voic Experimental in the Cell Phone Sample Response rates across the six conditions ranged from 39.6% to 41.0% with no significant differences in response rates across experimental conditions, neither within voic or compensation conditions alone nor interactions of the two. We also examined mean number of calls made to complete the survey and found no significant differences across the six conditions of the experiment. However, we did find significant differences in the percent of completed interviews among eligible cases, where eligible cases in the $5 and $10 compensation conditions who received a voic informing them of the potential compensation were more likely to complete the survey than those in the no compensation conditions who only received a standard voic informing them about the study. An additional area of interest within the experiment was an assessment of whether respondents eligible for (or requesting a compensation among those in the non-incentive condition) would provide contact info in order to receive the compensation, and how those who provided contact information differed from other respondents. There were statistically significant differences in the provision of contact information: 15.4% of respondents in the no incentive condition requested compensation of whom 8

9 73.6% provided contact information, as compared to 64.8% of respondents in the $10 condition and 54.7% of respondents in the $5 condition who provided contact information necessary to receive the compensation. Non-white respondents were more likely to request compensation and of those requesting compensation, those from Greater MN were more likely to provide contact information. Within the two incentive conditions, females, young adults, respondents with lower family income and lower education were likely to provide their contact information, while employed respondents and home owners were less likely to do so. In conclusion, these findings demonstrate that offering compensation in the cell phone frame did not lead to an increase response rates or efficiency as measured by the number of dials to obtain a completed interview, however, it does appear to increase efficiency as measured by the number of completes overall in each arm. Even if compensation is offered, not all respondents will provide the contact information necessary to receive compensation yet it is important to provide compensation to those who request it. 4. Survey Content The MNHA represents a version of the Coordinated State Coverage Survey (CSCS) developed by SHADAC to a significant extent based on the MNHA. Within each randomly selected household that agreed to participate, the person most knowledgeable about household members health insurance is asked to complete the survey. After identifying that the household is eligible to participate in the study (e.g., Minnesota is primary place of residence), the household is enumerated and gender, age and relationship information is gathered for all household members, and one person is selected at random to be the target of the survey. The majority of survey questions ask about health insurance coverage for the randomly selected individual within the sampled household the target. This is followed by questions about health insurance coverage for all other household members, and education and employment information about all adults in the household. Information is also collected concerning potential sources of insurance such as through the target s own or a family member s employer. Those lacking insurance are asked about their (their parents ) reason not to purchase coverage and about their awareness of public programs. The target s health status, access to health care and dental coverage is assessed, along with details about employment and marital status (requested for primary caregiver or wage earner if the target is a minor). County of residence, race/ethnicity, nativity and length of time living in the US and Minnesota are also asked. Finally, information about family income is requested along with questions relevant to weighting the data (e.g., number of phone lines 15 ). 15 Prior to 2011, the survey included questions about interruptions in telephone service used to adjust the weights to account for households with no telephone service. This adjustment has become less common over time; these items were dropped. 9

10 Some aspects of the survey have changed over time. For example, beginning in 2007 a question about homeownership was added in order to make adjustments to the data to account for response bias (see section on post-stratification weights below). The section dedicated to identifying whether elderly respondents carry supplemental Medicare coverage has evolved over time to account for federal changes in prescription drug policy. Additionally, questions about cell phone usage were included (modeled after the National Health Interview Survey (NHIS)). In 2009, the phone usage questions were expanded due to the addition of the cell phone sample frame, and questions were added to the introduction to screen out those driving at the time of the call and others for whom taking a call would not have been safe, as well as minors (in the cell phone frame unless they shared the cell phone with an adult). Questions were also added to capture emergency room (ER) use, appropriateness of ER use, and experiences of having premiums change or insurance coverage dropped due to a change in health status. For those with public insurance, several questions about knowledge of benefits were added. In addition to asking about the income of family members, an income question was included that focuses only on household members directly related to target. Finally, given the downturn in the Minnesota economy in 2009, a few questions were added to capture expectations about changes in income and experiences with transitions in health insurance coverage. In 2011 several questions were added to establish a baseline for monitoring the implementation of health reform (e.g., access barriers due to cost, difficulty obtaining a clinic appointment). Medicare specific questions that had been asked in 2007, but dropped in 2009, were added back in In order to construct the Household Insurance Unit (HIU) 16, a series of questions capturing marital status of all adult household members was added in Finally, the questions regarding household count were revised in November of 2011, following the discovery that some respondents were not accurately counting all people in the household. Overtime there were several omissions as well. For example, questions about premium costs asked in past versions of the MNHA were dropped in Beginning in 2004, in addition to asking ethnicity and the five broad race categories, follow-up questions were added that allowed respondents to provide more ethnic-specific self-identification. 17 For example, a respondent who self-identified as black/african American could further indicate Ethiopian, Somali, etc ethnicity. Similarly, a respondent who indicated their race was Asian could go on to indicate Vietnamese, Hmong, or other ethnicity identifiers. 18 These detailed self-identification questions were omitted in 2009 because sample size per race/ethnicity 16 A household insurance unit includes only those members of the household that would be eligible for the same health insurance policy. 17 This was consistent with Office of Management and Budget (OMB) suggestions to expand race/ethnic categories to tailor to the local context. Provisional Guidance on the Implementation of the 1997 Standards for Federal Data on Race and Ethnicity. Tabulation Working Group Interagency Committee for the Review of Standards for Data on Race and Ethnicity The 1998 and 2001 surveys expanded the list of race options to include Hmong and Vietnamese. 10

11 community in earlier years did not permit disaggregated reporting. Questions about the use of medication and health care for longer term conditions (conditions that lasted for at least 12 months), included in 2007, were not continued in In 2011 several questions were dropped that either had very small responses in the past or were never analyzed. These include detailed questions regarding spouse s employer coverage, knowledge of payment for doctor visits among those covered by public insurance, ER specific questions and questions about length of travel to receive care. Several phone and cell phone usage questions that were used in the past to adjust weights were dropped, as these questions had little impact on the weights. Also dropped because of time constraints were questions regarding income change and length of time residing in Minnesota. 5. Response Rates and Sample Coverage Over time response rates have dropped for all types of surveys. For telephone based surveys this general trend is attributable to 1) growth in the non-contact rate (e.g., fewer people answering their phone as a result of telephone screening devices) and 2) growth in refusal rates (e.g., households/individuals declining to participate in a survey due to frustration with fundraising and marketing phone calls and survey research in general). 19 Falling response rates and the implications for data quality are the subject of intense attention and scrutiny as demonstrated by special issues on non-response bias in the premier survey research journal, Public Opinion Quarterly, in 2006 and Response rates are a commonly used indicator of the quality of a survey. Traditionally, the response rate for a survey has been used as a proxy for the degree of systematic difference between respondents and non-respondents. 20 Therefore it makes sense that survey researchers spend resources to improve response rates such as repeated contact attempts to potential respondents, compensations, advance letters, and conversion of refusals. Fortunately, recent research indicates that lower response rates are not necessarily associated with greater response bias because surveys with high and low response rates demonstrate similar levels of absolute bias. 21 In a Pew survey of political attitudes, Keeter et al. (2006) 22 tested whether estimates derived from a rigorous method were similar to the estimates produced from the standard method even though the response rate of the rigorous method was twice as high (50% versus 25%). The 19 Curtin R, Presser S, Singer E. Changes in telephone survey nonresponse over the past quarter century. Public Opinion Quarterly, 69(1):87-98, 2005; Singer E. Introduction: Nonresponse bias in household surveys. Public Opinion Quarterly 70(4): , State Health Data Assistance Center (SHADAC). Are lower response rates hazardous to your health survey? Issue Brief 13. Available at: Accessed April Groves R. Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5): , Keeter S, Kennedy C, Dimock M, Best J, Craighill P. Gauging the impact of growing nonresponse on estimates from a national RDD telephone survey. Public Opinion Quarterly, 70(5): ,

12 estimates derived from the standard and rigorous methodology were in fact similar. This result was confirmed in a study of health insurance coverage. Davern et al. (2010) found that after adjusting for basic demographic characteristics, the estimates produced from a strategy characterized by multiple call attempts produced the same estimates of health insurance coverage and access as a less aggressive (and lower response rate) strategy. Since surveys are conducted within budget constraints, efforts to complete surveys of reluctant responders, instead of contacting new subjects who have a higher probability of response, decreases availability of sample and may not improve response bias. Therefore, some have suggested that expending limited resources to improve response rates beyond a certain point may not be cost-effective. 23 In general terms, the response rate is the ratio of the number of completed interviews divided by the number of eligible reporting units in a sample; the cooperation rate is the ratio of all interviewed cases to all eligible cases contacted. The response rates reported in Table 1 refer to AAPOR Response Rate #4 24 for the blended sample (cell and landline combined) which is the equivalent of the number of completed interviews divided by the total number of eligible phone numbers. 25 Consistent with the literature, MNHA response and cooperation rates have somewhat diminished over time and refusal rates have increased incrementally, but stabilized between 2009 and Davern M, McAlpine DD, Beebe TJ, Ziegenfuss J, Rockwood T, Call KT. Are lower response rates hazardous to your health survey? An analysis of three state telephone health surveys. Health Services Research 45(5): , The American Association for Public Opinion Research Standard definitions: Final dispositions of case codes and outcome rates for surveys. 5th edition. Lenexa, Kansas: AAPOR. Available at: 11.pdf 25 To estimate the number of eligible phone numbers among numbers with unknown eligibility (e.g., no answer), this rate applies the ratio of eligible to ineligible numbers among the numbers with known eligibility to the unknown numbers and includes the resultant number within the denominator of the response rate calculation. 12

13 Table 1. MNHA Sampling and Survey Description MNHA Total Completes Response Rate* Cooperation Rate* Refusal Rate** ,315 67% 78% 19% ,802 59% 68% 28% ,728 43% 57% 32% 2009^ 12,031 45% 53% 39% 2011 # 11,355 44% 45% 39% ~ represent landline sample frames; 2009 forward represent dual landline and cell phone sample frames. * Based on AAPOR RR4 response and cooperation rates from ; Based on AAPOR RR3 response and cooperation rates from 2009 forward which excludes partials. ** Based on AAPOR refusal rate 2 (REF2); includes estimates of eligible cases among unknown cases. For comparability with prior MNHA surveys, refusal rate calculations in 2009 and 2011 ignored screening that occurred (e.g., excluding minors both years and over sampling of cell only households in 2011). In 2011, the refusal rate was 27% and 58% for landline and cell frames respectively. ^ The 2009 total is comprised of 9,811 landline and 2,220 cell phone completes with response rates of 50% and 31% and cooperation rates of 58% and 40% for landline and cell frames respectively. # The 2011 total is comprised of 7,028 landline and 4,327 cell phone completes with response rates of 48% and 39% and cooperation rates of 49% and 40% for landline and cell frames respectively. 6. Weighting of Survey Responses The aim of weighting survey data is to adjust the results to account for sample coverage problems (the difference between respondents and non-respondents) and reduce potential bias associated with differential participation in the survey. Accounting for varying probabilities of selection and response rates through the application of weights enables the survey responses drawn from statistical samples to be representative of the entire population. Two types of weights were generated: 1) base weights and 2) post-stratification weights. The base weight takes into consideration that each respondent s probability of selection varied by sampling stratum, the number of phone lines connected to the household (or number of cell phones accessible to adults in the case of the cell phone frame), and the number of people living in the household. The post-stratification weights adjust the base weight to account for key characteristics of the state s population. Specifically, to more accurately reflect the population, sample weights were post-stratified by region, age, education, race, nativity (US versus foreign born), home ownership (beginning in 2007), household count, telephone usage, and telephone service interruption (dropped in 2011). An added complication in 2009 and 2011 for the computation of weights was the addition of a cell phone sample frame that was not limited to cell phone-only respondents. Therefore the weights needed to account for the probability of including individuals in the sample who live in dual landline and cell phone households. a. Base Weights Landline samples are associated with households and do not select individuals per se. This approach randomly draws telephone numbers associated with households within desired geographic areas (or 13

14 switch point in the case of the cell phone frame). By contrast, cell phone numbers are associated with individuals. The landline and cell RDD samples used in the MNHA were drawn from a sampling frame of Minnesota phone numbers in active area code/exchange groupings within geographic strata. 26 The first component of the base weight accounts for a person s known probability of selection based on the chosen geographic strata. This is necessary because some areas of the state were oversampled relative to others. The strata adjustment is calculated by dividing the total number of telephone numbers available in each region (regardless of whether or not they are in the sample) by the total number of interviews completed in that region. This indicates how many telephone numbers are represented by each telephone number that resulted in a complete. The strata weight component also accounts for differential response rates by strata. A second component of the base weight accounts for the number of people in the household. People in larger households have a smaller probability of being included than people in smaller households. Therefore, people in larger households receive, on average, larger base weights, correcting for their lower probability of selection. The second base weight component also illustrates that the purpose of weighting the MNHA is to develop ultimately person-level weights, essentially translating the response from randomly selected individuals in households into representative responses about Minnesotans in aggregate. Third, we adjust for the number of telephones in the household, as persons in households with more telephone lines (or cell phones in the cell sample) have a greater probability of being selected into the sample. For example, households with two telephones are twice as likely to be randomly selected as are single-telephone households; a weight of one-half appropriately adjusts for the two telephone household s greater probability of selection. In the case of households with cell phones, we account for the number of cell phones that could be answered by an adult in the household as we do not directly interview minors or cell phone numbers assigned to minors. 27 Below (see Dual Frame Weights) we describe adjustments made to account for the possibility that members of landline sample could also be captured in the cell phone sample. 26 As is common practice in survey research, the landline sample was drawn from banks of telephone exchanges that contained at least three listed household phone numbers (versus numbers assigned non-residential households). This increased the efficiency of the sample by increasing the likelihood of reaching an eligible household, study cost are reduced. 27 It is important to note that although the steps of the base weight calculation were the same in the landline and cell phone frame, the calculation was operationalized in separate analyses to ensure that the probability of selection was calculated specifically for each frame. In other words, the number of landline phones was considered for people who were reached in the landline frame (not the cell phones that respondents might have had access to). Similarly, in the cell frame no landlines were considered in calculating the probability of selection associated with access to working cell phones. 14

15 In 2009 and 2011, a fourth adjustment was made to the landline sample only, to account for the elderly screening conducted in the landline frame. 28 This adjustment was applied in a similar fashion to a poststratification adjustment and used American Community Survey (ACS) control totals for the proportion of households that are comprised of members 65 years of age and older (65+). This adjustment was made to bring the number of 65+ landline sampled households into alignment with the population screening alone did not accomplish this, as even after screening elderly respondents made up a greater portion of the survey respondents than the population overall (see section 2 above, Sampling Methodology). Because this adjustment relates to probability of selection for the landline frame, it is made prior to merging with the cell frame and applying post-stratification weights. Additionally, in 2011 we increased the probability of selecting a child as the target in households with children which required that we adjust these cases down in the base weight. b. Creating and Selecting the Dual Frame Weights Beginning in 2009, the MNHA incorporated a cell phone frame which introduced a new set of decisions and necessary weighting steps. This offered the opportunity to evaluate potential alternatives for correcting for cell phone under-coverage, including by determining: The relative data quality of responses from portions of the cell phone frame (distinguishing between cell-only respondents and cell mostly respondents); The sensitivity of population and sub-population estimates to including or excluding alternative portions of the sample frame; The relative statistical properties of survey estimates computed with alternative portions of the sample frame; and The cost implication of producing estimates with similar statistical properties but through various sampling strategies (e.g., how does the impact of screening for cell phone-only households and needing less sample compare with accepting all cell phones but needing a larger sample?). In 2009 SHADAC contracted with the National Opinion Research Center (NORC) to perform the evaluation of the dual frame weighting methodology and make recommendations for future waves of the MNHA. As part of this work, NORC computed weights based on four alterative combinations of survey frames, to test the efficiency of these survey frame combinations on the construction of final estimates: 28 Given that elderly have been shown to be over-represented in telephone surveys, the MNHA screened out a portion of households that were only composed of people 65 years of age or older. 15

16 1. Reject all of the cell phone interviews (basing estimates on the landline interviews alone); 2. In addition to landline completes, accept just the cell-only interviews; 3. Accept landline, cell-only and cell mostly interviews (exclude those who use cell phones rarely); or 4. Accept all of the landline and cell interviews. For two of these survey frame combinations (choices 3 and 4), NORC computed a series of weight adjustment factors to account for the overlap in both frames, i.e., the portion of households that could be theoretically reached in either the landline or cell phone frame. As mentioned earlier, the weight adjustment is focused on avoiding the creation of overstated weights for the overlap population, in which the separate sum of weights of the cell sample and the landline sample would be based on estimates of all landline numbers and cell phone numbers, effectively double-counting the overlap population. Again, multiple contacts of individuals across the two frames do not factor into this analysis. A common strategy of accounting for this overlap is to multiply the weights for the landline RDD interviews by a weighting adjustment factor, or λ (lambda), and multiply the weights for the cell phone interviews by 1- λ. Although some cases still have a chance of being included in either sample frame, the weights are adjusted so they are not overrepresented. This would be easier if information about the actual amount of overlap was available for the relevant geographic area and time frame (i.e., Minnesota in 2011). As this is not the case the adjustment factor, λ, for frames 3 and 4 was calculated five different ways to simulate alternative scenarios. Specifically, the approaches to calculating λ ranged from (1) the simple assumption that someone with landline and cell phones was twice as likely to be included (λ is set to 0.5) to setting λ to match to (2) observed, (3) effective or (4) weighted sample size per frame, to (5) the bias-correcting assumption where data from the National Health Interview Survey (NHIS) for the Midwest Census region was used to set the value of λ for cases with different patterns of telephone use. For the evaluation of the most appropriate weighting strategy, NORC produced a total of 12 composite weights (5 lambda weight adjustment factors applied to two of the four survey frame combinations, and two frame combinations without an overlap, e.g., landline only and landline plus cell phone-only). For each composite weight NORC computed the Mean Squared Error (MSE) on four key analytic variables to inform the selection of the most efficient weight. 29 The best performing weight adjustment in terms of MSE was the one that calculated lambda for the two overlap sample frames proportionately to the relative effective sample sizes (lambda adjustment 3 29 The MSE represents total survey error and is comprised of sampling error (variance) and non-sampling error (bias). Since in this theoretical study the true population means are not known, the true bias is not calculable. In two analyses, NORC assumed that unbiased estimates would be obtained from the sample frame that included only the landline and cell phone-only sample (option 2, above) or from the sample that included the landline and all cell phone completes using the the bias correcting lambda correction (option 5 above). 16

17 above; for details, see Appendix C). 30 The outcome of the evaluation suggests that the combination of landline and all cell phone interviews results in the lowest MSEs and outperforms the other three choices. It is important to note that this result is influenced in part by the sample design and the resulting distribution of completed interviews by telephone usage in 2009 (see Table 2), which resulted in a relatively low share of cell phone-only cases. In 2011, we intentionally screened to increase the proportion of cell phone-only cases in 2011 and used the same methodology for creating the lambda weight. A presented in Table 2, this screening resulted in a larger portion of cell phone-only completes the cell frame (22.5%) as compared to 2009 (7.4%), yet this is still below estimates for the Midwest of 29.9% in Despite the weighting adjustment, the cell phone completes, generally, had larger base weights than the landline completes simply because the universe of available cell phones per sample strata is larger than is true for landline strata. 32 Table , 2009 MNHA Telephone Usage Per Sample Frame (Unweighted) Sample Size by Frame Landline Cell Total Total % Landline Only 1,564 1,967 X X 1,564 1, Cell Phone-Only X X 2, , Cell Phone-Mostly 1,177 1, ,713 1, Landline Mostly 4,287 6,561 1, ,525 7, ,028 9,811 4,327 2,220 11,355 12, c. Post-stratification Weights While the base weights adjust for the known unequal probability of selection, post-stratification weights adjust for ways in which the sample s demographics and the resulting completed interviews differ from what is known about the population from which the sample was drawn. Population control total data are from an independent source (outside the survey); typically from the US Census Bureau. For example, if 20 percent of survey respondents were 65 years of age or older (with the base weights applied) yet the census data indicate that only 12 percent of the general population was elderly, a post- 30 Detailed results of the NORC dual frame weight evaluation are provided in Appendix C. The NORC report is available by request from Stefan Gildemeister at Stefan.Gildemeister@state.mn.us or Kathleen Call at callx001@umn.edu. 31 Blumberg SJ, Luke JV. Wireless substitution: Early release of estimates from the National Health Interview Survey, July- January National Center for Health Statistics. June Available from: Accessed June 28, In contrast to landline carriers, cell carriers have opened up scores of cell phone exchanges, many of which are not in use. Further landline frames can be restricted by excluding 100 blocks of phone numbers that have a limited number of phone numbers that appear in a listed directory. This increases the efficiency of the frame. This cannot be performed for cell frames as there are no directories. 17