Survey Methodology Program. Working Paper Series. Evaluation of Two Cost Efficient RDD Designs. Judith H. Connor Steven G.

Transcription

1 Survey Methodology Program Working Paper Series Evaluation of Two Cost Efficient RDD Designs Judith H. Connor Steven G. Heeringa N"0I7 Survey Methodology Program Institute for Social Research University of Michigan P.O. Box 1248 Ann Arbor, Ml Phone (313) Fax (313)

2 Evaluation of Two Cost Efficient RDD Designs Judith H. Connor, Steven G. Heeringa Survey Research Center, University of Michigan Paper presented at the AAPOR 47th Annual Conference St. Petersburg Beach, FL May 18, 1992 Introduction This paper will describe an analysis of the coverage of cost efficient random digit dialing (RDD) telephone sample designs which use as a frame a list of all hundred series with at least one listed household telephone number. The listed hundred series (LHS) frames, which are compiled from national files of directory listings, provide a cost efficient alternative to the traditional two-stage Waksberg-Mitofsky RDD design. RDD sample design methods were developed in the early to mid-1970s to address the noncoverage of unlisted telephone numbers that was inherent in directory sampling methods (Glasser and Metzger, 1972). The newly developed RDD methods addressed the unlisted number coverage problem but at the price of low working rates in the selected samples of telephone numbers (Groves, 1978). Working both with the RDD sampling frame and the sampling algorithm, new RDD designs were developed which achieved higher effective RDD working rates and the associated cost efficiencies (Sudman, 1973; Waksberg, 1978). Sudman's (1973) directory-based RDD method is in fact one form of a listed hundred series (LHS) design. True LHS RDD designs of the type evaluated in this paper first appeared in commercial and marketing research (Frankel and Frankel, 1977). Commercial research firms including A.C. Nielsen (A.C. Nielsen Company, 1976) and Chilton Research Services employed LHS-type designs in telephone survey research. Survey Sampling, Inc., one of the first major commercial suppliers of RDD telephone sampling services, offers the LHS design as one of several RDD sample design alternatives (Survey Sampling, 1991). Dale Kulp of Marketing Systems Group developed the Genesys Sampling System (Genesys, 1994), a PC-based system for selecting LHS samples. While LHS designs are now common in commercial and niarketing research, there is very little public information on the coverage properties of LHS RDD telephone samples compared to the higher cost, higher coverage Waksberg-Mitofsky design alternative. Casady and Lepkowski (1993) have examined the properties of the complete RDD sample frame (including zero-listed banks) and have developed cost and error models for two alternative LHS designs. The first section of this paper discusses the properties of the hundred series RDD frame and two LHS design alternatives that have been used at the Survey Research Center (SRC). The second part describes the design of a special telephone household coverage evaluation that uses the monthly sample control files of SRC's Surveys of Consumer Attitudes (SCA). The results of this special coverage study are presented in Part 3 of the paper. The paper concludes in Part 4 with general conclusions about the practicality of the new sample designs for social science research studies. l

3 [Readers should also see Brick et. al. (1994) for a second empirical study of the coverage properties of the LHS RDD sample design.] I. Properties of Telephone Sample Frames and Design Alternatives I.A. Bellcore Frame / Waksberg-Mitofsky Two-Stage RDD Design The Bellcore file, which can be purchased from Bellcore Research, contains a list of all Area Code/Central Office (AC/CO) combinations which are currently in use. This file is updated monthly. The number of AC/CO combinations which appear on the April 1990 Bellcore file is 41,308. If all possible combinations of four digits (the numbers ) were appended to these 41,308 AC/CO records, a sampling frame of41,308 * 10,000 or 413,080,000 possible telephone numbers is generated. This is the frame used in the first stage of the Waksberg-Mitofsky design. Although the percent of these numbers which are working residential numbers varies by area of the country, on average about twenty-two percent of the possible ten digit numbers generated from known AC/CO combinations are working residential numbers. However, because working residential numbers are clustered within banks of consecutive numbers, once a hundred series (the first eight digits of the phone number) is identified as having at least one working residential number, then on average over fifty percent of the numbers in the hundred series will be working household numbers. Both the Waksberg-Mitofsky design and the LHS single stage design take advantage of the clustering of working.residential telephone numbers to increase the probability of reaching working household numbers, thereby reducing the number of unproductive phone calls. In the Waksberg- Mitofsky design, the primary stage calling results in about 22 percent working household numbers, and the second stage about 57 percent working numbers. For a typical sample design with 1,500 respondents, a 70 percent response rate, and a cluster size of 8, about 470 clusters (working primaries) are needed. This would require a primary stage sample of about 2,136 numbers at the first stage and about 3,760 numbers at the second stage for a total of 5,896 numbers. Table 1 shows a set of typical sample design specifications for a two-stage Waksberg-Mitofsky design. I.B. "Listed Hundred Series" (LHS) Frame / One Stage RDD Design The samplingframe of all hundred series with at least one working household number for May 1990 had 1,656,627 records which is only forty percent of the 4,130,800 possible hundred series from the April 1990 Bellcore file. This hundred series frame, which is available commercially, is produced by aggregating all listed telephone numbers to the hundred series level. A count of the number of listed household numbers is added to each hundred series level record. Experience in using this frame has shown that on average between 52 and 55 percent of the numbers in hundred series with at least one working household number are working household numbers. The average number of listed numbers in the hundred series from the May 1990 file was Therefore for a typical sample of 1,500 respondents with a 70 percent response rate and a 52 percent working rate, about 4,121 telephone numbers are needed. The sample can be administered as a simple lisffsbnpfe HXAMFhES^ffiArMitoE^ FOR TWO TYPES OF RDD DESIGNS 2

4 ^^^ otwo^stage^fe:^1 i One Stage irom?eisted rwaksberg^mitofslcy^ ^Hundred Series -Frame;:. Number of Interviews: 1,500 1,500 Response Rate: Eligible Households: 2,143 2,143 Second Stage Working Rate: * Second Stage Numbers: 3,760 4,121 Cluster Size: (Working Household) 8 1 Number of Working Primaries: 470 _ First Stage Working Rate: 0.22 _» Number of Primary Numbers: 2,136 Total Telephone Numbers: 5,896 4,121 'Based on one SRC survey which used this design. working numbers which can be a problem in a short tum-around study. In a two-stage design, the numbers replaced toward the end of a study are more likely to be "no contact" cases and contribute to the nonresponse rate. Since the average number of calls required to assign a final disposition to a telephone number is greater than one, the difference in number of telephone calls required for the two types of designs is even greater than the 1,725 (5,896-4,121) in the example shown in Table 1. I.C. Two Alternative Designs Using the "Listed Hundred Series" Frame The University of Michigan Survey Research Center (SRC) has recently explored two variations of RDD sample designs using this listed hundred series frame: (1) a two-stage design in which the hundred series are selected with probability proportional to the number of listed household numbers and at the second stage, the cluster size is set inversely proportional to the number of listed household numbers, and (2) a one-stage design in which the hundred series with one or more listed numbers are selected with equal probability and a random two-digit number is appended to each selected hundred series. Both of these designs produce equal probability samples of households with telephones. Although the first alternative which selects hundred series with probability proportional to their number of listed numbers may seem more efficient, experience has shown that this is not the case. The working rate for the one-stage equal probability sample is approximately the same as that 3

5 for the second stage of the two-stage probability proportionate to size (PPS) design (Lepkowski, 1988). In two recent national telephone samples conducted by SRC, one selected "PPS-to listed count" at the first stage and one selected with equal probability at the first stage, the percents of working household numbers were 54.7 and 51.9 respectively. Given the additional complexity of the PPS selection and the clustering at the second stage which is required for an epsem sample but which decreases the efficiency of the sample, it would seem that the one-stage equal probability design may be the better of the two alternatives. This RDD design has no clustering within the hundred series and is equivalent to a simple random sample (or a stratified random sample). Both of these alternative designs have advantages over the traditional two-stage Waksberg- Mitofsky RDD design in cost and ease of aa'ministration. The one-stage equal probability design has the additional advantage of no clustering which increases the precision of the sample estimates. The disadvantage of the new designs is that the sampling frame does not guarantee the complete coverage of the telephone household population that is in theory achieved by the Waksberg-Mitofsky design. The undercoverage can come from two sources hundred series with listed numbers that are missed in compiling the frame and hundred series with only unlisted numbers. These possible lapses in coverage were expected to be minor. However it was reassuring to test this assumption empirically by checking the coverage of the new frame against sample control files from the Survey Research Center's Surveys of Consumer Attitudes. II. Design of the Coverage Evaluation The Surveys of Consumer Attitudes (SCA) is one of the oldest and most well known survey programs conducted by the Survey Research Center. It was started by Professor George Katona in the late 1940'sandwas conducted three times a year in the 1950's, quarterly from 1960 through 1977, and monthly since Since 1976, it has been under the direction of Dr. Richard Curtin, and in 1989 the Index of Consumer Expectations was chosen to be part of the Leading Indicator Composite Index which is used for forecasting by the Department of Commerce (Curtin, 1982). The SCA uses a rotating panel design. Each month's sample consists of a new selection of approximately 300 RDD cases and a reinterview of about 200 cases from the prior six month's survey. Because the Survey of Consumer Attitudes uses the Waksberg-Mitofsky design, sample control files from this study provide a standard for coverage against which the coverage of other designs can be compared. In order to test the coverage of the "listed hundred series" frame, sample control files containing all SCA sample telephone numbers for the Ihirty-month period from May 1989 to October 1991 were obtained from the SRC Control Office. In addition, access to the seven primary number samples used in the first stage of the design and to the data files used to compute the Index of Consumer Sentiment was obtained. Figure 1 shows the data processing steps in the coverage analysis. Figure 1: Data Processing for Coverage Evaluation 1. Merge of SCA Sample Control Files with "Listed Hundred Series" File 4

6 Sample Control Files Listed Hundred Series (30 files combined) File Time Period: May Oct May 1990 No. of Records: 24,647 1,656,627 Match ID: Match Criteria: Area Code, Central Office Code, Hundred Series (1st 8 digits of phone number) All records from Sample Control file whether or not there is a match from the Listed Hundred Series Frame. Result: The output file had 24,647 records which are in 1,352 separate hundred series. Of these, 1,300 hundred series were found in the Listed Hundred Series file and 52 were not covered bythefile. At the case level, 23,118 cases were covered by the hundred series file and 1,529 cases were not covered. 2. Merge of Primary Number Files with "Listed Hundred Series" File Primary Number Files Listed Hundred Series (7 Files combined) File Time Period: April April 1991 May 1990 No. of Records: 7,000 1,656,627 Match ID: Area Code, Central Office Code, Hundred Series Match Criteria: All records from Primary Number file whether or not there is a match from the Listed Hundred Series Frame. Result: The output file had 7,000 records of which 2,757 are covered by the listed hundred series file. 3. Merge of SCA Respondents and SCA Control File (with Coverage Flag) SCA Respondent Files SCA Control File with Coverage Flag (30 Monthly Survey Files (30 Sample Control Files combined) combined) Time Period: May October 1991 May October 1991 No. of Records: 8,867 (approx. 300/month) 24,647 (approx. 822/month) Match ID: Year and Month of Survey, Case ID Match Criteria: All interview cases will have a record in the sample control file. There are 8,867 cases in the output file. The merged files described above provided a basis for evaluating several aspects of the coverage of the listed hundred series frame. In order for a case to appear in the SCA sample 5

7 control file, it must have been in a hundred series generated from a working household primary number. The coverage and working rate of both the working and non-working primary numbers which form the first stage of the SCA sample were analyzed. Other analyses included: (1) an investigation of the special characteristics (region, urban versus rural exchange, estimated working rate) of hundred series generated from working primary numbers but not covered by the listed hundred series frame; (2) the characteristics of SCA respondents (age, race, sex, education, consumer attitude) not covered by the listed hundred series frame; and (3) the rate of obsolescence of the frame. All of these analyses use data from the monthly Surveys of Consumer Attitudes. m. Results of Coverage Evaluation m.a. Coverage Analysis of Primary Number Sample In the Waksberg-Mitofsky design, the primary number sample is a true RDD sampling of all possible telephone numbers in the Bellcore system. There were 7,000 primary numbers used for the SCA files during the period May 1989 through October These numbers were matched against the May 1990 listed hundred series file. Table 2 shows the results of this analysis. Of the 7,000 primary numbers, 1,544 or 22% were working household numbers. This is close to the expected 20% working rate for the first stage of the sample. Of the working primary numbers, 1,491/1,546 or 96.6% were covered by the listed hundred series frame. The total including both working and nonworking numbers which is covered by the listed hundred series frame is 2,757/7000 = 39.4% which is close to 1,656,627/4,130,800 or 40.1% - the percent of all possible Bellcore hundred series which is accounted for by the number of hundred series in the listed hundred series frame. An analysis of the listed rates for hundred series from working and nonworking primary numbers was performed. Figures 2 and 3 show these results. For hundred series defined by non-working primaries, 76.7% had no listed numbers and 85.4% had 0-25 listed numbers. Of the hundred series defined by working primary numbers, 3.5 % had no listed numbers 1 and 11.2% had 0-25 listed numbers. The mean number of listed numbers for working primary hundred series was 47.1 (st. dev.: 19.5); for non-working primaries, the mean number of listed numbers in the hundred series was 7.8 (st. dev.: 17.4). The overall mean for all primaries was 16.5 (st. dev.: 24.2). Table 2: Coverage of Working and Non-working SCA Primary Numbers 'These hundred series may have had listed numbers but they were not covered by the hundred series frame. They may have been new hundred series, or hundred series which had only unlisted numbers. For the most part, these hundred series were numbers which had Area Code changes, for example in the Chicago suburbs (from 312 to 708). 6

8 ^ Status ;i^be^?'t :5N6t! -Govered'tlk:) Covered A'-tkTo'tal,, 'j Not Working HH 4,190 1,266 5,456 (row %) (76.8) (23.2) (100.0) Working HH 53 1,491 1,544 (row %) (3.4) (96.6) (100.0) Total 4,243 2,757 7,000 (row %) (60.6) (39.4) (100.0) m.b. Coverage Analysis Using Sample Control File > The merged sample control file was used to get an estimate of the working rate for both 4 covered and not covered hundred series. For covered hundred series, the working rate was 53.5%; for hundred series not covered by the frame, the working rate was 19.6%. Table 3 shows the analysis of working rates for covered and not covered hundred series. ^.? Table 3: Waksberg-Mitofsky Secondary Number Working Rates for Covered and Non-Covered Hundred Series ^Status of SCA -;a ^IMmar^Number^^^v Working Percent Not Covered 1, % Covered 23, % Total 24, % 7

9 Figure 2 DISTRIBUTION OF LISTED NUMBER RATE FOR WORKING PRIMARY NUMBERS (SCA 5/89-11/91) 12 1» 10 m <D w 8 6 m * 2 / 0 O J> \ 0 # & <T # ^ «P # a «Q «* ^ \ N \ v ry nj> ^ ^ V <F <b N <<r A V <?T X v\v,r \'^^ ^ ^' <P «N ^ * <8* & J> Number of listed numbers 8

10 Figure 3 DISTRIBUTION OF LISTED NUMBER RATE FOR NON-WORKING PRIMARY NUMBERS (SCA 5/89-11/91) w O 0) TJ O 5 40 c v \ N \ v "V fj? > w sp V ^ <f? V «*r <P Number of listed numbers 9

11 m.c. Effect on Coverage of Excluding Low Density Listed Hundred Series An analysis was done of the percent of eligible SCA households which would not be covered by the listed hundred series frame if "low density" listed hundred series are excluded from the frame. Figure 4 shows the decrease in coverage which results if hundred series with 1, 2, 3, 10 listed numbers were to be excluded. If hundred series with no listed households are excluded, about 2.7% of eligible households are missed. If those with 3 or fewer listed numbers are excluded, the number of eligible households missed increases only by a tenth of a percent to 2.8%. When hundred series with 10 or fewer listed numbers are excluded, the percent missed is 4.3% m.d. Frame Aging Effects The closer the sample is to the date the listed hundred series frame was constructed, the better the coverage should be. Data for approximately 430 eligible SCA households per month for the thirty month period from May 1989 through October 1991 was available from the SCA control file. Four month moving averages were constructed of the percent of numbers not covered by the listed hundred series frame. The highest coverage was for the four month period immediately preceding the date of the frame, January - April The percent of eligible households not covered during that period was about one percent. By a year after the date of the frame, the period May - August 1991, the coverage had deteriorated to 95.4% - but then went up 96.2% for the last four month period, July through October Some of the variation in coverage rates from one period to another is accounted for by different primary number samples being introduced. The results of this analysis are shown in Figure 5. 10

12 Figure A Percent of Eligible SCA Households Excluded If Low Density Hundred Series Are Not Used-May 1989 to Oct i o 3 o 1-0 H <2 <3 <4 <5 <6 <7 <8 <9 <10 + Eligible household # ii

13 Figure 5 Obselescence of Hundred Series Frame (Four Month Moving Averages) O 1 s > > > > ^ 0^ ^ ^ > y / / S ^ / A <r ^ ^ ^ ^ / * Time Period of SCA Eligible HH Numbers s * Date of frame construction: May

14 m.e. Characteristics of SCA Respondents Not Covered by Listed Hundred Series Frame One important check on the coverage of the listed hundred series frame is whether the households which are not covered have any unusual characteristics which distinguish them from the SCA sample as a whole. The thirty months of SCA data provided a means of comparing the demographic characteristics of respondents in non-covered households with the distributions of the complete sample. There are some significant differences in the distributions on race, age group, and region between the covered and non-covered households. However there was almost no difference in distribution between the total sample and the covered households. The difference in the distribution by region between covered and not-covered respondents is related to the clustering by Area Code and exchange of the uncovered numbers. Table 4 shows a comparison of the demographic and regional characteristics of the complete SCA sample, the covered sample, and the 202 respondents whose phone numbers were not covered by the listed hundred series frame. Another important check on the possible effects of non-coverage of sample households if the new frame were to be used is the computation of the Index of Consumer Sentiment. This computation was done for each of the thirty months with and without the respondents with noncovered telephone numbers. Figure 6 shows that the Index is virtually identical for both groups. 13

15 Table 4 Characteristics of SCA Respondents Not Covered by Listed Hundred Series Frame Sex M F SCA all n = 8867 SCA covered n = 8665 SCA not covered n = 202 Age NA SCA all SCA covered SCA not covered Education NA No HS diploma HS diploma No coll. degree College degree College degree SCA all SCA covered SCA not covered Region West Midwest NE South SCA all SCA covered SCA not covered Race White Black Hispanic Asian Other/NA SCA all SCA covered SCA not covered

16 100 Figure 6 Index Of Consumer Sentiment May 1989-Oct 1991 With and Without the Cases Not Covered by Hundred Series Frame I *«T* *1F **T *9r >9T *9T A w ^QT,«qr ^ ^ ^ A\ <^ A V V- V V ^ kv V ^ v) ^ <^ ""ICS-ALL ICS-COVERED 15

17 IV. Conclusions The small amount of undercoverage in the listed hundred series frame appears to result from new hundred series that may have been added or changed after the creation date of the hundred series frame. Of the 53 hundred series which resulted from working primary numbers but were not covered by the listed hundred series frame, 10 were from suburban Chicago and had the old 312 area code. These primary numbers were from April, July, and December 1989 before the area code was changed to 708. Of the 53 unmatched primary numbers, 19 were from primary number samples selected in 1989 and 9 were from 1991 primary number samples. The coverage of the listed hundred series frame is very acceptable if the frame is current ~ not over six months old. The coverage analyses using SCA samples showed that there are no significant differences in the demographic distributions between the complete SCA sample and the covered respondents. Also, the Index of Consumer Sentiment, the main survey variable of interest, is virtually identical for both groups. Since this survey is conducted in a short time period every month, the one-stage equal probability listed hundred series design would be very practical. It would allow all sample numbers to be released at the beginning of the study so that all numbers can have an adequate number of calls. There should be fewer "no contact" cases and a better response rate. In addition, there is the statistical advantage of having no clustering within hundred series. 16

18 Bibliography Brick, J.M., Kulp, D.W., Starer, A., & Waksberg, J. (1994), "Bias in list-assisted telephone samples." Paper presented at Annual Conference of American Association of Public Opinion Research (AAPOR). Casady, Robert J., & Lepkowski, James M. (1993), "Stratified telephone survey designs," Survey Methodology, Vol. 19, No. 1, pp Curtin, Richard T. (1982), "Indicators of consumer behavior: The University of Michigan Surveys of Consumers," Public Opinion Quarterly, Vol. 46, pp Frankel, Martin R., & Frankel, Lester R. (1977), "Some recent developments in sample survey design," Journal of Marketing Research, Vol. XTV, pp GENESYS Sampling Systems (1994), "Methodology." Internal GENESYS document. Glasser, Gerald J., & Metzger, Gale D. (1972). "Random-digit dialing as a method of telephone sampling," Journal of Marketing Research, Vol. 9, pp Groves, Robert M. (1978), "An empirical comparison of two telephone designs," Journal of Marketing Research, Vol. 15, pp Lepkowski, James M., (1988). "Telephone sampling methods in the United States," in Telephone Survey Methodology, R. Groves et al. (eds.), pp New York: J.W. Wiley & Sons, Inc. A.C. Nielsen Company (1976). Total Telephone Frame, Chicago: A.C. Nielsen Co. brochure. SSI Information Sheet (1991), "Statistical characteristics of random digit telephone samples." Fairfield, CT: Survey Sampling, Inc. Sudman, Seymour (1973), "The uses of telephone directories for survey sampling," Journal of Marketing Research, Vol. 10, pp Waksberg, Joseph (1978), "Sampling methods for random digit dialing," Journal of the American Statistical Association, Vol. 73, No. 361, pp rdd.des 17