Methodology for the Princeton RDD Survey August, 2008

Transcription

1 Methodology for the Prnceton RDD Survey August, 2008 Background In the summer of 2008, Westat conducted a natonal random dgt dal (RDD) survey on behalf of Prnceton Unversty. Major topcs covered n the survey ncluded worker perceptons of the off-shoreablty of ther jobs, occupatonal lcensng, and adult lfetme work experence. Ths document dscusses the methods used by Westat to conduct the survey. Questonnare Development Prnceton provded Westat wth a draft of the desred questonnare at the start of the project. Prnceton and Westat staff collaborated n fnalzng the queston order and wordng. A number of the questons had been developed and tested n earler work by Prnceton and under pror task order contracts wth Westat. Several questons regardng the respondent s employer, job actvtes, and demographcs, were taken from the Current Populaton Survey. Westat programmed the questonnare and skp patterns for admnstraton by Computer Asssted Telephone Intervewng (CATI), n both Englsh and Spansh. A few days before data collecton, Westat staff pretested the nstrument wth several volunteer respondents. Ths pretest suggested several addtonal revsons for the questonnare, ncludng shortenng t so as to acheve the desred average ntervew length of 15 mnutes. The CATI ntervew began wth a short screener nstrument desgned by Westat n order to determne the elgblty of household adults for the extended ntervew (.e., the study questonnare). In order to be elgble for the study, persons had to be adults n the labor force for ths project the labor force was defned as persons who are ether a) currently workng at a job for pay or proft, or b) currently lookng for work and have worked at a job n the past. Households n whch no adults are currently n the labor force were not elgble for the study. If a household contaned more than one adult n the labor force, one was randomly selected by the CATI program for partcpaton n the extended ntervew. When the person chosen for the extended ntervew was someone other than the screener respondent, age and work force status was confrmed wth these persons before contnung wth the extended ntervew. Samplng and Data Collecton Standard lst-asssted random dgt dalng (RDD) technques were used to dentfy and select respondents for the survey. Under the lst-asssted RDD approach, a samplng frame of 100-number banks s created from all known area codes and telephone exchanges n the Unted States, where a 100-number bank s defned by the three-dgt

2 area code, three-dgt exchange, and next two dgts of the telephone number. The frame s then matched aganst publshed resdental telephone drectores, and a count of the number of lsted resdental numbers was determned for each 100-number bank.. As s the practce n most lst-asssted RDD surveys conducted by Westat, the workng banks (..e., those remanng n the samplng frame) were those wth 1 or more lsted resdental phone numbers. Whle we could have ganed operatonal effcency by lmtng the workng banks to those wth a greater number of lsted resdental phone numbers, dong so would ncrease potental undercoverage snce households n the excluded banks would have no chance of beng selected for the survey. An equal probablty sample of telephone numbers was then drawn from the frame of workng banks descrbed above. Relatvely nexpensve automated procedures were ntally be appled to the selected numbers to remove known nonresdental and nonworkng numbers, leavng for screenng and subsequent processng those numbers that are more lkely to be actve resdental numbers. An ntal sample of 71,000 telephone numbers was drawn, along wth a reserve sample of an extra 20,000 numbers. Durng data collecton the reserve sample was released to be worked, as t became clear that ths was necessary to acheve the desred target of 2,500 completed ntervews. Data collecton lasted approxmately sx weeks, begnnng on June 5 th, 2008 and endng on July 20 th. Westat traned approxmately 100 ntervewers for the project, but as data collecton proceeded less productve ntervewers were dropped from the study. We completed 2,513 ntervews. Table 1 presents detals on the outcomes of our data collecton and the response rates. 2

3 Table 1. Outcomes of Data Collecton Estmated rate Number n sample Telephone numbers (lst-asssted method) 91,000 Numbers determned to be nonresdental va busness purge 3.7% 3,409 Numbers avalable for telephone screenng 87,591 Fnalzed cases for whch resdental status s undetermned 15.3% 13,437 Rng no answer, no contact 74.1% 9,958 Answerng machne 25.8% 3,462 Other nonresponse 0.1% 17 Fnalzed cases for whch resdental status s determned 84.7% 74,154 Determned to be nonresdental 75.0% 55,634 Determned to be resdental (households) 25.0% 18,520 Households completng screener 33.3% 6,165 Inelgble households 11.3% 2,086 Nonrespondng households 66.7% 12,355 Households avalable for extended ntervewng 4,079 Completed extended ntervews 61.6% 2,513 Elgble nonrespondents to extended ntervew 26.2% 1,067 Inelgble 1.9% 76 Unknown elgblty 10.4% 423 Screener Response Rate 28.2% Condtonal Extended Intervew Response Rate 63.5% Fnal Overall Response Rate 17.9% Weghtng the Survey Data The RDD survey was weghted to compensate for varable selecton probabltes, dfferental response rates, and possble undercoverage of the samplng frame. The weghtng process was carred out n fve steps as descrbed below. Base weghts for sample telephone numbers. Snce the samplng process started wth the selecton of telephone numbers, a base weght was frst attached to each selected telephone number, followed by weghtng of subsequent samplng unts (.e., households 3

4 and persons). The base weght for a sampled telephone number s the nverse of the selecton probablty of the telephone number. Resdental status adjustment. Only resdental telephone numbers are of nterest and they were dentfed through screenng. Snce t was not always possble to determne resdental status, the weght of unknown cases (telephone numbers wth unknown resdental status) was dstrbuted so as to preserve the dstrbuton of the cases for whch resdental status was known. Ths adjustment was carred out as a nonresponse adjustment wthn a number of adjustment cells defned by the Census Regon, metropoltan statstcal area (MSA) status, and mnorty status. All of these varables are exchange-level varables avalable n the RDD samplng frame. A cross-classfcaton of these varables yelded a total of 16 adjustment cells. 4 Census Regons: (1) Northeast, (2) Mdwest, (3) South, and (4) West; 2 MSA statuses: (1) MSA, and (2) nonmsa; 2 mnorty-status categores: (1) low mnorty and (2) hgh mnorty, where the low mnorty group s defned as those telephone numbers belongng to the exchanges wth less than 60 percent whte populaton. Elgblty status adjustment. There were 18,520 resdental telephone numbers screened to be resdental. Among them, 4,079 households had elgble persons, and 2,086 dd not have an elgble person - these were households that had no adults n the labor force at the tme of ntervew. As antcpated, for some resdental telephone numbers (12,355), t was not possble to ascertan elgblty status. Therefore, an elgblty status adjustment was performed usng new adjustment cells defned by the Census Regon, MSA status, and medan ncome of the telephone exchange. Fve medan ncome categores were defned, and there were altogether 50 adjustment cells. The household screenng questonnare determned how many elgble adults lved n the household. There are three categores of nterest. The frst category ncludes households n whch one or more resdents are elgble for the extended ntervew. The second category s households n whch no resdent s elgble. The thrd category ncludes households n whch elgblty s not known. It s possble that some of the unknown cases were elgble. Therefore, the weght of the elgble cases was adjusted upwards wthn each of the adjustment cells defned above, Adjustment for multple telephone households and sample person base weght. A household wth multple telephones had a hgher probablty to be selected n proporton to the number of telephones, so ths should be reflected n weghtng. Let k be the number of telephone numbers assocated wth household. The elgblty adjusted weght obtaned n step 3 was then adjusted by dvdng t by k. However, k was capped at 3 (.e., k 3 for households wth 4 or more telephones n use) to avod too much varablty n weghts. 4

5 Only one person was selected for an extended ntervew from all elgble persons n a partcpatng household. If m s the number of elgble persons n household, then the probablty of selectng an elgble person from the household s 1 m. The weght attached to the sample person from household s then gven by: w m ( pre ) ( re ) w k The value of m was capped at 4 to avod excessve varablty n weghts. There were 4,079 households nvolved n ths step of weghtng. Person-level nonresponse adjustment. Ths adjustment compensated for nonresponse resultng from sample persons who agreed to partcpate n the study but for some reason dd not complete the extended ntervew. Furthermore, some people were found nelgble durng the extended ntervew, and also there were some nonrespondents whose elgblty could not be confrmed. To reflect these response statuses, a smlar adjustment done n the frst step descrbed above was performed usng the weghtng cell approach. The weghtng cells were frst defned by the followng varables: Census Regon (1, 2, 3, 4); MSA status (MSA, non-msa); Medan ncome level of exchange (5 categores); Employment status (employed, unemployed). Note that the frst three varables are exchange-level varables, whereas the last varable s a person-level varable avalable from the screenng ntervew. Sx fnal cells were determned by a CHAID analyss as shown n table 2. CHAID stands for Chsquared Automatc Interacton Detector, whch performs cell collapsng accordng to user crtera. We specfed the mnmum cell sze to be 30 and cells were merged f the dfference n cell response rates was not sgnfcant at the 5 percent sgnfcance level, based on the Ch-squared test. Note that MSA status dd not enter n the formaton of the fnal cells. 5

6 Table 2. The Fnal Adjustment Cells Determned by CHAID Cell Number Regon Medan Income Employment Status Counts 1 1 and 3 all Employed 1, and 4 all Employed 1,776 3 all 1 and 2 Unemployed all 3 Unemployed 73 5 all 4 Unemployed 68 6 all 5 Unemployed 44 Total 4,079 Calbraton Adjustments. Fnally, the nonresponse-adjusted person weghts were calbrated to known populaton control totals obtaned from the June 2008 Current Populaton Survey (CPS) data. Ths was done usng a rakng algorthm. The margnal dmensons of the rakng procedure were defned by the followng varables (CPS source varable names are gven n parentheses). Sex (PESEX); Age group (PRTAGE): 18-35, 36-49, 50+; Educatonal attanment (PREDUCA4): Some hgh school, some college; Census Regon (GEREG); MSA status (GTMETSTA): MSA, non-msa; Race/ethncty (PTDTRACE and PEHSPNON): recoded nto two groups, non-hspanc whte only and all others; Employment status (PRFTLF and PREMPNOT): full tme employed/part tme employed/unemployed; Type of Employer (PRCOWPG): prvate / non-prvate. Some of these varables were combned, and altogether 5 rakng dmensons were defned as shown n table 3. 6

7 Table 3. Fve Rakng Dmensons Rakng Dmenson Number of Categores Sex by Age group 6 Educaton level 2 MSA by Regon 8 Race/ethncty by Employment status 6 Employment type 2 We defned the correspondng varables usng the survey data. However, some of the varables nvolved were survey questons that had resulted n some degree of mssng values. Before runnng the rakng algorthm, the mssng values were mputed usng hotdeck estmaton procedures, whch use smlar values from the complete records n the data. The resultng weghts of the rakng procedure are the fnal weghts to be used n analyss. Industry, Occupaton, and Off-shoreablty Codng Tranng of coders. Westat conducted a one-day tranng sesson wth 4 coders for ths project. The tranng covered how to use the Standard Occupatonal Classfcaton Manual 2000 (SOC) for codng occupatons, the North Amercan Industry Classfcaton System 2002 (NAICS) for codng ndustres, the assgnng of an offshoreablty score to the occupaton, and how to use the Mcrosoft Access system developed for the project. Much of the tranng was devoted to classfcaton prncpals for SOC and NAICS codng. For off-shoreablty, the codng scheme and job characterstcs that need to be consdered n applyng a score were dscussed n-depth. A summary of these job characterstcs was ncluded n the tranng materals and also on a separate reference sheet that coders could refer to throughout the project. The remanng porton of the tranng conssted of exercses where coders examned raw data responses from the questons of nterest from the survey nstrument and assgned SOC, offshoreablty, and NAICS codes to them. The codes and the bass for assgnng these codes were then dscussed as a group. Access tool for codng. Usng Mcrosoft Access, Westat desgned a tool for use by the coders n readng the raw data responses and assgn a NAICS code, a SOC code, and an offshoreablty code from the questons of nterest from the CATI ntervew. The responses to questons Q4a or Q4b, Q5, and Q5a were used for assgnng the NAICS code. The responses to questons Q6a or Q6b, Q7, and Q8 were used for assgnng the SOC code and the offshoreablty code. However, responses to all of the questons lsted above were avalable to the coders no matter whch code they were assgnng, because 7

8 nformaton relevant for assgnng a code mght be found n any of the responses. Once loggng nto the system, the tool presented sx columns showng the raw data responses from the 6 questons noted above. For assgnng the SOC and offshoreablty codes the responses to questons Q4a or Q4b, Q5, and Q5a were presented to the left and the responses to questons Q6a or Q6b, Q7, and Q8 were off to the rght. For assgnng the NAICS code the order of the responses was reversed wth the responses to questons Q6a or Q6b, Q7, and Q8 presented on the left and the responses to questons Q4a or Q4b, Q5, and Q5a were off to the rght. If the coder was assgnng a SOC and off-shoreablty code, two addtonal columns conssted of tools for enterng these values. The box for codng the offshoreablty code was a drop-down box that lsted the sx offshoreablty codng optons (the 5-pont off-shoreablty scale plus a 0 code opton that meant nsuffcent nformaton or too vague for assgnng a code) and ther defntons. If the coder was assgnng the NAICS code, one addtonal box was provded for entry of the NAICS code. A fnal column for all NAICS, SOC and off-shoreablty codng presented a box for coders to enter a flag to ndcate supervsor revew or assstance s needed, or to enter an update flag for codng correctons. The off-shoreablty scores that had prevously been assgned by Westat to SOC categores were also ntegrated nto the codng too. For cases where a coder s assgned off-shoreablty score dd not match the score assgned to the SOC category, a report was generated for revew of the case by the Westat supervsor. Codng procedures. Each of the four coders receved a randomly assgned onefourth of the completed ntervews. The coders examned the raw responses presented by the Access tool and assgned the SOC, off-shoreablty, and NAICS codes. The SOC and off-shoreablty codng were completed as a unt for each case and the coders were nstructed to assgn both codes before proceedng to the next case. The NAICS codng was done separately from the SOC and off-shoreablty codng, although all responses from the questons of nterest from the survey nstrument were avalable for both types of codng, SOC and NAICS. The supervsor 100% verfed the codng on an on-gong bass throughout the codng process, provdng close oversght and ndvdual feedback about the coders accuracy of assgnng the NAICS, SOC, and off-shoreablty codes. Durng the codng perod the supervsor remaned readly avalable to answer questons and provde gudance as needed. Verfcaton of Reported Occupatonal Lcenses Westat staff attempted to verfy a randomly selected 1/12 th of all reported occupatonal lcenses. The frst step n ths process nvolved askng for the full name of the respondent, along wth the state (or cty or county) n whch the lcense was applcable. We collected up to three states/ctes/countes n whch the lcense was sad to applcable. However, almost one-quarter of the respondents who reported an occupatonal lcense refused to provde ther name to us we were unable to proceed further n the verfcaton of these cases. The next step nvolved searchng for a sutable database on the nternet by whch we could verfy that the respondent currently holds a 8

9 vald lcense. Ths generally meant examnng state government webstes, many (f not all) of whch provde readly accessble nternet-based tools for consumers and employers to verfy that a gven ndvdual has a professonal lcense. However, very few ctes or countes appear to provde such tools. We also used nternet search tools (google.com) extensvely n an effort to fnd relevant nformaton. For example, we performed searches on the ndvdual name, parng t wth addtonal nformaton such as the occupaton, the state or cty, and the term lcense. 9