A Smulaton Study to Compare Weghtng Methods for Nonresponses n the Natonal Survey of Recent College Graduates Amang Sukash, Donsg Jang, Sonya Vartvaran, Stephen Cohen 2, Fan Zhang 2 Mathematca Polcy Research. Inc., 600 Maryland Ave., SW, Sute 550, Washngton, D.C. 20024 2 Natonal Scence Foundaton, Dvson of Scence Resources Statstcs, 420 Wlson Blvd., Sute 965, Arlngton, VA 22230 Abstract Common methods to adjust samplng weghts to account for survey nonresponse are the weghtng cell technque, response propensty modelng, or a combnaton of both. Each rases several ssues; for examples, whch covarates to use to construct the weghtng cells or to model response propenstes, whether weghts are used n modelng, and whether to weght the adjustment factor. To address these ssues, we used a smulaton based on a data from the Natonal Survey of Recent College Graduates (NSRCG) mantaned by the Natonal Scence Foundaton to evaluate these weghtng methods. In the end, we expect that the weghtng adjustment wll have successfully accounted for possble nonresponse bas. Key Words: survey, nonresponse, weght adjustment, weghtng cell, propensty model.. Introducton.. Background Ths paper focuses on methods to compensate for unt nonresponse through weghtng adjustment n a survey data. A common method to compensate for unt nonresponse s to adjust the samplng weghts. There are several methods ncludng the weghtng cell technque, response propensty modelng, or a combnaton of both. When performng weghtng adjustments one may face wth several ssues as follows: (a) Choosng the method: weghtng cell, or propensty score modelng (b) If weghtng cell method s used, how to construct the cells: based on covarates, or based on propensty scores (c) If response propensty modelng s used, how to estmate the model: desgn based (weghted) fttng, or random sample (unweghted) fttng (d) Optons n calculatng adjustment factor: weghted response rate, unweghted response rate, or ndvdual propensty score. Weghtng cell technque classfes respondents and nonrespondents nto adjustment cells based on auxlary nformaton avalable for both groups so that samples wthn cells are homogeneous n ther response propenstes. Then, wthn each cell, respondents are 374
weghted by the nverse of the response rate n the cell (for example, Lessler and Kalsbeek 992). Response propensty modelng technque regress a bnary response ndcator to the survey on some predctors/covarates observed for both respondents and nonrespondents (Lttle 992). Then, the predcted response propenstes can be used further n weghtng adjustment process as descrbed n the next paragraph. There are, n general, two dfferent ways to form adjustment cells: one s based on a cross-tabulaton of covarates (Oh and Scheuren 983, Lttle 986), and another s based on decles of predcted propensty scores from a response propensty model (Eltnge and Yansaneh 997). As an alternatve to weghtng cell adjustment methods, one may consder usng the nverse of the estmated response probablty (propensty score) for each ndvdual respondent as the weghtng adjustment factor (Czajka 992). When adjustment s done through weghtng cell, the nverse of response rate can be used as the adjustment factor wthn each cell. Ths adjustment factor s then multpled to the respondent s samplng weght to produce the adjusted weght that accounts for the nonrespondents. The response rate can be calculated as a weghted rato of the respondent counts to the sample counts, or as an unweghted rato of the respondent counts to the sample counts (Lttle and Vartvaran 2003). In ths paper, we looked nto these ssues, and through a smulaton study we nvestgated whether they provde better nonresponse adjustments wth regard to correctng nonresponse bas and effcency of the estmates for unt nonresponse n the Natonal Survey of Recent College Graduates (NSRCG), a sample survey wth a two-stage cluster sample desgn..2. The Natonal Survey of Recent College Graduates The NSRCG s part of the Scentsts and Engneers Statstcal Data System (SESTAT) mantaned by the Natonal Scence Foundaton (NSF). SESTAT collects nformaton about employment, educatonal, and demographc characterstcs of scentsts and engneers n the Unted States through three natonal surveys of ths populaton: the Natonal Survey of College Graduates (NSCG), the Natonal Survey of Recent College Graduates (NSRCG), and the Survey of Doctorate Recpents (SDR). The NSRCG covers a populaton of ndvduals who recently obtaned bachelor's or master's degrees n a scence, engneerng or health feld (SEH). The NSRCG provdes nformaton to the educatonal planners wthn the federal government and n academa, as well as the employers n all sectors (educaton, ndustry, and government) to understand and predct trends n employment opportuntes and salares n SEH felds for recent graduates, as ths group of ndvduals has recently made the transton from school to the workplace or attendng graduate school. Dfferent than the NSCG and SDR that use a one-stage stratfed sample desgn and that both can be vewed as longtudnal surveys (sampled unts are followed n subsequent survey rounds wth a supplemental sample of new graduate cohorts), the NSRCG uses a two-stage sample desgn wth a sample from schools at the frst stage and a sample of graduates from selected schools at the second stage. At the frst stage, lsts of graduates who earned bachelor s or master s degrees durng certan academc years are collected from sampled nsttutons. These lsts are then used for constructon of the samplng frame from whch a sample of graduates s selected at the second stage. 375
In ths smulaton study we used a survey data from the 2006 NSRCG, whch sampled 300 schools at the frst stage and then sampled 27,000 graduates at the second stage. The lst of graduates requested from sampled schools covered three academc years: 2002 2003 (AY03), 2003 2004 (AY04), and 2004 2005 (AY05). Stratfcaton of graduates mplemented n the second-stage samplng was based on the followng, yeldng 756 samplng strata: three cohorts by degree year, two degree types (bachelor s and master s), 2 major felds of study, three race/ethncty groups (non- Hspanc whte; non-hspanc Asan ncludng Pacfc Islanders and unknown races; and mnorty, ncludng Hspanc, black, and Amercan Indan), and two gender groups. For more detals on the samplng desgn, see Jang et al. (2006). 2. Weghtng Methods In lookng for a more systematc weghtng adjustment procedure for the NSRCG, we proposed adjustment stages and weghtng methods as follows. Sampled unts can be classfed by ther dsposton statuses: locatng (located vs. not located), survey elgblty (elgblty known vs. elgblty unknown), and response (responded vs. not responded). Snce the reason for nonresponse for each of these three dchotomous statuses s not the same and cases n each status can be characterzed dfferently, ths research recommended three stages of weghtng adjustments to account for these three types of unt nonresponse separately:. Adjustment for unlocated sample persons. 2. Adjustment for sample persons located but wth unknown elgblty status. 3. Adjustment for sample persons located and elgble but dd not complete the survey. These adjustments are carred out sequentally (Iannacchone 2003). In these sequental adjustments, at any partcular step, when the weghted procedure s mplemented, weghts from the prevous step are used. The followng are the fve weghtng methods compared n the smulaton: Method (SM). Weghtng cell s based on cross-tabulaton of sgnfcant man effects wth an unweghted rato-adjustment factor Method 2 (SM2). Weghtng cell s based on decles of predcted propensty values from an unweghted model (not accountng for sample desgn) wth an unweghted rato-adjustment factor Method 3 (SM3). Weghtng cell s based on decles of predcted propensty values from an unweghted model wth a weghted rato-adjustment factor Method 4 (SM4). The nverse of ndvdual predcted propensty values from an unweghted model s the adjustment factor Ths research focuses only on weghtng adjustment for unt nonresponse and excludes any other post-nonresponse adjustments, such as post-stratfcaton, trmmng, or rakng. 376
Method 5 (SM5). The nverse of ndvdual predcted propensty values from a weghted (desgn-based) model s the adjustment factor. In both weghtng cell and model-based response propensty methods, fve samplng varables plus a varable that ndcates whether or not the sampled student s a non-u.s. resdent alen are used as the canddates for weghtng cell constructon and model buldng. These varables are: graduate cohort, degree level, degree feld, race/ethncty, gender, and resdency status. Not all varables may be ncluded n each weghtng stage; a modelng procedure s used to select varables used for weghtng. In method SM, frst, a man-effect model wth these sx predctors s estmated. Then, sgnfcant man effects are dentfed and only these sgnfcant effects are used to construct cross-tabulaton that creates weghtng cells. It turned out that all sx varables were sgnfcant. For small cells (cells wth sample sze less than 20) cell collapsng s carred out before nonresponse adjustment takes place. Cell collapsng process was desgned to be as systematc as possble, where choosng the varables to collapse reled on the order of sgnfcance level of covarates, and was automated to avod subjectve judgment. For weghtng methods based on response propensty model (SM2-SM5), the 2006 NSRCG samples provde nformaton on response propenstes and survey outcomes n the recent college graduate populaton. The propensty score s estmated through a logstc regresson model, where the logt functon of response/nonresponse ndcator varable s regressed wth a set of covarates observed for both respondents and nonrespondents (the sx man effects mentoned above and ther nteracton terms). Frst, CHAID (Ch-square Automatc Interacton Detecton) usng AnswerTree software s used to select potental two- and three-way nteractons (Magdson 993). Then, an ntal model that ncludes all man effects and CHAID dentfed nteracton terms s ftted usng a stepwse varable selecton. The resultant model from ths step s used as the fnal model n the smulaton methods SM2-SM5. Once ths fnal model s determned, response propensty scores are estmated usng ether unweghted fttng whch s a regular random sample modelng method (SM2, SM3, SM4) or weghted fttng (SM5) whch s a desgn based modelng method for clustered data. These estmated response propenstes are then used drectly as the adjustment factors n methods SM4 and SM5, or used to form 0 adjustment cells n methods SM2 and SM3 (Eltnge and Yansaneh 997). To form 0 adjustment cells based on estmated response propenstes, these estmated response propenstes are sorted n ncreasng order. Then, nne cut-off ponts based on the 0th,20th,30th,,90th percentles are estmated and used as the boundares for the weghtng cells. When weghtng cell method s used (SM-SM3), the nonresponse adjustment factor wthn each cell can be calculated as the nverse of unweghted response rates n methods SM and SM2; or as the nverse of weghted response rates n method SM3. 3. Smulaton Settng We used the 2006 NSRCG sample n smulatng data wth nonresponses. However, snce we do not have survey outcomes for the nonrespondents, we use respondents-only data and treat ths as f t were a full sample. We wll call these data sets the full-sample data. When weghted by the 2006 NSRCG fnal analyss weghts, ths set of respondents 377
represents the populaton of graduates for the NSRCG. These data serve as the benchmark when evaluatng smulated data. Gven these data, we generated replcates and smulated unt mssngness wthn each replcate. 3.. Response Rate We smulated the overall graduate response rate, broken down nto three components correspondng to the three stages of nonresponse adjustment. In the 2006 NSRCG, the unweghted rate of each component s as follows: Locaton rate = 76.3% Known-elgblty rate among located = 90.% Completon rate among elgble = 99.2% However, snce the last rate s large and close to 00%, we dd not nclude t n our smulaton. The combned rate for the frst two rates s 76.3% 90.% = 68.7%. In our smulaton we smulated three dfferent (unweghted) graduate response rates as follows: 60%, 68.7%, and 80%. However, the same conclusons hold, and we therefore only focus on the response rate of 68.7% n ths paper. 3.2. Nonresponse Mechansms The three dfferent nonresponse mechansms are consdered n ths smulaton: Mssng Completely at Random (MCAR). Snce MCAR does not depend on covarates ( con toss ), the response probablty/propensty s constant for everyone. Suppose R denotes ndcator of response/nonresponse for graduate ; that s, R = f respondng; R = 0 otherwse. Then, R (for each weght adjustment step) wll be generated as follow: R ~ Bernoull(P) for graduate, where the value of P s gven as follows (note that these response rates are smlar to those exstng n the orgnal NSRCG data): - Non-located adjustment: P = 0.763 - Unknown elgblty adjustment: P = 0.90 Mssng at Random (MAR). In MAR, the mssngness depends on observed values of covarates. That s, the probablty to respond wll dffer from case to case dependng on the values of ther covarates. Three optons were used as the response propenstes: () MAR. Unweghted response rate n weghtng cells constructed based on crossclassfcaton of sgnfcant varables, (2) MAR2. Unweghted response rate n 0 weghtng cells constructed based on the estmated propensty scores, (3) MAR3. Indvdual estmated propensty score calculated through a desgn-based logstc regresson. Ths response rate/probablty under the three MAR schemes above s attached to each case and ths value wll be used as the probablty parameter P to generate response ndcator n a Bernoull random number generator. Not Mssng at Random (NMAR). In the NMAR, the mssngness depends on covarates as well as (unobserved) values of survey outcomes. Suppose we assume 378
that nonrespondents n the NSRCG corresponded to the followng groups (based on survey outcomes WRKG and/or SALARY): Graduates who dd not have a job (WRKG = No ), Graduates who had a job wth hgh ncome (SALARY > 00,000). Frst we assgned response probablty from MAR3 to each ndvdual, and then adjust ths response probablty based on the values of WRKG and SALARY. We would expect to observe a lower response probablty for the cases n the above two groups and a hgher probablty for the rest. That s, for cases wth WRKG = No or WRKG = Yes wth SALARY > 00,000 we assgn an average response probablty of 0.45 n adjustment for locaton (we dd not change the probablty of known/unknown elgblty). Thus, n summary ths paper presents 5 data sets used n the smulaton: MCAR data set, three MAR data sets correspondng to three dfferent response propensty calculatons, and NMAR data set. 3.3. Computer Programmng and the Number of Replcates When choosng the number of replcates, we consdered not only the convergence of the statstcs beng evaluated, but also the length of tme requred to perform the whole process. We performed ths smulaton usng R, a software for statstcal computng and graphcs (www.r-project.org). All calculaton here, ncludng survey estmaton and desgn-based modelng, can be run under R. We ran the smulaton,000 tmes. A run based on a larger number of replcates (2,000) on some of the data sets produced smlar results. Therefore, we decded to use,000 replcates n all 5 data sets. 3.4. Evaluaton To measure bas correcton through weghtng, we compared the survey estmate calculated based on the full-sample survey data usng the fnal survey weghts (the true value ) to the estmate calculated based on the smulated respondents-only usng the nonresponse-adjusted weghts across,000 replcates. The survey estmates/statstcs to be compared are: Total estmates: overall, by degree feld and degree level. Medan salary: by degree feld and degree level. Mean salary: by degree feld and degree level. Proporton of employed: by degree feld and degree level. Proporton of unemployed looked for work: by degree feld and degree level. Suppose θˆ 0 denotes the estmate calculated based on the full sample (no unt mssng), and θˆr denotes the estmate calculated based on each smulated data under method (=,2,3,4,5) from replcate r (r =,, 000). We calculated the percentage of relatve dfference, defned as ˆ θ ˆ θ RELDIFF = r 0 r 00%, () ˆ θ 0 379
so that the magntude of dfference from the true value (the bas) can be measured as a percentage. We nvestgated the plots of θˆr where the horzontal axs represents ndex of ndvdual replcate and the vertcal axs represents the relatve dfferences of statstc beng compared (for example, see Fgure ). Also, the followng mean of dfferences (BIAS) and square root mean square error (RMSE) can be used to measure the magntude of bas and varablty of the estmate from weghtng adjustment for nonresponse: BIAS = 000 000 ˆ ( ˆ r θ 0 ) r= θ, (2) RMSE = 000 000 ˆ ( ˆ r θ0 ) r= 2 θ. (3) In addton, we evaluated the effect of weghtng adjustment to the varance of total estmate by comparng the desgn effects (DEFF) due to weght varaton (Ksh, 992) across,000 replcates. 4. Smulaton Results Dscusson on the smulaton fndngs wll focus on the followng: Response propenstes Bas and effcency of the Total, Medan, Mean and Proporton Desgn effects due to the weghts 4.. Response Propenstes The response propensty s calculated as the response rate wthn each weghtng cell n the weghtng cell adjustment method, or the estmate of propensty score gven the covarates for each case n the model-based adjustment method. Once calculated, ths response propensty s then used as the weghtng adjustment factor, whch s calculated as the recprocal of response propensty. Here, we assessed the varablty of response propenstes produced under the fve smulaton methods by usng a 95 percent confdence nterval of response propenstes p ±.96 s p, where p s the mean response propenstes and s p s the standard devaton of response propenstes for each replcate. In ths applcaton, when mssng data s MCAR, there s lttle varablty n the response propenstes, as expected. 2 Weghtng adjustment may not be a practcal concern n 2 Method SM produces response propenstes wth less varablty than methods SM2, SM3, SM4, and SM5. Ths s to be expected snce wth the covarate adjustment cells n SM we allowed for collapsng and often ended up wth few cells under MCAR; however, n the other methods we forced these weghtng methods to have 0 cells or as many as ndvdual covarate patterns whch lead to more varablty. Though, n the data wth MCAR such dfferences across methods are small. 380
MCAR. When mssng data s MAR (MAR, MAR2, and MAR3), n any adjustment method the varablty of response propenstes becomes large. Weghtng method SM produces response propenstes wth more varablty than other methods do, although these dfferences are not too strkng. Under the NMAR data, n any weghtng methods the response propenstes are moderate larger than those under MCAR data but smaller than those under MAR data. Method SM produces response propenstes wth more varablty than other methods do. When comparng between method SM2 and method SM3 (weghted and unweghted response rates wthn the same adjustment cells), n any mssng data mechansms, n general the unweghted response rates (method SM2) have slghtly smaller varablty than do the weghted response rates (method SM3), though such dfferences are almost neglgble. Also, when comparng between the unweghted (random sample) model and the desgn-based model (method SM4 vs. method SM5), there exst small dfferences between the two, especally on the rght tal of propensty dstrbuton where the desgnbased propensty scores tend to be larger than those based on the unweghted model. 4.2. Nonresponse Bas Correcton a. Total The sum of weghts across all samples represents an estmate of total populaton overall (frame total). When there are nonresponses and the samplng weghts for respondents are adjusted to account for these nonrespondents, the sum of the adjusted weghts s expected to be equal to the frame total. We frst dscuss total estmates overall, and then we dscuss total estmates wthn certan domans. Totals, Overall. Under MCAR, method SM3 produced adjusted weghts that sum exactly to the frame total for each smulaton replcaton; whle n method SM5 the sum of adjusted weghts actually vares a lttle to the frame total across replcates. The estmate of grand total based on these weghts s unbased 3 and very effcent (the estmate has small varablty across replcaton). On the other hand, methods SM, SM2, and SM4 produced adjusted weghts that stll result n an unbased estmate of grand total but are neffcent, as the varablty of the estmates across replcates s large. Under all MAR data (MAR, MAR2, MAR3), methods SM3 and SM5 also produce adjusted weghts that sum to the frame total. Under data wth MAR, there s a tendency for SM2 to produce weghts that underestmate the frame total, though ths underestmaton s mnor (see Fgure ). Recall that n the smulaton MAR response mechansm s generated wth response probablty calculated as the unweghted response rate wthn the weghtng cells based on the covarates. However, SM2 adjusts the weghts usng unweghted response rate calculated wthn the 0 cells (based on response propensty decles). Thus, t s mportant to know whether the (unweghted) response rate under SM2 s an unbased estmate of MAR response probabltes wthn each of 0 cells. If that s the case, then the adjustment under SM2 may produce unbased estmate of totals. However, when that s not the case, then the adjustment under SM2 may produce based estmate of totals. 3 Throughout the dscusson of smulaton result, the term unbased s used to ndcate the expected value or the average across sample replcatons (or repeated samples). 38
Fgure : Estmate of grand total by weghtng methods SM-SM5, for MAR data. Fgure 2: Response probabltes for MAR, MAR2, and average MAR response probabltes wthn 0 cells, 2006 NSRCG Data To nvestgate ths, n Fgure 2 we plotted the unweghted response rates based on the covarate based weghtng cells used to generate MAR presented by the crcles, and the unweghted response rates based on the propensty score based weghtng cells (method SM2) wthn 0 cells presented by the dotted lnes. To check whether unweghted response rate n SM2 s unbased estmate of MAR response probabltes, we calculated 382
the mean of MAR response probabltes wthn each of 0 cells, and plotted them as the sold lnes. One can see that n the frst and the last cells, the unweghted response rate (dotted lne) s based for the sold lne, whch means the adjustment factor under method SM2 s based. As a consequence, when the response propensty correlates wth samplng weght, unweghted response rate used as adjustment factor may produce based estmate. As an example n the NSRCG, Asan and Mnorty are oversampled thus have hgher samplng weghts than Whtes; however, the response rate n Asan and Mnorty s lower than that n Whte. Thus, when nonrespondents are cases wth larger samplng weghts; the estmate s underestmated. Under data wth MAR2 and MAR3, methods SM, SM2, and SM4 produced weghts that sum to the frame total (unbased) but less effcent varance, where the varablty of ths statstc across,000 replcates s less than that under the data wth MCAR. Under the NMAR data, smulaton result shows a dfferent pattern. Yet, methods SM3 and SM5 produced weghts that stll sum to frame total. However, methods SM, SM2, and SM4 resulted n a sum of weghts that clearly underestmates the populaton total. However, weghtng adjustments are not meant to address NMAR data, and so we do not address these results further. Note that n methods SM, SM2, and SM4, where the estmate of grand total s ether downward based or unbased wth large varaton, even though the absolute value of devaton of the sum of weghts from the frame total s not trval (rangng from 23,000 to 36,000 graduates), the percentage of relatve dfferences s small, less than ± 2 percent. For the grand total estmaton, methods SM3 and SM5 are superor than methods SM, SM2, and SM4 wth regards to the bas and varance of the estmate. We note that wth method SM3, usng the weghted response rates wthn cells forces the sum of weghts to equal the frame total, where the total s calculated as the sum of fnal weghts across all respondents n the orgnal data. In ths case t can be shown analytcally that the weghtng adjustment method SM3 always produces adjusted weghts that sum back to the true grand total n the followng paragraphs. However, ths may not apply to subgroup analyses. Let Y and w, respectvely, denote a surveyed varable and samplng weght for ndvdual sampled person. For a grand total estmaton, Y = for all. In addton, let n denotes the sample sze wthn weghtng cell c ( c =, L, C), and w denotes the c cs samplng weght for ndvdual s wthn cell c. The total estmate T ˆ defned as n n C Tˆ = wy = w = w cs = = c= s= n c (4) * s an unbased estmate of the correspondng overall total populaton T. Let n c denotes the number of respondents n weghtng cell c. When the adjustment factor A s c calculated as the nverse of weghted response rate (method SM3), then t can be shown * that the estmate calculated based on respondents only Tˆ (wth the nonresponse adjusted weghts) wll produce the same total estmate Tˆ as follows: 383
* * * c c wcs c c s = T wcs C n C n C n C n * Tˆ = Acswcs = Acs wcs = w * n cs = wcs c c= s= c= s= c= s= c= s= nc s= = ˆ. (5) When an unweghted response rate s used (method SM2), the same result s not guarantee but t can be easly obtaned by post-stratfcaton to the desred frame total. However, n ths study, we have not consdered the fnal stages of weghtng, whch typcally ncludes such post-stratfcaton. To analytcally prove that weghtng adjustment method SM5 also produces unbased wth very small varance grand total estmate s not trval. Ths proof wll nvolve evaluatng mathematcal expectaton of the nverse response propensty. Let R denote a bnary response ndcator, Χ s covarates, and φ denotes the true response propensty for ndvdual. Under a response propensty model and a mssng at random mechansm we assume that the expected value of R gven the covarates Χ and the sample desgn s equal to the true response propensty, or E ( R Χ, w ) = φ (6) m where E m denotes the expected value wth respect to the model. The true response propensty φ s estmated by ˆ φ, and n the weghtng method SM5 the ndvdual nverse of ˆ φ s used as the weghtng adjustment factors. The estmate of total usng the weghts adjusted by nverse of ndvdual response propensty s calculated as * n n ˆ* T = w = R ˆ ˆ w. (7) φ φ = = It can be shown that ths total estmate s an unbased estmate for populaton total. Frst, by mplementng Taylor Seres approach we can show that R E, m Χ w ˆ. (8) φ Then we can mplement a double expectaton ( E and E, respectvely, denote the d m expected value wth respect to the model and the desgn) as follows: n n * ( ˆ R, E d m ) d m, E T Χ w = E we Χ w E w T d ˆ = φ =. (9) NSRCG Totals, Wthn Domans. We calculated survey estmates (total graduates, medan and mean salary, proporton of employed, and proporton of unemployed looked for work) for each doman defned as a cross-classfcaton between two levels of degree (bachelor s and master s) and eght groups of degree felds resultng n 6 domans of analyses. For the MCAR data, when comparng across fve methods n any doman of analyss, any methods of the fve weghtng methods produced unbased total estmate wth a large varance across replcates. The varablty s about the same across fve 384
weghtng methods. Therefore, there s no dfference n the estmates across fve weghtng methods under data wth MCAR as the mssng mechansm. For the MAR and NMAR data, gven a partcular mssng data, n general the fve weghtng methods seem to produce the same pattern. There s no specfc pattern of bas/unbasedness that can be attrbuted to the specfc weghtng method. The bas/unbasedness depends on the doman of analyss and specfc mssng data mechansm we are lookng at. That s, whenever there s a bas n estmaton for a specfc doman of analyss and mssng data mechansm, all fve methods tend to produce a bas estmate wth the same drecton, ether underestmaton or overestmaton. b. Medan Salary In general, for a gven doman of analyss and mssng data mechansm, the estmate of medan salary and the varablty across,000 replcates are about the same across the fve weghtng methods. The estmates are unbased under the MCAR and MAR data. Under the NMAR data, however, the estmate of medan salary s underestmated for all domans. (Recall that our smulaton s set up to randomly exclude large porton of cases n the hgh salary group.) Ths explans that NMAR data cannot be taken care of through any of our weghtng methods SM-SM5. c. Mean Salary The concluson for estmate of mean salary s the same as for medan salary. In general, for a gven doman of analyss and mssng data mechansm, the estmate of mean salary s the same across the fve weghtng methods, wth the same varablty as well. In addton, for all domans of analyss the estmate of mean salary s unbased under the MCAR and MAR mssng data mechansms. Under the NMAR data, the estmate of mean salary s underestmated for all but for a few domans, where the estmate of mean salary s ether unbased or only slghtly underestmated. d. Proporton Employed Across domans, the proporton of employed n the SESTAT populaton s from moderate to large. For example, n the 2006 NSRCG, these numbers range from 67 percent to 93 percent across domans defned by degree level and degree feld. We compared the estmate resultng from the smulaton to the number based on the full sample. Note that n the NMAR data, the unemployed graduates were randomly excluded n each smulated data. Gven a specfc doman of analyss and under a specfc mssng data mechansm, the fve weghtng methods produced an estmate that s about the same. Under MCAR and MAR data the estmates based on all weghtng methods are unbased. In some domans these estmates of proporton have large varance, but n other domans the varablty s small. Snce the varance of proporton s a functon of the proporton tself and sample sze, the varablty that we saw n these plots could possbly due to ether the magntude of proporton or the sample sze, or both. Under the NMAR data, the proporton of employed graduates s overestmated; whch means that the weghtng adjustment was not able to correct nonresponse bas under the NMAR data. e. Proporton Unemployed Lookng for Work The denomnator for ths proporton s unemployed graduates, whch s only 2,278 cases n the 2006 NSRCG data. The estmates of proporton range from 2 to 48 percent across 385
our 6 domans of analyses. These estmates are based on as few as 3 cases n the smallest doman (master s graduates n the Health Scences), and as many as 457 cases n the largest doman (bachelor s graduates n the Socal and Related Scences). Note that n our NMAR data smulaton we randomly excluded a large porton of unemployed graduates and then further broke down ths sample by our 6 domans of analyses. Thus, estmaton of the proporton of unemployed looked for work under NMAR data sets can be consdered as small doman estmaton. Gven a specfc doman of analyss and a specfc mssng data mechansm, the fve weghtng methods produced estmates that are about the same. The plots show that under all mssng data mechansms and for all domans of analyses the estmates based on all weghtng methods are unbased, but wth large varance. Under the NMAR data, the varance s qute large. We conjecture that ths s because of the small doman sample sze rather than the weghtng adjustment method. 4.3. Desgn Effect Due to Weght Varaton When there s nonresponse n the sample, weghtng adjustment may add more varablty wthn the respondent weghts. We assessed possble varance nflaton due to the varablty added to the adjusted weghts that results from usng a partcular weghtng method. Varance nflaton due to weght varaton can be measured through the desgn effect (DEFF) due to weght varaton. The desgn effect computed here s only for the grand total estmate, whch s calculated based on all respondents. We compared the desgn effect calculated from the survey fnal weghts n the full-sample data to the nonresponse-adjusted weghts n the smulated data. Under the MCAR data, there s no ncrease n the desgn effect for any weghtng methods used, though varablty across replcatons s not trval. Under the MAR data, n general the desgn effect was ncreased, except for methods SM4 and SM5 when the data s MAR. Under the NMAR data, t s clear that all weghtng methods nflate the grand total varance as the desgn effect ncreased. 5. Concluson and Dscusson When the mssng data mechansm s MCAR, any method of weghtng adjustment should work. The pont estmates and ther varances show smlar results across weghtng methods for all doman analyses and types of statstcs. Ths mssng data mechansm s not a practcal concern, as a smple rato weghtng adjustment technque may provde satsfactory compensaton for unt nonresponse. On the other hand, when the mssng data mechansm s NMAR, the method of weghtng adjustment for nonresponse may not be successful n correctng nonresponse bas, as expected. Therefore, we present our concluson for mssng data under MAR assumpton as follows. Based on our smulaton nvestgaton usng the 2006 NSRCG, our man conclusons under MAR are as follows: Overall, all fve methods consdered for smulaton are comparable. Weghtng cells based on covarates can lead to ssues wth respect to small cell szes and requres collapsng. Though n practce such collapsng strategy can be ad hoc, n our study smulaton small cells were effectvely handled wth a systematc, 386
automated method of collapsng. As the number of covarates ncludng paradata avalable for use n the covarate cell creaton ncreases, the covarate cell adjustment becomes a less desrable method because of sparseness and the ncreased need to collapse cells, ultmately lmtng the ablty to ncorporate addtonal covarates. Alternatvely, nverse propensty estmate adjustments maxmze the utlzaton of auxlary nformaton and nonresponse bas reducton. However, there s often the concern that ths may cause the most varable weghts, and thus n turn, larger varances of estmates. In ths study, more varable weghts were observed, but the mpacts on survey estmates were mnmal n our NSRCG applcaton due to a few cases havng the largest weghts. In practce, some of ths weght varaton may be dealt wth va weght trmmng. As an alternatve to the covarate-based weghtng class adjustments and the ndvdual nverse propensty estmate adjustments, the hybrd technque propensty cell adjustments are attractve n a sense that ths method makes the response propensty dstrbuton smooth (and thus makng the weght varaton less) whle utlzng all auxlary nformaton. The propensty cell adjustments are able to avod the sparseness covarate cell adjustments face and the varablty that nverse propensty weghts face. Note, however, wth dsproportonate samplng rates, the propensty cell method wth unweghted adjustment factor, for estmates of the grand total, mght over/underestmate because each of the weghtng cells based on propensty values mght cut across many samplng cells wth dfferent samplng rates, and these rates may be related to nonresponse. In ths case, nverse of weghted response rate as adjustment factor s recommended. Ths smulaton dd not consder poststratfcaton, whch would also accomplsh unbased estmates of doman totals. Our smulaton concluded that when the weghtng cells are constructed based on decles of propensty scores, the weghted response rate results n unbased survey estmate wth mnmum varance for our man estmate of nterest, the grand total, though there was no clear pattern for doman totals, means, medans and proportons. Therefore, based on ths study and when consderng the overall total estmate, under the MAR mssngness whch s a common assumpton n practce the weghtng cells method based on groupng the propensty scores wth the adjustment factor calculated as weghted response rate (method SM3), or the weghtng method that uses the adjustment factor calculated as the nverse of ndvdual estmated propensty score wth the desgnbased fttng technque used to estmate model parameters (method SM5) provde a reasonable method for the NSRCG nonresponse adjustment. Because method SM3 uses weghted ratos for the adjustment factors, t produces adjusted weghts that sum back to the frame total. We also note that for the estmaton of grand total, method SM5 provdes total estmates close to the frame total. Ths s mportant to SESTAT snce grand totals are the man estmate of nterest. Further, methods that nvolve modelng can better handle future paradata, and ths s a strong reason to consder usng propensty methods n SESTAT that avod the ssues of cell collapsement. Fnally, knowng that all weghtng methods we examned n the smulaton study perform smlarly for other estmates 387
(doman totals, means, medans and proportons) s mportant as every weghtng method we assess should adjust for nonresponse bas under mssng at random (MAR). The method recommended here may not provde smlar results under dfferent data sets. Therefore, we suggest that the judgment to choose a partcular weghtng-adjustment method should be based on the specfc survey desgn used, as well as a thorough emprcal nvestgaton of the mssng data. Gven the data set, we also recommend that the statstcan who constructs the weghts should nvestgate whether a slght modfcaton of the procedures/methods (for example, weghted vs. unweghted) produces sgnfcantly dfferent results, and whether such dfferences (f pronounced) result n dfferent survey estmates. Readers can refer to Korn and Graubard (999) for dagnostc technques when usng survey weghts. Acknowledgements and Dsclamer Work on ths artcle was supported and funded by the Natonal Scence Foundaton, contract SRS-04-20325. The vews expressed here are those of the authors and not necessarly those of the Natonal Scence Foundaton or Mathematca Polcy Research. References Czajka, J. L., S.M. Hrabayash, R.J.A. Lttle, and D.B. Rubn. Projectng from Advance Data Usng Propensty Modelng: An Applcaton to Income and Tax Statstcs. Journal of Busness & Economc Statstcs, Amercan Statstcal Assocaton, vol. 0, no. 2, Aprl 992, pp. 7-3. Eltnge, J. L., and I.S. Yansaneh. Dagnostcs for Formaton of Nonresponse Adjustment Cells, Wth an Applcaton to Income Nonresponse n the U.S. Consumer Expendture Survey." Survey Methodology, vol. 23, 997, pp. 33-40. Iannacchone, V.G. Sequental Weght Adjustments for Locaton and Cooperaton Propensty for the 995 Natonal Survey of Famly Growth. Journal of Offcal Statstcs, vol. 9, no., 2003, pp. 3-43. Jang D., M. Satake, M.E. Bozylnsky, H. Xu, and X. Ln. Sample Desgn for the 2006 Natonal Survey of Recent College Graduates. Report submtted to Natonal Scence Foundaton. Prnceton, NJ: Mathematca Polcy Research, Inc., March 2006. Ksh L. Weghtng for unequal P. Journal of Offcal Statstcs. Vol 8, 992, pp. 83 200. Korn, E. L., and B. I. Graubard. Analyss of Health Surveys. New York: Wley, 999. Lessler, J.T., and W.D. Kalsbeek. Nonsamplng Error n Surveys. New York: Wley, 992. Lttle, R.J.A. Survey Nonresponse Adjustment for Estmates of Means. Internatonal Statstcal Revew, vol. 54, 986, pp. 38-57. Lttle, R.J.A. Models for Nonresponse n Sample Surveys. Journal of the Amercan Statstcal Assocaton, vol. 77, 992, pp. 237-250. Lttle, R.J., and S. Vartvaran. On weghtng the rates n nonresponse weghts. Statstcs n Medcne, vol. 22, 2003, pp. 589-599. Oh, H.L., and F.J. Scheuren. Weghtng Adjustment for Unt Nonresponse. In Incomplete Data n Sample Surveys, vol. 2, Theory and Bblographes, edted by W.G. Madow, I. Olkn and D.B. Rubn. New York: Academc Press, 983. 388