Random Group Variance Adjustments When Hot Deck Imputation Is Used to Compensate for Nonresponse 1

Transcription

1 Random Group Variance Adjustments When Hot Deck Imputation Is Used to Compensate for Nonresponse 1 Richard A Moore, Jr., U.S. Census Bureau, Washington, DC Abstract The 2002 Survey of Business Owners produced a comprehensive set of estimates of businesses owned by various race, ethnic, and gender groups. Variances were calculated using the method of random groups. At the time of sample selection, chosen units were assigned a random group number, in accordance with the design. Each random group approximated a subsample with the original sample design. When nonresponse is not uniformly or randomly distributed, hot deck imputation can reduce the bias in the universe estimates. However, the responding unit which is used as a donor most likely will not come from the same random group as the nonresponding unit. If the original random groups are used to calculate the variances, the resulting variance estimates may be severely biased. This paper compares three methods for accounting for hot-deck imputation in the estimates of variance: (1) adjusting the weights of the responding units using aggregate control totals; (2) using the original weights and random group numbers for all sampled records, and adjusting the variances using factors derived from the response rates; and (3) donating the random groups of donors to the corresponding nonrespondents, and calculating the variances using the modified random group structure. 1. The 2002 Survey of Business Owners The Survey of Business Owners (SBO) provides statistics that describe the composition of U.S. businesses by gender, Hispanic or Latino origin, and race. For the 2002 SBO, a sampling frame of nearly 23 million businesses was constructed using tax return data supplied by the Internal Revenue Service. After classifying each business by state of operation, industry, and employer/nonemployer status, other administrative information was used to place each business into one of the following possible ownership groups: American Indian, Asian, Black or African American, Hispanic, Non-Hispanic white men, Native Hawaiian and Other Pacific Islander, Publicly owned, Women, and Other. Although the assignment to one these groups is not always correct, businesses do usually show a stronger propensity to respond according to the group in which they are placed than those businesses placed in other groups. These subclassifications based on race, ethnicity and gender were the sampling strata. About 450,000 cases with large receipts, payroll, or employment, were selected into the sample with certainty. A stratified sample of about 2 million of the remaining businesses was then selected. All selected businesses were mailed a survey form that asked for the composition of the business ownership by gender, Hispanic or Latino origin, and race. Sample cases were weighted by the inverse of the probability of selection and ownership tabulations were made based on their responses. 2. The Problem of Nonresponse Response rates may vary substantially by race and Hispanic origin. To address this differential, the 2002 Survey of Business Owners used hot-deck imputation to impute race, Hispanic origin, and gender responses for each nonresponding business. Within each sampling stratum, selected businesses were first sorted by several characteristics in order to group together businesses which should give similar responses to the SBO questions. After sorting, response/nonresponse patterns are not randomly distributed. Response data from the nearest responding business on the list were then donated to each nonrespondent. Respondents could be used as donors up to five times. Flags were set in the SBO tabulation data set to differentiate the total respondent cases from those that did not respond. In addition, the unique identifying control number of each donor respondent was attached to every business receiving its donated responses. There was some partial (or item) nonresponse. Because donor control numbers were not attached to partial respondents, these cases were treated as respondents in all calculations in this 1 This paper is released to inform interested parties of ongoing research and to encourage discussion. Any views expressed on statistical, methodological, technical, or operational issues are those of the author and not necessarily of the U.S. Census Bureau. 262

2 paper. Strata having very few respondents were combined with other strata to form superstrata with sufficient numbers of responding cases to form a donation pool. Cases in the superstrata were sorted, as above. The response donation procedure was exactly the same as the procedure used in uncombined strata. 3. Computing the Estimate and Variance for a Publication Cell Estimates were calculated after the donor imputation was completed. To determine the number of Black-owned architectural firms in Alabama, for example, the sample weights for all architectural businesses in Alabama that responded or were imputed as Black-owned were summed. Computing estimates was the easy part. A more complicated task was devising a method for accurately estimating the variance of this estimate. The Survey of Business Owners has traditionally used the Random Group method (with 10 random groups) for producing the estimates. At the time of sampling, each unit selected with probability less than 1.00 was assigned to 1 of 10 random groups in a modular fashion. (Cases selected with certainty were assigned to a separate group.) This procedure forms 10 subsamples and a certainty component from which we can produce 10 independent estimates for each cell. We can then use the variance of the random group estimates to estimate the variance of the estimate from the full sample. If we had 100 percent response in the cell, we would be done. Some of the cases, however, have been imputed. Is there a way to adjust the random group variance estimate to more accurately reflect the effect of the imputation? The remainder of this paper discusses and compares two alternatives to the method used in previous SBOs. 4. Standard Method (WTD ADJ): Weight Adjustment of Noncertainty Respondents The most straightforward method (and the one used in past SBOs) for obtaining variances is to adjust the sampling weights of the responding cases to compensate for nonresponse and then compute the variances using these adjusted weights. After the hot-deck imputation procedure, we can calculate the following four estimates for each tabulation cell (race/gender/ethnicity by state by industry): (1) The firm count estimate using all cases inflated by the original sampling weight (FT), (2) the firm count estimate using only respondent cases inflated by the original sampling weight (FR), (3) the aggregate receipts total using all cases with receipts of each unit inflated by its original sampling weight (ST), and (4) the aggregate receipts total using only respondent cases with receipts of each unit inflated by its original sampling weight (SR). We can then calibrate the weights of the respondents by inflating the original sampling weight of the respondents by the factor of FT/FR. Summing the weight-adjusted respondent cases gives the same estimate of firm count as summing both the reported and imputed cases using the original sampling weights. Similarly, we can calibrate the receipts estimates by multiplying the original sampling weights by ST/SR. Summing the receipts of the respondent cases weighted by these receipts-adjusted weights gives the same estimate of receipts as the estimate obtained by summing the receipts of both the reported and imputed cases using the original sample weights. Using only the weight-adjusted respondent records and the original random group assignment, we can obtain an estimate for the variance of the firm count and receipts estimates for each publication cell. This method was used in past iterations of the SBO. It can be difficult to implement, because the adjustment factors must be calculated for every cell. Special tabulations of the SBO data are frequently requested, and the calibration method requires the calculation of new adjustment factors for each new tabulation, which can be cumbersome and laborious. A simpler alternative would be desirable. 5. Alternative Method 1 (OH-SCHEUREN): Scheuren-Oh Adjustment Oh and Scheuren (1983) studied the problem of variance estimation using hot-deck imputation, for a sample of size n with r respondents and m nonrespondents. They selected m of the r respondent cases with replacement and donated their responses to the m nonrespondents. They then used the Jack-Knife Method on the set of n = m + r observations to calculate a variance, using both imputed and respondent cases. They showed that, when there are a large number of respondents in the sample, the calculated variance understates the true variance (after hot deck imputation) by a factor of F = (n/r + m/n). SBO prefers the use of random group replication (instead of another replication method, such as jack-knife), because the random group method is much easier to use. Unfortunately, we could not find an extension of this work for random group estimation, but since the jack-knife is a close variant of the random group method, we decided to apply the Oh-Scheuren factor to our full sample random group variances and compare it to the variance obtained by the standard method. 263

3 This procedure is simpler to implement than the standard method. Although the Oh-Scheuren factors do change with every cell, and have to be recalculated to correspond with each requested tabulation, the calculation is much simpler. First, we don t have to exclude the imputed cases. Second, we only have to count the number of imputed and responding cases in each cell, as opposed to computing separate nonresponse adjustment factors for each variable (firm counts and receipts as discussed here, plus employment and payroll, which are also published). 6. Alternative Method 2 (DONOR): Donate the Random Group Number In Addition to the Responses Suppose that a respondent has a sampling weight of 5. This means that the business represents 5 firms in the universe. Additionally, suppose that the respondent s answers are donated to a nonrespondent with a sampling weight of 3. The responding unit now represents 8 units. If we only used respondents and decided to adjust the respondent weight to 8, 8 units would be tabulated in the same random group as the respondent and 0 units in the random group of the nonrespondent. The process would be equivalent to donating the random group number of the respondent to the nonrespondent. We will use this technique to calculate a variance for each estimate and compare it to the corresponding variance obtained using the standard method. This procedure is the easiest to implement of the three methods considered. The question we examined was whether this method gives a reasonably accurate estimate of the variance for most cells. 7. Defining the Cells to Be Used In order to get meaningful estimates for the purpose of this analysis, we restricted the analysis to publication cells (demographic class by state of operation by industry) which satisfied two conditions. First, the cell had to contain at least 10 respondents that were tabulated in the specified demographic class. Second, the ratio of the number of responses to the total number of cases tabulated in the cell had to be at least Each publication cell meeting these criteria was placed into one of 120 analysis groups, based on the following three characteristics: (1) four demographic classes -- Asianowned, Black-owned, Hispanic-owned, and Womenowned; (2) five response rate classes, based on the number which actually responded and which were tabbed with positive weight divided by the number of cases tabbed with positive weight in a publication cell to 55 percent; 55 to 65; 65 to 75; 75 to 85; and 85 to 95; and (3) six classes based on the number of cases imputed in each cell to 3 imputes; 4 to 5; 6 to 9; 10 to 19; 20 to 49; and 50 or more imputed cases. We then produced statistics for each publication cell and analyzed these statistics. In order to avoid results skewed by small sample sizes, all analysis groups that contained less than 25 publication cells were dropped from the analysis at this point. After all elimination, 4,711 publication cells in 60 analysis groups remained. Because we observed no substantial differences by demographic group, the data and analysis below are presented and discussed by response rate and imputation class groupings. 8. Analysis For each of the publication cells, we calculated variances using each of the three methods. We then formed the two ratios: the Oh-Scheuren method variance to the weightadjusted variance, and the Donor method variance to the weight-adjusted variance. We calculated the mean and standard deviation of the distributions of these ratios in each analysis group. Using the Central Limit Theorem, we estimated the standard error of each mean to be the standard deviation of the distribution divided by the square root of the number of publication cells in the analysis group. Table 1 of Attachment A shows the results of this analysis for the variances of firm count estimates. It provides information for the six ranges for the number of imputed cases and the five response rate ranges as well as the overall results using all 4,711 cells. The overall mean ratio of the firm count variance obtained from the Oh-Scheuren Method was 1.094, while the mean ratio obtained from the Random Group Donor Method was slightly higher at Both columns labeled R 1" are marked with a Yes. This means that a two-tailed 90-percent t- test indicates that each of these ratios are significantly different than When observed one at a time, each method gives statistically significantly higher variances than the Weight Adjusted for Noncertainty Respondents Method. From Table 1 we also notice that for all of the six ranges of the number of imputed cases, the Oh-Scheuren Method gives ratios greater than Within each analysis class, we can say with 90 percent confidence that any single ratio is significantly different from The data gives the appearance that this method gives ratios that will converge asymptotically to a value of 1.05 or lower as the number of imputed cases per cell increases. The table gives both the means of the ratios of variances and the corresponding standard errors. It is possible that some observations appear in more than one publication cell (e.g., Asian and Hispanic), so there may be some dependence between analysis class estimates, but the sum of variances provides an upper bound for the variance of the differences. Tests 264

4 for statistical significance of the trends between ranges can be computed. The opposite trend occurs for the Random Group Donor Method. The average ratios generally increase as the number of imputed cases increase. For cells in the ranges with less than 9 imputed cases the average ratios vary from to They increase monotonically thereafter, as the ranges for number of imputed cases increases. The 50 or more imputed case range shows an average ratio of Variances for each estimate are provided, so that the reader can test any comparisons. Table 2 of Attachment A shows the analogous information for the analysis of the variances of the receipts estimates. We see a much more substantial difference between the Oh-Scheuren and the Random Group Donor methods. On average, the Oh-Scheuren Method produces a 48.7 percent increase in the variance of the receipts estimate, while the Random Group Donor Method decreases the variance by 7.4 percent. Tested at 90 percent confidence independently, each of 6 ranges of number of imputed cases and each of five response rate ranges produce average ratios that are significantly higher than The Random Group Donor Method consistently produces average ratios in the 0.90 to 1.00 range. The Random Group Donor Method consistently produces average ratios in the 0.90 to 1.00 range. use of the Oh-Scheuren Method. For firm count estimates with the number of imputed cases in a cell less than 10, the Random Group Donor Method appears to be superior. The ratios for cells with a small number of imputed cases are closer to If obtaining estimates for receipts (or another similar variable such as employment or payroll) is the main purpose of the tabulation, we recommend using the Random Group Donor Method. From the information in Table 2, this method clearly out-performs the Oh- Scheuren Method for all response rate ranges and all ranges based on the number of cases imputed. 10. References Oh, H. L. & Scheuren F. J. (1983). Weighting Adjustment for Unit Nonresponse. In Incomplete Data in Sample Surveys, 2, Edited by W. G. Madow, I. Olkin,and D.B. Rubin, pp New York: Academic Press. Rao, J. N. K. & Shao J. (1992). Jackknife Variance Estimation with Survey Data Under Hot Deck Imputation. Biometrika, 79, No. 4 (Dec. 1992), pp Shao, Jun (2006). Notes from a short course on Analysis of Data with Missing Values. Because outliers can highly skew means and variances, the table also provides columns containing the percentage of cells where the ratio, R, of the two variance methods is within 45 percent of These are labeled, 0.55 < R < 1.45". For the variances of the firm count estimate, 74 percent of the 4,711 publication cells had a ratio in this range for the Random Group Donor Method, while only 71 percent fell in this range for the Oh- Scheuren Method. For variances of receipts estimates, 73 percent of the ratios fell in this range using the Random Group Method, compared to 67 percent with the Oh- Scheuren Method. 9. Recommendations Based on the results in Tables 1 and 2, we make the following recommendations as a possible alternative to the standard Weight Adjustment for Noncertainty Respondents Method for subsequent tabulations of the Survey of Business Owners. If the user is mainly interested in the accuracy of the firm count estimates and if many of the key cells have 20 or more cases imputed as a race, ethnicity, or gender to be tabulated, then we recommend 265

5 Attachment 1 Page 1 of 2 Table 1. Comparison of Variance Ratios --- Firm Count Estimates The Oh-Scheuren and Random Group Donor Methods With the Weighted Adjustment for Noncertainty Respondent Method # Cells Ratio Oh-Scheuren vs. Wt. Adjustment Ratio RG Donor vs. Wt. Adjustment Total 4, Yes 71% Yes 74% # Imputed 1 to Yes 90% No 93% 4 to Yes 82% Yes 88% 6 to Yes 76% Yes 80% 10 to Yes 68% Yes 73% 20 to Yes 68% Yes 70% 50 or more Yes 65% Yes 62% Response Rate Range 45% to 55% Yes 52% Yes 63% 55% to 65% 1, Yes 66% Yes 66% 65% to 75% 1, Yes 73% Yes 76% 75% to 85% Yes 84% Yes 86% 85% to 95% Yes 92% No 87% 266

6 Attachment 1 Page 2 of 2 Table 2. Comparison of Variance Ratios --- Receipts Estimates The Oh-Scheuren and Random Group Donor Methods With the Weighted Adjustment for Noncertainty Respondent Method Oh-Scheuren vs. Wt. Adjustment RG Donor vs. Wt. Adjustment # Cells Ratio Ratio Total 4, Yes 67% Yes 73% # Imputed 1 to Yes 85% No 87% 4 to Yes 75% Yes 85% 6 to Yes 65% No 77% 10 to Yes 64% Yes 71% 20 to Yes 68% Yes 69% 50 or more Yes 61% Yes 70% Response Rate Range 45% to 55% Yes 51% Yes 59% 55% to 65% 1, Yes 61% Yes 66% 65% to 75% 1, Yes 68% Yes 76% 75% to 85% Yes 80% Yes 86% 85% to 95% Yes 86% No 87% 267