Complex Survey Sample Design in IRS' Multi-objective Taxpayer Compliance Burden Studies

Transcription

1 Complex Survey Sample Design in IRS' Multi-objective Taxpayer Compliance Burden Studies Jon Guyton Wei Liu Micael Sebastiani Internal Revenue Service, Office of Researc, Analysis & Statistics 1111 Constitution Ave. NW, Wasington, DC Abstract Te Internal Revenue Service periodically conducts complex surveys to measure te prefiling and filing burden of individual taxpayers in response to te requirements of te U.S. federal tax system. Te sample design for te survey needs to balance tree major objectives. Te first is to ensure a sufficient number of respondents witin and across strata to meet te needs of te modeling of compliance burden. Te second is tat it must be efficient so tat te estimates are reliable. Te tird is to facilitate te comparisons between te current year study and te previous study. An iterative procedure for a stratified random sample design is proposed to searc for te optimal sample allocation. Te proposed procedure utilizes te optimality in te Neyman allocation metod, and incorporates te minimum sample size requirements for modeling and different nonresponse rates across strata. Our adjustment on te Neyman allocation causes loss of efficiency for descriptive analysis, but suc loss of efficiency is minimized so tat te estimates still meet te precision requirements. Furtermore, suc loss is well compensated by te gains in modeling and analytical capabilities. Key words: Complex survey sample design, Neyman allocation, Nonresponse rate, Sample size, Optimality, Sensitivity analysis 1. Introduction Te Internal Revenue Service periodically conducts complex surveys to measure te prefiling and filing burden of individual taxpayers in response to te requirements of te U.S. federal tax system. One of te callenges of tis type of researc is incorporating wat one learns from one study in te design of te subsequent study wile maintaining comparability between te studies. Our sample design specifications are developed to balance tree issues. Te first and most important is to ensure tat tere are a sufficient number of cases to meet te needs of te modeling tool to identify te determinants of burden, bot witin and across stratum. Te second is tat it must be efficient in te way te sample is distributed so tat estimates from te sample are reliable (i.e., meet confidence interval range requirements). Te tird is tat te design sould facilitate te comparisons between te studies. Tis paper discusses our approac to te sample design for Individual Taxpayer Burden (ITB) TY survey embedding te previous TY2007 survey sample design for comparability. 1 Te views expressed are tose of te autors and not te official positions of te Internal Revenue Service. 2 TY2010 refers to tax year We will use tis notation trougout te paper. 4675

2 An iterative procedure for a stratified random sample design is proposed to searc for te optimal sample allocation. Te proposed procedure utilizes te optimality in te Neyman allocation metod, and incorporates te minimum sample size requirements for modeling and different nonresponse rates across strata. Our adjustment on te Neyman allocation causes loss of efficiency for descriptive analysis, but suc loss of efficiency is minimized so tat te estimates still meet te precision requirements. Furtermore, suc loss is well compensated by te gains in modeling and analytical capabilities. To make te study of ITB TY2010 survey comparable wit te one of TY2007 survey, we continue to use te same design variable, total monetized burden, te same stratified random sampling approac, and te same stratification variables tat were used in te TY 2007 study. 3 Te Neyman allocation metod was used to determine te optimal sample size for eac stratum subject to te total sample size of 15,000 in te sample design for te ITB TY2007 survey. It aimed to minimize te variance of te estimated mean burden, but it left several strata wit too few observations to model. A common rule of tumb is tat a sample must include at least 10 or 15 observations per independent variable in a regression model (Stevens, 2002; Bartlett et al., 2001). We coose 15 to be conservative. Te expected number of independent variables is 15, so te minimum expected number of complete responses for modeling is 225 for eac stratum. Our objective is to minimize te variance of te estimated mean burden constrained on tis minimum expected number of complete responses for modeling, wit response rate incorporated. Te total sample size increases to 20,000 in te ITB TY2010 survey. We start wit te same total sample size of 15,000 as in TY2007, considering tis as our base sample. Te remaining 5,000 is te reserved sample to make any adjustments for te purposes of modeling and increasing te precision of te estimate. An iterative procedure is used to searc for our optimal sample allocation. Te final allocation wit key inputs is sown in Table 1. Tis design results in an overall CV of 1.01%, less tan 2%, te general requirement on precision. 3 Tis approac is discussed in furter detail in Brick, et al, 2009 and Contos, et al,

3 Table 1 Sample allocation for ITB TY2010 survey Monetized Burden Projected Pop Count Est. Mean Est. Std. Dev. Est. Final Allocation 11 paid, low 9,822, paid, low-medium 26,114, , paid, medium 15,940, , aid, medium-ig 15,732, , , paid, ig 10,685,596 1, , , self, low 3,503, self, low-medium 2,707, self, medium 1,695, self, medium-ig 770, self, ig 288, soft, low 10,478, soft, low-medium 15,971, soft, medium 10,942, , soft, medium-ig 6,336, , , soft, ig 1,639,707 1, , Total 132,630, ,000 Overall CV 1.01% 2. Iterative Procedure for Sample Allocation TY2010 population counts witin eac stratum are projected based on TY2009 population counts and te projected growt rates. TY2007 survey data are used to estimate te mean, standard deviation, and item response rate of monetized burden for eac stratum. Wit tese inputs, we are able to determine te sample allocation as follows. 2.1 Step 1 Te adjustment procedure in Step 1 is sown in Table 2. Te Neyman allocation metod is first applied to te base sample of 15,000, te sample size of te TY2007 ITB survey sample. Te expected numbers of respondents are obtained for eac stratum according to te estimated item response rates 4. We compare tese numbers wit 225, te minimum sample required to model eac stratum. Nine strata are identified tat do not contain enoug respondents. Te sample sizes for tese strata are adjusted so tat te expected numbers of respondents tat are igligted in Table 2 equal 225, wile te sample sizes for te remaining strata maintain te Neyman allocation. Tis procedure results in a total sample size of 18,540, indicating tat we ave more sample to be allocated to te base sample to increase te precision of te estimate. 4 Te estimated item response rates are te lower bounds of te 99% Wilson Score confidence intervals in TY2007, adjusted for te overall expected lower bound of te response rate at 40% in TY

4 Table 2: Adjusted Allocation Step 1: Adjustment of Neyman Allocation for Modeling Monetized Burden Neyman Allocation Est. Item Minimum Sample Size for Modeling Adjusted Allocation: Step 1 11 paid, low , , paid, medium 2, , paid, mediumig 2, , , paid, ig 4, , , self, low self, medium self, mediumig self, ig soft, low soft, medium 1, , soft, mediumig soft, ig Total 15, ,861 3,375 18, Step 2 In Step 2, we determine te maximum sample size for base sample for te Neyman allocation, given te nine strata identified for furter adjustment in te Step 1. Te following inequality (1) is used to find te maximum sample size for base sample. N S Α n b + ( 20,000 nb ) 5,600, (1) N S H Were nb is te total base sample size, N and S are te population count and standard deviation for stratum, Α = strata{ 11,21, 22, 23,24,25,31,32,35}, H is te entire set of all strata. Te minimum total sample size is 5,600 for all nine strata identified in Step 1 (te sum of te igligted numbers in te last column of Table 2). 4678

5 Te first component on te left-and side of inequality (1), n b Α H N N S S, represents te sum of te sample sizes for all nine strata in Α from te base sample, due to te Neyman allocation. Te second component, 20,000 n b, represents te sum of te sample sizes for all nine strata from te reserved sample to adjust for modeling. We solve Inequality (1) for n and 16, 693. b n b Te adjustment procedure in Step 2 is sown in Table 3. Te Neyman allocation for te base sample of size 16,693 is calculated, and ten te adjustment procedure from Step 1 is repeated. Step 2 results in a total sample size of 20,008. Tis is because te number of strata to be adjusted is reduced from nine to eigt since te base sample size increases in Step 2. Specifically, te sample size in stratum 32 tat results from Neyman allocation becomes sufficient for modeling and does not need to be adjusted. Te value for n b = 16,693 is obtained assuming tat nine strata need to be adjusted; owever te result sows only eigt strata need to be adjusted, indicating tat tis non-matcing or nonsteady state requires anoter iteration. Table 3: Adjusted Allocation Step 2 Monetized Neyman Burden Allocation Estimated Minimum Sample Size for Modeling Adjusted Allocation: Step 2 11 paid, low , , paid, medium 2, , , paid, medium-ig 3, , , paid, ig 4, , , self, low self, medium self, medium-ig self, ig soft, low soft, medium 1, , soft, mediumig 1, , soft, ig Total 16, , ,

6 2.3 Step 3 In Step 3, we adjust te sample size for base sample so tat te final total sample size reaces exactly 20,000, given te eigt strata identified for adjustment in Step 2. Inequality (2) is used to find te sample size for base sample. N S B n b + ( 20,000 nb ) 4,988, (2) N S H Were, Β = strata{ 11,21,22,23,24,25,31, 32}. 4,988 is te minimum total sample size for all eigt strata identified in Step 2--te sum of te igligted numbers in te last column of Table 3. We solve Inequality (2) for nb and n b 16, 684. We start wit te base sample of 16,684 for te Neyman Allocation, and repeat te same adjustment procedure as in Step 2. Te number of strata requiring adjustment remains te same in tis round, implying tat we ave reaced a steady state and te resulting total sample size reaces te exact 20,000 as required. Te adjustment procedure is sown in Table 4. Table 4: Adjusted Allocation Step 3: Final Allocation. Monetized Burden Neyman Allocation Estimated Minimum Sample Size for Modeling Adjusted Allocation: Step 2 11 paid, low , , paid, medium 2, , , paid, medium-ig 3, , , paid, ig 4, , , self, low self, medium self, medium-ig self, ig soft, low soft, medium 1, , soft, mediumig 1, , soft, ig Total 16, , In summary, te allocation adjustment procedure described above maximizes te optimality in te Neyman allocation metod subject to te minimum sample size necessary for modeling witin eac stratum, wile incorporating response rate. As we can also see, te adjustment for modeling in te above iterative procedure is minimized 4680

7 subject to te total sample size of 20,000. Even toug we expect a sacrifice in precision due to adjustment of te Neyman allocation, tis sacrifice as been minimized. 3. Evaluation of te Proposed Sample Design Table 5 presents a comparison between te final design (Design I) and te Neyman allocation of 20,000 witout any adjustment (Design II). Te expected number of respondents from our design satisfies te minimum sample size for modeling witin eac stratum, wile te expected numbers of respondents for seven strata from te Neyman allocation witout any adjustment are far less tan te minimum sample size. Te overall CV from te Neyman allocation witout any adjustment is 0.95%, wile te overall CV from our design is 1.01%. Te precision in our design decreases, as expected, wic is a trade-off for incorporating modeling and response rate. However, our overall CV is still comparable wit te overall CV from te Neyman allocation. Suc sacrifice in precision is minor and is well compensated by te gains in modeling. Te overall CV based on te expected number of respondents is 1.62% from our design, and again it is comparable wit te corresponding CV of 1.53% from te Neyman allocation. Table 5: Comparisons between Two Designs: Design I: te proposed design wit adjustment for modeling and response rate Design II: Neyman allocation witout any adjustment Monetized Burden Estimated Design I Design II 11 paid, low paid, medium paid, mediumig paid, ig self, low self, medium self, mediumig self, ig soft, low soft, low-medium soft, medium soft, mediumig soft, ig Total Overall CV 1.01% 1.62% 0.95% 1.53% 4681

8 Te Neyman allocation metod assumes population counts and standard deviations are bot known for all strata, owever in practice tey are often estimates. We evaluate te projection metod of te TY2010 population by comparing te actual count in TY2009 and te projected count in TY2009 using te same projection metod. Te average relative error rate among all te strata is , supporting our projected counts for TY2010. However, tere is not a straigtforward measure to see ow reliable our estimated standard deviations are. It is of interest to investigate ow robust our design is to different values of population standard deviations. We obtain different sets of standard deviation estimates, and assume tey represent te true population parameter values in TY2010 and te overall CVs can be calculated based on tese estimates and our sample design. Table 6 sows tat our proposed sample design can still satisfy te precision requirement even wen te population standard deviations are mis-specified. Terefore, our design is robust to different estimates on population standard deviations. Table 6: Robustness of te proposed sample design to different values of standard deviations Std. Dev. I: estimates used in our final design Std. Dev. II: estimates using multiple imputations Std. Dev. III: estimates including outliers Std. Dev. IV: estimates excluding te observations above te 99 t percentile Std. Dev. V: estimates from ITB TY99/00 survey Monetized Burden Std. Dev. I Std. Dev. II Std. Dev. III Std. Dev. IV Std. Dev. V 11 paid, low paid, medium paid, mediumig 1, , , paid, ig 2, , , , , self, low self, medium self, mediumig self, ig soft, low soft, medium soft, mediumig 1, , , soft, ig 1, , , , , Overall CV 1.01% 1.40% 1.29% 0.79% 1.02% Finally, we consider te implications of our proposed design on te oter tree relevant variables, total money, total monetized time, and total time wit respect to bot precision 4682

9 of te estimates and modeling. Since te estimated item response rates for tese tree variables are almost identical to te ones for total monetized burden, immediately it follows tat te expected number of respondents from our design will satisfy te minimum sample size requirement for modeling tese tree variables. Table 7 sows te overall CV for eac of te tree variables based on our design, compared wit te CV from te Neyman allocation for eac variable separately. As sown in Tables 7-1, 7-2, and 7-3, altoug our design based on te total monetized burden still can satisfy te modeling of all tree variables, te sacrifice in precision is greater comparing te Neyman allocation wit eac variable as te design variable. Tis indicates tat a more advanced algoritm is needed if we want to simultaneously estimate and model total money, total monetized time, and total time. We expect tat our proposed design will provide sufficient responses in te various strata to support furter refinements in future designs. However, te Neyman allocation witout any adjustment for any of te tree variables results in modeling issues. 4. Conclusions An iterative procedure is proposed to searc for te optimal sample allocation to optimize te Neyman allocation subject to te minimum sample size for modeling purposes, wit response rates incorporated. On one and, any adjustment on te Neyman allocation may cause loss of efficiency; on te oter and, te Neyman allocation witout any adjustment will leave us wit some strata wit too few observations to model. Our proposed sample allocation procedure sows tat suc loss of efficiency can be minimized wit appropriate adjustment; moreover, it can be well compensated by te gains from modeling and te ability to conduct valid predictive analysis. Contemporary surveys often serve multiple purposes bot descriptive and predictive analyses. How to balance between te objectives to acieve a certain level of precision on te estimates and to conduct valid predictive analysis requires more teoretical development. Our future researc will extend te approac described ere. We are also interested in exploring te optimal allocation algoritm wen te objective of te survey is to estimate and model multiple variables simultaneously. Finally, Neyman allocation is often criticized because it ignores possible different response rates across strata. Wen response rates are quite different across different strata, te objective function is to minimize te variance of te mean estimate based on te respondents, and te sample size can be calculated from te number of respondents and response rate. We expect tat a more rigorous optimization procedure reflecting tese response rate differences will furter improve te performance of te sample design. 4683

10 Table 7-1: Implications of te proposed design on total money Design I: our design wit te total monetized burden as te design variable Design II: Neyman allocation wit total money as te design variable Estimated Design I Design II 11 paid, low , , paid, medium ,656 1,040 1, paid, mediumig ,092 1,228 4,264 1, paid, ig ,582 1,784 7,423 2, self, low self, medium self, mediumig self, ig soft, low soft, medium , soft, mediumig , soft, ig Total 20,000 7,701 20,000 7,610 Overall CV for total money 1.26% 2.06% 1.08% 1.77% 4684

11 Table 7-2 Implications of our proposed design on total monetized time Design I: our design wit te total monetized burden as te design variable Design II: Neyman allocation wit total monetized time as te design variable Estimated Design I Design II 11 paid, low , , paid, medium ,656 1,040 3,550 1, paid, mediumig ,092 1,228 3,582 1, paid, ig ,582 1,784 4,810 1, self, low self, medium self, mediumig self, ig soft, low soft, medium , soft, mediumig , , soft, ig Total 20,000 7,701 20,000 7,863 Overall CV for total money 1.26% 2.01% 1.18% 1.89% 4685

12 Table 7-3: Implications of our proposed design on total time Design I: Our design wit te total monetized burden as te design variable Design II: Neyman allocation wit te total time as te design variable Estimated Design I Design II 11 paid, low , , paid, medium ,656 1,040 1, paid, mediumig ,092 1,228 8,269 3, paid, ig ,582 1,784 2, self, low self, medium self, mediumig self, ig soft, low soft, medium , soft, mediumig , , soft, ig , Total 20,000 7,701 20,000 7,888 Overall CV for total time 2.43% 3.84% 1.86% 2.98% References Andreoni, James, Brian Erard and Jonatan Feinstein. Tax Compliance. Journal of Economic Literature. Vol. XXXVI (June 1998) Brick, Micael, George Contos, Karen Masken, and Roy Nord. Mode and Bias Analysis in te IRS Individual Taxpayer Burden Survey, 2009 Joint Statistical Meeting Proceedings, August Cocran, William G. Sampling Tecniques, 3 rd edition. Jon Wiley & Sons. Contos, George, Guyton, Jon L., Langetieg, Patrick, and Vigil, Melissa. Individual Taxpayer Compliance Burden: Te Role of Assisted Metods in Taxpayer to Increasing Complexit IRS Researc Conference. Guyton, Jon L., Jon F. O Hare, Micael P. Stavrianos, and Eric J. Toder. Estimating te Compliance Cost of te U.S. Individual Income Tax. National Tax Journal, September 2003,