Improving Timeliness and Quality of SILC Data through Sampling Design, Weighting and Variance Estimation

Transcription

1 Thomas Glaser Nadja Lamei Richard Heuberger Statistics Austria Directorate Social Statistics Workshop on best practice for EU-SILC - London 17 September 2015 Improving Timeliness and Quality of SILC Data through Sampling Design, Weighting and Variance Estimation We provide information

2 Prerequisites: EU-SILC in Austria "Community Statistics on Income an Living Conditions" Carried out in Austria since 2003, cross-sectional and longitudinal survey since 2004 Rotating panel design with 4 subsamples and 4 year panel duration Sample of first wave 2014: one-stage stratified probability sample from population register (ZMR) Minimum effective sample size (net- value): households, household surveyed in cross-section 2014 Sample survey of private households with voluntary participation Unit response rate 2014 operation: 76% first wave: 64% follow-up waves: 84% slide 2 17 th September 2015

3 Prerequisites: Indicators Europe 2020 indicators on poverty and social exclusion: main indicator At-risk-of-poverty or social exclusion (AROPE) Measurement of Europe 2020 Targets from EU-SILC 2008 onwards Main income based indicator: At-risk-of-poverty rate (AROP) Net household income of year preceding the survey year Equivalised net household income (EPINC) Measurement period: year preceding the survey year Households with EPINC below 60% of national median of EPINC are AROP slide 3 17 th September 2015

4 Prerequisites: Income from administrative registers Fully implemented register data use started with EU-SILC 2012 About 85% of total household income from administrative register information (remaining income components from questionnaire) Back-calculation of EU-SILC to ensure unbroken time series from 2008 onwards (also in UDB) Linkage of register and survey data Encrypted personal identifier (bpk) provided by federal ministry of the interior Provision of bpk for gross sample and persons living in households Eurostat grant agreement on Improving Methodology on Sampling, Weighting, Imputation and Variance Estimation in the Austrian EU-SILC with regard to Administrative Data (final results in 2016) slide 4 17 th September 2015

5 1. Efficiency gains by sampling slide 5 17 th September 2015

6 Selection of first wave sample Type of sampling: one-stage stratified probability sample Sampling units: dwellings registered in the central residence register (ZMR) Sample size (gross): 3,229 households (EU-SILC 2014) Stratification criteria: - Interviewer units (geographical units below NUTS2 level) - Disproportional allocation per NUTS2 level according to expected response rates (based on average response of two preceding years) Large discrepancies of relative distribution of NUTS2-levels between population and net sample would lead to higher dispersion of weight adjustment factors slide 6 17 th September 2015

7 Efficiency gains by sampling 25,00 Relative distribution of private households by provinces (NUTS2) for first wave of EU-SILC 2014 in % 20,00 15,00 10,00 5,00 Population Net sample with disproportinal allocation % Hypothetical net sample with proportional allocation % 0,00 Source: EU-SILC 2014 (unpublished results) slide 7 17 th September 2015

8 Efficiency gains by sampling Income register data allow for an economic evaluation of (almost) every address in the sampling frame Selecting addresses by building strata according to household income (HINC_REG) percentiles based on register data is possible Optimal allocation (Neyman allocation, cf. Cochran 1977) based on register income distribution to gain smaller standard error of AROP Since AROP is a component of AROPE, this may also yield smaller standard error for main indicator AROPE Ongoing work will be finalised in slide 8 17 th September 2015

9 2. Improving weighting with income register data slide 9 17 th September 2015

10 Cross-sectional weighting procedure Three steps of SILC weighting procedure: household weight = = design weight * nonresponse weight * adjustment weight 1) Design weight: inverse selection probability (S strata) d s = N s n s 2) Unit nonresponse weight: Inverse estimated response probability rh (based on stepwise logistic regression) multiplied to design weight b h = d s rh s {1,, S} h {1,, n (r) } 3) Adjustment of weights to external sources Calibration to known marginal distributions => Calibrated household weights (first year wave): ( weight adjustment factors) slide th September 2015

11 Calibration using marginal distribution from administrative income registers Auxiliary information available for sampling frame and thus also for gross sample of first wave Same source for variables in sample and in marginal distribution New marginal distributions provided by variables from wage tax register: Number of persons receiving income from employment (at least 15 years old) Number of persons receiving income from old-age benefits slide th September 2015

12 Cross-sectional weighting procedure: Calibration to external marginal distribution Household level Province (NUTS2) Household size Tenure status Burgenland Carinthia Low er Austria Upper Austria Salzburg Styria Tyrol Vorarlberg Vienna Personal level 119,482 1 Person 1,391,569 Ow ner 1,870, ,103 2 Persons 1,120,747 Not ow ner 1,891, ,689 3 Persons 567, , Persons 682, , , , , ,619 Age & sex Age Men Women Austria Not Austria , ,868 6,240, , ,116,783 1,090, ,770,060 1,799, , ,378 Citizenship (persons aged 16+) Recipients of unemployment benefits Employees (at least 60 days) (persons aged 15+) Source: Statistics Austria, EU-SILC 2014, M icro-census 2014, social security and wage tax register 2013 Retirees 597,568 3,813,936 1,989,331 slide th September 2015

13 Unit nonresponse analysis Research question: Is there a bias in the main income based indicator (AROP) caused by selective unit nonresponse? Household income based only on income registers (HINC_REG) is used as study variable Y for nonresponse analysis Highly correlated with overall household income (HINC) Unit nonresponse rate first wave of EU-SILC 2014: 36% High rate of persons with potential data in income registers (99%) in EU-SILC slide th September 2015

14 Unit nonresponse analysis: definitions Nonresponse All observations are missing for units of a selected sample (Unit-Nonresponse) Nonresponse of a unit is a random variable Occurrence of unit nonresponse has a certain probability (cf. Groves et al. 2004) Bias Systematic deviation of the expected value of an estimator Y from the true value Y in the population (cf. Särndal 2003) Unit nonresponse (UNR) bias Bias caused by unit nonresponse: Systematic deviation of the expected value of an estimator Y based in respondent set r from the value Y in entire gross sample s (cf. Groves 2006, Särndal & Lundström 2005) slide th September 2015

15 Unit nonresponse bias analysis Comparison with design weighted estimate Estimate of absolute bias Estimate of relative bias weight dataset mean(hinc_reg) (1) d s gross sample (2) d s net sample % (3) (version 1) net sample % d s 1/rh d s 1/rh d s 1/ch 1/rh d s 1/ch 1/rh (4) (version 2) net sample % (5) (version 1) net sample % (6) (version 2) net sample % (7) Calibration, base weight: (2) net sample % (8) Calibration, base weight: (3) net sample % (9) Calibration, base weight: (5) net sample % (10) Calibration, base weight: (4) net sample % (11) Calibration, base weight: (6) net sample % Calibration, base weight (4) (only persons present in hh) net sample % Source: Statistics Austria, EU-SILC 2014 (unpublished results) rh - estimated respose rate ch- estimated contact rate rh - estimated adjusted response rate slide th September 2015

16 Unit nonresponse bias analysis Comparison with sampling frame dataset mean(hinc_reg) Estimate of absolute bias Estimate of relative bias none sampling frame (1) d s gross sample % (2) d s net sample % (3) (version 1) net sample % d s 1/rh d s 1/rh d s 1/ch 1/rh d s 1/ch 1/rh weight (4) (version 2) net sample % (5) (version 1) net sample % (6) (version 2) net sample % (7) Calibration, base weight: (2) net sample % (8) Calibration, base weight: (3) net sample % (9) Calibration, base weight: (5) net sample % (10) Calibration, base weight: (4) net sample % (11) Calibration, base weight: (6) net sample % Calibration, base weight (4) (only persons present in hh) net sample % Source: Statistics Austria, EU-SILC 2014 (unpublished results) slide th September 2015

17 Conclusions of unit nonresponse analysis Assuming missing completely at random (MCAR) nonresponse mechanism could lead to substantial bias Modeling unit nonresponse in two steps seems to have a slight effect on reducing bias Calibration seems to reduce bias even if applied directly to the design weights Calibrated weights using register income only for persons who where actually in the net sample shows very similar results to sampling frame for HINC_REG slide th September 2015

18 3. Measuring improvement of precision caused by calibration weights slide th September 2015

19 Precision of EU-SILC indicators Variance estimation of AROP and AROPE does not include calibration effect, yielding a conservative estimate Therefore new precision requirements demand a more precise estimation of standard error (SE) for estimates of indicators p SE(p ) < p (1 p ) a N + b Austrian case for AROPE of EU-SILC 2014 (p =19.2%) SE p = % > p(1 p) a N+b = % Taking into account calibration variables correlated with AROP may have a reducing effect on standard error (cf. Berger/Skinner 2003, Deville 1999, Deville/Särndal 1992) slide th September 2015

20 Precision of EU-SILC indicators: Evaluation Instead of AROP the residuals ε n of the regression of AROP on the K calibration variables x k are used (Cf. Eurostat 2013, p. 13) AROP n = K k=1 β k x nk ε n = AROP n -AROP n Results show a reduction of the estimated standard error Standard error of AROP on person level in % dichotomous indicator linearized variable bootstrapping (1000 resamples) indicator without calibration effect residuals including calibration effect Change of standard error (in%) caused by calibration effect Source: EU-SILC 2014 (unpublished results) slide th September 2015

21 Precision of EU-SILC indicators: Evaluation Using residuals AROPE n -AROPE n instead of AROPE shows a reduction of the estimated standard error by 14.7% Standard error of AROPE on person level in % dichotomous indicator indicator without calibration effect residuals including calibration effect Change of standard error (in%) caused by calibration effect Source: EU-SILC 2014 (unpublished results) Incorporating calibration in variance estimation yields a slightly smaller estimated standard error and could make it possible to meet new precision requirements slide th September 2015

22 Concluding remarks The availability of income register data opens various opportunities for improving quality of indicators by sampling, weighting and variance estimation For sampling, optimal allocation may reduce standard error For weighting, marginal distributions from income registers may reduce bias For variance estimation, using residuals from the regression of calibration variables (including variables form income registers) on indicators instead of point estimators make it possible to measure this improvement in efficiency correctly in order to meet precision requirements slide th September 2015

23 Bibliography Berger, Y. G.; Skinner, C. J. (2003): Variance Estimation for a low income proportion. Applied Statistics 52, Part 4, Cochran, W. G. (1977): Sampling Techniques. New York. Wiley Deville, J. C. (1999): Variance Estimation for complex statistics and estimators: linearization and residual techniques. Survey Methodology 25, Deville, J. C.; Särndal, C.-E (1992): Calibration estimators in Survey Sampling. Eurostat (2013): Standard error estimation for the EU-SILC indicators of poverty and social exclusion. Eurostat Statistical Working Papers. Luxembourg. Groves, Robert M.; Fowler Jr., Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004): Survey Methodology. Hoboken. Wiley. Groves, Robert M. (2006): Nonresponse Rates and Nonresponse Bias in Household Surveys. Public Opinion Quarterly 70 (5, Special Issue), DOI: /poq/nfl033. Särndal, C.-E.; Lundström, S. (2005): Estimation in Surveys with Nonresponse. West-Sussex. Wiley. slide th September 2015

24 Thank you for your attention! Contact: Guglgasse 13, 1110 Wien Tel: +43 (1) Any questions or comments? slide th September 2015