Challenges and Successes in Developing the new Survey of Income and Program Participation (SIPP)

Transcription

1 1 Challenges and Successes in Developing the new Survey of Income and Program Participation (SIPP) Jason Fields, David Johnson, Matthew Marlay, and Judith Eargle U.S. Census Bureau July 2013, European Survey Research Association Conference Ljubljana, Slovenia

2 2 Mission The mission of the Survey of Income and Program Participation (SIPP) is to provide a nationally representative sample for: evaluating annual and sub-annual dynamics of income, movements into and out of government transfer programs, family and social context of individuals and households, and interactions between these items.

3 Basics Current SIPP Design 3 National Panel Survey First SIPP panel began in 1984 Paper instrument and overlapping panel design panels Conversion to CAPI and abutting (end-to-end) panels beginning in 1996 CAPI revision for 2004 panel incorporated much more dependent data Delayed start to the 2008 panel due to budget (September rather than February 2008 start) New 2014 Panel start in February 2014 most significant change since survey began. Sample Around 11,000 interviewed households for overlapping paper panels. Expansion to about 45,000 interviewed households for CAPI multistage-stratified sample of the U.S. civilian non-institutionalized population

4 Basics Current SIPP Design 4 Details 4-month recall period 3 interviews / year Adults (age 15+) are interviewed in Wave 1, data collected for all people Follows all adults in subsequent interviews Interview all household members at each address with original Wave 1 adult The sample is divided into 4 rotation groups for monthly interviewing Interviews are conducted by personal visit and by decentralized telephone Panel Duration Paper about 2½ yrs CAPI - 4 yrs 2008 Panel Exception, extended to 16 waves (about 5 years)

5 5 SIPP Re-engineering Implement Improvements to SIPP Reduce costs Reduce respondent burden Improve processing system Modernize instrument Expand/enhance use of administrative records Key Design Changes: Annual interview, 12-month reference period from 4-month Event History Calendar (EHC) methods to facilitate respondent recall over longer reference period

6 Timeline of SIPP Re-engineering How did we get here? (The Order of the Phoenix) December 2005 January 2006 February 2006 June 2006 August 2006 December 2006/ January 2007 Stopped work on converting existing SIPP to BLAISE and reprogramming post-data collection processing to SAS Preparing for budget release, sessions with key federal stakeholders were held to discuss future SIPP plans Publicly launched the Program of Income, Wealth, and Health Insurance Measurement and possible end of SIPP data collection in Sept 2006 Brookings Workshop: Concerns expressed by stakeholders about possible data gap and usefulness of administrative data. Stakeholder matrices sent to major users in order for them to identify critical items. The meeting consensus suggested that administrative data would only be most useful in data quality evaluation. First open stakeholder meeting held at Census. Publicly stated the Bureau was refocusing effort to use admin data to supplement (not replace) data collection. New survey plans presented EHC and hooks to allow opportunities for reimbursable supplements. Announced plans to continue SIPP with reduced sample from 10/06-5/07 Another open stakeholder meeting at Census where more details were presented on content, structure, and plans for the new survey. Census presented plans to the National Academies Committee for National Statistics [CNSTAT] group empanelled to advise SIPP. (Karl Scholz [chair], Pat Ruggles [study director]) 6

7 Timeline of SIPP Re-engineering 7 March/May 2007 March 2007 April/May 2007 June/July 2007 November 2007 December 2007 January 2008 April 2008 September 2008 October The Center for Economic Policy Research [CEPR] circulated Save the SIPP letter. Census presented progress to date, and suggested using linked CPS data and administrative records data. Presentation to Council of Professional Associations on Federal Statistics [COPAFS] With additional funding, announced plans to continue SIPP with reduced sample through Sept, 2007 Topical content meetings held at Census with SIPP stakeholders Senate and House Appropriations Committees propose additional funding to continue SIPP collection in 2008 Association of Public Policy and Management SIPP Discussion Event History Calendar Conference See: American Economic Association, SIPP; The Last 20 Years session. Population Association of America, SIPP; Panel on Re-engineering the Survey of Income and Program Participation American Statistical Association Survey Research Methods Committee to advise SIPP is initiated SIPP Small Grants and Workshop See CNSTAT Panel Report Released

8 SIPP Re-engineering Field Test Plans 8 - Proof of concept test - Sample, design, results paper and pencil reinterview test - EHC CAPI test - Hurdles and highlights Integrated Blaise and C# instrument prototype - CAPI revised test 2011 Test improvements to the Wave 1 instrument, training, and expand sample to all regional offices. Interwave locating experiment 2012 Test Wave 2 concepts and instrument, examine movers and attrition issues, dependent interviewing methods and refine training Wave 3 interview allows household members to return and additional mover and dependent interviewing evaluation SIPP EHC-based instrument is the production SIPP instrument - A new Phoenix "A Bird of a Different Color"

9 Developing the CAPI SIPP-EHC : Challenges Faced 9 Developing new technical capacity Experimentation Limitations Evolution Crisis planning Limited lead time / preparation Changing goals demanded flexibility New procedures Training and acceptance by field staff Development and refinement of procedures

10 Current Status Wave 3 Test 2013 SIPP-EHC Closed-out at the end of April 2013 Testing mover locating, reinstatement capability, and instrument refinements SIPP Panel Interviewing Wave 15 Wave 15 started in May and Wave 16 starts in September These waves are collecting the calendar year 2013 overlap that will be available for comparison with the data collected by the 2014 Wave 1 instrument 2014 SIPP Panel Instrument is in revision and preparation for administration from February through May 2014

11 SIPP-EHC Development and Implementation for Sep 2009 Jan - May - Sep Jan - May - Sep Jan - May - Sep Jan - May - Sep Jan - May - Sep Jan Paper Test Eval. Analysis 2010 SIPP-EHC Instrument Dev SIPP-EHC Dress Rehearsal Ref. Period CY2009 SIPP 2008 Panel Waves 1 12 (Rotation 1 field months) Field work Processing and Evaluation 6 Regions 8k hhlds 10 States 2011 SIPP-EHC Inst. Dev. 2011SIPP-EHC Dress Rehearsal Ref. Period CY2010 Field work Wave 1 Processing and Evaluation Re-contact Activities 2012 SIPP-EHC Wave 2 Inst SIPP-EHC Ref. Pd CY SIPP-EHC CARI Field work Wave SIPP-EHC Wave 3 Inst SIPP-EHC Ref. Period CY Regions-4k hhlds-20 States Test of Wave 1, 2 & 3 [Feedback and movers] Extension w13-w SIPP Panel Inst. Refinement Production 2014 SIPP Panel Wave 1 Ref. Period CY2013 Processing and Evaluation Field work Wave 1 Materials Prep Processing and Evaluation Field work Wave 1 12 Regions-Full Production Panel 2010 based sample

12 Notes on the New Design Scope Similar to SIPP Broader than core / includes key topical module content in each wave 12 Better integration of concepts EHC generates integrated reporting across domains Topics previously implemented as add-on modules now integrated Facilitate hooks to enable supplements for additional content Increased efficiency in processing and producing data products Flexibility in administration (dynamic interview month and reference period) Dependent data incorporation into EHC instrument Reduced cost through annual administration Improve management through realigned structure and improved monitoring using all available tools especially paradata

13 SIPP EHC 2014 Contents and Dependent Data Content 13

14 The 2011 SIPP-EHC Instrument Design & Dependent Data 14

15 2011 SIPP-EHC Completed Calendar 15

16 2012 SIPP-EHC Calendar w/ Dependent Data 16 Dependent Text Complete spells representing information that was reported last time, but not active at the time of the interview. Should be left alone. The information displayed here changes depending on the dependent information. Provisional spells representing information that was reported last time, and current at the time of the interview. Needs to be extended.

17 Evaluating Field Test Data 17 Comparison of survey reports from SIPP-EHC, SIPP, and Administrative Records Comparison for CY2010 and CY2011. Waves 1 and 2 for the SIPP-EHC, Waves 5 through 11 for 2008 SIPP Forthcoming report compares across 19 topics including demographics, income and program participation Administrative records are available to evaluate 7 of those topics: employment and earnings, Medicare, Medicaid, OASDI, SSI, SNAP and TANF

18 Comparison to Administrative Records 18 Survey estimates generally differ statistically...but not always substantially or in economically important ways In almost all cases where administrative records are available, SIPP-EHC is at least as accurate as SIPP.

19 Monthly Values for CY2010 and CY2011 SIPP and SIPP-EHC 19 37% Medicaid 10% SSI 27% 21% Medicare 0% 10% TANF 11% 0% 19% OASDI 10% Unemployment Insurance 9% 28% SNAP 0% 56% Employment 18% %

20 Annual Values for CY2011 SIPP and SIPP-EHC Employment Poverty 0.41 Health Insurance SNAP Medicaid SSI Medicare TANF OASDI Unemployment Insurance

21 Median Monthly Amounts for CY2011 SIPP and SIPP-EHC 21 $785 OASDI $388 TANF $840 $400 $200 SNAP $900 Unemployment Insurance $209 $484 SSI Earnings $654 $1,947 $698 $2,033 $0 $1,000 $2,000 $0 $1,000 $2,000

22 False Negative Rates Compared to Administrative Data S I P P E H C Medicaid Medicare OASDI SNAP SSI

23 SIPP-EHC False Negative Rates By Year Medicaid Medicare OASDI SNAP SSI

24 False Positive Rates Compared to Administrative Data S I P P E H C Medicaid Medicare OASDI SNAP SSI

25 SIPP-EHC False Positive Rates By Year Medicaid Medicare OASDI SNAP SSI

26 Transition Rates SIPP, SIPP-EHC, and Administrative Records S I P P E H C A R Medicaid Medicare OASDI SNAP SSI

27 Medicaid Transitions SIPP-EHC and SIPP J F M A M J J A S O N D

28 Transitions on Seam SIPP-EHC and Administrative Records E H C A R Medicaid Medicare OASDI SNAP SSI

29 29 Conclusions SIPP-EHC and SIPP produce estimates that are not substantially different. SIPP-EHC agrees with admin data at least as well as SIPP. Transitions fall disproportionately on seams.

30 Additional Evaluation Plans 30 Key indicators by topic evaluated against as many sources as possible False transitions and mis-timed transitions recall or seams Instrument usability and last minute fixes for flow and wording Issues related to successful use of dependent data Flexibility for Interviewer/Respondent interaction Mover individuals locating procedures and tools Changing respondents Data quality comparisons Respondent Identification Policy Paradata evaluation and integration

31 Successes 31 Integrated Blaise/C# survey instrument that allows: Data storage through a common Blaise data structure Conversational EHC navigation Dependent data incorporated into EHC for Wave 2 and beyond Computer Assisted Recorded Interviewing (CARI) Development of a SAS based data processing system with new ground up edits Responsive and integral stakeholder involvement Survey realignment to a new management structure

32 Lessons Learned Rapid (Agile) development, complexities in design and implementation need frequent reviews revisions benefit from prototyping 32 Responsive management allowed us to determine necessary adjustments to plans, requiring additional flexibility and time Need early and continued stakeholder involvement Importance of engaging field staff. Implement respondent and interviewer debriefings, item and case notes for issues.

33 SIPP-EHC Milestones and Plans Fieldwork for the 2013 SIPP-EHC Data collection completed at the end of April 2013 Finalizing the instrument, procedures, and training for 2014 production. 33 User and stakeholder information meetings to present evaluation findings Evaluations from 2010, 2011, and 2012 SIPP-EHC comparisons will be presented at National Academies on July 10, 2013 Summer of 2013, final revisions for production instrument 2014 sample selection based on most recent 2010 sampling frame and cost constraints Fieldwork for the 2014 SIPP Panel Wave 1 Data collection February through May 2014 First data release from 2014 planned by the end of Wave 2 data collection in 2015.

34 34 Thank you! Contact:

35 Crude Response Rates for Tests, 2004 and 2008 Panels SIPP Panel 2008 SIPP Panel Wave Wave Wave Wave SIPP Field Test 2012 SIPP Field Test 2011 Sample CARI Sample SIPP Field Test Combined Sample 66.2 Wave 1 Wave 4 Wave 7 Wave Sample CARI Sample Combined Sample