Contents I. INTRODUCTION...1 II. OBJECTIVE AND SCOPE...1 III. METHODOLOGY AND QUESTIONS ANSWERED...2 IV. RESULTS OF ANALYSIS...11 V. CONCLUSION...

Transcription

1 Research Project to Understand the Medicaid Undercount: The University of Minnesota s State Health Access Center, the Centers for Medicare and Medicaid Services, the Department of Health and Human Services Assistant Secretary for Planning and Evaluation, The National Center for Health Statistics, and the U.S Census Bureau Phase II Research Results: Examining Discrepancies between the National Medicaid Statistical Information System () and the Current Population Survey (CPS) Annual Social and Economic Supplement (ASEC) Final June 8, 2007 Note: This paper was made possible by grant from the Robert Wood Johnson Foundation to the State Health Access Data Assistance Center (Michael Davern, PI) with additional support supplied by the Office of the Assistant Secretary for Planning and Evaluation (ASPE), the Center's for Medicare and Medicare Services (CMS), and the U.S. Census Bureau. This paper has undergone a limited review by all the participating organizations in accordance to existing agreements among these organizations. The views expressed are those of the author(s) and not necessarily those of ASPE, CMS, or the US Census Bureau.

2 Contents I. INTRODUCTION...1 II. OBJECTIVE AND SCOPE...1 III. METHODOLOGY AND QUESTIONS ANSWERED...2 IV. RESULTS OF ANALYSIS...11 V. CONCLUSION...18 VI. LIMITATIONS...18 VII. APPENDICES:...20 I. Description of Four Project Phases...20 II. Key Decisions Leading to Phase II Research Design...20 III. Files and Methods Used...20 IV. Meta Data in Phase II...20 V. Detailed Tables and Documentation...20 i

3 I. Introduction This paper describes the results of the second phase of a four-phase research project of University of Minnesota s State Health Access Data Assistance Center (SHADAC), Centers for Medicare and Medicaid Services (CMS), Assistant Secretary for Planning and Evaluation (ASPE), National Center for Health Statistics (NCHS), and the US Census Bureau. The research is designed to explain why discrepancies exist between survey estimates of enrollment in the number of enrollees reported in state and national administrative data. The research done for this project includes both national and state-level analysis. National files include the Medicaid Statistical Information System (), the Medicaid Analytic extract (MAX), and the Medicare Enrollment Database (EDB). Survey files include the Current Population Survey (CPS) and the National Health Inventory Survey (NHIS) 1. Additionally, the following states have been invited to participate in this study: Florida, Maryland, California, Pennsylvania, New Jersey, Louisiana, and Minnesota. Participating states will provide data from their State Children's Health Insurance Program (SCHIP) enrollment files to CMS, and CMS will, in turn, provide these data to the U.S. Census Bureau. Understanding differences between enrollment data and survey data will benefit the Census Bureau and other participating agencies by suggesting possible improvement to CPS and other surveys. It will also engender a better understanding of existing CPS insurance data that provide a more accurate view of other insurance coverage for U.S. residents. As such, this research will enable a revised computation of the number of U.S. residents without health insurance. We consider the results presented in this paper to form only a basis a jumping-off point for making such an analysis. Without a sophisticated strategy to reassign surveyed persons categorized by CPS as uninsured but established by this study to have Medicaid coverage that considers the effect of having non-randomly missing data and also the offsetting effect of false positive reporting, such a computation would likely be significantly inaccurate. We expect additional work released by team members to speak directly to this issue. II. Objective and Scope The objective of Phase II is to investigate the size and source of the discrepancy between estimates derived from the Current Population Survey (CPS) March Supplement (known also as the Annual Social and Economic Supplement or ASEC) and counts of enrollees derived from. The input for this research is the validated and MAX files for calendar year (CY) 2000 and 2001 matched respectively to the 2001 and 2002 Current Population Survey March Supplements (ASEC), and the Census Bureau s Master Address File Auxiliary Reference File (MAF-ARF) and Person Characteristics File (PCF). We also use enrollment data for the first four months subsequent to each survey s reference period (which is the calendar year immediately prior to the year of survey administration) because we conjecture that current and 1 Description of these files can be found in Appendix III. 1

4 recent enrollment status (though not germane to the CPS questions about Medicaid coverage as worded) helps explain respondent reporting of Medicaid coverage. To address explanations for discrepancies between CPS and enrollment reported in, we produced summary statistics intended to suggest or discredit plausible explanatory factors for it. We constructed a series of logistic models to evaluate these factors in a multivariate setting. Additionally, we provided a summary data file for team members to corroborate and extend our analysis. Phase II Metadata can be found in Appendix IV and detailed tables in Appendix V. The Appendix V tables appear each in two presentations: the first using the original CPS survey weights and the second using adjusted weights developed by a re-weighting process described in Appendix III, which also describes the research files and other methods used to produce the results. III. Methodology and Questions Answered File Processing to Create the Research File 1. Validate records using a modified version of the Census Bureau s Person Identification Validation System (PVS). The validation technique compares the date-ofbirth and sex shown on the record to the date-of-birth and sex shown for that SSN on the Social Security Administration s Numident file. The process works under the assumption that if the SSN were incorrect, it would be highly unlikely for the and Numident date-of-birth and sex to be similar. For validated records, convert the SSN to an anonymous Protected Identity Key (PIK) to protect the security of the personal information. 2. Use the MAF-ARF, which contains PIKs and Master Address File Identifiers (MAFIDs, each representing a specific place of residence) to determine persons who may have been out of the CPS universe because they lived in institutional group-quarters. 3. For each person in, determine the months during which that person was enrolled in output a record showing this to the Summarized Enrollment History File (MSEHF), which has a single record for each validated SSN on and a single record for each client account without a validated SSN. 4. Process CPS records through the PVS, which includes verification and search procedures, to assign a PIK. 5. Account for the CPS cases that are un-linkable because the (represented person s) SSN is unknown or unverifiable. Create a new set of CPS weights by dropping these un-linkable cases and re-assigning their weight proportionally to records within the same reweighting strata that do have identifying data. While the re-weighting strata are developed in a way meant to minimize bias in derived estimates, re-weighting would cause completely no bias only if person-identification data are missing-at-random 2 : that 2 Little, R. J. A., Rubin D.B., Statistical Analysis with Missing Data. 2nd Edition.Wiley,

5 is to say, if the missing-ness of the identification data is statistically independent, within each re-weighting stratum, of any characteristic being measured. In the real world, this is unlikely to be true, and so, to the degree that the measured characteristics vary according to the missing-ness of the identity data, re-weighting could introduce bias into the generated estimates just as similar post-stratification adjustments made to all Census Bureau demographic households surveys can. Nevertheless, we believe that re-weighting is a way to project the results from the identified CPS reportees 3 to the entire CPS frame. 6. Process the MAX data for the analysis year by validating the SSN (by comparison to matching record) and replacing it with a PIK. 7. Link MSEHF to CPS, the Person Characteristics File (PCF), and the processed MAX file, joining on PIK. Produce tabulations shown in Tables 1 and 2 (Appendix IV). Produce an analysis file with a single record for each person in CPS. Use this analysis file to produce tabulations for Tables 3 and 4 (Appendix IV) and to perform the regression analysis. Assessing the feasibility of using record linkage to identify survey errors causing the discrepancy between the CPS March Supplement and the national Q1: What is the status of the raw match between CPS observations and records? Prior to accounting for universe differences and duplicate records, how many CPS and records have person-identifying data (PIK substituting for SSN) so we can determine the record s type of match outcome and therefore use it in the analysis of the count estimate discrepancy? How does the analytical universe differ after we use adjusted weights to account for being unable to correctly determine type of match outcome for CPS records lacking identifying information? We tally raw un-weighted and weighted counts of the number of CPS and records by type of match outcome. Counts are tallied overall and by selected characteristics. We use CPS demographic characteristics, poverty level, and health insurance coverage responses for CPS only and CPS-to- matched records. For only, we use PCF (or when PCF characteristics are unavailable, ) demographic characteristics, and type of benefits. These tallies are presented in Appendix V, Table 1, which provides an overview of the match process. 3 In this document, reportee refers to all household members of reporting households; respondent refers only to the household members (one for each household) that took part in the survey interviews. 3

6 Assessing the scope of the survey undercount Q2: What is the size of the discrepancy between CPS and? How does the CPS estimate of enrollees compare to the count of enrollees? How does it compare after making adjustments to account for universe differences? What is the distribution of the discrepancy between counts and CPS estimates by demographic category? How does this distribution vary among residents of different states? We make the raw count of Medicaid enrollees by totaling the number of client accounts excluding persons known to be deceased prior to March of the corresponding survey year, and we make the CPS estimate by weighting each member of the households in CPS by the final survey weight and summing. However, before comparing the count to the CPS estimate, we account for the differences in what each measure covers. Direct comparability is inappropriate for several reasons: Differences in the definition of enrollment - The raw count includes individuals in the State Children's Health Insurance Program (SCHIP) if it is run through Medicaid (an arrangement called Medicaidexpansion SCHIP ) and it only sometimes does include individuals in SCHIP in run separately from Medicaid (an arrangement called stand-alone SCHIP ). States may offer one or both of these types of plans. - In contrast, the CPS estimate made from the variable MCAID 4 does always include individuals who are reported to have SCHIP coverage, regardless of their state of residence and type of SCHIP program. Our preferred measure of Medicaid coverage, (which is an augmented version of the CPS variable CAID 5 (called subsequently Augmented-CAID 6 )) does not count Medicaid persons identified covered through SCHIP but not explicitly identified as enrolled in Medicaid 7. 4 MCAID is set to show Medicaid coverage for a reportee if the respondent indicates (for the reportee) coverage on Medicaid or SCHIP when first asked (in response to direct questions about each possible type of coverage), indicates Medicaid, SCHIP, Public, or in response to being asked about additional coverage (subsequent to answering initial questions about each type), or indicates Medicaid, SCHIP, or Public in response to a verification question asked about each household member who had not been reported (earlier in the health insurance question sequence) having any coverage. The Census Bureau uses MCAID in computing published CPS estimates of prior year Medicaid enrollees. 5 CAID is set to show Medicaid coverage for a reportee if respondent indicates coverage on Medicaid in response to several direct questions (prior to being asked about any other coverage or verification of non-coverage) about Medicaid enrollment: whether any household members had Medicaid coverage, and if so, who in household had it. 6 Augmented-CAID includes all reportees CAID shows enrolled and, in addition, reportees indicated with Medicaid coverage in response to questions asked about additional coverage or in response to a verification question asked about each household member who had not been reported having any coverage. 7 Many times SCHIP and Medicaid share the exact same program name (e.g., Hoosier Healthwise in Indiana) and as a result are inseparable in the self-reported health insurance data. 4

7 - The raw count includes individuals receiving partial benefits (i.e., coverage only for selected services). In contrast, it is unclear if some respondents in the survey understand coverage as meaning having received at least some medical care provided by Medicaid. See Appendix II for details on how we decided to define coverage. Differences in the definition of universe - The raw count includes residents of institutional group quarters. In contrast, the CPS estimates do not account for them because they are not part of the CPS sampling universe. Differences in the definition of the unit of measure - The raw count is a measure of the total number of Medicaid client accounts and so individuals who have multiple client accounts (sometimes within the same state and sometimes in multiple states) are counted more than once. In contrast, the CPS estimates are a measure of the total number of individual enrollees. For these reasons we adjusted the raw count to make the measures more comparable. Appendix V, Table 2 shows a progression of adjusted counts, working from the one least comparable to the one most comparable to CPS estimates. Each subsequent column represents an exclusion being made to work towards a universe more directly comparable and the exclusions are cumulative: Total A excludes only persons known dead prior to March of the survey year (For 2000-CPS/ASEC 2001, 853,226 dead persons were excluded; for CPS/ASEC-2002, 905,525 dead persons were excluded) from the raw counts of client accounts. With Total B SCHIP clients are excluded from the counts. With Total C partial benefit clients are excluded from the counts. With Total D clients likely to reside in institutional group quarters are excluded from the counts. Our methodology to exclude persons in group quarters reviews other administrative data files and excludes individuals at addresses we know to be of institutions. This methodology has several known shortcomings: 1. We cannot determine the identity (PIK substituting for SSN) of about 10% of enrollees shown on and therefore cannot look for institutional addresses for these persons. 2. Many persons (31.2% for 2000 and 29.2% for 2001) who are identified (by PIK) nevertheless cannot be located in our alternative administrative data. 3. Many residents of institutions are recorded in administrative records at previous addresses or at addresses of persons responsible for their affairs. 4. We cannot identify all institutional addresses as such. 5

8 Because of these shortcomings, we believe this exclusion is of limited efficacy in identifying residents of institutional group quarters. Our intention is to more thoroughly attempt to account for these persons in our Phase III analysis. With Total E duplicative client accounts are excluded, and this count is of unique individuals as best as we can determine 8. We believe that Total E is the most similar to the estimate of Medicaid enrollees coming from CPS. Nonetheless, we did make a final exclusion: with Total F we exclude cases with missing person-identifying information to simplify reckoning of the source of discrepancy. CPS data offer several different ways to compute estimates of Medicaid coverage, depending on the data users needs. We consider the discrepancy with in terms of three CPS estimates 9 : The first estimate [shown in Table 2 as CPS Total A ] is a weighted tally of the MCAID variable. This is the variable that Census uses to generate Medicaid enrollee estimates. It is derived from a re-coding of several health insurance questions and categorizes persons identified as enrolled in Medicaid, SCHIP, other public coverage, and in some cases, simply as Medicaid enrollees. The estimate from MCAID can be considered a ceiling estimate for the number of Medicaid enrollees. The second estimate [shown in Table 2 as CPS Total B ] is a weighted tally of the Augmented-CAID 10 variable. Augmented-CAID is set to show Medicaid coverage whenever a reportee is explicitly described as having had Medicaid coverage or can be expected to have been so described if the respondent properly completed the health 8 Dual state reportees are counted only once and are not assigned a state for the state-level tallies. 9 It should be noted that, in some instances, respondents reported as Medicaid enrolled in the reference year children born subsequent to the reference year but prior to the survey interview. Certainly, these children could not have actually been enrolled in Medicaid during the reference year since they were not yet born, and therefore these are false positive reports. The original-weighted tally of these children is as follows: Survey Year Reference Year CPS Column A CPS Column B CPS Column C , ,324 44, , ,680 54, One significant difference between the Augmented-CAID variable (tallied in CPS Column B) and the MCAID variable (tallied in CPS Column A) is that MCAID variable is set (indicating Medicaid coverage) through the CPS coding procedure when a respondent indicates a child is enrolled in a State Children's Health Insurance Program (SCHIP), whereas this is not the case for the Augmented-CAID variable. In fact, some SCHIP programs, called Medicaid-expansion SCHIPs, are part of Medicaid, but others, called stand-alone SCHIPs, are not. Sometimes the CPS questionnaire fills are the exact same for SCHIP and Medicaid programs and sometimes not. Not all standalone SCHIP programs have a separate fill and not all Medicaid-expansion SCHIP programs share the same program name. Some states have one or the other of these types of programs, but some also have both, so not only is it quite likely that some respondents are uncertain about the Medicaid status of their children enrolled in these programs, but it is not straight-forward for analysts to determine if a CPS response indicating coverage in SCHIP should be identified as indicating Medicaid coverage as well. This is a limitation of our analysis, and should be considered when reviewing the tabulations in this report, particularly for persons under 18 years of age. It also suggests that neither MCAID (and Table 2, CPS Column A) nor Augmented-CAID (and Table 2, CPS Column B) can be assumed to be better aligned conceptually with actual coverage in Medicaid, and likely there is some type of in-between classification and corresponding estimate that would be better. 6

9 insurance question battery (in this case the response is said to be edited or imputed). The estimate from Augmented-CAID can be considered a floor estimate for the number of Medicaid enrollees. The third estimate [shown in Table 2 as CPS Total C ] is a weighted tally of Augmented-CAID but only for explicit responses. Persons with Augmented-CAID set from an edit or imputation are specifically excluded from this tally so the tally should not be considered a valid estimate of the number of Medicaid enrollees. Instead it should be used to evaluate the degree to which the values under CPS Total B are derived from explicit reporting. Assessing the cause of the count-estimate discrepancy Q3: How did CPS respondents report health insurance status for individuals who are known Medicaid enrollees? For individuals shown in to have had Medicaid coverage, how do their CPS Medicaid reports differ overall and categorically by various demographic characteristics, Medicaid enrollment patterns, other program participation, medical service utilization, and relationship to reference person 11? How do these vary among enrollees whose Medicaid enrollment status was reported, edited, or imputed, and between enrollees with full benefits or any benefits? From linked /CPS/MAX records, we generate Table 3 (Appendix V). Table 3 is presented in eight versions to show how reporting varies by source of health insurance data (reporting, imputation, or editing) and weighting schemes (using original weights or modified weights developed from the re-weighting process): Table 3 Table 3 Version: Page # Enrollees with Full Benefits: A. Full Benefit Enrollees B. Full Benefit Enrollees Whose CPS Insurance Status Was Edited C. Full Benefit Enrollees Whose CPS Insurance Status Was Imputed D. Full Benefit Enrollees Whose CPS Insurance Status Was Reported Enrollees with Any Benefits: E. All Enrollees F. All Enrollees Whose CPS Insurance Status Was Edited G. All Enrollees Whose CPS Insurance Status Was Imputed H. All Enrollees Whose CPS Insurance Status Was Reported The reference person is the first person identified by the respondent as having their name on the title or lease of the residence. 7

10 Q4: How did CPS respondents report health insurance status for individuals who are not known Medicaid enrollees? For CPS reportees not known to be Medicaid enrollees (primarily if no matching record is found, but also if record found shows no Medicaid enrollment for example, if the represented person is enrolled only under a stand-alone SCHIP), how does Medicaid enrollment reporting differ overall and categorically by various demographic factors, other program participation, and relationship to the reference person? How do these vary among reportees whose Medicaid enrollment status is reported, edited, or imputed and between matchable reportees (those with known SSN) and unmatchable reportees (those without known SSN)? From CPS records unmatched to or matched to but not showing enrollment, we generate Table 4 in Appendix V. For the original weight presentation, we present Table 4 in eight versions. For the re-weighted presentation, since unmatchable reportees are assigned a zero weight, the versions for unmatchable reportees are suppressed because otherwise they would be filled with zeros. The remaining versions are identified identically to those in original weight presentation, but page numbering is altered. 8

11 Orig. Wgt. Re-weighted Presentation Presentation Table 4 Version: Page # Page # Unmatchable Reportees: A. All Unmatchable Reportees N/A B. Whose CPS Insurance Status Was Edited N/A C. Whose CPS Insurance Status Was Imputed N/A D. CPS Insurance Status Was Reported N/A Matchable CPS Reportees not Known Receiving Medicaid: E. All Reportees Not Known Receiving Medicaid F. Whose CPS Insurance Status Was Edited G. Whose CPS Insurance Status Was Imputed H. Whose CPS Insurance Status Was Reported Q5: What does multivariate modeling tell us about the relative strength of factors related to the odds of a discrepancy between CPS and? What factors affect the odds a known enrollee is misreported as not having Medicaid coverage? What factors affect the odds a known non-enrollee is misreported as having Medicaid coverage? What factors affect the odds a known enrollee is misreported as having been uninsured? What factors affect the odds a person reported as not having coverage was found to be an enrollee? What factors affect the odds a person reported as having coverage was found to be an enrollee? We specified a series of logistic regressions to model the odds of response error in reporting about Medicaid coverage in the CPS and to impute actual Medicaid enrollment status from CPS data: Model 1a (n=38,388) Modeling Reported Not on Medicaid for Known Full Benefit Enrollees Model 1a is a behavioral model of the event that the CPS respondent explicitly reported no Medicaid coverage (Augmented-CAID is not set) for an enrollee. The universe is CPS reportee whose insurance status was explicitly reported (i.e., not imputed or edited) and for whom shows received full Medicaid benefits during the survey reference period. Model 1b (n=265,839) Modeling Reported on Medicaid for Reportees Whose Enrollment Count Not Be Confirmed in Model 1b is a behavioral model of the event that the CPS respondent explicitly reported Medicaid coverage (Augmented-CAID is set) for a non-enrollee. The universe is CPS reportees with an explicit survey report about a PIK but no matching record found in. While this model is intended to explain the occurrence of false positive 9

12 Medicaid reporting, it is important to note, that false positive reporting is only one reason why we could not confirm enrollment for these persons; this universe also includes persons whom we could not confirm enrolled because the records showing their enrollment are un-linkable (i.e., the SSN on those records were missing, wrong, or unverifiable). It is also possible that these persons had some other government sponsored health insurance coverage, such as from a stand-alone SCHIP, which are not actually part of the Medicaid program, but work in a similar way. For this reason, results from this model are more suggestive than conclusive about factors related to false positive reporting. Model 2 (n=38,388) Modeling Reported Not Insured for Known Full Benefit Enrollees Model 2 is a behavioral model of the event that the CPS respondent explicitly reported no insurance coverage (Medicaid or other types) for an enrollee. The universe is CPS reportees whose insurance status was explicitly reported (i.e., not imputed or edited) and for whom shows received full Medicaid benefits during the survey reference period. This model is identical in set-up to Model 1a, except the dependent variable is reported with no insurance rather that reported with no Medicaid coverage. Model 3a (n=311,308) Modeling Enrolled with Full Benefits for Persons Reported Not Enrolled Model 3a is an imputational model of the event that persons with no CPS indication (explicit, edited or imputed) of Medicaid coverage are found in to have received full Medicaid benefits. The universe is CPS reportees with Augmented-CAID not set. Model 3b (n=35,241) Modeling Enrolled with Full Benefits for Persons Reported Enrolled Model 3b is an imputational model of the event that persons with a CPS indication (explicit, edited, or imputed) of Medicaid coverage (Augmented-CAID is set) are found in to have received full Medicaid benefits. The universe is CPS reportees with a CPS indication of Medicaid coverage. Covariates (showing file source and models used in) entered into the equations are: Age (Source: CPS; in Models 1a, 1b, 2, 3a, and 3b) Enrolled in Survey Month (Source: ; in Models 1a and 2) Intensity of (Source: ; in Models 1a and 2) Last Month of (Source ; in Models 1a and 2) Male (Source: CPS; in Models 1a, 1b, 2, 3a, and 3b) Medicare Crossover (Source: MAX; in Models 1a and 2) Private Insurance (Source: MAX; in Models 1a and 2) Race/Hispanicity (Source: CPS; in Models 1a, 1b, 2, 3a, and 3b) Ratio to Poverty Level (Source: CPS; in Models 1a, 1b, 2, 3a, and 3b) Relationship to Survey Reference Person (Source CPS; in Models 1a, 1b, 2, 3a, and 3b) Service (Source: MAX; in Models 1a and 2) Source of CPS Insurance Data (Source: CPS; in Models 3a, and 3b) 10

13 SSI Recipient (Source: ; in Models 1a and 2) State (Source: CPS, ; in Models 1b, 2, 3a, and 3b) TANF (Source: MAX; in Models 1a and 2) Zero Family Income Reported (Source: CPS; in Models 1a, 1b, 2, 3a, and 3b) See Appendix III for complete variable specification. IV. Results of Analysis Note that results are analyzed in terms of enrollment during calendar year 2000 as reported in CPS/ASEC We believe that results for the subsequent year largely are consistent and can be explained in terms of the same factors. Question 1: Sample loss is a serious problem. Without creating new weights to account for invalid or missing SSNs, we would not be able to calculate a legitimate estimate of the undercount. More than one-fifth of the CPS sample cannot be used in the match analysis because respondents refused to provide an SSN for them or provided one that cannot be validated. There are 218,269 reportees (representing million persons) for CPS/ASEC For 16,096 (representing 20.2 million persons), the SSN provided is invalid and for 28,206 (representing 45.5 million persons), the respondent did not provide one or allow a look-up. For this reason, we generated a second set of statistical tables that assign the weight of the unmatchable CPS sample to the matchable sample in a way intended to limit biases caused by re-weighting. Question 2: For CY 2000, the gross CPS underestimate of Medicaid enrollees is either 34.4% 12 (if CPS MCAID variable is used) or 42.1% 13 (if the Augmented-CAID variable is used). These rates are computed using a total count of 45,039,478 uniquely identified clients in the CY 2000 file (without a known date of death prior to March 2001). Direct comparability is obscured for several reasons discussed above (Under Section III, Q2). To make the count more comparable to the CPS Medicaid enrollment estimates; adjustments are made as follows for 2000: 1. Remove from count clients enrolled under SCHIP: reducing the count to 43,654,866 (Table 2, CY 2000, Original Weight, U.S. Total, Total, Total B). 2. Remove clients not receiving full benefits: reducing the count to 39,735,764 (Table 2, CY 2000, Original Weight, U.S. Total, Total, Total C) ,533,238 (CY 2000, Table 2, Original Weight, U.S. Total, Total, CPS Total A) / 45,039,478 (CY 2000, Table 2, Original Weight, U.S. Total, Total, Total A) ,072,203 (CY 2000, Table 2, Original Weight, U.S. Total, Total, CPS Total B) / 45,039,478 (CY 2000, Table 2, Original Weight, U.S. Total, Total, Total A). 11

14 3. Removing clients residing in group-quarters: reducing the count to 39,576,713 (Table 2, CY 2000, Original Weight, U.S. Total, Total, Total D). 4. Removing duplicate client accounts: reducing the count to 38,170,103 (Table 2, CY 2000, Original Weight, U.S. Total, Total, Total E). Using this count, we find that CPS underestimates Medicaid enrollment by either 22.6% 14 with MCAID or 31.7% 15 with Augmented-CAID. An additional value of 36,216,390 (Table 2, CY 2000, Original Weight, U.S. Total, Total, Total F), excludes UN-identified clients from the previous. This simplifies reckoning the source of discrepancy by excluding non-linkable clients from consideration. By correspondingly excluding unlinked CPS reportees as well as those with edited or imputed responses for Augmented-CAID, the Augmented-CAID count is reduced to 18,416,133 (Table 2, CY 2000, Original Weight, U.S. Total, Total, CPS Total C). Excluding non-matchable records and non-matched CPS records, the discrepancy between the count and the Augmented-CAID estimate is 36.2 M M = 17.3 M. Of this, M 18.9 M = 14.5 M is due to the misclassification of Medicaid enrollment status among known Medicaid enrollees. This accounting is non-authoritative, but shows that the predominating factor associated with the discrepancy is the CPS misclassification of Medicaid enrollment status. We can disaggregate the misclassification among known Medicaid enrollees as follows: Reported Enrollment Misclassified M 19 Imputed Enrollment Misclassified M 20 Total Enrollment Status Discrepancy M Because imputation is meant to match reality at the aggregate level and not the individual level, the discrepancy attributable to imputation is to be expected and is offset by persons imputed to be enrolled, but not enrolled, which may be as high as 2.5 M 21 Also, because imputation is ,533,238 (CY 2000, Table 2, Original Weight, U.S. Total, Total, CPS Total A) / 38,170,103 (CY 2000, Table 2, Original Weight, U.S. Total, Total, Total E) ,072,203 (CY 2000, Table 2, Original Weight, U.S. Total, Total, CPS Total B) / 38,170,103 (CY 2000, Table 2, Original Weight, U.S. Total, Total, Total E). 16 CY 2000, Table 2, Original Weight, U.S. Total, Total, Total F ,355,301 (CY 2000, Table 3, Re-Weighted, Version A, Total Weighted Count, Medicaid Only) + 5,733,020 (CY 2000, Table 3, Re-Weighted, Version A, Total Weighted Count, ). 18 CY 2000, Table 3, Re-Weighted, Version A, Total Weighted Count, Total ,725,677 (CY 2000, Table 3, Re-Weighted, Version D, Total Weighted Count, Total) - 11,706,737 (CY 2000, Table 3, Re-Weighted, Version D, Total Weighted Count, Medicaid Only) 4,011,708 (CY 2000, Table 3, Re-Weighted, Version D, Total Weighted Count, ). 20 5,146,577 (CY 2000, Table 3, Re-Weighted, Version C, Total Weighted Count, Total) 875,551 (CY 2000, Table 3, Re-Weighted, Version C, Total Weighted Count, ) 928,069 (CY 2000, Table 3, Re-Weighted, Version C, Total Weighted Count, ) ,008 (CY 2000, Table 4, Re-Weighted, Version G, Total Weighted Count, ) + 1,599,353 (CY 2000, Table 4, Re-Weighted, Version G, Total Weighted Count, ). 12

15 modeled solely from CPS interview responses, the discrepancy likely results directly from respondent misreports. This failure to correctly report Medicaid for known enrolled reportees causes CPS to overestimate the number of U.S. residents with no insurance. Since these misreported reportees are counted (on a weighted basis) as non-insured rather than Medicaid-insured, an improved estimate of non-insured residents would re-classify them out of the uninsured category. In addition to the enrolled reportees we know are misclassified by CPS, there likely are enrollees whom we cannot identify as misclassified because we do not know their identity (that is PIK, replacing SSN) so cannot see that they too had been enrolled. But, it is also reasonable to expect that there are reportees who are classified as insured when they are not, which partially offsets the underestimation caused by those for whom Medicaid coverage is not reported. An improved estimate of uninsured persons would need to take into proper account all these factors. The team will release such an accounting subsequently to this report. Question 3: Table 3, CY 2000 Expanded Sample, Re-Weighted, Version D (in Appendix IV) shows that misreporting increases as poverty, recentness of enrollment, and length of enrollment diminish. The chart below shows a clear trend relating reported enrollment and poverty level: Percentage Reported Enrolled Among Known CY 2000 Medicaid Enrollees Receiving Full Benefits Percentage Reported Enrolled % 50-74% 75-99% % % % % 200%+ Ratio to Poverty Level Also, we see marked trends showing that enrollment is better reported the earlier that enrollment was initiated, or if the person is no longer enrolled, the later enrollment was terminated: 13

16 Percentage Reported Enrolled Among Known CY 2000 Medicaid Enrollees Receiving Full Benefits Percentage Reported Enrolled Not Enrolled When Surveyed (As of) Enrolled When Surveyed (Since) CY 2000 Quarter Related to the above trend, there appears to be a relationship between the number of days that a person was enrolled in the correct reporting of that enrollment for CPS: Percentage Reported Enrolled Among Known CY 2000 Medicaid Enrollees Receiving Full Benefits Percentage Reported Enrolled < > 180 Enrolled in Survey Year Not Enrolled in Survey Year Days Enrolled CY 2000 Also notable is that enrollment is better reported for persons enrolled in the year the survey was administered compared to reportees who were no longer enrolled. Another blatant result emerging from review of Table 3 is that the correct reporting rate of enrollees in a fee-for-service plan receiving medical services in calendar year 2000 is 61.1%, but for those not receiving medical services it is 27.2%. Similar differences are seen for persons enrolled in managed care: 14

17 Percentage Reported Enrolled Percentage Reported Enrolled Among Known CY 2000 Medicaid Enrollees Medical Service Not Noted (*Not all Managed Care Services are Captured in MAX) No Managed Care Type of Some Managed Care* Medical Service Received To attempt to isolate the effect of the factors the tables above show are important in the bivariate setting, we perform multivariate analyses and report the results below Question 5: The results from the regression analysis are presented in Appendix V, Table 5. They show we can readily distinguish sample persons who have a high likelihood of having their coverage misreported from those with a low likelihood, and we can predict with some accuracy who was enrolled in Medicaid during the reference period and who was not. Model 1a: Modeling Reported Not on Medicaid for Known Full Benefit Enrollees: The results from Model 1a show that level of poverty and last month of enrollment have the biggest effects on the odds 22 that a CPS respondent misreported an enrollee as not having Medicaid coverage. The poverty relationship appears monotonic: higher poverty being associated with lower odds of misreport. Service (received under Medicaid), specified based on enrollment in managed care or fee for service, also has a strong effect on odds. For example, the odds of misreport among individuals who received no medical services and were not enrolled in a managed care program are more than five times the odds of misreport among enrollees who were in managed care and had medical services noted. Another notable result is the lower odds of misreport for young enrollees and those with more intense coverage. 22 Ratio of highest odds to lowest odds among categories defined by a variable. Note that for the last month of enrollment, the specified odds ratio must be taken to the eleventh power (representing the difference from 1 month to 12 months) to determine a comparable value. 15

18 Model 1b Modeling Reported on Medicaid for Reportees Whose Enrollment Could Not Be Confirmed in : In general, the effect of any one factor appears larger than in Model 1a, presumably due to fewer factors being included in the model. As in 1a, we see that poverty level has the biggest effect on the odds of respondent misreport. Here, this variable has the opposite effect: lower income is associated with higher risk of misreporting. Age also has a large effect on the odds of respondent misreport. In contrast with Model 1a, there appears to be no consistent pattern in misreport rates across age groups. There is a striking difference in odds for the lowest age category compared to all other age groups except 65 and older, which suggests the possibility that children enrolled in stand-alone SCHIP are being falsely reported as enrolled on Medicaid. There is more evidence for this possibility in the higher odds associated with being in the Child category compared to other categories of the variable for Relationship to Reference Person. Also, the results show respondents have substantially higher odds of misreporting an old person (age 65 and older) than a younger adult (age 18 to 44 and age 45 to 64), maybe because respondents are confusing Medicaid with Medicare. Also there are substantially higher odds of misreport for black and Hispanic non-enrollees than for others. Model 2 Modeling Reported Not Insured for Known Full Benefit Enrollees: Old and young enrollees have lower odds of being reported as having no insurance. This may relate to respondents being more aware of coverage opportunities for these groups because there are public insurance programs such as Medicare and SCHIP, dedicated largely to covering old and young people. Also, enrollees with more intense and more recent coverage have lower odds of being misreported as uninsured. Similarly, enrollees who received services have lower odds. It is interesting to note that poverty has substantially less effect on odds in this model than in Model 1a. This suggests the possibility that poverty represents a proxy for being less likely to have had private insurance in the reference period, with those persons having had both private insurance and Medicaid often being reported as having had only the private insurance. Model 3a Modeling Enrolled with Full Benefits for Persons Reported Not Enrolled: Not surprisingly, the poverty variable stands out as the best for imputing true enrollment status from CPS data for people who were reported to have no coverage. 23 This reflects 23 In terms of effect on odds ratio. See previous note. 16

19 income limits in Medicaid eligibility rules. There are also much higher odds of actual enrollment among young children than for other age groups. This could reflect program rules and the circumstances surrounding children s enrollment in Medicaid, namely the inability of their parents to find work while caring for them, which in-turn decreases their family s income. Model 3b Modeling Enrolled with Full Benefits for Persons Reported Enrolled: As in Model 3a, the poverty variables stand out as the best among the 3b variables for imputing true enrollment status from CPS data for people who were reported as having coverage, but again, it is difficult to distinguish how much this result relates to group differences in rates of enrollment (in in other assistance programs that respondents may confuse with Medicaid) versus differences in rates of respondent error. Also, in this model, more so than the others, state of residence has a sizeable impact on the odds. Since what is being modeled is true positive reporting and states vary in how they administer and name their subsidized insurance programs (Medicaid, SCHIP, MEDCAL, MEDIKAN ) and other types of other of assistance programs that may be confused with Medicaid, this suggests that how states administer and name programs may impact reports about coverage. Summarizing the results across the models, we see that respondents appeared to report more accurately in the following circumstances: Reporting about white, non-hispanic people. Reporting about people they can be expected to know more about (i.e., their spouse as opposed to their grown child or other person). Reporting about enrollees with more intense and more recent Medicaid coverage. Reporting about enrollees who can be expected to have more contact with the Medicaid program, given their type of coverage and receipt of medical services. The best CPS variables for imputing true Medicaid status are the following: Poverty Race/ethnicity Relationship to reference person Age We recognize that we are limited in how we interpret these results. Most obviously, the results are dependent on the particular variables in each model, both in terms of how they relate to one another and to hidden factors. For example, the results showing more error in reporting about blacks and Hispanics may relate to a hidden factor such as differences in household composition. Since we have not yet replicated these findings in studies of other surveys, we do not accept the results as conclusive evidence for any particular hypothesis. 17

20 V. Conclusion Our research shows the CPS underestimates are mainly a reporting error problem: Reportees on Medicaid during the reference period are often reported as not having been covered by it. Some even report being uninsured but the majority of the misclassified cases report some other type of coverage just not Medicaid. Our regression analysis shows that the major predictors of coverage are length of time, recency and poverty level. Our research is limited by the fact that we were not able to attempt a link with all the cases due to missing identifiers on about 10% of the records. We could also not attempt to link all of the CPS cases due to missing identifiers (26% or so). Additional work is being done to see (though modeling) how our results reported here would be different if we had linking data for all data records. Also, we are conducting an investigation into how the universes from the CPS and match up with respect to institutional group quarters (Phase III of this project). With Phase IV, we plan to compare the results from the CPS to the NHIS. VI. Limitations Our ability to draw conclusion is limited by the quality of our input files, particularly, the success and accuracy of the algorithms (within the Person Identity Validation System PVS) used to identify people on and CPS, and CPS sample error. In Phase I of this study we investigated the quality of the file. The has some known quality limitations: incomplete and incorrect person identification (using the SSN field), person duplication (within and among states), and, from the perspective of our analysis, the inclusion of institutionalized persons, who are out of scope for the Current Population Survey. We have attempted to mitigate these factors in this analysis, but to a certain degree they impact results of the analysis presented here. This issue of non-identification is serious and occurs also on CPS data and will we discuss this subsequently in this section (two paragraphs down). Duplication on is largely manageable by counting only once for persons with multiple client accounts, when the duplication is apparent (that is, the records have a common PIK, which replaces SSN). To the degree that we cannot perform this un-duplication because we do not recognize that several client accounts belong to the same person, our count of enrollees will be somewhat overstated. Also tending to cause an overstatement is the inclusion of institutionalized Medicaid enrollees within the file. We have sought to mitigate this issue through a procedure that looks to see if addresses available for enrollees on alternative available administrative record data are for institutions. However, we believe that this procedure was largely inadequate (see the discussion in Section III under Q2) and will attempt a better accounting for these CPS-out-of-scope persons in the Phase III analysis. 18

21 In addition to the known quality factors, the accuracy of in describing the enrollment status of Medicaid enrollees is critical to the validity of this analysis. To some degree we were able to review the quality of the in this regard in the Phase I analysis by looking at the quality of its geographic and dual-eligibility coding. These suggest that enrollment data are of high quality. However any inaccuracies in enrollment data will propagate to our generated statistics. We believe that our person-identification routines (which seek to identify each person record with a unique PIK) perform with little error because the quality of their results have been continually evaluated over several years of use with varying types of data. Nevertheless, the lack of complete link-identification data (SSN re-coded as PIK) on both the CPS and mean that statistics derived from the status of the match of these files need to be carefully interpreted. Nonidentified CPS reportees cannot have their enrollment status confirmed or established by reference to. For this reason, prior to re-weighting, non-identified CPS records are not useful for analyzing the accuracy of CPS reported Medicaid enrollment status, and the identified CPS records do not represent the whole CPS frame: non-institutionalized U.S. residents. Reweighting corrects for this incomplete representation by shifting weight from the non-identified to identified reportees, but this introduces bias to match-derived statistics to the (unknown) degree that the missing-at-random assumption does not hold within the re-weighting strata. Nonidentification within means that persons reported Medicaid-enrolled to CPS but not confirmed so by the match, may potentially be actual enrollees, and should not be assumed necessarily to be false positives. Like all estimates derived from sample, ours are subject to the uncertainty engendered by the sampling: that is, the sample error. We do not present measures of possible sample error within the tables. Instead, we reference the Census Bureau s standard procedure for estimating error through generalized variance functions: (see particularly, page 7). These functions give their users an idea about the level of error associated with each statistic but on a statistic-by-statistic basis may be substantially inaccurate. We should also point out that the logistic regression techniques we used are in application as much an art as a science, and therefore it is likely that with additional effort or expertise that they can be improved upon. Still we think they are useful for identifying factors associated with reporting accuracy and for predicting enrollment status from survey results. 19

22 VII. Appendices: I. Description of Four Project Phases II. Key Decisions Leading to Phase II Research Design III. Files and Methods Used IV. Meta Data in Phase II V. Detailed Tables and Documentation 20

23 APPENDIX I: Project Background In 2004, Dr. Mike Davern from the State Health Access Data Assistance Center (SHADAC), University of Minnesota was funded by the Robert Wood Johnson Foundation to match enrollment data from seven states to the Current Population Survey (CPS) to study why the large discrepancy exists between CPS estimates and Medicaid enrollment counts. During the summer of 2004, Dr. Davern approached Census Bureau and Center for Medicare and Medicaid Services (CMS) officials about participating in the study. Realizing the amount of time that it would take to acquire the state data, Census officials offered the idea of conducting a national match first using a file they had already acquired: Medicaid Statistical Information System () file. Census officials further suggested that a precursor to the national study could be a simple quality check on both the and the Medicare Enrollment Database (MEDB) files, which was in scope of current agreements and systems of records notices with CMS. The Federal Health and Human Services Department (HHS) Office of the Assistance Secretary for Planning and Evaluation (ASPE) and SHADAC provided additional funding for the process. This money allowed for the national match and provided an opportunity to analyze the National Health Interview Survey (NHIS) and therefore bring its sponser, the National Center for Health Statistics (NCHS), into the study. Note that since NHIS asks a point-in-time question about Medicaid enrollment versus the CPS s Have you been on Medicaid anytime in the calendar year?, the impact of timing (both for the reference period and its relationship to the moment of survey fielding) can be examined by comparison of results. Hence, the SNACC Team emerged, which is an acronym for the first initial of each of the participating agencies. After the first faceto-face meeting on the project, the SNACC team agreed to break the study into four distinct, but related, phases: Phase І: Merging the National Level CMS Databases In Phase I, we create a national database of health-insurance enrollment and evaluate the quality of the information it contains. We create the database by merging the CMS Medicaid Statistical Information System () file with the CMS Medicare Database (MEDB) file (see Appendix III for an explanation of the research and supporting files). We evaluate the quality of the database by assessing our ability to accurately merge the input files and by comparing the characteristics of the individuals in the database to expectations based on Medicaid eligibility rules and characteristics of the U.S. population. (For more information, please see Phase I Research Results: Overview of Medicare and Medicaid Files, February 2007.) Phase II: Matching the to the Current Population Survey (CPS) In Phase II, we match data from and the Current Population Survey (CPS) according to a unique person-identifier (the Protected Identity Key or PIK, which replaces Social Security Number or SSN to protect reportee privacy). We supplement the matched records with information from the Medicaid Analytic extract (MAX), the Person Characteristic File (PCF), and the Master Address File Auxiliary Reference File (MAF-ARF) and examine why there are discrepancies between records of enrollment and CPS reports of Medicaid coverage (See hypotheses outlined below). 1