THE MEASUREMENT OF MEDICAID COVERAGE IN THE SIPP: EVIDENCE FROM CALIFORNIA, David Card Andrew K. G. Hildreth Lara D.

Transcription

1 THE MEASUREMENT OF MEDICAID COVERAGE IN THE SIPP: EVIDENCE FROM CALIFORNIA, David Card Andrew K. G. Hildreth Lara D. Shore-Sheppard This project was made possible by the cooperation of the California Department of Health Services (DHS) and the U.S. Census Bureau. We are grateful to Gene Hiehle and James Klein of DHS and David Illig of the California Health and Human Services Agency for assistance and guidance in the use of DHS data. We also thank John Abowd, Julia Lane, Ron Prevost, and Lars Vilhuber of the U.S. Census Bureau for their assistance with SIPP data. Barrett Graf provided outstanding research assistance. Funding for part of this project was provided by the California Program on Access to Care (grant number ) and by the Joint Center for Poverty Research/U.S. Census Bureau Research Development Grant Program. All opinions are those of the authors and do not reflect the views of the U.S. Census Bureau or the California Department of Health Services.

2 The Measurement of Medicaid Coverage in the SIPP: Evidence from California, David Card, Andrew K.G. Hildreth, and Lara D. Shore-Sheppard September 2001 JEL No. I38, C81 ABSTRACT This paper studies the accuracy of reported Medicaid coverage in the Survey of Income and Program Participation (SIPP) using a unique data set formed by matching SIPP survey responses to administrative records from the State of California. Overall, we estimate that the SIPP underestimates Medicaid coverage in the California population by about 10 percent. Among SIPP respondents who can be matched to administrative records, we estimate that the probability someone reports Medicaid coverage in a month when they are actually covered is around 85 percent. The corresponding probability for low-income children is even higher at least 90 percent. These estimates suggest that the SIPP provides reasonably accurate coverage reports for those who are actually in the Medicaid system. On the other hand, our estimate of the false positive rate (the rate of reported coverage for those who are not covered in the administrative records) is relatively high: 2.5 percent for the sample as a whole, and up to 20 percent for poor children. Some of this is due to errors in the recording of Social Security numbers in the administrative system, rather than to problems in the SIPP. David Card Andrew K. G. Hildreth Lara D. Shore-Sheppard Department of Economics California Census Research Department of Economics 549 Evans Hall Data Center Willams College UC Berkeley 2538 Channing Way Williamstown, MA Berkeley, CA Berkeley, CA and NBER and NBER

3 One of the most widely debated policy concerns in the United States is the adequacy of health insurance coverage for low-income families. Since 1965 Medicaid has been the primary health insurance program for public assistance recipients and other low-income groups. Medicaid was originally designed to provide health insurance for single parent families and the aged, blind, and disabled. Over the past two decades, however, the program has gradually expanded to cover children in lower-income families that are not participating in other welfare programs. 1 Despite these expansions, measured health insurance coverage rates remain far below 100 percent. Data from the Census Bureau s Current Population Survey, for example, show that about 23 percent of people under age 18 who lived in poor households lacked health insurance coverage in the mid-1990s (U.S. Bureau of the Census, 1997). In the wake of recent federal and state-level welfare reforms there have been renewed efforts to maintain and expand Medicaid coverage. 2 Nevertheless, measured health insurance coverage rates have actually drifted downward over the past decade. 3 While analysts agree that expansions in the potential availability of Medicaid have not led to equivalent increases in measured coverage (Shore-Sheppard, 1999; Gruber, 2000), there is less 1 See Gruber (2000) for a description of these expansions and a review of the literature on their effects. 2 Most prominent is the State Children s Health Insurance Program (CHIP), implemented in California as the Healthy Families Program. Not all legislative changes have been in the direction of expanding Medicaid coverage: the 1996 federal welfare reforms substantially restricted Medicaid eligibility for immigrant children (see US House of Representatives Committee on Ways and Means Green Book, 2000, pp ). 3 In 1992, 86.1 percent of all individuals and 87.3 percent of children under 18 had health insurance coverage. In 1999 the corresponding rates were 84.5 percent of all individuals and 86.1 percent of children. (U.S. Census Bureau, 2000). These data are based on tabulations from the Current Population Survey. 1

4 consensus on the reasons for this phenomenon. One simple explanation is that some people fail to report their true Medicaid status to the Current Population Survey (CPS) or the Survey of Income and Program Participation (SIPP) the two key sources of data on health insurance coverage in the U.S. Available evidence suggests that under-reporting of participation in meanstested government programs is significant. For example, estimates reported in the SIPP Quality Profile (U.S. Bureau of the Census, undated) suggest that SIPP respondents under-report participation in Aid to Families with Dependent Children (AFDC), Food Stamps, and Supplemental Security Income (SSI) by 25 percent. Such a high rate of under-reporting could easily account for the absence of measured health insurance coverage among low-income populations who are eligible for Medicaid. 4 In this paper we present new evidence on accuracy of reporting of Medicaid coverage in the SIPP among California residents in the early 1990s. Our analysis is based on a unique matched data set composed of survey information from the 1990, 1991, 1992, and 1993 SIPP panels and administrative data on Medicaid eligibility from the State of California s Medi-Cal Eligibility File (MEF). As described below, MEF data were matched to SIPP records for the roughly three quarters of individuals in the SIPP who provided a valid Social Security number (SSN), or granted permission for the Census Bureau to look up their SSN. Using this matched data file it possible to compare reported versus actual Medicaid eligibility on a month-by-month basis for about 20,000 individuals. 4 For example, Gruber (2000) reports that 32 percent of California children were eligible for Medicaid coverage in 1996, whereas data from the 1993 SIPP show that 29 percent of children (age 16 or younger) were covered by Medicaid in late If Medicaid coverage is underreported by 25 percent, however, then the SIPP data actually imply a takeup rate of over 100 percent. 2

5 The next section of the paper provides a brief overview of the Medicaid program, with special reference to California. In Section II we describe the SIPP survey and present a variety of data on measured Medicaid participation patterns in the SIPP California sample. Section III describes the eligibility file that is the source of our administrative data, and summarizes the matching process. We also present information on the characteristics of the matched sample versus the overall SIPP California sample. Section IV contains our main results, including crosstabulations of reported Medicaid status in the SIPP survey and the MEF for the overall matched sample, and various subsamples, including children. Finally, Section V presents our main conclusions. I. The California Medicaid Program in the Early 1990s Medicaid is a joint state-federal program that pays for medical services for eligible lowincome individuals, including elderly, blind, and disabled recipients of SSI 5 ; the medically needy (people who have recently incurred large medical expenses); and people in low income families. Historically, the latter group was comprised of cash welfare recipients in the AFDC program. Starting in the mid-1980s, however, a series of federal law changes expanded Medicaid eligibility to families with incomes above the AFDC threshold, and others that did not meet the family composition rules of AFDC. The 1989 Omnibus Budget Reconciliation Act (OBRA) mandated that states offer Medicaid coverage to pregnant women and children up to age 6 with family incomes below 133 percent of the federal poverty threshold. OBRA 1990 further 5 California operates a state supplemental program known as the State Supplemental Payment (SSP) program that parallels the federal SSI program. 3

6 expanded coverage to all children born after September 30, 1983 and living in families with incomes below 100 percent of the poverty line. Other legislative changes in the late 1980s and early 1990s allowed states to expand Medicaid coverage beyond these minimum mandates. Many states responded by raising the income thresholds for coverage: California, for example, raised the family income limit for pregnant women and infants to 200 percent of the federal poverty line. Enrollment patterns in the California Medicaid program known as Medi-Cal have closely tracked national trends. During the period from 1990 to 1995 the state accounted for a steady 15 percent of the total Medicaid caseload for the U.S. Further, the ratio of per-capita expenditures in California to the nation as a whole remained relatively constant. In light of this relative stability and the size and diversity of the California population, we believe that the state provides an excellent testing ground for evaluating the quality of survey data on Medicaid coverage. Table 1 reports the various Medi-Cal eligibility categories in effect in California as of late 1995, along with estimates of the number of people and the fraction of the eligible population in each category. Despite the coverage expansions in the late 1980s and early 1990s, more than three quarters of individuals covered by Medi-Cal in 1995 were adults or children who were enrolled in AFDC or SSI. The majority of this group about 60 percent of total Medi-Cal eligibles consisted of AFDC recipients. Another 10 percent were medically needy adults and children; 5 percent were refugees and undocumented aliens; and 5 percent were medically indigent adults and children. Only about 3 percent of Medi-Cal cases in 1995 were women or 4

7 children who were receiving coverage as a result of the poverty-relaterd expansion programs. 6 Given the high fraction of Medi-Cal enrollees whose eligibility is linked to welfare participation, it is not surprising that changes in Medi-Cal enrollment are strongly related to changes in the welfare caseload. Indeed, as shown in Figure 1, changes in Medi-Cal enrollment over the early 1990s were roughly proportional to changes in the AFDC rolls. 7 The recession of the early 1990s led to a steep rise in California s welfare rolls and similar proportional increases in the Medi-Cal-eligible population. Since the middle of the decade welfare rolls and Medicaid coverage have both declined sharply in California, with evidence that most of the fall in Medi- Cal enrollment has been attributable to the fall in the number of families receiving cash welfare (Broaddus and Guyer, 2000). II. Medicaid Coverage Among California Respondents in the SIPP Panels In this paper we study the reporting of Medicaid coverage by California respondents in the 1990, 1991, 1992, and 1993 SIPP panels. Table 2 provides an overview of the four panels. Each SIPP panel consists of four rotation groups who were interviewed on a staggered schedule every four months. Individuals in the 1990 and 1991 panels were interviewed 8 times, individuals in the 1992 panel were interviewed 10 times, and those in the 1993 panel were 6 One reason for the low fraction of the caseload arising from the poverty-related expansions in California is that California has very generous AFDC benefit rates. Consequently, the number of children in families with incomes above the AFDC threshold but below the poverty line is lower than in most states. 7 Over the period, each additional person in the average monthly AFDC caseload in California was associated with 6 additional Medi-Cal enrollees at some point during the calendar year. 5

8 interviewed 9 times. Overall, the four panels contain information for a total of 238,938 people over the period from October 1989 to December Just over 10 percent of the sample (24,681 individuals) were in California in their first interview. A larger fraction about 13 percent of the sample or 31,336 people spent at least one month in California. Table 3 presents some descriptive statistics for the sample of individuals who were in California at the first SIPP interview, and for various subsets of this population including young children (age 6 or under), all children, people living in poor and near-poor families, and people who reported that they were covered by Medicaid in the first survey month. About one quarter of the SIPP California sample are children, and just over 10 percent are elderly. Consistent with national patterns, children are over-represented among the populations of poor and near-poor, and make up close to one-half of Medicaid enrollees. The diversity of the California population is evident in the ethnic composition of the SIPP sample. White non-hispanics account for only 60 percent of Californians, and make up even smaller fractions of children, people in poverty, and Medicaid recipients. Hispanics (of all racial groups) make up one-quarter of the overall California population, and larger shares of children, people in poverty, and Medicaid recipients. An important feature of the SIPP is sample attrition: about 10 percent of individuals who were in the first SIPP interview left the sample by the 6 th interview, and another 3 percent left by the 8 th interview. Attrition rates are about the same for children as for the overall sample, but are higher for people who were in poor or near-poor families in the first interview, or were enrolled in Medicaid. The non-randomness of sample attrition has some implications for our estimates of SIPP response accuracy which we hope to be able to address in future work. In the early and mid-1990s California had somewhat higher poverty rates than the nation 6

9 as a whole, and higher welfare recipiency and Medicaid enrollment rates. 8 On average about 15 percent of the SIPP California sample was poor (i.e., had family income below the federal poverty line), with an even higher poverty rate among children. In view of Medicaid eligibility criteria, it is not surprising that over one-half of Medicaid enrollees are poor, and over 85 percent live in families with incomes less than 200% of the federal poverty line. The bottom rows of Table 3 present Medicaid enrollment data for the SIPP sample. About13 percent of the sample report that they were covered by Medicaid in the first interview. Consistent with the trends in Figure 1, enrollment is rising within the panels over time. More detail is shown in Figure 2, which plots the fraction of SIPP respondents who were enrolled in Medicaid by panel and month. 9 For reference, we have super-imposed a time line that represents the ratio of average monthly Medi-Cal enrollment in each year to the population of the state i.e., the true fraction of the California population enrolled in Medicaid. 10 The data in Figure 2 suggest that Medicaid enrollment is under-reported in the SIPP. Taking a weighted average over all months from October 1989 to December 1996 (weighted by the number of people in the 8 In the early 1980s California poverty rates were below the national average. By the late 1980s, however, the state s poverty rates consistently exceeded the national average. See Card (2001) for a comparative analysis of labor market and poverty trends in California over the 1980s and 1990s. 9 The rates are weighted coverage rates, using a fixed weight for each individual based on his or her weight for the first calendar year of the appropriate panel. We compare alternative weighting procedures below. 10 State population is measured as of July 1 from U.S. Census Bureau data. Medi-Cal enrollment is the average monthly number of people who were ever enrolled during the calendar year. Comparisons for show that average monthly enrollment is very similar to the July enrollment count. One caveat to this calculation is that the state population probably grew faster over the 1990s than was estimated by the Census Bureau. 7

10 combined SIPP California sample in each month) we estimate that the SIPP sample underestimates Medicaid enrollment by 12 percent. A prominent feature of Figure 2 is that average Medicaid enrollment rates vary from panel to panel of the SIPP. For example, the average enrollment rate in January 1993 was 13.3 percent for people in the 1991 panel, 14.0 percent for people in the 1992 panel, and 16.1 percent for people in the 1993 panel. 11 By comparison, the ratio of MEF enrollment to the total California population in January was (approximately) 16.2 percent. We suspect that most of the variation across panels is attributable to variation in the characteristics of people included in the California sample that are correlated with eligibility for Medicaid. 12 In particular, inter-panel comparisons of the fraction of people who live in poverty suggest that the 1990 panel underrepresents poorer families relative to the 1991 and 1992 panels, whereas the 1993 panel overrepresents poorer families. 13 Cross-panel differences in Medicaid coverage rates are narrowed substantially once the comparisons are conditioned on age, ethnicity, and family poverty status The underlying sample sizes are 4,432, 6,105, and 6,597. Using conventional sampling errors, the rates are significantly different from each other. 12 The SIPP sample is not designed to be representative of the California population, and some variation will arise across panels in the composition of the sample relative to the underlying population. 13 We fit a regression for the incidence of poverty to a pooled sample of person-months from the four SIPP panels and included a full set of indicators for the calendar month and dummies for the different panels. Using the 1992 panel as a base, average poverty rates are 3.35% lower in the 1990 panel (standard error 0.19), 0.30% lower in the 1991 panel (standard error 0.15), and 4.00% higher in the 1993 panel (standard error 0.15). 14 Specifically, if we fit a regression model for the incidence of Medicaid coverage to a pooled sample of person-months from the four SIPP panels and include indicators for the calendar month and dummies for the different panels, we find that the probability of Medicaid coverage is 1.94% lower (standard error 0.18) in the 1990 panel relative to the 1992 panel, and 2.91% higher 8

11 The strong link between family income and Medicaid coverage is illustrated in Figure 3, which shows the fraction of individuals in different age ranges who report being covered in a given month as a function of their relative family income (the ratio of their family income to the appropriate poverty threshold). Apart from the lowest income category, the probability of Medicaid enrollment falls steadily with relative income. As would be expected from the nature of the Medicaid eligibility rules summarized in Table 1, enrollment rates conditional on relative family income are highest for young children, somewhat lower for older children, lower still for adult women, and lowest for adult men. The rise in Medicaid coverage rates in Figure 3 between the lowest income group (with family incomes under 50% of the poverty line) and the next category (with family incomes between 50% and 75% of the poverty line) illustrates an important feature of the enrollment process. For a variety of institutional and behavioral reasons (including asset limits, spend-down provisions, and the lengthy and complex Medicaid application process) Medicaid enrollment rates vary with both current and long run or permanent income. It turns out that families in the SIPP whose current monthly incomes are less than 50% of the poverty line have higher long-run average incomes than those in the next highest income group. 15 Thus, the relatively low (standard error 0.15) in the 1993 panel relative to the 1992 panel. Rates in the 1991 panel are not significantly different from those in the 1992 panel. When we augment the model with controls for ethnicity and interactions of a family poverty indicator with dummies for 5 age ranges, the difference between the 1990 and 1992 panels becomes insignificant, and the difference between the 1993 and 1992 panels narrows to 0.91% (standard error 0.13). 15 The average ratio of monthly family income to the poverty threshold for people in the lowest income category is 1.28, whereas the average ratio for people in the next category is Many families that have very low incomes in a month actually report zero or even negative incomes in the month, but much higher incomes in adjacent months. 9

12 Medicaid coverage rate for the lowest income category is explained by the relatively high permanent incomes for people in this category. Apart from this anomaly, average relative incomes over all months of the SIPP are in the same rank order as relative income in any given month, and comparisons across groups based on current monthly income effectively compare Medicaid coverage rates between groups that have similar differences in current and permanent relative income. A well-known feature of longitudinal data collected in retrospective surveys is that measured spells of program participation tend to exhibit so-called seam biases (see e.g., U.S. Census Bureau, undated, chapter 6; Czajka and Olsen, 2000). SIPP participants are interviewed every four months about activities in the immediately preceding four months. Because individuals have difficulty recalling precise dates in the past (Groves, 1989, chapter 9) and for other unknown reasons there is a tendency for changes in status to be measured at the interview seams between the 4 th and 5 th months for example. Not surprisingly, this is true for changes in reported Medicaid coverage. Figure 4 shows the transition rates into and out of Medicaid coverage in each month of the combined SIPP panels. (The sample underlying this figure includes only individuals who were living in California in the current and previous month.) In addition, the figure shows the average fraction of individuals who report Medicaid coverage by SIPP interview month in the pooled sample. 16 Roughly two-thirds of all spell 16 The apparent rise in Medicaid coverage rates in interview months is attributable to the influence of observations from the 1993 SIPP panel. Only the 1992 and 1993 panels have data beyond month 32, and only the 1992 panel has data beyond month 36. Medicaid coverage is about 3-4 percentage points higher in the 1993 panel than in other years. The shift in the fraction of data from this panel (from one-quarter in months 1-32 to one-half in months 33-36) would be expected to raise the average coverage rate by 1 percentage point, which is about the increase observed in Figure 4. 10

13 starting and ending events occur at a seam (versus an expected frequency of 25 percent). Moreover, there is a small but noticeable seam pattern in the rate of Medicaid coverage, with a tendency for higher coverage rates in the most recent months just before the SIPP interview (i.e., interview months 4, 8, 12,...). These patterns provide prima facie evidence of measurement error in SIPP-reported Medicaid coverage. III. Matching SIPP and Administrative Eligibility Data a. Potential Matching of Medi-Cal Data to SIPP Our analysis of reporting errors in Medicaid coverage is based on a matched file composed of SIPP records and administrative data for individuals who appear in the State of California s Medi-Cal Eligibility Data system at some point between June 1988 and June Records were matched using Social Security Numbers (SSNs). Since not all individuals in the SIPP have a valid SSN (or allow the Census Bureau to use their SSN for research purposes), it is important to understand the characteristics of the subsample of SIPP respondents who are eligible for matching. In the first SIPP interview household respondents are asked to provide names and SSNs for all people in the household. Respondents can provide SSNs, or they can refuse to allow their SSN to be used, or they respond that they don t have an SSN or don t know it. 17 In subsequent waves the interviewers try to obtain SSNs for individuals who have not yet provided one. Information for respondents who have not explicitly refused the use of their SSN is forwarded to the Social Security Administration for SSN validation. An attempt is made to assign an SSN 17 This information is provided by parents or guardians for children. 11

14 (using name, sex, and address information) to respondents whose numbers were not reported. For purposes of this project a list of validated SSNs for all individuals in the SIPP panels was searched for matches with SSNs in the administrative file described below. Among the 31,296 individuals in the four SIPP panels who lived in California for at least one month, 76.1 percent had a valid SSN. The fraction is higher (82.2 percent) for the 24,681 people who were living in California at the first SIPP interview. The difference is potentially explained by two factors. First, most of those who are observed in California at some point but not in the first interview are people who joined a household that is already in the SIPP only 7 percent were first interviewed in some other state and then followed to California. The joiners include new-born infants and young children who are presumably shifting between households. 18 As noted below, children are less likely to have a valid SSN. Second, the joiners tend to be in the SIPP for a relatively short time: the median number of months covered by the SIPP interviews is only 10. Thus, interviewers have less chance of obtaining a valid SSN. Table 4 compares the characteristics of three samples of SIPP respondents who were living in California at the first SIPP interview. Column 1 reports the characteristics of the overall sample. Column 2 reports the characteristics of the subsample who have a valid SSN, and are therefore potentially eligible to match to the administrative Medicaid records. Finally, column 3 reports the characteristics of the subsample without a valid SSN. Just over one-third of this group consists of people who refuse to allow the use of their SSN, while the remainder consists of people who either do not have an SSN, or report a invalid SSN that cannot be validated using 18 About 11 percent of the joiners were born during the panel. Another 13 percent were between the ages of 1 and 16 when they joined the panel. 12

15 their name and address information. As shown in Appendix Table 1, a key factor determining whether a valid SSN is available is age: only 69 percent of children under the age of 6 (at the first interview) have a valid SSN, compared to 79 percent of older children and youths (ages 6-24), and over 85 percent of older adults. Because of these differences, the valid-ssn subsample under-represents children relative to the overall sample. Ethnicity is also related to the probability of a valid SSN. About 86 percent of white non-hispanics have a valid SSN, compared to 77 percent of black non- Hispanics, 82 percent of Asian non-hispanics, and 76 percent of Hispanics. Thus, the valid-ssn subsample slightly over-represents white non-hispanics relative to other groups. Individuals with valid SSNs are also less likely to be poor or near-poor than those without an SSN. Sample attrition rates, however, are fairly similar for the two groups. Most importantly for this study, reported Medicaid coverage rates are fairly similar in the subsamples with and without valid SSNs. This equality may seem rather surprising, given that the subsample without SSNs includes a higher fraction of children (who have higher Medicaid coverage rates) and a higher fraction of individuals with low incomes (who also have higher coverage rates). It should be noted, however, that the public assistance and Medicaid systems require SSNs for those who are covered. Thus, nearly all adults and children who receive Medi- Cal coverage should actually have an SSN. On balance, this requirement seems to offset the lower rate of SSNs for children and poor adults, leading to a roughly proportional representation of people covered by Medicaid in the valid-ssn subsample. Based on the comparisons in Table 4 we conclude that the group of individuals in the SIPP who can be matched to administrative Medicaid records via their SSNs is not completely representative of the California population, 13

16 but does include reasonable fractions of children and people from low-income families. b. The Medi-Cal Eligibility File Eligibility for Medicaid coverage in California is established at county social welfare offices through on-line access to a state-wide database maintained by the state s Health and Welfare Data Center. 19 This file has a record for each individual who is currently eligible for Medi-Cal, or was eligible at any time over the previous 15 months. Around the 24 th of each month a snapshot is taken of the eligibility data base: these snapshots are known as the Medi- Cal Eligibility Files (MEFs). Each monthly MEF includes individual identifying information (sex, date of birth, ethnicity, address, Social Security Number) and two key pieces of information regarding eligibility for Medi-Cal coverage in each month: an eligibility code summarizing eligibility status, and an aid code giving the type of program the individual is covered by. 20 We use these codes to determine Medicaid eligibility for each person in each month. For this project the California Department of Health Services granted access to a series of 17 MEFs drawn every six months from July 1989 to July Each file contains data for the current and previous 12 months. These files provide data for the 109 month period from June 1988 (the earliest date covered by the July 1989 MEF) to July The files were shipped 19 Eligibility as used by the state denotes that an individual is enrolled in the program and may receive services paid for by Medi-Cal. 20 Medi-Cal, like other state Medicaid systems, offers different types of coverage to different classes of eligible people. Some individuals expenses are fully covered whereas others have to share costs, or spend a certain amount before they are covered. This introduces some ambiguity in the interpretation of Medicaid eligibility: individuals who are ineligible until they reach a certain level of expenses could be considered covered by health insurance but would be classified as ineligible for Medicaid in the MEF. 14

17 directly to the U.S. Census Bureau, where all records for individuals with Social Security Numbers that matched those of people in the SIPP panels were retained, and coded with an identifier that could be matched to the public use versions of the SIPP data. The MEF records for this subset of matched people were then shipped to the California Census Research Data Center, where we matched them to the SIPP files. An important feature of the MEF records is the overlap in information provided for each person. For example, eligibility information for December 1991 is contained in the January 1992 and July 1993 MEF s, while data for January 1992 is contained in the January 1992 MEF, the July 1993 MEF, and the January 1993 MEF. For a variety of reasons, the information for a given month is not necessarily consistent across MEFs. As noted in Klein (1999), one important source of inconsistency is the fact that Medicaid eligibility can be established after the fact this is particularly likely to affect eligibility under medically needy and medically indigent programs. One simple way of combining data across MEF files is to adopt the rule that the latest information is best : thus, eligibility in any given month is assigned based on the last MEF that covers that month. 21 After close examination of the monthly eligibility patterns in overlapping MEF files we decided on a variant of this rule. Specifically, for any calendar month we use the eligibility data in the most recent MEF, with the exception that we did not use the information provided for the 12 th previous month. 22 Even with this rule, however, the administrative coverage data exhibit a seam bias pattern, suggesting that there is some remaining 21 This is the procedure used by State analysts to construct longitudinal coverage histories. 22 We found many cases where it seemed that eligibility information for the 12 th previous month had been over-written with information from the current month. Use of data for the 12 th previous month led to a relatively high number of 1-month gaps in spells coverage or non-coverage. 15

18 measurement error in assigned coverage. In particular, a relatively high fraction of transitions into or out of coverage occur at months that represent seams between data drawn from different MEFs. Using MEF records for individuals who can be matched to the SIPP, we estimate that roughly 40 percent of spell transitions in any six month period occur at a seam date (months 6, 12, 18,... of the 109 month sample period) versus an expected frequency of 16.7 percent if transitions occurred equally across months. At least some of this administrative data seam bias appears to be due to the way the matching between the MEFs and the SIPP public-use identifiers was done. Each record in a MEF that had an SSN was linked individually to the SIPP, and the matched records were shipped to the California Census Data Research Center. Consequently, if an individual received an SSN after beginning Medi-Cal (so that initial MEF records for that individual did not have an SSN but later records did), then later MEF records for that individual would be included on the matched file, but early records would be missing. Thus it would appear in the matched data as if that individual had started Medi-Cal at the time of a seam. 23 This is most likely to have occurred in children, as they are less likely to have an SSN but would be required to obtain one as they participated in Medi-Cal. We are unable to determine the magnitude of this source of seam bias as we have access only to those MEF records that were successfully matched to a SIPP record. However it is likely that the MEF data probably contain some errors a fact that must be taken into consideration in evaluating the reliability of SIPP-reported coverage. 23 Similarly, if the SSN was missing or incorrectly coded in a later MEF but was present and correct earlier, exit from Medi-Cal would appear to have occurred at a seam. 16

19 IV. Analysis of The Matched File To analyze the accuracy of Medicaid coverage in the SIPP we compare reported coverage rates in the SIPP and MEF file for the same individual in the same calendar month. The analysis is restricted to individuals who report a valid SSN and who were living in California (according to SIPP records) in the month in question. Appendix Table 2 provides a brief summary of the resulting sample. Overall, there are 23,850 individuals in the SIPP panels who reported living in California in at least one month, and who provided a valid SSN. On average, each person in this sample has just under 28 months of valid SIPP interview data, and provided just under 27 months of data while living in California, leading to a total of 642,859 personmonths of potential Medicaid Coverage. The average fraction of months recorded as covered by Medicaid is percent. For individuals who match with a MEF record, we derive MEF-based coverage for each month directly from their administrative data. In months when an individual appears not to be in the MEF system (but is still living in California) we assume that the individual is not covered by Medicaid. Similarly, for individuals who report a valid SSN and are living in California but never appear in the MEF s during the period from July 1988 to July 1997, we assume that the individual was never covered by Medicaid. It is worth emphasizing that any errors in the matching process will lead us to underestimate Medicaid coverage in the MEF. For example, if the wrong SSN is assigned to an individual in the SIPP there is relatively little chance of finding a match in the MEFs, and an 17

20 individual who is covered by Medicaid will be incorrectly coded as uncovered. 24 Perhaps more importantly, if an individual s SSN is mis-coded or missing in the MEF, then no match will be found between the SIPP and MEF records, and an individual who is actually covered by Medicaid will be assigned an uncovered status. We discuss some circumstantial evidence on the extent of this problem below. Table 5 presents the cross-tabulations of MEF and SIPP Medicaid coverage for the overall sample of California residents with valid SSNs, and for various subsamples, including children, individuals in poor or near-poor families, and children in lower-income families. The table contains two sets of entries the upper entry in each cell is based on unweighted data, while the lower entry (in italics) is estimated using the first year sample weights for each person to weight their person-month observations. 25 The first two columns of Table 5 provide some information on the particular subsample the fraction of overall person-months contributed by the subsample, and the fraction of personmonths for the subgroup as a whole attributable to people with a valid SSN. The latter ratio is over 80 percent for the overall sample, but is lower for children and people in poor families, as would be expected given the results in Table 4 and Appendix Table 1. The next four columns show the fractions of person-months in each of four possible categories: covered by Medicaid in 24 There is some possibility that an individual with an incorrect SSN will be matched to a MEF record that falsely indicates Medicaid coverage. Since only about 15% of people are covered in any given month, however, a random match is far more likely to lead to a false negative report of coverage than a false positive report of coverage. 25 These weights do not take account of differential probabilities of having a valid SSN, or of differences in sample attrition rates. In future work we plan to implement alternative weighting schemes that adjust for these two phenomena. 18

21 both MEF and SIPP; uncovered in both files; covered in MEF but not in SIPP; and uncovered in MEF but covered in SIPP. On average just over 4 percent of the person-month observations appear in the two conflicting categories. Interestingly, for the entire sample and for almost all subgroups, the fraction of people who report coverage in the SIPP but are uncovered in the MEF exceeds the fraction who are covered in MEF but fail to report coverage in the SIPP. As a result of this inequality the estimated Medicaid coverage rate for individuals with valid SSNs is higher using SIPP-reported coverage than using the MEF records. Given the evidence in Figure 2 that overall Medicaid coverage rates are understated in the SIPP, this is a surprising conclusion. Under the assumptions that the MEF data are accurate, and that there are no errors in the matching process, the only explanation for this pattern is that SIPP respondents without valid SSNs substantially under-report their coverage. As we discuss below, however, we believe that at least part of the explanation for the lower rate of coverage in the administrative data is incorrect or missing SSNs in the MEF system. The three right-hand columns of Table 5 present statistics that summarize the accuracy of Medicaid reporting in the SIPP. The first is the probability of reported Medicaid coverage in the SIPP survey for people who are currently covered in the MEF. In the overall sample this fraction is just over 85 percent using the unweighted data, and slightly higher using the weighted data. The second is the probability of reported Medicaid coverage in the SIPP for people who are currently not covered in the MEF this false positive response rate is about 2.8 percent for the unweighted sample, and about 2.5 percent using the weighted results. Finally, the last column of the table presents the so-called reliability of SIPP-reported Medicaid coverage: this is the 19

22 regression coefficient of MEF coverage status on observed SIPP coverage status. 26 The reliability index measures the degree of attenuation bias that would arise if observed SIPP coverage status were used as an explanatory variable in a regression model in place of true MEF coverage. For the overall sample the reliability of SIPP Medicaid coverage (as a measure of MEF status) is about 80 percent, or slightly higher using weighted data. Looking at the patterns of SIPP and MEF coverage for children and people in low income families in Table 5, the probability of SIPP-reported coverage for those who are covered in MEF is fairly stable at percent. By comparison, the probability of a false positive SIPP response varies substantially across groups, with a rate up to percent (depending on the use of weights) for children under the age of 5 in poor families. These results suggest that SIPPreported coverage is relatively accurate for those who are actually covered by Medicaid. However, the SIPP has a higher rate of errors for people who are not currently covered by Medicaid, but belong to groups with high average probabilities of coverage. An important caveat to the conclusion that SIPP has a high rate of false positive coverage responses for children is the possibility that MEF coverage status is incorrect. There are two potential sources of errors in MEF coverage status. First, judging by the presence of seam patterns in the coverage transition rates, the MEF records themselves appear to contain some errors. Second, and potentially more important for understanding the apparently high rate of false positive coverage responses in the SIPP, errors in the merging process caused by invalid SSNs in either the SIPP or MEF will lead to cases where an individual is recorded as uncovered in the MEF (because we were unable to find a record for their SSN in the MEF) but is actually 26 The reliability index is P(MEF=y SIPP=y) P(MEF=y SIPP=n). 20

23 covered. Although we are uncertain about the overall probability of such errors, several factors suggest that the error rate is likely to be higher for children. On the SIPP side, SSNs for children are provided by their parents, and may be less accurately reported than the numbers for adults. On the MEF side, many children enter the MEF system without an SSN and are processed while an application for an SSN is pending. Consequently, early months of participation in Medicaid by such individuals, though correctly reported in the SIPP, will appear to be false positives. In addition, if the SSN is not subsequently updated in the MEF, it will be impossible to find any MEF record for the child, even though the child has a valid SSN and is covered by Medi-Cal. The MEF contains a field which indicates the validity status of the SSN. Over 80 percent of the 5.9 million SSNs in the MEF in June 1991 are coded as validated. However, 7 percent of MEF records are coded as having no valid input for the SSN field, and 8 percent are coded to indicate that there was no SSN at the date of entry into the system. 27 By comparison with these statistics, virtually all of the MEF records that we can match to individuals in the SIPP are coded as having a validated SSN. The very low match rate for MEF records without a validated SSN suggests that the number entered in the SSN field for these records is invalid. To get a sense of the likely effect of such errors, suppose that 4 percent of all SSNs in the MEF are incorrect, with a 2 percent error rate for adults and a 10 percent error rate for children Another 2 percent of records have an unvalidated SSN, and 2.5 percent indicate that the individual is an undocumented alien. We are grateful to Lars Vilhuber for his assistance in processing the SSN validity codes on the MEF records. 28 Anecdotal evidence on the validity of SSNs in other administrative data sets (such as unemployment insurance records) suggests that error rates of 2-6 percent are plausible. 21

24 Suppose further that 25 percent of the children with incorrect SSNs are actually covered by Medicaid (roughly consistent with the data for children age 0-15 in Table 5) and that 10 percent of all adults with incorrect SSNs are covered by Medicaid. Finally, assume that people with incorrect SSNs report their true coverage status to the SIPP. 29 Under these assumptions, 3.2 percentage points of the person-months recorded as MEF=no and SIPP=no should be eliminated from the sample of valid SSN person-months (these are the observations for individuals with incorrect SSNs who are not covered by Medicaid) and another 0.8 percentage points of the person-months recorded as MEF=no and SIPP=yes should be eliminated from the sample (these are observations for people with incorrect SSNs who are actually covered by Medicaid). These changes have no effect on the estimated probability of SIPP coverage among those who are covered in the MEF, but they lead to a substantially lower estimate of the probability of SIPP coverage for those without MEF coverage. In particular, the false positive rate for the unweighted data falls to 1.97 percent, versus 2.81 percent in Table Carrying out parallel calculations for children age 0-15, the estimated probability of SIPP-reported coverage for those covered in MEF is again unaffected, but the probability of a false positive SIPP response falls to 2.17 percent (versus 5.22 percent in Table 5). Given the limitations of the data, it is difficult to determine the true magnitude of matching errors caused by incorrect or missing SSNs in the MEF. However, we are able to examine likely errors among one group: children whose mothers report Medicaid coverage in 29 If actual misreporting rates are no higher than 15 percent (as suggested in Table 5) then ignoring misreporting has little effect on the calculations. 30 If we double the assumed error rates to 4 percent for adults and 20 percent for children, the implied false positive rate falls to 1.05 percent. 22

25 both the SIPP and the MEF. Since most mothers who are covered by Medicaid are eligible through AFDC, their children should also be covered by Medicaid. In Table 6 we show an alternative version of Table 5 in which we have assumed that all children who are recorded as covered by Medicaid in the SIPP but not in the MEF and whose mothers report Medicaid coverage in both data sets are actually enrolled in Medicaid, but erroneously missing from the MEF (either because of missing SSNs or other matching problems). We believe that this is a relatively conservative assumption that gives a lower-bound estimate of the effect of matching errors and missing SSNs. Unfortunately, it only affects some of the children who are potentially missing in the SIPP-MEDS match: those who are living with their mother, whose mothers are themselves covered by Medicaid, and whose mothers also provided valid SSNs to both SIPP and MEDS. Nevertheless, use of this assumption raises the coverage rate for children in the MEF to a level slightly above the SIPP coverage rate. The probability of a false positive response for all children falls by about one-fifth, from 5.22 percent to 4.18 percent. Not surprisingly, the decrease in measured false positive responses is greatest for poor children. Based on these results, we believe that the false positive response rates recorded in Tables 5 and 6 should be viewed as upper bounds on the rate of over-reporting of Medicaid coverage in the SIPP. 31 As noted earlier, the presence of the high rate of false positive coverage responses in Table 5, taken together with the evidence in Figure 2 of overall under-reporting of Medicaid coverage in the SIPP, implies that people without valid SSNs must be substantially under- 31 As the approach taken in Table 6 will miss child-only AFDC or Medicaid cases, we also tried using a child s sibling s report of Medicaid coverage as additional information, however this approach made little difference in the results, probably because there were few children with a sibling who was recorded as receiving Medicaid in both data sets. 23

26 * reporting their true coverage to the SIPP. To see this, let R V represent the fraction of individuals * with valid SSNs who are covered by Medicaid according to the MEF, let R I represent the corresponding fraction among those with invalid SSNs, and let λ denote the fraction of people who report a valid SSN to the SIPP. The fraction of the state population covered by Medicaid * according to the MEF is λ R V + (1-λ) R I*. Similarly, let R V and R I represent the fraction of individuals with and without valid SSNs, respectively, who report Medicaid coverage in the SIPP. Then the estimated Medicaid coverage from the SIPP for the entire population is λ R V + (1-λ) R I. Finally, let γ represent the ratio of the Medicaid caseload measured in the SIPP to the actual number of Medicaid recipients in California. These definitions imply that * (1) λ R V + (1-λ)R I = γ(λ R V + (1-λ) R I* ). * The fractions R V, R I, and λ are all observable from SIPP data, and R V is also observable as the MEF-based coverage rate of people in the SIPP with valid SSNs. For the population as a whole the data from Figure 2 suggest that γ is approximately Equation (1) can therefore be solved for the true rate of Medicaid coverage of people with invalid SSNs in the SIPP (R I* ), and the net under-reporting rate of Medicaid coverage in the SIPP for people with invalid SSNs (R I /R I* ). Using the unweighted data in the top row of Table 5 and an estimate of R I = 15 percent (see * * Table 4), the implied estimate of R I is 26.1 percent and the implied estimate of R I /R I is 57.6 percent. The very conservative corrections in Table 6 (which affect a limited fraction of children * * in the sample) reduce the estimate of R I to 25.1 percent and increase the estimate of R I /R I to 59.8 percent. If an adjustment is made for the possibility of incorrect or missing SSNs for people other than the children whose records were updated in the construction of Table 6, however, the 24