The Implications of Differences Between Employer and Worker Employment/Earnings Reports for Policy Evaluation

Transcription

1 PAM264_08_20291.qxd 8/15/07 1:03 AM Page 737 The Implications of Differences Between Employer and Worker Employment/Earnings Reports for Policy Evaluation Geoffrey L. Wallace Robert Haveman Abstract Differences in administrative (UI) and survey (S) records on employment and earnings have substantial implications for assessing the impact of a variety of public interventions, such as welfare-to-work and employment training programs, and especially the state-oriented welfare reform legislation of We use data from the 1998 and 1999 waves of the Child Support Demonstration Evaluation (CSDE) Resident Parent Surveys to explore individual differences between survey and UI employment and earnings reports for a Wisconsin sample of current and former welfare recipients. After exploring the potential causes of misreports from both sources, we document the degree of discrepancy between survey and UI earnings and employment measures and assess the difference between the two earnings measures in estimates of simple human capital (earnings) functions. Last, we evaluate the correspondence of the two measures with hardship indicators of economic well-being. INTRODUCTION Between August 1996, when President Clinton signed the landmark welfare reform legislation known as the Personal Responsibility and Work Reconciliation Act of 1996, and August 2000, the number of families receiving cash benefits associated with state AFDC-TANF programs decreased by more than 50 percent. Numerous research studies have attempted to track the economic progress and well-being of these women after leaving public assistance. 1 Because national survey data systematically and severely undercount public assistance receipt, most of these studies have relied on state survey and administrative data to monitor the post reform outcomes of target populations. 2 1 Examples of such studies published in this journal include Pavetti and Acs (2001); Loeb and Corcoran (2001); Ong (2002); Danziger, Hefflin, Corcoran, Oltmans, & Wang (2002); Bloom, Hill and Riccio (2003), Johnson and Corcoran (2003), and Cancian and Meyer (2004). 2 For reasons that are not well understood, the problem of underreporting of cash assistance recipiency and income has increased dramatically in recent years. For instance, Meyer and Sullivan (2006) calculate that for 1993, the March CPS failed to capture 12 percent of cash transfers associated with the Aid to Families with Dependent Children (AFDC) program. By 2003, the March CPS failed to capture 38 percent of cash transfers associated with the state Temporary Assistance to Needy Families programs that replaced AFDC after Journal of Policy Analysis and Management, Vol. 26, No. 4, (2007) 2007 by the Association for Public Policy Analysis and Management Published by Wiley Periodicals, Inc. Published online in Wiley InterScience ( DOI: /pam.20291

2 PAM264_08_20291.qxd 8/15/07 1:03 AM Page / Differences in Employment and Earnings Reports These state data take two primary forms. In some studies, state data is obtained from surveys administered to a subset of a state s caseload at a particular point in time, with follow-up surveys administered at subsequent intervals. These surveys typically collect information on demographic characteristics, living arrangements, earnings, employment, and often broader indicators of hardship and/or well-being. Additional information on employment and earnings from state administrative records, gathered as part of employer reports to the Unemployment Insurance System (UI), is matched to survey respondents in a number of these studies. Other studies have used data from strictly administrative sources. The administrative records of state welfare offices are used to provide information on demographic characteristics and recipiency status, again with information on employment and earnings coming from UI reports. The extensive use of these alternative pure forms of state data on individuallevel employment and earnings outcomes in state-specific studies has made it difficult to compare outcomes both within and across states. 3 More important, because survey and administrative measures of employment and earnings may differ substantially, the reliability of any overall assessment of labor market outcomes based on these alternative approaches is open to question. Indeed, for numerous reasons, these two data sources, both intended to measure the same labor market outcomes, may provide substantially different assessments. Employer-submitted UI reports tend to miss some portion of earnings (and hence employment) due to: (1) work and earnings of workers in employment categories exempt from reporting requirements (for example, independent contractors), 4 (2) the underreporting of certain types of income (particularly tips), (3) employment in a state other than the state of residence, and (4) errors in recording Social Security numbers, matching UI wage records, and non-reporting. These errors may reflect either inadvertent noncompliance or the intentional response due to reporting incentives. Although the Bureau of Labor Statistics (BLS) claims that 97.1 percent of 2001 jobs were covered by UI, relatively little is known about the extent of erroneous reports of work and earnings in UI data. Evidence from the few studies that have examined this issue suggests that the extent of underreporting to the UI system is substantial. Blakemore, Burgess, Low, and St. Louis (1996) and Burgess, Blakemore, and Low (1998) conclude that about 45 percent of employers failed to report earnings of some UI covered employees; 13.6 percent of their covered workers had no reports, and 4.2 percent of wages were excluded. Nearly half of all unreported workers were improperly classified as independent contractors. There is evidence that underreports are concentrated in small firms and firms with high turnover. Firms are only responsible for paying UI taxes on employees up to an earnings threshold, so firms with high turnover face a higher effective UI tax rate and thus have a greater incentive to systematically underreport employment and earnings. Individual survey responses regarding work and earnings also have errors. Employment and earnings from illegal activities, irregular work, odd jobs, or reciprocal tasks 3 Few studies have used a combination of administrative and survey data in assessing outcomes of interest. Acs and Loprest (2001) provide a detailed review of 49 studies of welfare leavers that use state data. Of these 49 studies, 27 use survey data exclusively, 18 use administrative data exclusively, with the remaining 4 using both survey and administrative sources of data. 4 Categories of workers not covered by the UI reporting requirement include self-employed workers, independent contractors, farm laborers, domestic workers, military personnel, government workers, some part-time employees of nonprofit institutions, employees of religious orders, and some students employed by their schools. It is estimated that UI records cover about 91 percent of Wisconsin workers.

3 PAM264_08_20291.qxd 8/15/07 1:03 AM Page 739 Differences in Employment and Earnings Reports / 739 for friends, family, and neighbors may be misreported in survey responses. Misreporting may also occur if respondents view the survey as eliciting information that may affect them adversely (for example, earnings reports for recipients of cash assistance), or have difficulty recalling past or intermittent work. The use of imputed values in the case of missing or incomplete data may also introduce error in survey data. 5 A number of recent studies have focused on differences between self and employer reports of employment and earnings; most of them focus on working-age males, and all find substantial differences in individual level and mean earnings between the survey and UI sources (Hotz & Scholz, 2001; Rodgers, Brown, & Duncan, 1993; Bound & Krueger, 1991). Using matched work and earnings data from a sample of National Job Training Partnership Act (JTPA) trainees, Kornfeld and Bloom (1999) (KB) provide estimates of the extent of the discrepancy for a number of working-age population groups and contains the most comprehensive analysis of the implications of this problem for evaluating public policy interventions. In this study, 26 percent of adult men and nearly 15 percent of adult women had quarterly survey and UI earnings values that varied by more that $1,000; mean survey earnings for the entire sample were approximately 30 percent higher than mean UI earnings. Although the differences between self and employer reports of employment and earnings have been studied extensively, to our knowledge there has not been a systematic effort to analyze these differences in the context of welfare-affected populations. Indeed, in most instances such comparisons are not even possible because the timing of the survey does not correspond with the availability of UI reports. Surveys administered to welfare recipients, typically intended to capture point in time employment and hourly or monthly wages, do not provide employment and earnings measures that are directly comparable to quarterly UI reports. There are many reasons to suspect that discrepancies between survey and employer reports of earnings to the UI system would be more pronounced for welfare-affected women than for the working-aged populations included in prior studies. For instance, Edin and Lien (1997) document a substantial degree of off the books, informal sector, and underground economy work among welfare-reliant mothers in Chicago, Boston, San Antonio, and Charleston. Because Wisconsin taxes the earnings of welfare recipients at a 100 percent rate, the incidence of off the books, informal sector, and work in the underground economy may be pronounced for the women included in our data. While the earnings associated with such work may appear in measures of survey earnings, they will not be captured in employer reports to the UI system. Welfare-reliant women are also more likely to work intermittently, leading to problems of recall in survey data. Additionally, they may be more likely to work for employers that have a strong incentive to underreport employment and earnings to the UI system. In particular, welfare recipients may be more likely to work for employers with high turnover. Given the potential for substantial discrepancies in survey and employer reports of earnings and employment measures, it is important to assess the relative accuracy of these two alternative data sources for both interpreting the results of past studies based on one of these sources, and for the design of future studies of the 5 The Appendix summarizes the likely direction of bias in survey and UI employment/earnings measures relative to some unknown true value. All appendices are available at the end of this article as it appears in JPAM online. Go to publisher s Web site and use the search engine to locate article at interscience.wiley.com/cgi-bin/jhome/34787.

4 PAM264_08_20291.qxd 8/15/07 1:03 AM Page / Differences in Employment and Earnings Reports labor market experiences of welfare-affected populations. Although surveys may be better at capturing work that occurs off the books, in the informal sector, and in the underground economy, they are costly to administer. If UI data proves reliable and differences between survey- and UI-based employment and earnings outcomes are not large, UI data on earnings and employment may be a viable substitute for costly surveys. In this paper, we use data from the 1998 and 1999 waves of the Child Support Demonstration Evaluation Resident Parent Survey (CSDE) to explore individual differences between survey and UI employment and earnings reports for a Wisconsin sample of current and former welfare recipients. After providing evidence of misreports from both sources, we document the degree of discrepancy between survey and UI earnings and employment measures, and assess the difference between the two earnings measures in estimates of simple human capital (earnings) functions. Last, we evaluate the correspondence of the two measures with a hardship indicator of economic well-being. DISCREPANCIES IN EMPLOYMENT AND EARNINGS REPORTS We rely on employment and earnings information from UI employer reports matched to an extensive and careful survey of low-wage women in Wisconsin. Our sample consists of 1,882 mothers in Wisconsin who received cash assistance at some point in late-1997 or 1998, and who have nearly complete 6 job and earnings information in both the 1998 and 1999 waves of the CSDE Resident Parent Survey. The CSDE is unique in its efforts to secure reliable information on work and earnings responses of low-wage, welfare-affected women. The special circumstances of this population are reflected in extensive explanations of the questions asked regarding the nature and extent of their work and earnings. For example, in seeking information about earnings, it was explained that the question referred to the total income you earned from all jobs combined during... [the year]. The respondent was explicitly told to exclude any money that was received from the public workforce/welfare agency, even though that payment required work, and that money received in the form of salaries, tips, commissions, and as payment for odd jobs is to be included in the earnings response. Self-employment was explained, and respondents were told that income from this activity is also to be included. The project merged quarterly reports compiled by the Wisconsin Unemployment Insurance (UI) program, indicating whether a person has worked during a quarter, and the earnings of the person. Discrepancies in Employment Reports Table 1 presents survey and UI earnings and employment indicators for our CSDE sample. For purposes of comparison, we also present estimates of these indicators 6 Included in the sample are 198 and 134 women who didn t know their earnings in 1998 and 1999, respectively (with an overlap of 37). Women who indicated they did not know their earnings in either year were asked a series of questions designed to determine their approximate earnings. They were first asked whether their earnings were above or below $20,000. Contingent on the answer to this question they were then asked whether their earnings fell into one of four intervals for the below $20,000 group ($0 $5,000; $5,000 $10,000, $10,000 $15,000, and $15,000 $20,000), and intervals of $20,000 $25,000, $25,000 $30,000, $30,000 $35,000, or over $35,000 for the above $20,000 group. Survey earnings for women who responded to these questions were interpolated as the midpoint of their $5,000 earnings category.

5 PAM264_08_20291.qxd 8/15/07 1:03 AM Page 741 Differences in Employment and Earnings Reports / 741 Table 1. Survey versus UI reports of earnings and employment status, 1998 CSDE sample and KB sample. All Observations Probable False Sure (including Workers Nonworkers Workers Nonworkers) Annual Employment Rate: CSDE sample, n 1,882 Survey 0.74 UI 0.84 S/UI 0.89 Quarterly Employment Rate: KB sample of adult women, n 4,943 (or 27,191 person quarters) Survey 0.58 UI 0.56 S/UI 1.02 Annual Earnings: CSDE sample, n 1,882 Survey Earnings ($000s) $5,760 $7,830 $5,740 UI Earnings $3,490 $6,630 $5,140 S/UI ($000s) Quarterly Earnings: KB sample of adult women, n 4,943 (or 27,191 person quarters) Survey Earnings $1,825 $2,234 $1,244 UI Earnings ($000s) $838 $1,973 $1,007 S/UI for the adult women in the KB sample. 7 Sure workers are those for whom positive earnings are indicated in both survey and UI reports they constitute slightly more than 70 percent of the CSDE sample. About 17 percent of the CSDE women have conflicting employment information from the two sources. We classify such observations as either false nonworkers (those for whom employers report work despite their own self-reports of nonemployment) or probable workers (those without employer reports of work who self-report earnings). The group of false nonworkers, 13 percent of the CSDE sample, is the more problematic; these women either forgot that they worked or misrepresented their earnings to survey interviewers. Because of these discrepancies, the survey and UI reports indicate quite different employment rates; 74 percent from the survey versus 84 percent from UI reports for an S/UI employment rate ratio of It is interesting that the quarterly survey/ui ratio reported by KB for the 4,943 adult women JTPA trainees is 1.02, indicating a survey employment rate in excess of the UI rate, the reverse of the pattern for our sample. 8 7 All the numbers taken from KB were computed on the basis of quarterly, as opposed to annual, data. 8 One possible explanation for this reversal in employment patterns is the fundamental difference in reporting incentives facing women in the KB study (applicants for a federal job training program) and the CSDE women (low-skill current and former cash assistance recipients). Because of their close ties to Wisconsin s Temporary Assistance to Needy Family Programs (Wisconsin Works) program, the CSDE women face incentives to hide their earned income, as earnings in this program are taxed at 100 percent. In this context, any perception that a survey administrator is an agent of welfare administrators may lead to under and nonreporting of earnings and an associated increase in the prevalence of false nonworkers. See Wallace and Haveman (forthcoming) for an exploration of this hypothesis.

6 PAM264_08_20291.qxd 8/15/07 1:03 AM Page / Differences in Employment and Earnings Reports 60,000 50,000 40,000 Survey Earnings 30,000 20,000 10, ,000 20,000 30,000 40,000 50,000 60,000 UI Earnings Figure 1. Survey versus UI earnings. Note: The sample consists of 1,882 women who were interviewed as part of the 1998 and 1999 Child Support Demonstration Evaluation (CSDE) resident parent survey. Discrepancies in Earnings Reports Consistent with the large disparities in reports of employment between the UI and survey, earnings differences are also substantial. Figure 1 presents a scatter-plot of the two earnings values for the entire sample of 1,882 CSDE women. The y-axis shows reports of earnings from the CSDE survey and the x-axis UI reports of earnings actually paid. The 238 women with zero earnings in both data sources are concentrated at the origin of the figure. The 68 probable workers (women with zero UI earnings but positive survey earnings) are shown along the y-axis. The 247 false nonworkers (zero survey earnings but positive UI earnings) are displayed along the x-axis. The 1,329 sure workers with positive earnings in both data sources are shown in the interior of the figure. While there is a substantial degree of nonconformity between survey and UI earnings, the sample correlation between the two earnings values is 0.66 for both the entire sample and for the subsample of sure workers. 9 Table 1 also shows mean earnings estimates for both our sample of women and the adult women from the KB study, distinguishing among the various earnings groups. In both samples, survey earnings for probable workers are substantially 9 There are two substantial outliers associated with sure workers clearly visible in Figure 1. Both of these outliers are characterized by having one source of earnings well in excess of $40,000 and the other source of earnings at around $10,000. The empirical results that follow and the conclusions that we reach are robust to the exclusion of these outliers (results available from authors upon request).

7 PAM264_08_20291.qxd 8/15/07 1:03 AM Page 743 Differences in Employment and Earnings Reports / 743 higher than UI earnings for false nonworkers, suggesting that UI data is likely to miss near-typical employment whereas the survey misses infrequent and/or lowwage employment. For the sure workers among the CSDE sample, mean survey and UI earnings are $7,830 and $6,630, respectively, for an S/UI ratio of For the adult women who are sure workers in the KB sample, the S/UI earnings ratio is CAN THE SOURCE OF THE EARNINGS DISCREPANCY BE EXPLAINED? The CSDE survey provides detailed information that allows us to explore the impact of a number of conjectures regarding the source of the survey-ui earnings discrepancy. In particular, the CSDE survey provides information on whether or not the county of residence of the respondent borders on another state, whether the respondent lived out of state for part of the survey year, as well as the number and characteristics of jobs held over the year. With respect to job characteristics, the CSDE survey indicates whether a respondent worked an odd job, whether she was hourly versus salaried, and whether she received income from tips, bonuses, or commissions. From UI records, we obtain quarterly work histories for each of the survey years, including the number of employers that reported earnings; from administrative records on welfare use we obtain monthly welfare histories. Table 2 provides a listing of conjectures associated with variables available in the CSDE survey or the matched UI records, along with a rationale for each conjecture. We speculate that some of the conjecture variables will be associated with having a high value for one of the earnings values relative to the other, while others will increase the magnitude of the discrepancy with no clear directional bias. We hypothesize that living out of state, living in a border county, working odds jobs, or receiving income from tips and commissions, all of which allow for actual earnings not to be reported to the UI system, will increase survey earnings relative to UI earnings. Conversely, we hypothesize that having many employers, intermittent employment, last working well before the survey was administered, not knowing precise earnings, and being on welfare for a large fraction of the year will increase the magnitude of the survey-ui differences without a clear directional bias. Using the sample of sure workers in 1998, we regress the survey-ui earnings difference (S UI), on a variety of conjecture variables coded from the CSDE, and matched administrative data (and a set of control variables); see Table 3. These conjecture variables generally have the expected sign, and most of them are statistically significant. For example, living out of state or in a border county, or having earnings from tips, commissions, or an odd job, all of which are not reported or likely to be underreported in UI records, are positively related to the magnitude of the discrepancy. Being a nonsteady worker 10 (and hence less likely to provide reliable earnings reports) and not knowing precise earnings values are also positively and significantly associated with the magnitude of the discrepancy, suggesting that recall problems lead to high values of survey earnings relative to UI earnings. In sum, with but few exceptions, our conjectures regarding the sources of the discrepancy between survey and UI earnings reports are confirmed in these estimates. In addition to the regression results shown in Table 3, we estimated a number of alternative specifications, including specifications using both the log ratio of survey and UI earnings and the ratio of survey and UI earnings as dependent variables in 10 Nonsteady workers are those who fail to work three quarters in the year or have more than two jobs (survey measure) or employers (UI measure) in a calendar year.

8 PAM264_08_20291.qxd 8/15/07 1:03 AM Page / Differences in Employment and Earnings Reports Table 2. Conjectures regarding the source and magnitude of discrepancy between survey and UI reports of earnings. Conjecture Rationale for Conjecture 1. On average, respondents who report UI records do not include earnings from earnings from odd jobs, tips, and odd jobs. Thus, earnings from odd jobs commissions will have larger would increase S relative to UI, leading to discrepancies, all else equal. larger discrepancies. Additionally, earnings from tips and commissions are more likely to be underreported in UI and survey data. Tip earnings are likely to be underreported in UI data because they are both taxed and difficult to verify. Earnings from tips and commissions may be misreported in a survey because their more varied nature results in recall problems or imprecise estimation by sample respondents. 2. On average, respondents who were on Because Wisconsin reduces welfare welfare for more months during 1998 benefits dollar for dollar with earned will have larger discrepancies, all else income, there are strong incentives for equal. working welfare recipients to conceal their work activities. This may result in reduced reports of survey or UI earnings depending on the nature of concealment.* 3. Respondents who lived out of state for Out of state employers do not report some portion of 1998 or lived in a border earnings to Wisconsin s UI system. county will have larger discrepancies, all else equal. 4. Steady workers those with earnings Workers with continuous work on a few in at least 3 quarters of the year, and jobs or for few employers are more likely with no more than two employers or to provide reliable earnings reports to jobs during the year will have smaller survey interviewers. discrepancies, all else equal. 5. Respondents who indicate that they Respondents indicating that they don t don t know their earnings (and for know their earnings in 1998 are more whom an imputed value is provided) likely to report their earnings to surveyors will have larger discrepancies than with error (either intentionally or those who do respond to the earnings unintentionally). Additionally, errors question, all else equal. introduced in the imputation process may increase the discrepancy. 6. On average, those whose last quarter Those whose last quarter of employment of employment was early in the year will was far from the date of the survey are have larger discrepancies than those who likely to forget their prior year s earnings, worked in the last quarter of the year, all and to report them with error. else equal. *Concealing work activities may take two forms. Recipients may be inclined to underreport (or to not report) their earnings to surveyors if they believe this information will result in adverse administrative decisions such as loss of benefits. In this case, UI earnings would be higher than survey earnings, all else equal. Alternatively, recipients may provide false social security numbers to employers in an attempt to disguise their earnings. Assuming these earnings are reported to surveyors, such concealment would lead to higher survey than UI earnings, all else equal.

9 PAM264_08_20291.qxd 8/15/07 1:03 AM Page 745 Differences in Employment and Earnings Reports / 745 Table 3. OLS estimates of the determinates of the S-UI earnings difference among sure workers. N 1,329 (Standard errors in parentheses.) Variable Coefficient Estimate Age (0.1252) Age squared (/100) (0.2010) High school graduate (vs. less than high school) (0.2768) Some college (vs. less than high school) * (0.5077) Black (vs. white) (0.3298) Hispanic or other (vs. white) (0.5032) Milwaukee county (vs. rural county) Other urban county (vs. rural county) (0.3850) * (0.4659) Border county ( 1) ** (0.4702) Out of state in 1998 ( 1) (1.8419) Fraction of 1998 receiving cash assistance Steady worker ( 1) Paid hourly (vs. salaried) Arrangements for overtime pay ( 1) * (0.5187) ** (0.3058) (0.3996) (0.2807) Earnings from odd jobs ( 1) ** (0.3996) Earnings from tips or commissions ( 1) (0.4029) Didn t know earnings ( 1) ** (0.4526) Last worked in quarter 1 or 2 (vs. 3 or 4) * ( ) Worked two quarters in year (worked one quarter 1) (0.4487) Worked three quarters in year (worked one quarter 1) (0.509) Worked four quarters in year (worked one quarter 1) (0.508) R-squared *Statistically significant at the 0.10 level. **Statistically significant at the 0.05 level. place of the survey-ui difference. The results of models estimated using the log ratio of survey and UI earnings were very similar to those shown in Table 3. The models estimated using the ratio of survey and UI earnings were characterized by high standard errors and a lack of statistically significant effects. These results reflect the

10 PAM264_08_20291.qxd 8/15/07 1:03 AM Page / Differences in Employment and Earnings Reports large number of observations for which one source of earnings is substantially different from the other, resulting in exceedingly high variability in the dependent variable. In our sample, the ratio of survey to UI earnings ranges from to 518. Given the extreme variability of the ratio of survey to UI earnings, the similarity of the results using the log ratio of survey earnings to UI earnings to those using the survey-ui difference, and the fact that the difference results can be used to decompose the mean squared discrepancy (see below), we present results from the model that uses the survey-ui difference as the dependent variable. We next use regression estimates shown (Table 3) to simulate the contribution of these factors to the actual survey-ui earnings discrepancy. By setting conjecture variables to certain fixed values, we are able to simulate the change in the 1998 mean squared discrepancy (MSD) associated with these variables. 11 For example, we can compute an estimate of the MSD assuming that no sample members worked odds jobs or received income from tips and commissions by turning these variables off and performing a least squares decomposition. Consistent with the conjectures, living in a border county increases the MSD, as does having an odd job, receiving tips and commissions, not being a steady worker, not knowing earnings, and last working prior to the third quarter of Overall, the conjecture variables that we have been able to study because of the detailed information available in the CSDE data account for about 9 percent of the total discrepancy in Other factors, chiefly fixed differences in survey and UI reports not related to independent variables in the model and simple random variation, account for the bulk of the total discrepancy. 12 DO COEFFICIENT ESTIMATES IN LABOR MARKET MODELS VARY BY S AND UI? In the preceding sections we demonstrated that there are substantial and unexplained differences between earnings and employment measures constructed from survey data and those constructed from employer reports to the UI system. Although differences between employment and earnings measures are substantial, it remains to be seen whether measurement of the systematic relationship between human capital variables (for example, age, education, and race) and earnings and employment depends upon the survey or UI source of information. In Tables 4 and 5, we show regression coefficients (earnings) or probability differences (employment) related to these human capital variables for both 1998 and 1999 using the two data sources. We also report the difference in these estimates across the two sources of data and standard errors for these differences. In Table 4, we compare OLS estimates of a simple human capital model across data sources using the subsample of sure workers. There are only small differences in the coefficient estimates across the survey and UI models, and these differences are not statistically significant. We conclude that coefficient estimates describing the relationship of standard human capital variables to the earnings of women who work are independent of the choice of data source. In Table 5, we report similar logit estimates of the impact these same human capital variables had on the probability of employment, again for both 1998 and Impact is measured as the percentage point change in the predicted probability of employment for a 27-year-old black high school dropout implied by a unit increase 11 We define the mean squared discrepancy (MSD) as (survey earnings i UI earnings i) Six percent of the 1998 MSD can be accounted for mean differences in survey and UI earnings in the subsample of sure workers. The corresponding figure for the full sample is 1.4 percent.

11 PAM264_08_20291.qxd 8/15/07 1:03 AM Page 747 Differences in Employment and Earnings Reports / 747 Table 4. Ordinary least squares estimates of log earnings; sure workers in both the 1998 and 1999 samples N 1,103. (Standard errors in parentheses.) Survey 1998 UI 1998 Difference Survey 1999 UI 1999 Difference Age ** ** (0.0333) (0.0340) (0.0476) (0.0314) (0.0337) (0.0461) Age squared (/100) * * (0.0556) (0.0569) (0.0796) (0.0507) (0.0546) (0.0745) High school ** ** ** ** (vs. less than high school) (0.0686) (0.0702) (0.0982) (0.0627) (0.0674) (0.0920) Some college ** ** ** ** (vs. less than high school) (0.1031) (0.1055) (0.1475) (0.0941) (0.1012) (0.1382) White (vs. nonwhite) * (0.0672) (0.0688) (0.0962) (0.0614) (0.0660) (0.0901) Intercept ** ** ** ** (0.4731) (0.4839) (0.6767) (0.4623) (0.4972) (0.6788) R-squared *Statistically significant at the 0.10 level. **Statistically significant at the 0.05 level.

12 PAM264_08_20291.qxd 8/15/07 1:03 AM Page / Differences in Employment and Earnings Reports in the relevant independent variable. Predicted employment rates for an individual with these baseline characteristics using both S and UI measures of employment are also shown in the table. For both years, the estimated relationship of education and being white to the probability of being employed is greater when survey data are used, and in nearly all cases the difference in these coefficients is statistically significant. For both years, likelihood ratio tests reject the null hypothesis that models of S and UI employment are equivalent. Hence, we conclude that estimates of the determinants of earnings of women in our sample who work are not sensitive to the source of data, whereas estimates of the determinants of employment across the two data sources are widely divergent. DO TEMPORAL EMPLOYMENT AND EARNINGS PATTERNS VARY BY SURVEY AND UI? The employment and earnings differences related to the use of survey and administrative data reported in Table 1 suggest that the use of survey information understates employment levels and substantially overstates earnings among low-skill female workers, relative to UI data. Additionally, the results reported in Table 5 indicate that there are substantial differences between the effects of education and race on survey and UI employment. In this section, we compare the changes over time in both employment and earnings levels for the full sample and for subgroups using the two sources of data on these outcomes. These comparisons are shown in Table 6. Examining Table 6, the survey employment measure indicates slightly higher employment growth than the UI measure, but the differences between employment growth measures are not large and are only statistically significant for the entire sample and one of the subgroups. Turning to the earnings growth estimates in the bottom panel of Table 6, there is a remarkable degree of correspondence across measures. For the entire sample and for three out of four subgroups, the ratio of growth indicators is close to one. The one anomaly in this pattern occurs among whites without a high school diploma or equivalent. For these women, the UI data suggest earnings growth on par with other demographic groups, whereas the survey data suggest much lower earnings growth. EARNINGS SOURCES AND ECONOMIC HARDSHIP Although results presented above provide evidence that choice of either survey or UI data matter in assessing employment and earnings levels, the determinants of employment, and earnings growth, we have thus far offered no prescription regarding which source of information is better suited for tracking the labor market outcomes of welfare-affected populations. One criterion for choosing between the two sources is their relative accuracy in tracking individual well-being. Here, we assess the relative performance of these measures for our welfare-affected population by relating constructed survey- and UI-based income measures to survey reports of material hardship. We construct survey- and UI-based income measures by adding cash assistance received as part of Wisconsin s TANF program (W2) to each earnings measure. For each income measure, we compute the ratio of income to needs for each family unit, where need is defined by the poverty threshold for the observed family size, computed as one plus the number of biological children. Because we are concerned that other adults may be contributing unmeasured income to the respondents households, thus confounding the results, we eliminate women from our analysis

13 PAM264_08_20291.qxd 8/15/07 1:03 AM Page 749 Differences in Employment and Earnings Reports / 749 Table 5. Logit estimates of employment. a N 1,882. (Standard errors in parentheses.) Survey 1998 UI 1998 Difference Survey 1999 UI 1999 Difference Age ** ** (0.0168) (0.0018) (0.0177) (0.0166) (0.0020) (0.0176) High school ** ** ** ** * (vs. less than high school) (0.0228) (0.0178) (0.0289) (0.0214) (0.0165) (0.0271) Some college ** ** ** ** ** ** (vs. less than high school) (0.0316) (0.0269) (0.0415) (0.0272) (0.0236) (0.0360) White (vs. nonwhite) ** ** ** ** ** (0.0243) (0.0203) (0.0317) (0.0247) (0.0213) (0.0326) Baseline Employment Rate b Pseudo R-squared *Statistically significant at the 0.10 level. **Statistically significant at the 0.05 level. a The estimates indicate the effect of unit changes in the independent variables on the probability of employment. b The employment rate for a 27-year-old black high school dropout.

14 PAM264_08_20291.qxd 8/15/07 1:03 AM Page / Differences in Employment and Earnings Reports Table 6. Change in employment and earnings from 1998 to S99/S98 UI99/UI98 ( ) S99 S98 ( ) UI99 UI98 Employment White, High School or more (N 382) ** 1.04* White, Less than High School (N 212) * 1.02 Nonwhite, High School or more (N 597) Nonwhite, Less than High School (N 691) 1.09* 1.03* 1.06 Total (N 1882) 1.03* * Earnings White, High School or more (N 382) 1.35** 1.33** 1.02 White, Less than High School (N 212) ** 0.84* Nonwhite, High School or more (N 597) 1.32** 1.37** 0.96 Nonwhite, Less than High School (N 691) 1.39** 1.39** 1.00 Total (N 1,882) Note: The null hypotheses for the reported statistical test are that the differences between the 1998 and 1999 sample means are zero (the first and second column) and the differences in these differences are zero (the last column). S98 and S99 refer to 1998 and 1999 survey measures and UI98 and UI99 refer to 1998 and 1999 UI measures. *Statistically significant at the 0.10 level. **Statistically significant at the 0.05 level. sample who reported living with other adults for more than 6 months during the relevant survey year. These restrictions leave a subsample of 916 women who lived with their children, but without other adults, for 6 or more months in 1998 and Using these survey- and UI-based income-to-need ratios, we compare hardship rates for women at various points in the distribution of income-to-needs. The sample is quite disadvantaged, with the 75th percentile of income-to-needs ranging from 0.75 to 0.90, depending on the year and the income measure used. The material hardship measures that we use are based on survey responses to questions regarding the experience of material hardship. We employ the following four hardship measures: (1) whether there is sometimes not enough to eat, (2) whether the respondent has gone without electricity because they could not afford to pay the bill, (3) whether the respondent had to live with friends or relatives because they could not afford to pay the rent or mortgage, and (4) whether the respondent went without phone service because they could not afford it. We also study a fifth measure, whether any hardship is recorded. We compare the difference in hardship rates between women in the high and low portions of the distributions of survey- and UI-based income-to-needs. Examples of the split samples for these difference-in-differences comparisons are: 1) poor versus nonpoor, 2) top quartile versus bottom three quartiles, and 3) top quartile versus bottom quartile. For all of these splits, statistical tests indicate that being higher in the distribution of income-to-needs is associated with lower rates of hardship, irrespective of the survey or UI income measure used. Although most of the difference estimates are large, positive, and statistically significant when the UI-based income measure is used, fewer of the difference estimates using the survey-based income measure are positive and statistically significant, and in general the differences-in-differences (the UI difference less the survey difference) estimates are positive. While these positive difference-in-difference

15 PAM264_08_20291.qxd 8/15/07 1:03 AM Page 751 Differences in Employment and Earnings Reports / 751 estimates suggest that that the UI-based income measure tracks reported hardship over the distribution of income-to-needs more closely than the survey-based income measure, they are small, and are not generally statistically distinguishable from zero. For example, using a split involving the top 25 percent and the bottom 75 percent of the distributions of income-to-needs, all five of the difference-in-differences estimates are positive for 1998 and the estimate for the rent/mortgage hardship is statistically significant; for 1999, three of the five measures are positive and that on any hardship is statistically significant. When the difference-in-differences estimates are generated by comparing the poor to the nonpoor, four out of five are positive and statistically significant in 1998; for both 1998 and 1999, 6 out of the 10 difference-in-difference estimates are positive and statistically significant. From these several estimates, we conclude that both survey- and UI-based income measures are strongly related to hardship (as an indicator of economic well-being), and that the UI-based income measure tends to track reported hardship over the distributions of income-to-needs somewhat more closely than the survey-based measure. CONCLUSIONS Our analysis of the reliability of data on employment and earnings from alternative data sources is both sobering and reassuring for policy analysts and researchers. Using data on a large sample of low-skill single mothers, we find substantial disparities in employment and earnings reports between a uniquely high-quality survey of low-skill workers and employer-based reports of earnings. While average survey earnings are 18 percent ($600) higher than average UI earnings, the UI-based employment rate is 13 percent higher than the survey-based rate. Furthermore, there are a substantial number of women (13 percent of the sample) who reported not working for pay in the survey, but for whom employers reported employment and a smaller number of women (5 percent of the sample) who report working for pay, but do not appear in UI records. After documenting the degree of discrepancy in self and employer reports of earnings, we examine the extent to which individual 1998 survey-ui differences can be explained by differences in coverage (receipt of income from odd jobs, tips, commissions; living out of state; or living in a border county) or characteristics of employment (being a nonsteady worker ; not knowing earnings precisely, and last working prior to the 4th quarter of 1998) that are likely to lead to inaccuracies. We find that about 9 percent of the mean squared discrepancy can be explained by these variables. Having established that there are significant differences between earnings and employment measures associated with the two data sources that cannot be fully accounted for by differences in coverage or characteristics, we document the extent to which the choice of data source affects coefficient estimates of the determinants of earnings, employment, and earnings growth. We find that coefficient estimates from a simple version of the empirical human capital model, estimated over women with positive earnings in both sources, are not sensitive to the source of earnings data, but that estimates from a logistics model of employment status are. In particular, the effect of educational attainment and being white on employment are substantially (and statistically significantly) higher when survey-based employment measures are used. Estimates of employment growth from 1998 to 1999 are fairly consistent between the two sources. Estimates of earnings growth over the same period are fairly consistent for most demographic groups, the exception being whites without

16 PAM264_08_20291.qxd 8/15/07 1:03 AM Page / Differences in Employment and Earnings Reports a high school diploma. For this group, earnings grow by 35 percent using the UI measure and by only 14 percent using the survey measure. In sum, we are left with two competing sources of data that lead to different conclusions concerning mean earnings employment levels and the effect of age, education, and race/ethnicity on employment. In an effort to gain insights into which data source is more appropriate for tracking the level of economic well-being, we form alternative survey- and UI-based income-to-need measures for each family, and then compare rates of hardship for these families at various ranges in the distribution of survey- and UI-based income-to-needs. The results indicate that being higher in both the distributions of survey- and UI-based income-to-needs is associated with lower rates of hardship. Additionally, the UI-based income-to-needs measure appears to track the reported hardship indicator of economic well-being at least as well its survey-based alternative. Given the similarity of most of the results across the two sources of earnings employment data, the fact that UI earnings seem to track hardship measures of well-being at least as well as survey earnings, as well as the ready availability and consistency of administrative data across states, we conclude that UI is preferred to survey data for monitoring labor market outcomes and tracking the economic wellbeing of welfare-affected populations. GEOFFREY L. WALLACE is Assistant Professor at the Robert M. La Follette School of Public Affairs and Department of Economics, University of Wisconsin-Madison. ROBERT HAVEMAN is Professor Emeritus at the Robert M. La Follette School of Public Affairs and the Department of Economics, University of Wisconsin-Madison. REFERENCES Acs, G., & Loprest, P. (2001). Studies of welfare leavers: Data, methods, and contributions to the policy process. In Studies of Welfare Populations: Data Collection and Research Issues, National Research Council: National Academy Press, Blakemore, A. E., Burgess, P. L., Low, S. A., & St. Louis, R. D. (1996). Employer tax evasion in the Unemployment Insurance Program. Journal of Labor Economics, 14, Bloom, H. S., Hill, C. J., & Riccio, J. A. (2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management, 22, Bound, J., & Krueger, A. B. (1991). The extent of measurement error in longitudinal earnings data: Do two wrongs make a right? Journal of Labor Economics, 9, Burgess, P. L., Blakemore, A. E., & Low, S. A. (1998). Using statistical profiles to improve unemployment insurance tax compliance. Research in Employment Policy, 1, Cancian, M., & Meyer, D. R. (2004). Alternative measures of economic success among TANF participants: Avoiding poverty, hardship, and dependence on public assistance. Journal of Policy Analysis and Management, 23, Danziger, S., Hefflin, C. M., Corcoran, M. E., Oltmans, E., & Wang, H. (2002). Does it pay to move from welfare to work. Journal of Policy Analysis and Management, 21, Edin, K., & Lein, L. (1997). Making ends meet: How single mothers survive welfare and lowwage work. New York, NY: Russell Sage Foundation. Hotz, J. V., & Scholz, J. K. (2001). Measuring employment and income outcomes for lowincome populations with administrative and survey data. In Studies of Welfare Populations: Data Collection and Research Issues, National Research Council: National Academy Press, pp

17 PAM264_08_20291.qxd 8/15/07 1:03 AM Page 753 Differences in Employment and Earnings Reports / 753 Johnson, R. C., & Corcoran, M. E. (2003). The road to economic self-sufficiency: Job quality and job transition patterns after welfare reform. Journal of Policy Analysis and Management, 22, Kornfeld, R., & Bloom, H. S. (1999). Measuring program impacts on earnings and employment: Do unemployment insurance wage reports from employers agree with surveys of individuals? Journal of Labor Economics, 17, Loeb, S., & Corcoran, M. (2001). Welfare, work experience, and economic self-sufficiency. Journal of Policy Analysis and Management, 20, Ong, P. M. (2002). Car ownership and welfare-to-work. Journal of Policy Analysis and Management, 21, Meyer, B., & Sullivan, J. X. (2006). Consumption, income and material well-being after welfare reform. National Bureau of Economic Research Working Paper 11976, Cambridge, MA. Pavetti, L., & Acs, G. (2001). Moving up, moving out or going nowhere? A study of the employment patterns of young women and the implications for welfare mothers. Journal of Policy Analysis and Management, 20, Rodgers, W. L., Brown, C., & Duncan, G. J. (1993). Errors in survey reports of earnings, hours worked, and hourly wages. Journal of the American Statistical Association, 88, Wallace, G. L., & Haveman, R. (forthcoming). Work and earnings of low-skilled women: Do employee and employer reports provide consistent information? Journal of Economic and Social Measurement.