2 Project Description

Transcription

1 1 Project Summary This project aims to enhance understanding of labor supply and program participation decisions by: 1) developing and estimating a behavioral model of labor supply using survey and administrative data from a randomized experiment, 2) assessing the importance of precise measurement of agents choices and incentives on the accuracy of the model s predictions, and 3) evaluating the ability of the estimated model to predict the results of other experiments in di erent populations. The potential intellectual merit of the proposed research is to generate accurate quantitative models of labor supply and program participation. Such models can be used to evaluate the impact of reforms to the social insurance and tax system and to make quantitative statements about the welfare e ects of proposed and existing policies. Our estimation approach combines observational and experimental variation in incentives and opportunities. The experimental variation in welfare rules allows us to relax the key exogeneity assumptions typically made when estimating behavioral responses to changes in program rules. Our ability (or inability) to accurately predict the results of markedly di erent randomized experiments provides a policy-relevant metric for evaluating our modeling framework in general and, more speci cally, our ability to account for self-selection into program and labor market participation. The project will deliver two research papers. The rst paper focuses on assessing the extent to which the ability of labor supply models to match experimental impacts depends upon measurement issues and modeling complexity. We will begin by developing and estimating a model of welfare participation and labor supply using data from the California Work Pays Demonstration Project (CWPDP) a large scale randomized welfare reform experiment implemented in California in the early 1990s. The longitudinally merged administrative and survey data available from this experiment allow us to measure the budgets (and hence incentives) of agents in substantially more detail than previous studies. Moreover, the various components of the data contain independent repeated measurements on a number of key variables allowing us to detect and model recording and reporting problems in a more satisfactory manner than has previously been attempted in this literature. We will then examine how our estimates change when we coarsen the choices available to agents, use approximate rather than exact policy rules, or ignore measurement problems. Our focus will be on how these changes in uence our ability to match experimental impacts on quantities of direct policy interest such as total welfare payments and program participation. The analysis will also provide a quantitative unbundling of the incentive e ects associated with the CWPDP treatment, allowing us to ascertain, for example, the relative importance of changes in welfare eligibility rules vs. changes in earnings disregards. The second paper is motivated by the concern that randomized experiments may have little external validity. The CWPDP experiment, for instance, was conducted on a sample of on-going welfare recipients residing in four California counties during a sustained boom in the state job market. To what extent can what we learn from the California experiment be generalized to other populations, time periods, and program mixes? The many state welfare experiments conducted during the 1990s provide us with the opportunity to answer this question. We will use our estimates from the CWPDP sample to generate predictions about the results of randomized experiments in two other states. This will entail developing methods to re-estimate distributions of unobservable preferences and skills from the control observations available in each state s sample. We will conclude with an assessment of the practical advantages (if any) of access to experimental variation in generating credible out of sample policy predictions. The broader impact of our project will be to develop methods for enhancing what can be learned from social experiments through use of a behavioral model. We seek to illustrate how to build and estimate economic models capable of pooling together and interpreting the results of multiple experiments in the presence of unobserved individual-level heterogeneity. We believe such methods will become increasingly important as social and eld experiments continue to proliferate in economics. 1

2 2 Project Description This project aims to enhance understanding of labor supply and program participation decisions by: 1) developing and estimating a model of labor supply and program participation using experimental data, 2) assessing the importance of precise measurement of agents choices and incentives on the accuracy of the model s predictions, and 3) evaluating the ability of the estimated model to predict the results of other experiments in di erent populations. The project will deliver two research papers which we brie y summarize here. The rst paper focuses on assessing the extent to which the ability of labor supply models to match experimental impacts depends upon measurement issues and modeling complexity. We utilize data from the California Work Pays Demonstration Project (CWPDP) a large scale randomized welfare reform experiment conducted in California in the early 1990s providing a variety of nancial incentives for welfare recipients to work. Longitudinal administrative and survey data components from the experiment are available providing detailed information on program participation, eligibility, hours of work, and transfer amounts. We develop and estimate a model of labor supply and program participation on this sample using exact program rules, allowing for detailed hours and participation choices, and accounting for non-random measurement error in both survey and administrative sources. We then estimate versions of the model eliminating these features one by one i.e. using approximate program rules, coarsening agents choice sets, and ignoring measurement errors. We also estimate versions of the model with and without the use of experimental variation. We then compare the performance of each model variant in accurately predicting the experimental impacts of the CWPDP reform. Our goal is to assess which features of the modeling process matter most for generating accurate forecasts of policy e ects. The second paper builds on the rst and asks what is necessary to generate accurate forecasts in di erent populations of policy reforms that have not yet occured. We do this by using our previous CWPDP based model estimates to generate predictions of the e ects of two di erent randomized experiments conducted by the Manpower Demonstration Research Corporation (MDRC) in di erent states and time periods and involving markedly di erent incentive schemes. While some attempts have been made to extrapolate between randomized samples (Hotz and Imbens, 2005) or to provide a meta-analysis of experimental results (Grogger, Karoly, and Klerman, 2004) there has not yet been a systematic assessment of the ability of formal behavioral models to rationalize disparate experiments. This exercise requires us to acknowledge (and deal with) the heterogeneity in unobservable tastes and abilities likely to be present across these samples. We do this by re-estimating the distribution of heterogeneity in the validation samples using pre-treatment and control observations. We also deal with the fact that the model s variables are measured di erently in these new datasets by re-estimating and re-tailoring the measurement structure of the model. We then examine the ability of variants of this procedure to match the experimental impacts found in these samples with an emphasis on determining which modeling and implementation choices matter most. The broader impact of our project will be to develop methods for enhancing what can be learned from social experiments through use of a behavioral model. We seek to illustrate how to build and estimate economic models capable of pooling together and interpreting the results of multiple experiments in the presence of unobserved individual-level heterogeneity. We believe such methods will become increasingly important as social and eld experiments continue to proliferate in economics. 2.1 Does Measurement Matter? Estimating a Model of Labor Supply and Welfare Participation using Survey and Administrative Data Motivation Developing models capable of quantitatively forecasting the e ects of changes in social welfare and tax policy has been an objective in the economics literature at least since the pioneering work of 1

3 Marschak (1953), Orcutt and Orcutt (1968), and Kosters (1969). Early contributions to the labor supply and program participation literature including Ashenfelter and Heckman (1974), Burtless and Hausman (1978), and Ashenfelter (1983) focused on incorporating basic restrictions from economic theory into relatively simple statistical models. Since then structural models of labor supply and program participation have increased substantially in complexity by incorporating multiple program participation (Keane and Mo tt, 1998; Hoynes, 1999; Heim and Meyer, 2003), dynamics (Keane and Wolpin, 2002a, 2002b, 2007; Fang and Silverman, 2009), and endogenous family formation (Keane and Wolpin, 2006). The development and estimation of any model of labor supply and program participation confronts the researcher with two fundamental challenges. First, any behavioral model must specify the policy environment, the choice set of the agent, and the non-policy determinants of the agent s choices (her preferences, opportunities, and constraints). Second, estimation of the proposed model has to deal with the limitations of the available data. The rst challenge is complicated by the fact that the modern welfare system consists of a variety of interconnected programs (TANF, Food Stamps, SSI, Medicare, etc.) each with its own eligibility criteria. The further addition of tax regulations results in a highly non-linear budget set, a large number of choice variables, and a plethora of program-speci c determinants of eligibility and support amounts. As rst pointed out by Burtless and Hausman (1978) and Hausman (1985), estimation in the presence of non-linearities requires an explicit decision-theoretic approach. Accounting for all or most of the interacting programs means that the decision model is bound to be demanding to estimate. The second challenge concerns measurement of the above incentives, choices, and choice determinants. Both survey and administrative data have important limitations in this respect. On the one hand, survey data (a) can only supply crude measures of eligibility status and of the individual- and household-level characteristics that determine the support amounts given eligibility, and (b) su er from recall and rounding bias. On the other hand, administrative data on program participation (a) is typically narrow in scope, (b) records information only for periods when the individual - or the relevant group of individuals - are receiving assistance, and (c) is subject to a potentially non trivial amount of non-random mis-reporting re ecting attempts by the agent to maximize the support received. This mis-reporting may be economically important and, if disregarded, may lead to an incorrect assessment of the impact of the program on the (overall) well-being of the agent. Researchers have yet to deal with the above challenges in a uni ed framework. Indeed, the existing literature employs two alternative, and to a large extent exclusive, approaches to handling the complexities of specifying and estimating models of labor supply and program participation. One approach involves a crude simpli cation of the budget set. This is accomplished by focusing on a small subset of the programs and by smoothing away the kinks associated with the few programs considered. For instance, MaCurdy, Green, and Paarsch (1990) argue in favor of approximating the nonlinear implicit tax schedule generated by public assistance using low order polynomials. Fraker, Mo t, and Wolf (1985) and McKinnish, Sanders, and Smith (1999) take similar regression based approaches to reducing the dimensionality of the policy environment. The smoothing of the budget set and the reduction of the choice set allows the researcher to (a) model choice functions directly (instead of deriving them from primitives) and (b) work with a very rich set of hours alternatives e.g. by estimating continuous hours of work supply functions. The second approach provides a di erent answer to the above challenges. It strives to specify the policy rules and the budget set as accurately as the data permit. However, it applies a severe coarsening of the choice alternatives vis a vis hours of work. For instance, Keane and Mo tt (1998) and Keane and Wolpin (2002 a,b, 2006, 2007 a,b) reduce the set of labor supply alternatives to only full-time, part-time, and no work and deal with a limited set of program participation options. This coarsening reduces substantially the computational/estimation complexities. However, when considering a population at risk of welfare dependence it may also miss important behavioral responses along the intensive margin of labor supply. Such models also potentially miss important interactions between hours choices and eligibility requirements for 2

4 program participation. Furthermore, both approaches rarely account for measurement or reporting errors, often because the survey or administrative data being used make doing so impractical. When measurement and reporting errors are taken into account, they usually play an ancillary role. Keane and Wolpin (2006), for instance, estimate a dynamic model of labor supply, program participation, fertility, and marriage incorporating measurement and classi cation errors. However, these errors are introduced for technical reasons, their stochastic processes are highly restricted, and identi cation is indirect in the sense of relying exclusively upon speci cation of the rest of the behavioral model. In sum, while most researchers recognize the existence of measurement limitations, in practice, such limitations are typically not explored or accounted for. The immediate consequence of this paucity of analysis is that we do not know how much measurement matters. In this paper we strike a di erent set of compromises than previous work. We model budget sets, choices, and measurement issues in great detail at the expense of assuming that agent s decisions are myopic. This allows us to sidestep the computational demands associated with dynamic discrete choice models and a ords us a great deal of latitude in incorporating a multiplicity of programs and hence an accurate speci cation of the budget and choice sets. It also allows us to work with a more detailed collection of hours of work alternatives. The merged administrative and survey data at our disposal allow us to not only model the budget set accurately, but to also measure it accurately. We propose an estimation method that takes advantage of the repeated measurements in our data. Our identication of the measurement structure is therefore more direct than in previous model-based work. Finally, the experimental component of the data allows us to judge the estimated model based on its ability (or lack thereof) to reproduce the impacts of randomized policy changes. This is the metric that we employ to decide whether measurement matters a question we examine by estimating nested versions of the model that disregard some of the data sources and/or do not account for measurement and reporting error. The Work Pays Experiment and Data In 1992, prompted by a statewide downturn in the economy, California responded to scal crisis by reducing the cash bene ts associated with AFDC. Because the proposed reductions exceeded the federal regulatory limit, a federal waiver had to be obtained and a randomized evaluation conducted. This resulted in the California Work Pays Demonstration Project (CWPDP) a large scale randomized welfare reform experiment providing reduced cash assistance and nancial work incentives to welfare participants. In 1992, 15,000 AFDC cases were sampled for the study, two thirds of which were family group or one parent (FG) cases and the rest of which were unemployed parent or two parent (UP) cases. Of these 15,000, 10,000 were allocated to an experimental group which, along with the rest of the state, experienced a reduction in AFDC cash bene ts while the remaining 5,000 were allocated to a control group which was consistently exposed to the set of AFDC rules in place as of September The CWPDP data are organized into administrative and survey modules measuring the behavior of study participants at di erent levels of aggregation. The administrative records contain monthly data from December 1992 through September 1997 on sampled welfare cases, the assistance units inside of those cases, and the individuals within those assistance units. The ability to directly measure the assistance unit (AU) is a key advantage of the administrative data since AU composition is a key determinant of the transfers program participants receive. There is also a retrospective administrative component covering the period from January 1987 through December The administrative modules were collected directly from state and county welfare o ces with monthly information on program participation, eligibility, hours of work, earnings, payment amounts, and the relevant program rule inputs (unearned income, assets, assistance unit size, etc.). In addition to the large administrative samples, two rounds of household survey data were collected between 1993 and 1996 on roughly 3,000 cases, providing contemporaneous and retrospective information on a variety of topics including program par- 3

5 ticipation, earnings, hours of work, and demographic information. Finally, a data validation was conducted by the California Review and Evaluation Bureau (REB) which involved interviewing case workers, inspecting les, and collecting non-computerized administrative information including hours worked. Data from 7,000 cases validated by the REB are available providing data through Incentives and Reduced Form Impacts The CWPDP reform consisted of a bundle of treatments. The key elements of the reform were: 1 (1) Maximum aid payments were reduced gradually by 5-25% depending on AU size, (2) The 100 hour work limitation for AFDC-U cases was removed; (3) The four month time limit on the $30 and 1/3rd income disregard was eliminated, which changed the statutory tax rate on earnings after four months of participation from 100% to 33%. Each of the three reforms provided incentives to work through income and substitution e ects. They also provided incentives to participate in other programs such as food stamps both because of income e ects (less transfer income post-reform) and substitution e ects (food stamp payments increase when AFDC payments fall). Focusing on AFDC-FG cases, Hotz, Mullin, and Scholz (2002, 2003) nd relatively large experimental impacts of the CWPDP on employment rates with treated cases experiencing a nearly 10% increase in employment after four years and similar sized increases in mean earnings. However, they nd no e ect on AFDC participation. Our own preliminary analysis con rms these results but suggests substantial heterogeneity in the program s impact. Figure 1 plots participation rates in AFDC and food stamps (on a reverse scale) in the experimental and control groups among cases participating in AFDC-U or AFDC-FG in For the AFDC-FG sample there are no detectable di erences in the behavior of the experimental and control groups. The dominant pattern in the data is mean reversion as sample cases in both groups (who were all on the welfare rolls in 1992) eventually nd employment and exit. For the AFDC-UP sample, however, we nd sizable reductions in welfare participation and increases in food stamp participation suggesting important behavioral responses to treatment. In results not shown we also nd large reductions in mean AFDC transfer amounts and increases in mean food stamp transfers in each of these subpopulations. Another outcome of interest is earnings while on welfare which we can measure using administrative records. The reduction in implicit tax rates accompanying the reform provide incentives for AFDC recipients to work part time. Figure 2 plots the 85th and 98th percentiles of earned income reported to caseworkers in the two samples over time by treatment status. 3 Though the earnings quantiles of both groups increase over time due to mean reversion, the AFDC-FG experimental group experiences slighty larger earnings growth than controls. The AFDC-U sample exhibits substantially larger e ects especially at the 98th percentile. The di erences in both samples between impacts at the two quantiles is suggestive of important treatment e ect heterogeneity as in Bitler, Gelbach, and Hoynes (2005). In results not shown we also nd that these e ect patterns di er by education status with high school graduates exhibiting larger earnings responses in both samples. We conclude from this evidence and the results of Hotz, Mullin, and Scholz that large (yet substantively di erent) behavioral responses are present in both samples, that important interactions exist between the public assistance programs, and that heterogeneity is substantively important in interpreting the CWPDP e ects. We turn now to a description of a model capable of formalizing these reduced form ndings. The Model The model builds on the static joint labor supply and program participation models of Keane and Mo tt (1998), Hoynes (1999), and Heim and Meyer (2003). Decision making is modeled at the household level and at monthly frequencies. All the choices are discrete. Households make decisions sequentially but 1 See Hotz et al (2002) for a more in depth discussion. 2 Our sample is composed of all cases with at least one federal adult in Participation rates are time averaged across individuals and the con dence intervals account for non-independence of individual observations across time. 3 Lower percentiles are dominated by zeros. 4

6 because they are myopic their discount rate equals zero. For ease of exposition we next present the key features of the model for a household containing at most one assistance unit per program. Preferences Households have preferences over consumption, leisure, and program participation. The preferences of household i in calendar month t are represented by a utility function u(c it ; H h it ; P it ; U it; " U it; U i ) where C it is household consumption, h it is a vector of monthly hours worked by the adults in the household (possibly zero, at most H), P it is a vector of size M whose elements are the program participation indicators Pit m, and U it collects pre-determined and possibly time-varying household-speci c preference shifters such as demographics. The stochastic error " U it represents idiosyncratic taste shocks and U it represents unobserved and time-invariant household tastes towards consumption, leisure, and program participation. The speci cation of u (:) is chosen to allow for rich income e ects, as in Keane and Mo tt (1998) and Van Soest, Das, and Gong (2002). We also allow for state dependence as re ected by the fact that U it may contain past choices. State dependence in preferences is a way to capture psychic and monetary costs associated with, for instance, program take-up, such as the stigma e ect rst estimated by Mo tt (1983). These costs have proven to be quantitatively important to reproduce empirical program participation patterns such as nonuniversal take up by eligible households as well as persistence in welfare dependency. Finally, the speci cation of u (:) incorporates participation costs as in Heim and Meyer (2003). Constraints: Budget Constraint The household is subject to a static budget constraint: its aggregate composite consumption in period t equals per-period income (from various sources), minus expenses in child care, medical care, and housing, and minus a monthly fraction of last year taxes. Thus, the per-period budget constraint is given by C it + X it = Y it T it + B it where X it are total expenses, Y it is income from all sources other than transfer programs, B it is the sum of the transfers received from all the programs taken up during the period, and T it is the tax bill. Income from all sources is the sum of earnings by all the adults in the household and unearned non-transfer income: Y it = X E ijt + Yit U. Individual earnings E ijt are the product of the hourly wage rate and the number of hours worked in the month: E ijt = w ijt h ijt. The amount of transfer income is given by B it = B P it ; Eit R; hr it ; Y U;R it ; X it ; A B t ; B it where A B t summarizes the program rules in e ect at time t for all M programs. Total transfer income is the sum of transfer income from each program, Bit m, but the function B is not simply the sum of the program-speci c program functions because programs interact. AFDC grants, for instance, are treated as income for determining the FS grant amounts. Likewise, receipt of Supplemental Security Income (henceforth SSI) makes a person ineligible for AFDC while AFDC receipt entails categorical eligibility for FS. The function B captures these interactions in eligibility and grant amounts. The program rules are imposed exactly, without any approximation or smoothing. The speci cation of B re ects that total transfer income depends on earnings, unearned income, and hours of work as reported to the program agencies (Eit R; hr it ; Y U;R it ). That is, we allow for the possibility of under reporting of income and hours to the welfare agencies and incorporate the fact that di erent programs depend on di erent de nition of countable gross income and di erent de nitions of assistance unit. This is re ected in Eit R; it hr being vectors collecting earnings and hours of work across programs and household members. Previewing the discussion below, we note that a unique feature of our data is that it contains reported earnings, hours, and unearned income. Only household characteristics that are relevant for the M programs, B it, are included in B. Elements of B it are, for instance, the assistance unit size AU it m. We do not allow for discrepancies between actual and 5

7 reported values of this variable. Finally, the tax bill T it derives from the application of the exact federal and state income tax rules, including EITC; it is an element of the state space of the household in month t. Constraints: Wage O er Functions and Hours Constraints The labor supply choices available to an agent are constrained by the existing production and search technologies. Speci cally, let H denote the set of possible hours alternatives: from zero hours per month to H hours per month. Zero hours correspond to no o er being received in month t and H hours correspond to a full time job o er. The granularity of H entails one hour increments. In each month, household member j may receive up to H job o ers corresponding to any of the hours alternatives in the set H. Let G H be a distribution function with support equal to the (ordered) collection of all subsets of H. At the beginning of the month, member j receives a draw from G H where G H is allowed to depend on individual characteristics and aggregate labor market conditions: G H :ja H t ; H it. :ja H t ; H it The presentation above reveals that we do not model job search explicitly but we think of G H as a reduced form decision rule in an environment in which search is stochastic and in which there are complementarities in workers hours for output production as in Chetty et al. (2009). Exclusion restrictions are implicitly imposed on this (constrained) search decision rule as re ected in the fact that G H depends only on a subset of the agent s information set. In implementation the function G H may be further restricted by, for instance, assuming independence across o ers. In what follows it is convenient to denote by " G ijt the source of randomness in the hours draw. Before describing wage o ers we comment on why we choose to allow for hours constraints. On the one hand, these constraints generate frictions in the response of labor supply to policy changes. Hence this modeling choice is capable of rationalizing observed cross-sectional bunching of monthly hours at particular values as in Dickens and Lundberg (1993). On the other hand, and more practically, by constraining the hours choice we do not have to select in advance a particular coarsening of the choice set and we preserve computational tractability despite the very complex and non linear budget set. Conditional on member j receiving at least a job o er, the wage o ered to her is common across job o ers and is given by log w ijt = 0 W ijt + " W ijt + W ij where W ij is a time invariant and person-speci c skill endowment and "W ijt is the non-predetermined component of the wage o er. In accordance with the ndings in Card and Hyslop (2005), and consistent with the relatively short period during which agents are observed in our data, we do not allow for skill accumulation. Instead, all the elements of W ijt are assumed to be time invariant, they are education, sex, and age. Choice Set and Objective Function Let ( it ; " it ; i ; A it ) represent the union of the elements in, respectively, W it ; B it ; U it ; n o H it, " U it ; " W ijt ; "G ijt, U i ; W i, and A B it ; A H it. Given (it ; " it ; i ; A it ), the household chooses in each month t its labor market and program participation (P it ; h it ). Decisions are made subject to the budget constraint, the wage o er functions, and the hours constraints and with the objective of maximizing expected discounted utility. Due to the assumed myopia, the choice problem is static and decisions involve a simple comparison of per-period alternative-speci c utility values where the (unconstrained) set of alternatives is given by the Cartesian product of H and P. To reduce complexity, we choose not to explicitly model as choice variables the assistance unit sizes and other characteristics of the household such as the presence of a spouse and the number and ages of the children. The same applies to expenditures choices and unearned income. Instead, we posit restricted reduced form decision/outcome rules. Such rules incorporate a dependence on the permanent household traits U i ; W i. Let N it be the subset of it containing such variables. We specify a vector valued rule Q such that N it = Q N it 1 ; U i ; W i where some of the elements of N it 1 may be excluded for some of the elements of N it. Because agents are myopic they do not need to know Q(:). 6

8 Estimation: The Behavioral Structure Estimation of the parametric model requires functional form assumptions for u; Q; G H as well as distributional assumptions for the time-invariant household traits and the idiosyncratic stochastic shifters (; "). The measurement structure is then overlaid on the behavioral structure. We describe each component next. We conclude the discussion of estimation by presenting the likelihood function for a generic set of distribution functions. In doing so we also explain how we will deal with the problem of missing state space elements stemming from the non-overlap in the time span of the various data sources. Unobserved Heterogeneity We allow the households in the population to be (permanently) heterogeneous in their skills and their preferences ( W ; U ). These dimensions of heterogeneity are assumed to be jointly distributed according to the cumulative function F. It is assumed that is independent of ". Irrespective of the speci cation of F, and due to the choice-based nature of the sample, the distribution of permanent traits that is recovered pertains to the population of households which are welfare recipients at the time of treatment assignment. Idiosyncratic Shocks We assume that the stochastic processes governing the job o ers of the various adult members of the household are independent conditional on aggregate conditions and observable individual n and household o characteristics: that is " G ijt is independent of "G ij 0 t. All of the elements of " it = " U it ; " W ijt ; "G ijt are serially n o uncorrelated. Their joint distribution is denoted by F ". Typically, " U it ; " W ijt are assumed to be jointly normal and restrictions are imposed on the variance-covariance matrix. We will do the same. Estimation: The Measurement and Reporting Structures Before illustrating the speci cation of the measurement and reporting structures, it is useful to consider which of the variables appearing in the decision problem are available in the data (though possibly measured with error), in which speci c source they can be found, and at which frequency and over which spells of time they are recorded/gathered. Table 1 summarizes this information. The table highlights one of the remarkable features of our merged administrative and survey data. We have access to multiple measures for all the key variables of the model and these measures are available at several points in time for the same person/household. Thus, all the model variables are observed but for the time-invariant household traits and the idiosyncratic stochastic shifters, namely (; "). Speci cation of the Measurement and Reporting Structures The presence and features of measurement error in survey data are extensively analyzed in Bound, Brown, and Mathiowetz (2001). The possibility of mis-reporting of program-relevant variables to program agencies is documented in e.g. Hotz, Mullin, and Scholz (2003). If measurement and reporting errors are present, disregarding them will lead to incorrect inference which may translate into incorrect predictions. Accordingly, we exploit our merged data, in combination with the model, to account for either type of errors. Due to space limitations, we next present only the measurement and reporting equations for earnings. We have two measures: earnings contained in the administrative data, E AD;m ijt, and earnings reported in the survey, Eijt SV. We assume that EAD;m ijt coincide with earnings reported to the welfare agencies, E R;m ijt. In turn, E R;m ijt depends on actual earnings E ijt, the policy rules, and a random factor that captures variation in, among other things, the ability to conceal earnings information, E;R;m ijt. Finally, Eijt SV equal actual earnings up to an addittive mean-zero serially uncorrelated measurement error, E ijt. Formally, E AD;m ijt = E R;m ijt = E m (E ijt ; R;E;m ijt ; A B;m t ), Eijt SV = E ijt + E ijt. The precise speci cation of the (mis-)reporting function E m (:) extends the one employed in van der Klaauw and Wolpin (2009) which pertains to under-reporting of assets in survey data. Our speci cation allows the mis-reporting to depend on the policy parameters. 7

9 Identi cation of the measurement error structure relies heavily on (a) the assumed parametric behavioral model, and (b) the speci cation of the measurement error process. We share this identi cation strategy with all model-based estimation approaches that lack measures with and without error of the same variable and that can rely on no or just a very limited set of exogenous variables (e.g Keane and Wolpin, 2001). Our approach to accounting for measurement and reporting errors implies that parametric (mis-) reporting functions such as E m (:) need to be estimated along with the variance-covariance matrix of. We always assume that survey and adminstrative data error are independent and that the survey measurement errors for di erent variables are independent. Estimation: The Likelihood Function The behavioral and measurement/reporting models imply a joint distribution of observables. Next we provide a generic illustration of an individual s contribution to the likelihood function. Let O denote the variables produced explicitly by the model, we refer to these variables as outcomes. Outcomes include the choice variables, labor supply and program participation, and those variables that are functions of choice variables such as earnings and grant amounts. The model accounts for endogeneity of e.g. unearned income and wages but neither are, according to our de nition, outcomes. Let 1 denote the set of variables that we assume to be fully exogenous. Namely, treatment status, age, race, education, county of residence of the individual and the time series of aggregate economic and policy variables. 1 are always observed and have no error. Let 2 = collect the permanent unobserved household traits. Let 3 collect the remaining state variables. We use O 3 to denote measures of 3 and R the vector of indicators denoting which measures of 3 are available. R varies over time for the same household due to data availability The behavioral model can thus be synthetically represented as O = O ( 1 ; 2 ; 3 ; b ). Accounting for selection into the CWPDP sample, we can write the conditional joint distribution of the observables as f O; R O AF DC 3 ; Rj 1 ; P1 = 1; where = ( b ; mr ) collects the model as well as the measurement- and reporting-related parameters and where P AF DC 1 is the indicator of program participation at the time of random assignment. Exploiting the structure of the behavioral model (in particular the implied exclusion restrictions), the distributional assumptions (e.g. lack of serial correlation in "), an assumption of missing at random in connection with di erences in period coverage across sources, and the measurement and reporting structures, we can rewrite the conditional joint distribution as the product of two factors. The rst factor is Model for Missing 3 z } { f (Rj 1 ; 1 ) (1) and the second factor is Z f O 3 j 3 ; 1 ; 2 {z } Measurement/Reporting Model for 3 f 0 ( 1 ; 2 ; 3 ; b ); {z } 3 AdF Behavioral Model {z } Measurement/Reporting Model for O AF DC 2 ; 3 j 1 ; P1 = 1; 4 {z } Model for Unobservables (2) where the union of 1, 2, 3, 4, and b coincides with. Our main focus is in recovering b, the parameters of the behavioral model, free of the sample selection. Our ability to correct for the initial sample selection (except for the dependence of the distribution of the permanent traits ) rests on (1) the exclusion restrictions imposed in the behavioral model, (2) the observation that, post-assignment, households transit in and out of welfare, and (3) the purging through the model of the dependence of behavior on aggregate conditions. On a related note, the data o ers the possibility of incorporating a 9 year monthly history of program participation prior to random assignment. We will explore ways of using this data. 8

10 In practice, the presence of missing state variables (due to the survey not covering all the periods contained in the administrative data) makes the use of simulation methods the preferred approach. The model will thus be estimated via Simulated Maximum Likelihood. Because of the presence of missing state variables and measurement and reporting error, the implementation of the estimation follows the approaches in Keane and Sauer (2009 a, b) and Keane and Wolpin (2001). Finally, estimation will rst be conducted with only CWPDP control group observations as in Todd and Wolpin (2006) and then with both control and experimental group observations. The objective is to explore the consequences of exploiting the exogenous variation in the policy induced by the experiment. Validation and Speci cation Analysis The experimental component of the data provides us with the opportunity to carry out a stringent validation exercise. We will quantitatively assess the extent to which the estimated model is able to reproduce the experimental impacts of the policy changes. Our focus will be on assessing how use of exact rules, detailed modeling of choices, and the measurement structure of the model in uence our ability to t the experimental e ects. We will answer these questions by considering nested versions of the estimated behavioral model including versions that rely on a coarser speci cation of the choice set, versions that use only survey data and ignore reporting errors, and version that ignore programs such as food stamps and approximate eligibility criteria. We will also compare the results of predictions generated by versions of the model estimated on the control group only to those estimated on the pooled sample and hence utilizing experimental variation. We will rely on di erent metrics for the purpose of validation. To stay close to the experimental literature, we will look at quantile treatment e ects on transfer amounts and earnings for various subpopulations (Bitler, Gelbach, and Hoynes, 2005). We will also employ metrics of direct policy relevance such as the grant cost of the programs, which the administrative data allows us to measure exactly under each of the policy scenarios, the size of the AFDC caseloads, and the extent to which changes in AFDC rules induce changes in other programs caseloads. The Next Step The above exercise will allow us to determine what modeling and measurement features matter for accurately predicting treatment e ects in a given population. A more ambitious goal, however, is to be able to predict the results of interventions in di erent populations. Without a behavioral model researchers have little basis on which to forecast the e ects of such interventions particularly if they involve di erent bundles of incentives than previously studied. The second paper, which we now brie y describe, will assess the extent to which these goals can or cannot be met by use of the sort of detailed choice model developed in the rst paper. 2.2 Labor Supply Models and the External Validity of Randomized Welfare Experiments Motivation A common criticism of randomized experiments is that their results are not easily generalized to new environments (Heckman and Smith, 1995; Deaton, 2009). Large scale social experiments such as the Negative Income Tax rst gained popularity in the economics literature in the study of labor supply. The original ambition of that literature as exempli ed by Orcutt and Orcutt (1968) was to use the experimental policy variation to identify structural preference parameters which could be used to forecast the results of national policy changes. Three decades after the rst studies of the Negative Income Tax experiments we have access to a plethora of sophisticated experimental datasets from which a number of qualitative conclusions have been reached. 4 Until recently, however, there have been relatively few attempts to build quantitative structural models exploiting the variation in these samples (Lise Seitz, and Smith, 2004; Todd and Wolpin, 2006; Attanasio, Meghir, 4 See Grogger, Karoly, and Klerman (2002) for a recent review. 9

11 and Santiago, 2005) and to our knowledge there have been no attempts to synthesize the results of diverse experiments through the use of such models. We can think of two main reasons for this state of a airs. First, there is the belief that extrapolations based on parametric models rely too heavily on the untested (and untestable) assumptions embedded in those models. Second, there is the belief that most experimental datasets are not amenable to a model-based approach. Most experiments are short-lived, the data collection is limited in scope and in time coverage, and the samples subject to randomization are typically choice based. This criticism is exempli ed for the case of welfare experiments by Fang and Keane (2004) who contend that the state waiver experiments are so deeply awed as to be uninformative in either forecasting future participation patterns or understanding the mechanisms driving the changes associated with the 1996 PRWORA reforms. This paper examines the relevance of both sets of concerns directly. We will assess the extent to which a common model (and the implied estimated parameters) can rationalize the results of multiple welfare experiments. The exercise entails using the model parameters estimated on the CWPDP data to generate forecasts of the results of a set of experiments that were conducted in di erent states and time periods the very same experiments criticized by Fang and Keane (2004). Experimental Samples We consider two experimental datasets: Jobs First (JF) and the Minnesota Family Investment Program (MFIP) both of which contain administrative data on earnings and welfare payments assembled by the Manpower Demonstration Research Corporation (MDRC). The Jobs First program provided large nancial work incentives by substantially raising the earnings disregard and reducing the implicit tax rate on earnings for welfare participants. Bitler, Gelbach, and Hoynes (2005) found a pronounced pattern of quantile treatment e ects on earned income and transfer amounts in this sample which they interpret as being qualitatively consistent with fairly elastic labor supply among program participants. The MFIP program involved two experimental samples: one with nancial incentives and one with nancial incentive and work requirements. The MDRC nal report nds impacts of both treatments on a variety of outcomes. Building Policy Forecasts A number of challenges are involved in the out-of-sample prediction exercise that we will carry out. Importantly, these challenges are bound to be encountered in any situations in which one is to combine data sources that were collected independently. The rst challenge stems from the fact that the CWPDP, JF, and MFIP interventions were applied to di erent populations. For instance, inclusion in the MFIP experiment was conditional on long-term welfare recipiency, as of the assignment date, while a minimal history of a month on welfare was su cient for inclusion in CWPDP. Even in the absence of this di erence in selection rules, being welfare recipients in California and in Connecticut at the assignment date may be very di erent because of the underlying di erences in these states welfare generosity and economic conditions. The second challenge stems from the fact that there are di erences in the content of the data available across experiments. The administrative assistance data available for the JF and MFIP experiments is limited to AFDC and FS grant amounts and does not contain the variables that enter the grant functions such as assistance unit size and income sources. Likewise, several key variables in the public use survey data accompanying the JF and MFIP experiments were top coded or otherwise coarsened to protect privacy. 5 Moreover, there are di erences in the survey instruments employed across experiments. UC Data was responsible for designing and implementing the CWPDP survey while MDRC was in charge of the survey component for the JF and MFIP experiments. Returning to our likelihood schematic in (1) and (2), it becomes apparent that the only estimates that 5 For instance, the JF survey data does not allow us to determine the exact number of children below the age of 17 that are present in the household at the time of the survey. Also, exact dates were blanked out before the public release of the data. This prevents an exact month by month merge of the survey to the administrative monthly records. 10

12 can be ported for prediction purposes to the new samples are the behavioral parameters b. The remaining parameters governing the measurement structure of the data ( 1 ; 2 ; 3 ) and the distributions of unobserved heterogeneity (a subset of 4 ) need to be reestimated. We will do this by making use of pre-treatment and control observations in the forecast samples. After retailoring the measurement equations to the new datasets we will then maximize the simulated likelihood subject to the restriction that the behavioral parameters b equal the value estimated in the CWPDP. With new estimates of ( 1 ; 2 ; 3 ; 4 ) we will then be able to simulate the impacts of the relevant intervention and compare them to the reduced form patterns of treatment e ects found in those samples. An important issue we hope to examine is how our conclusions from the previous paper change when studying out-of-sample forecasts. How important for example is allowing for measurement errors or using exact policy rules for forecasting the results of new experiments? And how important is access to experimental variation in the estimation sample for this exercise? Finally, because the data available in the two validation samples is coarser than that in CWPDP it is interesting to determine the advantages (if any) that improved measurement in our estimation sample will provide for forecasting patterns of treatment e ects in these new samples. Welfare Implications and Unbundling Provided that some of the models are able to yield reasonably accurate predictions, we will conclude with an assessment of the positive and normative implications of the model for evaluating di erent policy interventions. We provide here a brief list of policy relevant questions that we will seek to answer: (1) What are the e ects of pairing the work requirements of the MFIP intervention with the milder nancial incentives of the CWPDP intervention? (2) Are treated populations made better o by the various experimental interventions? (3) What combination of welfare incentives is socially optimal (i.e. raises welfare at minimum cost)? and (4) What are the likely impacts on welfare and program participation of recently proposed changes in the generosity of the California state welfare system? 11

13 Fraction of Cases Active (AFDC) Fraction of Cases Active (Food Stamps) Fraction of Cases Active (AFDC) Fraction of Cases Active (Food Stamps).5 Figure 1 Welfare Participation (AFDC FG Sample) Experimental vs. Controls Year AFDC (Controls) Food Stamps (Controls) AFDC (Experimentals) Food Stamps (Experimentals) note: Shaded Regions and Brackets Represent 95% Confidence Intervals Welfare Participation (AFDC U Sample) Experimental vs. Controls Year AFDC (Controls) Food Stamps (Controls) AFDC (Experimentals) Food Stamps (Experimentals) note: Shaded Regions and Brackets Represent 95% Confidence Intervals 12