Transcription

1

2

3

4

5

6

7 TABLE L-7 Explanatory Variables Used in Previous Studies MDTA Data CLMS-Based Studies Dickinson, Johnson Bryant Study Ashenfelter (1978) Ashenfelter and Card (1985) and West (1987) and Rupp (1987) Program, Year, MDTA classroom trainees 1976 CETA trainees CETA trainees enrolled Two cohorts of CETA Outcome Variable enrolled in first 3 months and 1978 annual social in annual trainees and 1978 of annual security earnings. security earnings. annual social security social security earnings. earnings. Local Labor Market No No No No Information Age, Race, Sex Yes Yes Yes Yes Education No Yes Yes Yes Training History No No No No Children No No No Yes Employment Histories No No Yes (recent) Yes (recent) Hours Worked No Yes Yes Yes Unemployment No No Yes (recent) Yes (recent) Histories Welfare Receipt? No No Yes No Earnings Histories 5 years pre-program 2 years pre-program Same as Ashenfelter 4 years pre-enrollment 5 years post-program 2 years post-program and Card (1985)

8 Matching Criteria None specified (a) 1975 earnings # $20K Matching based on a Matching on 1976 earnings, and household income # $30K; metric over a vector of change in earnings , (b) In labor force in March, predictors of labor force status, 1976; earnings including family income, in labor force (c) Age greater than 20. lagged earnings (1970- in 1975 or at March ), unemployment in interview and demographics. 1975, worked in public sector, in labor force in March 1976 and demographics. Source: Heckman, Ichimura and Todd (1997), Table 1. Notes: TABLE L-7 (continued) 1. "Used" means employed in the matching process and/or included in the outcome equation. 2. CLMS = CETA Longitudinal Manpower Survey. The CLMS data matched Social Security longitudinal earnings records to CETA participants and CPS comparison group members from the March 1976 and 1977 CPS. All of the CLMS-based studies use the social security earnings data except for Bassi (1983), which also uses the CPS earnings data. All of the personal and family information available in the CPS, including short-term employment and labor force participation histories, are available in the CLMS but not necessarily used in the analyses based upon it. 3. "Matching Criteria" indicate the criteria for membership in the comparison group. This is sometimes referred to as "screening" in the literature.

9 TABLE L-8 Explanatory Variables Used in Previous Studies CLMS-Based Studies NSW (Supported Work) Data NJS Data Study Bassi (1983, 1984) LaLonde (1986) Fraker and Maynard (1987) Heckman, Ichimura, Smith and Todd (1998) Program and Outcome Variable CETA, 1977 and 1978 annual social security earnings NSW, 1978 annual social security and PSID survey NSW, 1977, 1978 and 1979 annual earnings for AFDC women and youth NJS, monthly survey earnings in the 18 months after random assignment or measured eligibility Local Labor Market No No No Yes Information Age, Race, Sex Yes Yes Yes Yes Education? Yes Yes Yes Training History No No No Yes (partial) Children? Yes Yes Yes Employment Histories? No No Yes Hours Worked? No No Yes Unemployment Histories? No No Yes (one year) Welfare Receipt? Yes Yes Yes (one year) Earnings Histories Same as Bryant and Rupp (1987) Two years pre-program Two years pre-program Five years pre-program Two years post-program Two years post-program Two years post-program

10 Matching Criteria (Criteria for Membership in Comparison Sample) Same as Bryant and Rupp (1987); also uses a random sample from the March 1976 CPS. PSID: household head in CPS: March 1976 CPS earnings matched to SSA earnings; screens based on 1976 personal and household income. Three samples: I: eligible in sample period; screen out in-school youth; AFDC women match on age of youngest child and welfare receipt. II: eligible in sample period; cell matching based on predictors of 1979 SS earnings including prior earnings, change in earnings, education, family income, and demographics. III: eligible in sample period; match on earnings estimated on eligible non-participant sample, age, and sex. Within age and sec groups, match on propensity score based on site, race, age, schooling, marital status, labor force status history, number of recent jobs, training history, house-hold size, and house-hold recent earnings. Source: Heckman, Ichimura and Todd (1997), Table 1. Notes: 1. "Used" means employed in the matching process and/or included in the outcome equation. 2. CLMS = CETA Longitudinal Manpower Survey. The CLMS data matched Social Security longitudinal earnings records to CETA participants and CPS comparison group members from the March 1976 and 1977 CPS. All of the CLMS-based studies use the social security earnings data except for Bassi (1983), which also uses the CPS earnings data. All of the personal and family information available in the CPS, including short-term employment and labor force participation histories, are available in the CLMS but not necessarily used in the analyses based upon it. 3. "Matching Criteria" indicate the criteria for membership in the comparison group. This is sometimes referred to as "screening" in the literature. 4. "?" indicates that the study does not specify the variables used.

11

12

13

14

15

16

17

18

19

20 Nominal Dollars 1300 FIGURE 1 Mean Self-Reported Monthly Earnings National JTPA Study Controls and Eligible Non-participants (ENPs) and SIPP Eligibles Male Adults k-20 k-15 k-10 k-5 k k+5 k+10 k+15 Month Relative to Random Assignment (Controls) or Eligibility (ENPs and SIPP Eligibles) Source: Heckman and Smith (1998b) SIPP Eligibles JTPA ENPs JTPA Controls

21 FIGURE 2 Mean Annual Earnings Prior, During, and Subsequent to Training for 1964 MDTA Classroom Trainees and a Comparison Group: White Males $6,000 $5,000 $4,000 $3,000 Earnings $2,000 $1,000 $ Year Source: Ashenfelter (1978). Trainees Comparison Group

22 $6,000 FIGURE 3 Mean Annual Earnings for 1976 CETA Trainees and a Comparison Group Males $5,000 $4,000 $3,000 Earnings in 1967 Dollars $2,000 $1,000 $ Year Source: Ashenfelter and Card (1985). Trainees Comparison Group

23 2500 FIGURE 4 National Supported Work (NSW) Average Annual Earnings Treatments, Controls, and Matched CPS Comparison Group AFDC Recipients Enrollment Period Earnings Year Source: Fraker and Maynard (1987) NSW experimentals NSW Controls Matched CPS Comparison Group

24 160 FIGURE 5 Earnings of Participants in Swedish UI Training in 1991 and Two Comparison Groups Adult Males--Ages Mean Annual Earnings in Thousands of 1995 SEK Source: Regner (1997) Year Trainees Comparison Group 1 Comparison Group 2

25 FIGURE 6 Earnings of 1991 Participants in Norwegian Labor Market Training Programme and a Randomly Assigned Control Group All Participants Mean Annual Earnings in 1994 NOK Year Source: Raaum and Torp (1997) Treatment Group Control Group

26 i_ * ih?@ U?},t 4@ Lht L4T@htL? BhLTG 5 5@4T*ic #iti?_i? V@h@M*iG +i@*,@h??}t? b.h E '?i@hit?i}mlhc 5 ' UL44L? tttlh hlti?t ) i@? f f wlu@* w?i@h wlu@* w?i@h 5ULhi L_i* #g %L 5 %L EM ' f EM ' ef w@*l?_i 5@4T*i b.d. DDD 2.f HH 2bb Hf e #` hlt 5ULhi E2DD EDbS Eeb ES2H ED2b Ee. I Lu mhhs 4T@U f I SI fi bdi e.i DSI S2I #` 5@4T*i f2b ef. D 2. 2S HH S. bs#` hlt 5ULhi EfS ESbH ES.2 E.2 EDb ESf I Lu m.be 4T@U D.eI 2I fi DI DI DI ei,@h*) + t@4t*i f..h S2 De. 2bS e2. 2 b #` hlt 5ULhi EeS E 2eD E De E ef. E D2 E b2. E I Lu m2.eh 4T@U efei 2HI 2I b.i H.I 2DI HfI w@*l?_i 5@4T*i f22. Sf DHS 2e2 DS2 2.fH w@*l?_i hlt 5ULhi E2bS E edb E 2bb E ef. E SD EbSb I Lu mhhs 4T@U DeI efsi 2efI efdi 2SeI ef2i fsi

27 TABLE ST-2 Descriptive Statistics for Adult Male Experimental and Comparison Group Samples NSW Experimental Samples Comparison Groups Variable Lalonde Dehejia- Wahba Early Random Assignment CPS sample PSID sample Age (6.63) (7.10) (6.75) (11.05) Education (1.70) (1.79) (1.6) (2.87) Black (.40) (0.37) (0.38) (0.26) Hispanic (0.31) (0.28) (.30) (0.26) Married (0.37) (0.37) (0.40) (0.45) No H.S. Degree (0.41) (0.41) (0.43) (0.46) Real Earnings in 1974 (6221) (5364) (6718) (9570) Real Earnings in 1975 (5066) (3151) (3894) (9270) Real Earnings in 1978 (6253) (6632) (7582) (9647) Real Earnings in 1979 (11028) Zero Earnings in 1974 (0.50) (0.44) (0.50) (0.32) Zero Earnings in 1975 (0.49) (0.48) (0.49) (0.31) (10.44) (3.08) 0.25 (0.43) 0.03 (0.18) 0.87 (0.34) 0.31 (0.46) (13407) (13597) (15555) 0.09 (0.28) 0.10 (0.30) Experimental Impact (1978 earnings) 886 (488) 1794 (670) 2748 (1005) Sample Size 297 Treatments 425 Controls 185 Treatments 260 Controls 108 Treatments 142 Controls Notes: Estimated standard deviations in parentheses. Robust standard errors are reported for experimental impact estimates.

28 TABLE ST-3 Dehejia and Wahba (1998,1999) Sample Composition Month of Random of Random Assignment and Earnings Months Before Random Assignnment Number in Cell, Row Percentage and Overall Percentage Shaded Area Indicates DW Sample Month of Random Zero Earnings in Months Assignment Before RA August July January December November October April March February January Non-Zero Earnings in Months Before RA

29 TABLE ST-4 Deheja and Wahaba (1999a) Propensity Score Model Coefficient Estimates (Estimated Standard Errors in Parentheses) LaLonde Experimental DW Experimental Sample Sample Early RA Experimental Sample Variable CPS PSID CPS PSID CPS PSID Age (0.2146) (0.0739) (0.2681) (0.0932) (0.3288) (0.1095) Age squared (0.0068) (0.0011) (0.0085) (0.0014) (0.0104) (0.0017) Age cubed / (0.0678) (0.0837) (0.1029) Years of schooling (0.1909) (0.2433) (0.2249) (0.3028) (0.3739) (0.4639) Years of schooling squared (0.0099) (0.0124) (0.0120) (0.0160) (0.0193) (0.0246) High school dropout (0.1929) (0.2564) (0.2588) (0.3481) (0.3003) (0.3877) Married (0.1471) (0.1747) (0.1932) (0.2416) (0.2149) (0.2545) Black (0.1445) (0.2004) (0.2000) (0.2998) (0.2451) (0.3211) Hispanic (0.1879) (0.3687) (0.2637) (0.5426) (0.3188) (0.5441) Real earnings in ( ) ( ) ( ) ( ) ( ) ( ) Real earnings in 1974 squared 1.54e-09 (5.0e-10) 1.64e-09 (6.87e-10) 1.86e-09 (6.32e-10) Real earnings in ( ) ( ) ( ) ( ) ( ) ( )

30 Real earnings in 1975 squared 2.97e-11 (3.9e-10) 5.28e-10 (5.68e-10) 4.10e-10 (5.30e-10) Zero earnings in (0.1693) (0.3788) (0.2209) (0.4340) (0.2398) (0.4360) Zero earnings in (0.1703) (0.3547) (0.1994) (0.3871) (0.2329) (0.3932) Schooling * Real earnings in e-06 (4.4e-06) (6.14e-06) 6.25e-06 (7.15e-06) Zero earnings in 1974 * Hispanic (0.7193) (0.7882) (0.8670) Intercept (2.4165) (1.6405) (3.0398) (2.0629) (3.9116) (2.7713)

31 TABLE ST-5 LaLonde Propensity Score Model Coefficient Estimates (Estimated Standard Errors in Parentheses) LaLonde Experimental DW Experimental Sample Sample Early RA Experimental Sample Variable CPS PSID CPS PSID CPS PSID Age (0.0588) (0.0716) (0.0689) (0.0801) (0.0924) (0.1070) Age squared (0.0010) (0.0011) (0.0011) (0.0012) (0.0015) (0.0017) Years of schooling (0.0362) (0.0502) (0.0435) (0.0568) (0.0550) (0.0729) High school dropout (01972) (0.2306) (0.2438) (0.2664) (0.2986) (0.3108) Black (0.1623) (0.1911) (0.2126) (0.2363) (0.2687) (0.2855) Hispanic (0.2150) (0.3282) (0.2889) (0.3887) (0.3713) (0.4387) Married (0.1531) (0.1804) (0.1908) (0.2115) (0.2355) (0.2513) Working in (0.1396) (0.1635) (0.1739) (0.1861) (0.2104) (0.2203) Number of children (0.0648) (0.0777) (0.0809) (0.0898) (0.0986) (0.1014) Missing children variable (0.3512) N.A (0.3813) N.A (0.4422) N.A. Intercept (0.9800) (1.1894) (1.1759) (1.3367) (1.5639) (1.7471)

32 Sample and Propensity Score Model TABLE ST-6A Bias Associated with Alternative Cross-Sectional Matching Estimators Comparison Group: CPS Adult Male Sample Dependent Variable: Real Earnings in 1978 (bootstrap standard errors shown in parentheses, trimming level used to determine common support is 2%) (1) (2) (3) (4) (5) (6) (7) (8) (9) Mean Diff. 1 Nearest Neighbor Without Common Support 10 Nearest Neighbor Estimator without Common Support 1 Nearest Neighbor Estimator with Common Support 10 Nearest Neighbor Estimator with Common Support Local Linear Matching (bw = 1.0) Local Linear Matching (bw =4.0 ) Local Linear Regression Adjusted Matching a (bw =1.0 ) Local Linear Regression Adjusted Matching (bw =4.0 ) Lalonde Sample with DW Prop. Score Model (255) -555 (596) -270 (493) -838 (628) (529) (437) (441) (490) (441) as % of $886 impact -1101% (29) -63% (67) -30% (56) -95% (71) -147% (60) -156% (49) -162% (50) -159% (55) -150% (50) DW Sample with DW Prop. Score Model (306) 407 (698) -5 (672) -27 (723) -261 (593) -88 (630) -67 (611) -127 (709) -96 (643) as % of $1794 impact -574% (17) 23% (39) -0.3% (37) -1.5% (40) -15% (33) -5% (35) -4% (34) -5% (40) -7% (36) Early RA sample with DW Prop. Score Model (461) (1245) (1354) (1407) (1152) (1927) (1069) (3890) (1124) as % of $2748 impact -404% (17) -283% (45) -132% (49) -197% (51) -87% (42) -125% (70) -80% (39) -112% (142) -123% (41) Lalonde Sample with Lalonde Prop. Score Model (296) (1459) (1299) (1407) (1165) (3969) (1174) (4207) (1178) as % of $886 impact -1154% (33) -406% (165) -240% (147) 405% (159) 264% (131) 402% (448) 306% (133) 388% (474) -266% (133) a. Regression adjustment includes race and ethnicity, age categories, education categories and married.

33 Sample and Propensity Score Model TABLE ST-6B Bias Associated with Alternative Cross-Sectional Matching Estimators Comparison Group: PSID Adult Male Sample Dependent Variable: Real Earnings in 1978 (bootstrap standard errors shown in parentheses, trimming level used to determine common support is 2%) (1) (2) (3) (4) (5) (6) (7) (8) (9) Mean Diff 1 Nearest Neighbor Without Common Support 10 Nearest Neighbor Estimator without Common Support 1 Nearest Neighbor Estimator with Common Support 10 Nearest Neighbor Estimator with Common Support Local Linear Matching (bw = 1.0) Local Linear Matching (bw = 4.0) Local Linear Regression Adjusted Matching a (bw = 1.0) Local Linear Regression Adjusted Matching (bw = 4.0) Lalonde Sample with DW Prop. Score Model (264) (898) (787) -166 (959) -898 (813) (747) (633) -587 (1059) -817 (681) as % of $886 impact -1882% (30) -331% (101) -239% (89) -19% (108) -101% (92) -140% (84) -145% (71) -66% (120) -92% (77) DW Sample with DW Prop. Score Model (330) 361 (924) -82 (1200) 447 (827) -85 (1308) -122 (1362) 143 (633) 693 (2092) 777 (833) as % of $1794 impact -947% (18) 20% (52) -5% (67) 25% (46) -5% (73) -7% (76) 8% (35) 39% (117) 43% (46) Early RA sample with DW Prop. Score Model (555) (1237) (1315) (1487) (1222) (1079) (936) (1036) (1010) as % of $2748 impact -618% (20) -223% (45) -130% (48) -196% (54) -121% (44) -71% (39) -119% (34) -256% (38) -249% (37) Lalonde Sample with Lalonde Prop. Score Model (262) (872) (1080) (872) (985) (976) (964) (1179) (1042) as % of $886 impact -1858% (30) -438% (98) -345% (122) -433% (98) -336% (111) -416% (110) -397% (109) -419% (133) -396% (118) a. Regression adjustment includes race and ethnicity, age categories, education categories and married.

34 Sample and Propensity Score Model TABLE ST-7A Bias Associated with Alternative Difference-in-Difference Matching Estimators Comparison Group: CPS Adult Male Sample Difference Between Real Earnings in 1978 and Real Earnings in 1975 (bootstrap standard errors shown in parentheses, trimming level used to determine common support is 2%) (1) (2) (3) (4) (5) (6) (7) (8) (9) Mean Diff 1 Nearest Neighbor Without Common Support 10 Nearest Neighbor Estimator without Common Support 1 Nearest Neighbor Estimator with Common Support 10 Nearest Neighbor Estimator with Common Support Local Linear Matching (bw =1.0 ) Local Linear Matching (bw = 4.0) Local Linear Regression Adjusted Matching a (bw =1.0 ) Local Linear Regression Adjusted Matching (bw = 4.0) Lalonde Sample with DW Prop. Score Model 867 (314) (563) (520) -929 (554) (539) (483) (472) (524) (475) as % of $886 impact 98% (35) -172% (64) -149% (59) -105% (63) -120% (61) -137% (55) -143% (53) -137% (59) -143% (54) DW Sample with DW Prop. Score Model 2093 (365) 45 (781) -101 (689) -607 (784) -417 (681) -88 (629) -75 (621) -88 (848) -75 (719) as % of $1794 impact 117% (20) 3% (44) -6% (38) -34% (44) -23% (38) -5% (35) -4% (34) -5% (47) -4% (40) Early RA Sample with DW Prop. Score Model 598 (549) 1398 (1342) 1041 (1166) 1689 (1212) 3200 (1108) 2993 (3152) 2909 (917) 1876 (4021) 1461 (1521) as % of $2748 impact 22% (20) 51% (49) 38% (42) 61% (44) 116% (40) 109% (115) 106% (33) 68% (146) 53% (55) Lalonde Sample with Lalonde Prop. Score Model 897 (333) -463 (1290) 1317 (878) -21 (1092) 1229 (862) 192 (1102) 927 (801) 193 (3970) 928 (1466) as % of $886 impact 101% (38) -52% (146) 149% (99) -2% (123) 138% (97) 22% (124) 105% (90) -16% (448) 105% (165) a. Regression adjustment includes race and ethnicity, age categories, education categories and married.

35 Sample and Propensity Score Model TABLE ST-7B Bias Associated with Difference-in-Difference Matching Estimators Comparison Group: PSID Adult Male Sample Difference Between Real Earnings in 1978 and Real Earnings in 1975 (bootstrap standard errors shown in parentheses, trimming level is 2%) (1) (2) (3) (4) (5) (6) (7) (8) (9) Mean Diff 1 Nearest Neighbor Without Common Support 10 Nearest Neighbor Without Common Support 1 Nearest Neighbor Estimator with Common Support 10 Nearest Neighbor Estimator with Common Support Local Linear Matching (bw =1.0) Local Linear Matching (bw =4.0) Local Linear Regression Adjusted Matching a (bw =1.0) Local Linear Regression Adjusted Matching (bw =4.0) Lalonde Sample with DW Prop. Score Model -383 (318) (1033) -148 (931) 608 (1070) -568 (939) 188 (823) 79 (686) -344 (1127) -318 (733) as % of $886 impact -43% (36) -186% (117) -17% (105) 69% (121) -64% (106) 21% (93) 9% (77) -39% (127) -36% (83) DW Sample with DW Prop. Score Model 797 (362) 537 (1031) 725 (1208) 568 (906) 737 (1366) 286 (1414) 803 (792) 287 (2173) 803 (1058) as % of $1794 impact 44% (20) 30% (57) 40% (67) 32% (51) 41% (76) 16% (79) 45% (44) 16% (121) 45% (59) Early RA Sample with DW Prop. Score Model -133 (629) -46 (1131) 1135 (1266) 316 (1276) 1153 (1273) 2118 (1016) 1018 (993) 207 (1375) 111 (1352) as % of $2748 impact -5% (23) -2% (41) 41% (46) 11% (46) 42% (46) 77% (37) 37% (36) 8% (50) 4% (49) Lalonde Sample with Lalonde Prop. Score Model -427 (311) -381 (980) 263 (1115) -364 (929) 238 (1063) -204 (969) 39 (1009) -204 (1323) 39 (1172) as % of $886 impact -48% (35) -43% (111) 30% (126) -41% (105) 27% (120) -23% (109) 4% (114) -23% (149) 4% (132) a. Regression adjustment includes race and ethnicity, age categories, education categories and married.

36 Sample and Propensity Score Model TABLE St-8A Bias Associated with Alternative Regression-Based Estimators Comparison Group: CPS Adult Male Sample Dependent Variable: Real Earnings in 1978 (estimated standard errors shown in parentheses) (1) (2) (3) (4) (5) (6) (7) (8) (9) Mean Diff. Regression With LaLonde Covariates a Regression With DW Covariates b Regression With DW Covariates Without RE74 Regression With Rich Covariates c Difference- In- Differences Difference- In- Differences With Age Included Unrest. Difference- In- Differences a Unrest. Difference- In- Differences With Covariates LaLonde Sample (470) (410) (388) (393) -974 (451) 868 (379) -522 (371) (357) (388) as % of $886 impact -1101% -182% -148% -165% -110% 98% -60% -271% -215% DW Sample (600) -690 (505) -34 (486) -238 (489) 625 (555) 2092 (481) 802 (470) (454) (479) as % of $1794 impact -574% -38% -2% -13% 35% 117% 45% -94% -61% Early RA Sample (811) (655) (620) (629) -301 (707) 1136 (649) -5 (634) (608) (625) as % of $2748 impact -373% -50% -41% -43% -11% 41% -0% -85% -63% a) The LaLonde Covariates are the variables from the LaLonde propensity score model. b) The DW Covariates are the variables from the Dehejia and Wahba (1999a) propensity score model. c) The Rich Covariates model includes indicators for age categories, interactions between the age categories and racial and ethnic group, education categories, a marriage indicator, interactions between the marriage indicator and race and ethnicity, real earnings in 1975 and its square, an indicator for zero earnings in 1975, number of children, and number of children interacted with race and ethnicity. d) Unrestricted difference-in-differences refers to a regression with real earnings in 1978 on the left-hand side and real earnings in 1975 on the right-hand side. In the specification with covariates, the covariates are age, age squared, years of schooling, high school dropout, and indicators for black and hispanic. This specification follows that in LaLonde (1986).

37 Sample and Propensity Score Model TABLE ST-8B Bias Associated with Alternative Regression-Based Estimators Comparison Group: PSID Adult Male Sample Dependent Variable: Real Earnings in 1978 (estimated standard errors shown in parentheses) (1) (2) (3) (4) (5) (6) (7) (8) (9) Mean Diff. Regression With LaLonde Covariates a Regression With DW Covariates b Regression With DW Covariates Without RE74 Regression With Rich Covariates c Difference- In- Differences Difference- In- Differences With Age Included Unrest. Difference- In- Differences a Unrest. Difference- In- Differences With Covariates LaLonde Sample (668) (783) (756) (751) (808) -427 (543) (573) (580) (665) as % of $886 impact -1810% -297% -287% -276% -238% -48% -207% -368% -360% DW Sample (846) -920 (940) (960) (920) -492 (993) 797 (683) -497 (704) (720) (791) as % of $1794 impact -992% -51% -72% -60% -27% 44% -28% -121% -110% Early RA Sample (1311) (1161) (1072) (1057) -820 (1139) -159 (920) (929) (936) (981) as % of $2748 impact -617% -67% -71% -63% -30% -6% -49% -107% -103% a) The LaLonde Covariates are the variables from the LaLonde propensity score model. b) The DW Covariates are the variables from the Dehejia and Wahba (1999a) propensity score model. c) The Rich Covariates model includes indicators for age categories, interactions between the age categories and racial and ethnic group, education categories, a marriage indicator, interactions between the marriage indicator and race and ethnicity, real earnings in 1975 and its square, an indicator for zero earnings in 1975, number of children, and number of children interacted with race and ethnicity. d) Unrestricted difference-in-differences refers to a regression with real earnings in 1978 on the left-hand side and real earnings in 1975 on the right-hand side. In the specification with covariates, the covariates are age, age squared, years of schooling, high school dropout, and indicators for black and hispanic. This specification follows that in LaLonde (1986).

38 Sample and Propensity Score Model TABLE ST-9A Bias Associated with Alternative Cross-Sectional Matching Estimators Comparison Group: CPS Adult Male Sample Dependent Variable: Real Earnings in 1975 (bootstrap standard errors shown in parentheses, trimming level used to determine common support is 2%) (1) (2) (3) (4) (5) (6) (7) (8) (9) Mean Diff. 1 Nearest Neighbor Without Common Support 10 Nearest Neighbor Estimator without Common Support 1 Nearest Neighbor Estimator with Common Support 10 Nearest Neighbor Estimator with Common Support Local Linear Matching (bw = 1.0) Local Linear Matching (bw =4.0 ) Local Linear Regression Adjusted Matching a (bw =1.0 ) Local Linear Regression Adjusted Matching (bw =4.0 ) Lalonde Sample with DW Prop. Score Model (254) 972 (314) 1047 (258) 91 (399) -235 (342) -168 (315) -160 (333) -194 (280) -58 (270) as % of $886 impact -1199% (29) 110% (35) 118% (29) 10% (45) -27% (39) -19% (36) -18% (38) -22% (32) -7% (30) DW Sample with DW Prop. Score Model (172) 362 (248) 96 (199) 580 (339) 156 (268) 0 (196) 8 (203) -39 (274) -21 (212) as % of $1794 impact -690% (10) 20% (14) 5% (11) 33% (19) 9% (15) 0% (11) 0% (11) -2% (15) -1% (12) Early RA Sample with DW Prop. Score Model (354) (1769) (1132) (1357) (953) (3903) (939) (1427) (982) as % of $2748 impact -426% (13) -334% (64) -170% (41) -259% (49) -204% (35) -234% (142) -186% (34) -180% (52) -177% (36) Lalonde Sample with Lalonde Prop. Score Model (224) (1845) (1090) (1889) (1078) (4507) (1103) (1679) (770) as % of $886 impact -1255% (25) -354% (208) -388% (123) -402% (213) -403% (122) -424% (509) -410% (124) -409% (190) -371% (87) a. Regression adjustment includes race and ethnicity, age categories, education categories and married.

39 Sample and Propensity Score Model TABLE ST-9B Bias Associated with Alternative Cross-Sectional Matching Estimators Comparison Group: PSID Adult Male Sample Dependent Variable: Real Earnings in 1975 (bootstrap standard errors shown in parentheses, trimming level used to determine common support is 2%) (1) (2) (3) (4) (5) (6) (7) (8) (9) Mean Diff 1 Nearest Neighbor Without Common Support 10 Nearest Neighbor Estimator without Common Support 1 Nearest Neighbor Estimator with Common Support 10 Nearest Neighbor Estimator with Common Support Local Linear Matching (bw = 1.0) Local Linear Matching (bw = 4.0) Local Linear Regression Adjusted Matching a (bw = 1.0) Local Linear Regression Adjusted Matching (bw = 4.0) Lalonde Sample with DW Prop. Score Model (238) (673) (524) -442 (631) (524) -161 (547) (456) -243 (507) -499 (435) as % of $886 impact -1839% (27) -145% (76) -222% (59) -50% (71) -165% (59) -18% (62) -154% (51) -27% (57) -56% (49) DW Sample with DW Prop. Score Model (194) -176 (443) -807 (676) -121 (304) -822 (746) -408 (518) -660 (435) 406 (962) -26 (644) as % of $1794 impact -992% (11) -10% (25) -45% (38) -7% (17) -46% (42) -23% (29) -37% (24) 23% (54) -1% (36) Early RA sample with DW Prop. Score Model (374) (771) (778) (984) (770) (690) (701) (853) (811) as % of $2748 impact -611% (14) -221% (28) -171% (28) -208% (36) -163% (28) -148% (25) -156% (26) -268% (31) -255% (30) Lalonde Sample with Lalonde Prop. Score Model (213) (624) (712) (779) (740) (597) (629) (604) (623) as % of $886 impact -1810% (24) -395% (70) -374% (80) -392% (88) -363% (84) -393% (67) -402% (71) -395% (68) -401% (70) a. Regression adjustment includes race and ethnicity, age categories, education categories and married.

40 TABLE BIAS-1 Decomposition of Differences in Mean Earnings for Adult Participants in the U.S. National JTPA Study [Mean monthly earnings differences between experimental controls and comparison sample of eligible nonparticipants during the 18 months following the baseline in four sites] Adult Males Adult Females Mean Difference in Earnings = B (47) (26) Nonoverlapping Support = B (35) (13) [-88] [318] Different Density Weighting of Propensity Scores = B 2 (42) (20) [195] [-355] Selection Bias = B (28) (26) [-7] [136] Average Selection Bias When Matching Only in Regions of Common Support Selection Bias as a Percent of Treatment Impact Control Group Sample Size Comparison Group Sample Size Source: Heckman, Ichimura, Smith, and Todd (1996), Table 1, p Notes: 1. The numbers in parentheses are the bootstraped standard errors. They are based on 50 replications with 100% sampling. 2. The numbers in square brackets are the percentage of the mean difference in earnings (row 1) attributable to each component of the bias.

41 TABLE BIAS-2 Decomposition of Differences in Mean Earnings in the U.S. National JTPA Study [Mean monthly earnings differences during the 18 months following the baseline in four sites] Experimental Controls and Treatment Group Dropouts Experimental Controls and SIPP eligibles. Adult Males Adult Females Adult Males Adult Females Mean Difference in Earnings = B (38) (23) (56) (23) Nonoverlapping Support = B (12) (6) (30) (19) [-45] [9] [-104] [206] Different Density Weighting of Propensity Scores = B 2 (16) (10) (44) (16) [11] [-99] [287] [-367] Selection Bias = B (37) (26) (33) (15) [135] [190] [-83] [260] Average Selection Bias When Matching Only in Regions of Common Support (40) (29) (57) (26) Selection Bias as a Percent of 97% 68% 440% 676% Treatment Impact Source: Heckman, Ichimura and Todd (1997), Table 2. Notes: 1. Bootstrap standard errors appear in parentheses. They are based on 50 replications with 100% sampling.

42 2. The numbers in square brackets are the percentage of the mean difference in earnings (row 1) attributable to each component of the bias. 3. Treatment group dropouts (or "no-shows") are persons randomly assigned to the experimental treatment group who failed to enroll in JTPA. 4. The SIPP eligibles are persons in the 1998 SIPP full panel who were eligible in month 12 of the 24 month panel using eligibility definition "B" from Devine and Heckman (1994).

43 FIGURE BIAS - 1 Density of Estimated Probability of Program Participation For Adult Male Controls and Eligible Nonparticipants in the National JTPA Study Density Controls Nonparticipants >0.80 Propensity Scores

44 FIGURE BIAS - 2 Density of Estimated Probability of Program Participation For Adult Female Controls and Eligible Nonparticipants in the National JTPA Study Density Controls Nonparticipants >0.80 Propensity Scores

45 FIGURE BIAS - 3 Density of Estimated Probability of Program Participation For Adult Male Controls and No-Shows in the National JTPA Study Density >0.80 Controls No-shows 0.4 Propensity Scores

46 TABLE R-1 Treatment Group Dropout and Control Group Substitution in Experimental Evaluations of Active Labor Market Policies [Fraction of Experimental Treatment and Control Groups Receiving Services] Fraction of Treatments Fraction of Controls Study Authors/Time Period Target Group(s) Receiving Services Receiving Services 1. NSW * Hollister, et al. (1984) Long Term AFDC Women 0.95~ 0.11 (9 months after RA) Ex-addicts NA year old H.S. dropouts NA SWIM Friedlander and AFDC Women: Applicants and Recipients Hamilton (1993) (Time period not reported) a. Job Search Assistance b. Work Experience c. Classroom Training/OJT d. Any activity AFDC-U Unemployed Fathers a. Job Search Assistance b. Work Experience c. Classroom Training/OJT d. Any activity JOBSTART Cave, et al. (1993) Youth High School Dropouts (12 months after RA) Classroom Training/OJT Project Kemple, et al. (1995) AFDC Women: Applicants and Recipients Independence (24 months after RA)

47 a. Job Search Assistance b. Classroom Training/OJT c. Any activity New Chance Quint, et al. (1994) Teenage Single Mothers (18 months after RA) Any education services Any training services Any education or training NJS Heckman and Self-reported from Survey Data Smith (1998c) (18 months after RA) Adult Males Adult females Male youth Female youth Combined Administrative and Survey Data Adult males Adult females Male youth Female youth Notes: RA = random assignment. H.S. = high school. Service receipt includes any employment and training services. The services received by the controls in the NSW study are CETA and WIN jobs. For the Long Term AFDC Women, this measure also includes regular public sector employment during the period. Sources: Masters and Maynard (1981), p. 148, Table A.15; Maynard (1980), p. 169, Table A14. Friedlander and Hamilton (1993), p. 22, Table 3.1; Cave, et al. (1993), p. 95, Table 4-1; Kemple, et al. (1995), p. 58, Table 3.5; Quint, et al. (1994), p. 110, Table 4.9; Heckman and Smith (1998c) and calculations by the authors.

48 Figure R-IA Percentage Receiving Classroom Training Adult men Percent Treatments Controls Month after random assignment The percentages are the proportion of persons among the sample who report the receipt of classroom training in each month following random assignment. The sample includes only those persons who responded for the entire 32 months of the survey. Month 0 is the month of random assignment. Standard error bars indicate +/- 2 standard errors about the mean.

49 Figure R-IB Percentage Receiving Classroom Training Adult women Percent Treatments Controls Month after random assignment The percentages are the proportion of persons among the sample who report the receipt of classroom training in each month following random assignment. The sample includes only those persons who responded for the entire 32 months of the survey. Month 0 is the month of random assignment. Standard error bars indicate +/- 2 standard errors about the mean.

50 TABLE SIMULATION-1 Bias in Non-experimental Estimates of the Impact of Training Unmatched Comparison Group Samples Base Case with Base Case where agent Base Case with Base Case Common Coefficient Base Case Knows E(α), not α i Increased Variance of α Parameter of Parameter of Parameter of Parameter of Parameter of Interest: Interest: Interest: Interest: Interest: E(α D=1) = E(α D=1) = E(α) = E(α D=1) = 98.4 E(α D=1) = Estimator (1) (2) (3) (4) (5) Cross Section: Mean SD (61.9) (47.9) (66.6) (47.9) (59.2) Diff-in-Diff (-1,3): Mean SD (58.0) (54.4) (61.4) (54.4) (56.1) Diff-in-Diff (-3,3): Mean SD (60.7) (52.3) (64.5) (52.3) (58.5) Diff-in-Diff (-5,3): Mean SD (63.1) (51.6) (67.1) (51.6) (57.9) AR(1) Regression: Mean SD (203.5) (203.3) (202.9) (210.3) (197.4) IV Estimator: Mean Median SD (4829.0) (231.5) (4829.1) (227.7) (5131.2) Corr(Z,D)

51 Ashenfelter (1979): Mean SD (59.3) (53.3) (62.7) (53.7) (58.8) Heckman (1979): Mean Median SD (3595.3) (221.4) (12811) (269.1) (6618) Kitchen Sink: Mean SD (54.9) (54.8) (58.6) (55.0) (52.4) Notes: 1. Estimates are based on 100 simulated samples of 1,000 observations each. The "mean" row presents the mean of the estimates from the 100 samples while the "SD" row presents the standard deviation of the estimates from the 100 samples. The "Corr(Z,D)" row for the IV estimates gives the average correlation between the participation indicator, D, and the instrument, Z. 2. The base case has θ - N(0,300), - N(0,280), Z - N(0,300), V - N(0,200), ρ = 0.78 and α = N(0,300). This case is based on estimates of the size of the permanent and transitory components of earnings from Ashenfelter and Card (1985) and of the variance in the impacts of training from Heckman, Smith and Clements (1997). In column (2), α = 100. In column (5), α = N(0,500). In the base case in columns (1) and (3), the fractions of Var(Y k+4 D=1) accounted for by α and θ are and , respectively. In column (2), they are and , respectively. In column (4), they are and , respectively. In column (5), they are and , respectively. 3. The cross-section estimator is the simple difference between participant and non-participant earnings in period k+4. The difference-in-differences estimates are based on the periods indicated, so that (-1,3) is the difference between the change in participant earnings from period k-1 to period k+3 and the change in nonparticipant earnings over the same interval. The difference-in-differences (-3,3) estimator is symmetric. The AR(1) estimates are based on a regression of Y k+4 on Y k+3 and D, with the estimate consisting of the coefficient estimate on D divided by (1 - ρ), where ρ is estimated by the coefficient on Y k+3. The IV estimates use Z as an instrument for a regression of Y k+4 on D. The Ashenfelter (1979) estimator is described in Section The dependent variable for this estimator is Y k+4 - Y k. The Heckman (1979) estimator is a special case of the class of control function estimators presented in Section In columns (1) and (5) the estimate is calculated as shown in Section In columns (2), (3) and (4) the estimate is the coefficient on D when the estimated control functions are included. The dependent variable for the Heckman (1979) estimator is Y k+4. The kitchen sink estimates are based on a regression of Y k+4 on Y k-1, Y k-2 and Z.

52 TABLE SIMULATION-2 Bias in Non-experimental Estimates of the Impact of Training Matched Comparison Group Samples Base Case with Base Case where agent Base Case with Base Case Common Coefficient Base Case Knows E(α), not αi Increased Variance of α Parameter of Parameter of Parameter of Parameter of Parameter of Interest: Interest: Interest: Interest: Interest: E(α D=1) = E(α D=1) = E(α) = E(α D=1) = 98.4 E(α D=1) = Estimator (1) (2) (3) (4) (5) Cross Section: Mean SD (80.3) (70.4) (81.8) (70.4) (74.4) Diff-in-Diff (-1,3): Mean SD (77.5) (77.1) (78.8) (77.1) (77.5) Diff-in-Diff (-3,3): Mean SD (82.5) (70.9) (84.4) (70.9) (82.2) Diff-in-Diff (-5,3): Mean SD (80.3) (76.1) (83.0) (76.1) (77.9) AR(1) Regression: Mean SD (333.7) (479.3) (333.7) (588.9) (323.3) IV Estimator: Mean Median SD (4001.6) (175.8) (4000.8) (191.0) (3643.9) Corr(Z,D)

53 Ashenfelter (1979): Mean SD (83.0) (85.0) (84.7) (84.7) (79.3) Heckman (1979): Mean Median SD ( ) (177.2) ( ) (198.5) (3739.7) Kitchen Sink: Mean SD (80.2) (91.5) (81.1) (94.9) (75.4) Notes: 1. Estimates are based on 100 simulated samples of 1,000 observations each. The "mean" row presents the mean of the estimates from the 100 samples while the "SD" row presents the standard deviation of the estimates from the 100 samples. The "Corr(Z,D)" row for the IV estimates gives the average correlation between the participation indicator, D, and the instrument, Z. 2. Matching consists of nearest neighbor matching on Y k-2 with replacement. The average number of unique observations in a matched sample is 92.1 in columns (1) and (3), 79.8 in columns (2) and (4) and 92.3 in column (5) 3. The base case has θ - N(0,300), - N(0,280), Z - N(0,300), V - N(0,200), ρ = 0.78 and α = N(0,300). This case is based on estimates of the size of the permanent and transitory components of earnings from Ashenfelter and Card (1985) and of the variance in the impacts of training from Heckman, Smith and Clements (1997). In column (2), α = 100. In column (5), α = N(0,500). In the base case in columns (1) and (3), the fractions of Var(Y k+4 D=1) accounted for by α and θ are and , respectively. In column (2), they are and , respectively. In column (4), they are and , respectively. In column (5), they are and , respectively. 4. The cross-section estimator is the simple difference between participant and non-participant earnings in period k+4. The difference-in-differences estimates are based on the periods indicated, so that (-1,3) is the difference between the change in participant earnings from period k-1 to period k+3 and the change in non-participant earnings over the same interval. The difference-in-differences (-3,3) estimator is symmetric. The AR(1) estimates are based on a regression of Y k+4 on Y k+3 and D, with the estimate consisting of the coefficient estimate on D divided by (1 - ρ), where ρ is estimated by the coefficient on Y k+3. The IV estimates use Z as an instrument for a regression of Y k+4 on D. The Ashenfelter (1979) estimator is described in Section The dependent variable for this estimator is Y k+4 - Y k. The Heckman (1979) estimator is a special case of the class of control function estimators presented in Section In columns (1) and (5) the estimate is calculated as shown in Section In columns (2), (3) and (4) the estimate is the coefficient on D when the estimated control functions are included. The dependent variable for the Heckman (1979) estimator is Y k+4. The kitchen sink estimates are based on a regression of Y k+4 on Y k-1, Y k-2 and Z.

54 TABLE R-3 Bias in Non-experimental Estimates of the Impact of Training Unmatched Comparison Group Samples Base Case with Base Case with Base Case with Sample Size = 2500 Sample Size = 5000 Sample Size = Parameter of Interest: Parameter of Interest: Parameter of Interest: E(α D=1) = E(α D=1) = E(α D=1) = Estimator (1) (2) Cross Section: Mean SD (35.4) (24.7) (18.8) Diff-in-Diff (-1,3): Mean SD (35.1) (25.3) (17.0) Diff-in-Diff (-3,3): Mean SD (38.5) (26.7) (18.4) Diff-in-Diff (-5,3): Mean SD (33.8) (22.0) (16.8) AR(1) Regression: Mean SD (137.1) (85.8) (65.2) IV Estimator: Mean Median SD (837.4) (470.9) (322.2) Corr(Z,D)

55 Ashenfelter (1979): Mean SD (30.1) (21.8) (15.5) Heckman (1979): Mean Median SD (931.8) (584.1) (380.8) Kitchen Sink: Mean SD (30.2) (20.9) (15.9) Notes: 1. Estimates are based on 100 simulated samples of the indicated size. The "mean" row presents the mean of the estimates from the 100 samples while the "SD" row presents the standard deviation of the estimates from the 100 samples. The "Corr(Z,D)" row for the IV estimates gives the average correlation between the participation indicator, D, and the instrument, Z. 2. The "base case" has θ - N(0,300), - N(0,450), Z - N(0,300), V - N(0,200), ρ = 0.78, α = N(0,300). Estimates for the base case with samples of size 1000 appear in Table 8.3A. This case is based on estimates of the size of the permanent and transitory components of earnings from Ashenfelter and Card (1985) and of the variance in the impacts of training from Heckman, Smith and Clements (1997). In column (1), the fractions of Var(Y k+4 D=1) accounted for by α and θ are and , respectively. In column (2), the fractions are and , respectively. In column (3), the fractions are and , respectively. 3. The cross-section estimator is the simple difference between participant and non-participant earnings in period k+4. The difference-in-differences estimates are based on the periods indicated, so that (-1,3) is the difference between the change in participant earnings from period k-1 to period k+3 and the change in nonparticipant earnings over the same interval. The difference-in-differences (-3,3) estimator is symmetric. The AR(1) estimates are based on a regression of Y k+4 on Y k+3 and D, with the estimate consisting of the coefficient estimate on D divided by (1 - ρ), where ρ is estimated by the coefficient on Y k+3. The IV estimates use Z as an instrument for a regression of Y k+4 on D. The Ashenfelter (1979) estimator is described in Section The dependent variable for this estimator is Y k+4 - Y k. The Heckman (1979) estimator is a special case of the class of control function estimators presented in Section The estimates in all three columns are calculated as shown in Section The dependent variable for the Heckman (1979) estimator is Y k+4. The kitchen sink estimates are based on a regression of Y k+4 on Y k-1, Y k-2 and Z.

56 TABLE SIMULATION-4 Bias in Non-experimental Estimates of the Impact of Training Unmatched Comparison Group Samples Base Case Base Case Base Case Base Case Base Case with with Reduced with Reduced with Reduced with Reduced ρ = 0 and No Variance of θ Variance of Variance of Z Variance of V Fixed Effect, θ Parameter of Interest: Parameter of Interest: Parameter of Interest: Parameter of Interest: Parameter of Interest: E(α D=1) = E(α D=1) = E(α D=1) = E(α D=1) = E(α D=1) =615.3 Estimator (1) (2) (3) (4) (5) Cross Section: Mean SD (58.0) (50.7) (55.6) (61.0) (57.8) Diff-in-Diff (-1,3): Mean SD (69.2) (6.2) (53.3) (58.7) (89.2) Diff-in-Diff (-3,3): Mean SD (71.7) (6.2) (59.9) (59.9) (78.0) Diff-in-Diff (-5,3): Mean SD (72.3) (5.9) (58.9) (60.7) (91.2) AR(1) Regression: Mean SD (157.1) (4510.8) IV Estimator: Mean Median SD (3630.6) (2020.2) ( ) (2678.8) (1962.8)

57 Corr(Z,D) Ashenfelter (1979): Mean SD (64.7) (6.6) (55.5) (56.6) (83.6) Heckman (1979): Mean Median SD (5429.7) (3310.5) ( ) (3162.0) (3546.4) Kitchen Sink: Mean SD (59.2) (6.2) (48.2) (53.8) (58.3) Notes: 1. Estimates are based on 100 simulated samples of 1,000 observations each. The "mean" row presents the mean of the estimates from the 100 samples while the "SD" row presents the standard deviation of the estimates from the 100 samples. The "Corr(Z,D)" row for the IV estimates gives the average correlation between the participation indicator, D, and the instrument, Z. 2. The base case has θ - N(0,300), - N(0,280), Z - N(0,300), V - N(0,200), ρ = 0.78 and α = N(0,300). In column (1), θ - N(0,30) and - N(0,337), in column (2), θ - N(0,538) and - N(0,30), in column (3), Z - N(0,30) and V - N(0,359), in column (4), Z - N(0,359) and V - N(0,30) and in column (5), ρ = 0 and there is no fixed effect, θ. The base case is based on estimates of the size of the permanent and transitory components of earnings from Ashenfelter and Card (1985) and of the variance in the impacts of training from Heckman, Smith and Clements (1997). In column (1), the fractions of Var(Y k+4 D=1) accounted for by α and θ are and , respectively. In column (2), the fractions are and , respectively. In column (3), the fractions are and , respectively. In column (4), the fractions are and , respectively. In column (5), the fractions are and , respectively. 3. The cross-section estimator is the simple difference between participant and non-participant earnings in period k+4. The difference-in-differences estimates are based on the periods indicated, so that (-1,3) is the difference between the change in participant earnings from period k-1 to period k+3 and the change in non-participant earnings over the same interval. The difference-in-differences (-3,3) estimator is symmetric. The AR(1) estimates are based on a regression of Y k+4 on Y k+3 and D, with the estimate consisting of the coefficient estimate on D divided by (1 - ρ), where ρ is estimated by the coefficient on Y k+3. The IV estimates use Z as an instrument for a regression of Y k+4 on D. The Ashenfelter (1979) estimator is described in Section The dependent variable for this estimator is Y k+4 - Y k. The Heckman (1979) estimator is a special case of the class of control function estimators presented in Section The estimates in all five columns are calculated as shown in Section The dependent variable for the Heckman (1979) estimator is Y k+4. The kitchen sink estimates are based on a regression of Y k+4 on Y k-1, Y k-2 and Z.