Curbing cream-skimming: Evidence on enrolment incentives

Transcription

1 Curbng cream-skmmng: Evdence on enrolment ncentves Pascal Courty, Do Han Km, and Gerald Marschke 1 January 2008 Abstract: Usng data from a large, U.S. federal ob tranng program, we nvestgate whether enrolment ncentves that exogenously vary the shadow prces for servng dfferent demographc subgroups of clents nfluence case workers ntake decsons. We show that case workers enroll more clents from subgroups whose shadow prces ncrease but select at the margn weaker-performng members from those subgroups. We conclude that enrolment ncentves curb cream-skmmng across subgroups leavng a resdual potental for cream-skmmng wthn a subgroup. JEL: H72, J33, L14. Keywords: Performance measurement, cream-skmmng, enrolment ncentves, bureaucrat behavor, publc organzatons. 1 Pascal Courty, Department of Economcs, European Unversty Insttute, Vlla San Paolo, Va Della Pazzuola 43, 50133, Frenze, Italy, emal: Pascal.Courty@eu.eu; Do Han Km, Department of Publc Admnstraton and Polcy, Unversty at Albany, State Unversty of New York, Albany, NY 12222, emal: dk7535@albany.edu; and Gerald Marschke, John F. Kennedy School of Government, Harvard Unversty, 79 John F. Kennedy Street, Cambrdge, MA 02138, NBER and IZA, on leave from Unversty at Albany, SUNY, emal: Jerry_Marschke@harvard.edu. The bulk of ths paper was completed whle Marschke was at the Unversty at Albany. We thank partcpants at the 2008 Vouchers, Contractng-out and Performance Standards Conference n Nuremberg.

2 1-Introducton The recent ntroducton of performance ncentves n several branches of the publc servce sector, such as n ob tranng, educaton, and health, has rased concerns as to ther mpact on enrolment decsons, and on equty and effcency outcomes. At the center of ths debate s the ssue that ncentves nduce case workers to cream-skm, that s, to select applcants on the bass of performance on measured outcomes nstead of value added accordng to the program s stated obectves (Anderson, Burkhauser, and Raymond 1993, Cragg 1997, Heckman, Henrch, and Smth 2002). A soluton that has been proposed to retan control over the recpent populaton s to adust the measures that are used to assess performance, effectvely settng dfferent shadow prces for dfferent soco-economc subgroups of enrollees. Although such methods are used n practce, and ther theoretcal underpnnng s uncontroversal, there s no evdence that adustment models actually have an mpact on enrolment decsons. Our case study s a large government ob tranng program that changed the adustment method used to assess performance three tmes durng our sample perod, usng four dfferent sets of shadow prces. We use ths varaton to produce the frst econometrc evdence on whether t s possble to nfluence case worker ntake choces. We study the federal program created under the Job Tranng Partnershp Act (JTPA), whch, between 1982 and 2000, provded ob tranng to the economcally dsadvantaged. Under JTPA, ob tranng servces were admnstered by over 620 sem-autonomous substate tranng agences each evaluated accordng to a set of performance measures defned at the federal level. Specfcally, a tranng agency s yearly performance was adusted upwards or downwards to account for the partcular mx of persons the agency enrolled. 1

3 To llustrate, consder the adustment made to the employment at termnaton measure for enrollng adults who never receved a hgh school degree. 2 By enrollng more hgh school dropouts a tranng agency lowered the mnmum performance (the mnmum fracton of partcpants employed at termnaton) necessary to avod sanctons and qualfy for a performance award. We refer to ths mnmum performance threshold as the performance standard. The adustment to the standard for enrollng hgh school dropouts vared over tme. We test whether case workers respond to the changes n these adustments. We quantfy the mpact of the adustment method both on ntake populatons and on performance outcomes. There are good reasons to thnk that JTPA s adustment methods may not change enrolment patterns n practce. Frst, case workers preferences may vary over socoeconomc subgroups, or case workers may be subect to pressures by local nfluence groups that overrde the typcally weak ncentves backng the adustments (Heckman, Smth, and Taber 1996). Second, Heckman and Smth (2004) have shown that most of the selecton occurs at the early stages of the partcpaton process, such as between elgblty and awareness, over whch the program staff has lttle or no control. Thus, even f case workers respond to changes n the shadow prces, ther response may be neglgble. Thrd, adustment methods may have lttle mpact n practce because they are complex. In our case study, for example, the adustment model can potentally dstngush over 16 mllon dfferent demographc subgroups for each of four dfferent performance 2 The employment at termnaton measure, the most mportant measure n the early years of JTPA, was defned as the fracton of program termnees who termnated wth a ob. 2

4 measures. 3 It may be mpossble, or not worthwhle, for a tranng agency to attempt to factor nto ts enrolment strategy so many shadow prces. The paper proceeds as follows. Secton 2 descrbes how performance adustment was mplemented n JTPA, defnes the concept of performance adustment weght (PAW), and revews the lterature on performance adustment n non-ob tranng areas of the publc and non-proft sectors. Most mportantly for our emprcal study, we argue that the changes n PAW over tme n JTPA are exogenous to the tranng agency s enrolment decson. Secton 3 dscusses how award maxmzng case workers should respond to changes n PAW and derves predctons on changes n enrollee populaton and performance outcomes. We test these predctons usng mcro-level data on case workers enrolment choces and performance outcomes n JTPA, leveragng three exogenous changes n the PAW. We estmate the mpact of these changes n PAW followng a dfference-n-dfference approach (tme and demographc subgroups) at the agency level. Our emprcal analyss establshes two sets of results. Frst, we fnd that changes n the ncentve for enrollng members of a subgroup sgnfcantly change the fracton of enrollees from the subgroup. Second, we demonstrate the exstence of wthn-subgroup heterogenety. Case workers ncrease the number of enrollees from a specfc subgroup by enrollng at the margn applcants that perform worse on the measure. Ths fndng s consstent wth the cream-skmmng hypothess that case workers use ther prvate nformaton about the elgble populaton whch they use to select enrollees that perform well on the performance measures. In contrast wth the lterature, whch focuses on the mpact of ncentves on overall enrolment at the tranng agency level (dfferences across 3 24 dfferent adustment factors have been used durng our sample perod (see Table 1, and the later dscusson). Each adustment factor takes bnary values mplyng 2 24 dfferent subgroups. 3

5 subgroups), we demonstrate that prvate nformaton carres through even wthn the demographc subgroups defned by PAW. Lterature Our results are of nterest to polcy makers and academcs for three man reasons. Frst, our evdence sheds new lght on the lterature on cream-skmmng. Interestngly, the evdence on cream-skmmng s mxed (Heckman et al. 1996, Cragg 1997) and one mght be tempted to conclude that enrolment decsons are not nfluenced by ncentves. But most prevous JTPA studes have focused on the enrolment ncentves due to performance measurement wthout factorng n the role of the adustment weghts n the performance standard. They test varants of the followng hypothess: Does rewardng (or sanctonng) a tranng agency based on the fracton of ts clents who obtan employment, dssuade t from servng hgh school dropouts and other persons wth poor labor market prospects? But the JTPA adustment model forces the tranng agency to consder how a person s attrbutes not only affects the performance outcome but also the standard: the agency knows that enrollng a hgh school dropout lowers ts employment outcome but t also lowers ts standard. Thus, the absence of strong evdence of creamskmmng may be because the performance standard adustment procedure was dong ts ob that s, reducng the ncentve to cream-skm and not because JTPA case workers dd not respond to ncentves. Our study tells a more complex pcture where both effects are at play: local agences respond to enrolment ncentves but cream-skmmng stll takes place due to unobservable characterstcs wthn demographc subgroups. 4

6 Second, the ssue of allocaton of publc servces receves much attenton n the polcy lterature (Heckman and Smth 2004). Ths debate s fueled by a general concern over equty and also because polcy makers often have specfc target populatons that they would lke served. For example, JTPA tself ntroduced several constrants on the allocaton of JTPA enttlements across soco-economc subgroups (Dckenson et al. 1988). The U.S. Department of Labor (DOL), whch admnstered JTPA, defned the concept of elgble populaton to restrct the pool of people who could be served. In addton, budget compartmentalzaton capped the resources that could be used on adults and establshed a mnmum expendture for youth enrollees. The DOL was also desrous that the hard-to-serve and most-n-need not be neglected and supported the states that ntroduced ncentves to target resources toward these sub-populatons (Barnow 1992, Courty and Marschke 2003). Lke quotas and budget compartmentalzaton, PAW are obectve and transparent, but n contrast to these schemes they leave some dscreton to local decson makers to explot potental trade-offs between sub-populatons. A drawback s that they may convey very complex ncentves, and also, perhaps, grant too much dscreton over unobserved heterogenety wthn subgroups. Our work can help polcy makers understand whether PAW can help acheve equty obectves and/or correct dstortons due to performance ncentves, or whether other methods are needed. Thrd, ths work contrbutes to the lterature on the effectveness of ncentves n the publc sector. Many polcy analysts now beleve that such systems can mprove accountablty and management (Osborne and Gaebler 1992, Gore 1993) 4 and such systems have become polcy through the Government Performance and Results Act 4 Greater use of performance measurement systems n the publc sector has also receved support n academc crcles (Natonal Academy of Publc Admnstraton 1991, Wholey and Hatry 1992, Bouckaert 1993, Kravchuk and Schack 1996). 5

7 (GPRA) of Our fndngs can help n the desgn of future adustment systems. For example, the Workforce Investment Act (WIA), whch supplanted JTPA n 2000, replaced the JTPA s regresson based adustment model wth negotated performance standards. For ths and other reasons, polcy-analysts have expressed concerns about cream-skmmng under WIA (Barnow and Smth 2004, U.S. Government Accountng Offce 2004, Henrch 2004, Barnow and Henrch 2008). Some analysts have called for the rentroducton of adustment models n the upcomng reauthorzaton of WIA. One contrbuton of ths paper s to provde some evdence the frst that we know of that ob tranng staff respond to JTPA-style adustment models and to quantfy these responses. In fact the lterature has repeatedly ponted out the dffcultes n separatng bureaucrat and applcant motves n explanng partcpaton (Heckman and Smth 2004). Our evdence crcumvents ths challenge by usng a natural experment that permts one to dentfy the relaton between PAW and enrolment choce. We show that bureaucrats respond to sophstcated contracts that nvolve a large number of mplct prces and requre the ablty to compute complex trade-offs between alternatve enrolment strateges. Although there may exst bureaucratc preferences over the choce of allocatons of publc resources as suggested by Heckman et al. (1996), our fndngs show that t s possble to nfluence bureaucratc preferences over ntake choce. 6 2-Performance Adustment Weghts: Background and Case Study 5 GPRA requres federal agences to formulate measures of performance and set performance goals to mprove publc accountablty and permt scrutny by congressonal oversght commttees and the publc. 6 Heckman et al. fnd that JTPA case workers were more lkely to enroll the applcants wth the lowest prospects for employment after tranng. Heckman et al. call ths behavor cream avodance whch they attrbute to a socal worker mentalty n tranng agency staff. 6

8 Much of the lterature on PAW has focused on ther use as a means to complement and fne tune performance ncentve systems. 7 The dea s that PAW can help to correct enrolment dstortons due to the ntroducton of outcome based performance ncentves. Performance ncentves stmulate agency efforts to produce value added, but they may also dstort the characterstcs of the populaton the agency selects. Ths problem has emerged wth ncentve schemes n educaton that measure school performance usng standardzed test scores (Jacob and Levtt 2003), n ob tranng that evaluate performance usng labor market outcomes of tranees (Heckman et al. 2002), and n health care where doctors and hosptals are evaluated usng report cards (Dranove, Kessler, McClellan, and Satterthwate 2003). The lterature on PAW has been mostly conceptual or prescrptve n nature. Rubensten, Schwartz, and Stefel (2003) and Brooks (2002) lay out ratonales for adustng performance standards, compare and contrast dfferent adustment strateges that one mght employ, and offer recommendatons to polcy-makers on how to adust standards. Courty, Henrch, and Marschke (2005) stuate the problem n the prncpalagent framework, and dscuss how performance outcome measures should be adusted. Another strand of the lterature documents how PAW have been used n the context of specfc applcatons. Trott and Ba (1987), Barnow (1992), Henrch (2004), and Barnow and Smth (2004) dscuss applcatons to ob tranng programs, Sedleck and Kng (2005) to workforce development programs, Berne (1989), Stefel, Rubensten, and Schwartz (1999) and Stefel, Schwartz, Rubensten, and Zabel (2005) to educaton, and Iezzon (2003) to healthcare. 7 PAW can also be used n the absence of outcome based ncentves, and the pont would then be to correct possble bas due to bureaucratc preferences. In fact, polcy makers may reward bureaucrats for enrollng certan groups f they feel that these groups would be otherwse underserved. 7

9 Case study: JTPA A large lterature dscusses varous aspects of the JTPA program (e.g. Johnston 1987), and offers descrptons of ts ncentve system (e.g. Courty and Marschke 2003) and the bureaucratc responses they nduce (e.g. Heckman and Henrch, forthcomng). To reduce unnecessary repetton, we present here only those features of the organzaton that are essental to our analyss of PAW, and drect the reader to more comprehensve sources when requred. The JTPA program was hghly decentralzed: the 620 plus tranng agences admnstered the program wth sgnfcant dscreton over whom to enroll. Whle applcants had to meet an ncome test to be elgble, JTPA was not an enttlement. Gven the JTPA annual budget (approxmately $4.1 bllon n 1993), and the large populaton that was elgble for tranng, agences could serve only one to three percent of the elgbles (550,000 new partcpants were enrolled n 1993). 8 The decson of whch elgbles to enroll consttutes the focus of ths paper. The Act called for fnancally-backed performance ncentves that would measure and reward tranng agency s success n developng partcpants human captal, the prmary goal of the program accordng to the Act (JTPA, secton 106(a)). Congress gave the DOL the responsblty of developng a workable set of performance measures that would reflect the Act s msson. The JTPA fscal year, or program year, ran from July 1 to June 30 of the next calendar year. At the end of each program year, tranng agences were rewarded (or sanctoned) on the bass of ther performance relatve to these DOL 8 See Dckenson et al. (1988) for a complete descrpton of the JTPA elgblty rules. 8

10 measures. For the average tranng agency, the award amounted to about seven percent of the operatng budget. Our emprcal analyss focuses on program years and on the adult JTPA populaton. For the perod, the DOL used four performance measures constructed from two labor market outcomes, employment and earnngs, to evaluate tranng agences. A tranng agency s employment rate at follow-up (ER) for a partcular program year was calculated as the fracton of enrollees termnated durng that year who were employed 13 weeks after termnaton. The average weekly earnngs (WE) was calculated as the average weekly earnngs durng the nnety days followng termnaton for those enrollees who were employed 13 weeks after termnaton. From the ER outcome, two performance measures were constructed: one ER measure was based on the performance of all adult enrollees and another was based on the performance of only the welfare-recevng subset of adult enrollees. Smlarly, separate adult and adult welfare performance measures were constructed based on WE. Each measure had assocated to t a separate standard. The DOL set lower standards for the welfare versons of the measures. Meetng these standards was a condton for recevng an award and n many states most of the award a tranng agency was elgble for was pad out for smply meetng the standard. Thus, the structure of the ncentves under JTPA meant that a tranng agency nterested n avodng sanctons and maxmzng ts award, should focus on meetng ts standards. For each of the four standards, the DOL developed an adustment model to establsh a tranng agency-specfc standard that accounted for the partcular agency s enrollee choces (demographc characterstcs of the enrollee pool) and local labor market 9

11 crcumstances (soco-economc condtons outsde the control of the agency). For example, t was determned that tranng agences that enrolled few hgh school dropouts should be handcapped relatve to those that enrolled more, and that tranng agences should not be penalzed for operatng n partcularly adverse labor markets. In ths study, we focus exclusvely on the set of factors n the DOL adustment models that are based on the demographc characterstcs of the enrollee populaton, as only these factors can nfluence enrolment decsons. PAW n JTPA To llustrate how the adustment methodology works, assume two demographc factors, gender (female, male) and race (black, non-black). The tranng agency s rewarded on the bass of excess performance, that s, performance above the performance standard. The DOL model adusts the performance standard around an exogenously gven baselne level, that we denote m 0, dependng on the characterstcs of the enrollee populaton. Suppose an agency enrolled x f percent of females, x b percent of black and denote by β the adustment weght for demographc characterstc =f,b. A stylzed performance adustment model can be wrtten as M 0 (x f,x b )= m 0 -(β f x f +β b x b ) (1) where M 0 s the adusted performance standard. The hgher the standard, the greater s the dffculty obtanng an award. We defne an adustment factor as a soco-economc varable (e.g. x f ) that s used to correct the standard and an adustment weght as the numercal value that s mputed to correct the standard (e.g. β f ). For example, f β f s postve, then the agency s more lkely to receve an award, ceters parbus, f t enrolls more females. 10

12 The DOL chose dfferent sets of factors for each performance measure based upon ther avalablty, ther statstcal relaton wth the performance measure, and poltcal consderatons. The frst lne n Table 1 presents the baselne level (m 0 ), the frst column n the bottom panel presents the set of adustment factors (x) for the adult ER and WE standards, and the core of the table reports the value of the adustment weghts (β) correspondng to these factors. 9 The columns report the weghts for dfferent program year cycles. The adustment weghts reman n force for two consecutve program years before they are updated. Thus for example n program years 1992 and 1993, the adustment weghts for the ER standard for females was.072; n program years 1994 and 1995 t was.056; and so on. They are constructed before the begnnng of a new two year cycle, usng nformaton on demographc characterstcs and outcomes observed n the prevous cycle, as the coeffcent estmates from a regresson of performance outcomes on demographc and labor market characterstcs. 10 Table 1 shows that the enrolment ncentves embedded n the DOL adustment model can sgnfcantly mpact the performance standard and therefore the agency s lkelhood to receve an award. For example, an agency n ether 1992 or 1993 enrollng only applcants that emboded all of the characterstcs assocated wth postve weghts would face a negatve performance standard on the employment measure (the adustment, 100Σ β, s greater than the baselne level mplyng M 0 <0), meanng that t would not be penalzed even f none of ts termnees were employed at termnaton. Although ths example s extreme, Table 1 reveals that many of the weghts can lower the 9 We obtaned the adustment weghts from Gude to JTPA Performance Standards for Program Years , , , and publshed by Socal Polcy Research Assocates (see footnote n table 1 for full ctes). 10 For more explanatons on the process of estmatng the coeffcents n the regresson model, see U.S. Department of Labor (1987), Barnow (1992) or Socal Polcy Research Assocates (1999). 11

13 employment standard by 10 percent or more. Table 2 focuses on the employment measure and presents summary statstcs on the dstrbuton across agences of the actual adustment to the baselne (Σ β x ) by program year. Lne 1, for example, says that the ER standards n 1993 vared across tranng agences from 37 percent (86-49) to 74 percent (86-12) to suggestng that a tranng agency s enrolment pool whch s a choce varable could greatly nfluence ts standard. The adusted performance standards for the earnngs measure (not reported) show the same degree of varaton. The lkely mpact of PAW on enrolment s dffcult to assess on theoretcal grounds alone. On the one hand, the magntude of the changes n the weghts mples that the enrollee ntake composton may have a sgnfcant mpact on the standard. Meetng the standard was an ssue n practce and could have fnancal consequences. 11 In fact, over the perod , on average about 23% of tranng agences faled to meet the employment rate standard, 6% faled to meet the earnngs standard, and 6% faled to meet to meet both standards. 12 On the other hand, the number of demographc subgroups, and thus the number of mpled shadow prces, ncreases exponentally wth the number of factors. For example, there were 10 factors actve n 1993 for the employment at termnaton measure (Table 1) whch requred the agency to dstngush among 1024 subgroups. In addton, the PAW vared across performance measures. As a result, PAW 11 The award for the successful tranng agency averaged about seven percent of ts budget. In some states, the hghest awards amounted to about sxty percent of the tranng agency s budget. The reader who s nterested n the detals of the ncentves confrontng JTPA tranng agences should see Courty and Marschke (2003). 12 If the performance standards were set too hgh, so that all tranng agences would fal no matter how they tred, then the ablty to modfy a standard usng the enrolment composton would not matter much, and one would not expect to see enrolment choces affected by PAW. Ths s also true f performance standards were set too low so that all tranng agences exceeded ther standards whether they enrolled purposefully or not. 12

14 ntroduced very complex trade-offs and may have had lttle consequence n practce. In the end, whether PAW nfluenced ntake choces s an emprcal ssue. Table 1 shows that there are sgnfcant changes n the adustment weghts over tme. For example, to meet ts employment standard n program years 1992 or 1993, an agency that enrolled no hgh-school dropouts would have to acheve an employment rate 18.4 percent hgher than an agency that enrolled only hgh-school dropouts (assumng that all other characterstcs are equal across the agences). In program year 1998 or 1999, however, the dfference drops to 6.6 percent, an order of magntude of about three. In addton, some adustment factors eventually dsappear from the adustment worksheets and new factors are ntroduced. 13 To wrte a mcro model of enrolment and for the emprcal work as well, t s more convenent to work wth demographc subgroups nstead of demographc characterstcs. There s a smple correspondence between subgroups and characterstcs. In our example, the two factors determne four demographc subgroups (black female, black male, and so on). Denote s=(s bf,s bm,s nf,s nm ) the enrolment vector measured n percentage of overall populaton over demographc subgroups where s bf, for example, represents the percentage of enrollees who are black and female. We can rewrte the performance standard as M 0 (n)= m 0 -(ω bf s bf +ω bm s bm +ω nf s nf +ω nm s nm ) (2) where ω bf =β b +β f, captures the decrease n standard due to ncreasng the fracton of black female by one percent, and can be nterpreted as the shadow prce for that demographc subgroup. The other coeffcents are smlarly derved, ω bm =β b, ω nf =β f, ω nm =0. In the rest of ths paper, we also call the ω adustment weghts, keepng n mnd the dstncton between 13 Ths lfecycle phenomenon of adustment weghts was observed earler n JTPA s hstory by Barnow and Constantne (1988) who attrbute t to ncreased profcency due to learnng by the tranng agences n selectng enrollees on the bass of factors not ncluded n the model. 13

15 the β n (1) and ω n (2). 3-Theoretcal Predctons We present n the appendx a mcroeconomc model of the tranng agency choce of enrollee populaton. We derve predctons on how the agency should respond to changes n the adustment weghts: how enrolment decsons and performance outcomes should change for dfferent demographc subgroups. Ths secton dscusses the ntuton behnd the model and summarzes ts predctons. To smplfy, we assume there s a sngle performance measure and I dstnct demographc subgroups. The cost of tranng s assumed subgroup-specfc and ncreasng and convex n the number of enrollees. Smlarly, average performance decreases wth the number of enrollees from a specfc subgroup. These assumptons are consstent wth the followng nterpretaton. Applcants dffer wthn a subgroup. Some applcants are easer to tran and are more lkely to acheve successful outcomes than others. It s optmal for the tranng agency to select frst the most promsng applcants. If the tranng agency hres more applcants of a gven subgroup t wll hre those who cost more to serve (cost s ncreasng) and who are less lkely to perform well (performance ncreases at a decreasng rate). These assumptons are reasonable f there s some heterogenety wthn demographc subgroups that s observed by the agency. Cream-skmmng becomes possble because the agency enrolls those applcants, wthn a demographc subgroup, who are lkely to perform well on the measure, rrespectvely of how well they perform on the true obectve of ob tranng. The tranng agency allocates ts budget across the demographc subgroups to maxmze ts award and may also have ts own preferences over enrolment choces. We frst show that 14

16 under general assumptons about the cost and performance outcome functons, the tranng agency responds to an ncrease n the adustment weght of demographc subgroup by enrollng more applcants of subgroup and fewer applcants of subgroup k. The proposton holds ndependently of the tranng agency s own preferences over enrollee choces. We then consder the mpact of a change n the adustment weght on the average subgroup performance outcome. As the adustment weght of subgroup ncreases, the number of enrollees of subgroup ncreases and the average performance outcome of subgroup decreases. The reason s smply that to ncrease ts enrolment of applcants of type, the tranng agency has to enroll less attractve applcants. Margnal enrollees acheve lower performance outcomes than average ones. 14 Under addtonal assumptons on the model s prmtves, we can derve general predctons on the mpact of any changes n the performance weghts. Specfcally, the ncrease n the number of enrollees from subgroup (Δn ) s greater than the same change for subgroup k (Δn k ) f subgroup s adustment weght ncreases by a larger amount (or decreases by a lower amount) than subgroup k. Formally, Δn >Δn k f and only f δ >δ k for any and k, where δ denotes the change n weght. The result also apples to changes n the fracton of enrollees and ths consttutes the focus of our emprcal nvestgaton (Hypothess H1). The result on average performance outcomes also generalzes to any change n weght: the change n average performance of subgroup s lower than the change n average performance of subgroup k f δ >δ k (Hypothess H2). The remander of ths paper tests hypothess H1 and H2. 14 To smplfy, we assume that the tranng treatment s constant across groups. 15

17 4-Data and Emprcal Strategy The varables we wsh to explan wth our analyss, measured at the level of demographc subgroup-agency-year, are the enrolment shares and the performance outcomes. To compute these varables, we use data from the Standardzed Program Informaton Report whch are collected by the U.S. Department of Labor and dstrbuted by the W.E. Upohn Insttute of Employment Research. Appendx 2 explans n detal how we constructed our panel data of demographc subgroups, agences, and years. Consstent wth the JTPA ncentves, we form the subgroups based on the termnees n the program year, not the enrollees. We do ths because the adustment model modfes the standards based on the characterstcs of the program year s termnees. 15 Recall that usng all 24 factors would generate more than 16 mllons subgroups. Snce the JTPA enrollee populaton s much smaller (for the enrolment analyss, for example, we have nformaton on 682,515 termnees over the 6 program years), we elmnate all the subgroups for whch we have no or few enrollees. In the end, we select 13 factors and construct 1,670 dfferent subgroups for whch we have nformaton over all 6 years n at least one agency. Ths yelds an average of 291 subgroups per agency-year. Table 3 presents descrptve statstcs for our man varables (PAW, enrolment shares, and performance outcomes). Table 1 and 2 demonstrate that there s much varaton across years n our explanatory varables. Table 4 shows that there s also much varaton from year to year n the enrolment 15 Of course, the termnee populaton closely resembles the enrollee populaton. We are explctly assumng that the tranng agences antcpate the effects of ts enrolment decsons on the performance standards whch s the case when the standard remans unchanged. Ths s reasonable because the average length of tranng (a few months) s short relatve to the perod durng whch the weghts reman constant (two years). We have consdered the possblty of delays n the enrolment responses after the three changes n the PAW that took place n our sample perod and ths dd not change the results. 16

18 sze of the demographc subgroups dentfed by the DOL adustment factors. We nvestgate whether ths varaton n enrolment can be explaned by the changes n adustment weghts as predcted by (H1). Mult-dmensonalty Each of the four standards had ts own adustment weghts that could potentally nfluence the enrollee ntake choce (H1) and also the performance outcomes (H2). Under JTPA, states were responsble for desgnng the ncentve contracts usng the four measures proposed by the DOL. Although these contracts vary greatly from state to state (dfferent emphass on the dfferent measures and dfferent choce of the award functon), we can leverage three patterns that are common to all contracts to cope wth the mult-dmensonal nature of the ncentve system. To start, the employment measure receved a dsproportonate emphass n determnng the award (Courty and Marschke, 2003). Moreover, awards were largely allocated for meetng standards and tranng agences were more lkely to fal the employment standards. Thrd, the PAW for the two welfare measure standards apply only to welfare subgroups and therefore should not nfluence the choce of non-welfare enrollees. Gven these consderatons, we proceed as follows. We ntally test H1 and H2 usng the adustment weghts on the adult ER performance measure. In focusng on the employment measure, ths approach follows the polcy evaluaton lterature (e.g., Anderson et al.1993) and s ustfed by the frst two characterstcs of the ncentve contracts mentoned above. Later, we ntroduce the adustment weghts on the WE standard. Snce we do not have nformaton on the contracts, we employ a general specfcaton for how ER and WE could nfluence enrolment and outcomes that allows for nteracton effects. Ths frst set 17

19 of analyses s vald under the assumpton that the change n weghts that apply to the welfare measures are ndependent from the change n weghts that apply to the two measures we consder. As a robustness check, we reproduce the prevous analyses wthout the welfare subgroups. The thrd characterstc of the ncentve system mples that H1 and H2 hold for ths subset of the sample even f the above assumpton does not hold. Exogenety of the changes n the PAW The PAW were changed three tmes n our sample perod (see Table 1). In the emprcal analyss, we assume that these changes are exogenous to contemporaneous enrolment decsons. Several arguments support ths assumpton. Recall that the PAW were computed as coeffcent estmates of a regresson of performance outcomes on demographc factors usng performance data (from all tranng agences) n the prevous two year cycle. The DOL regresson model used to compute the PAW was unstable and ths was due to mult-colnearty between factors. Consstent wth ths vew, Table 1 shows that the choce of demographc factors vared greatly over tme (only 9 out of 24 demographc factors were used throughout our perod). Ths choce was partally drven by the concern to keep the PAW postve (snce all the selected factors represented prorty target subpopulatons) and by current poltcal consderatons. The change n the PAW are exogenous f they are manly drven by the arbtrary choce of the factors ncluded and the sample realzaton of the two year cohort used to compute the regresson coeffcents. But changes n the PAW could also be drven by changes n labor market condtons and/or changes n enrolment strateges. We argue that ths s not an mportant ssue for 18

20 our emprcal exercse, and f anythng, t can only create an under-estmaton of the agency responses. Consder frst the later pont. The concern s endogenety of the PAW through strategc nter-temporal enrolment behavor. The behavor of all agences as a group nfluences changes n the weghts, because the DOL used the nformaton collected on past enrolment choces and outcomes to update the weghts, but an ndvdual agency can be assumed to maxmze the current perod award myopcally snce the mpact of ts enrolment decsons on future weghts s neglgble. Consder next changes n labor market condtons. To start, assume that these changes are condtonally uncorrelated (e.g. random walk or permanent changes). Such changes would nfluence the PAW (through the regresson model) but ths would not ntroduce an endogenety problem snce the change n next perod labor market condtons s uncorrelated wth the current change n PAW. The only concern are trends n labor market condtons. Such trends could bas the nference aganst our hypothess. Assume for example that the labor market potental of a subgroup starts to degrade. Ths ncreases that subgroup s PAW but the ncrease undercompensates for the contnung degradaton n the subgroup s potental so we would under-estmate the enrolment response relatve to the response that would take place wth a truly exogenous change n PAW. Emprcal strategy Denote s at as the share of enrollees of demographc subgroup n agency a n year t and w t the adustment weght, common to all agences, for subgroup and year t. 16 H1 mples an ncreasng relaton between changes n relatve weghts and changes n relatve 16 s at s defned as nat n k kat where n at s the number of enrollees of group n agency a and year t. 19

21 shares of subgroups. We test ths relaton usng the three changes n adustment weghts that took place n our sample perod (the weghts changed at the end of 1993, 1995, and 1997). We propose dfferent specfcatons to test H1 that are varatons around the followng approach. Assumng that the ncreasng relaton mpled by H1 s lnear and does not vary across subgroups, agences, or years gves (s at -s at )-(s kat -s kat )=γ[(w t -w t )-(w kt -w kt )] for all,k,a,t,t (H1 ) where the parameter of nterest γ s postve. Instead of comparng pars of demographc subgroup-years, whch does not naturally ft a regresson framework, we aggregate ths hypothess to obtan a relaton that can be estmated usng a fxed-effect regresson framework. 17 Formally, H1 s averaged over subgroups k and years t to obtan s at = -(s a - γw a )+(s a -γw a )+(s at -γw at )+γw t for all,a,t where s a denotes the average share n agency a across all years and subgroups, s a the average share n agency a across all years and smlarly for s at and the w averages. We can rewrte ths relaton as a dfference n dfference (tme and subgroup) equaton at the agency level s at = α a +α a +α at +γw t for all,a,t. The observed shares could vary randomly because they are measured wth error (whch s the case n our applcaton snce only a representatve sample of 62 percent of total populaton s ncluded n our dataset). 18 We obtan the followng emprcal model s at = α+α a +α at +γw t +ε at (3) where α s a constant, α a s a subgroup-agency fxed effect and α at s an agency-tme fxed effect. We assume ε at s normal, mean zero, and dstrbuted ndependently across tranng 17 Alternatvely we could choose a subgroup to serve as the reference subgroup aganst whch we compare all other subgroups but the choce of the reference subgroup s arbtrary. 18 Alternatvely, we could derve the econometrc model followng a random utlty approach, assumng that agences have group preferences that vary randomly over tme. 20

22 agences. The theory makes no predcton on (α,α a,α at ) but predcts that γ should be postve. Specfcaton (3) tests an averaged verson of H1. We nterpret γ as the average effect over all tranng agences, all year changes, and subgroups. We cluster the errors at the tranng agency level to permt arbtrary forms of autocorrelaton and heteroscedastcty wthn tranng agency panels. To test H2, we follow a smlar procedure. We estmate the performance of each subgroup holdng constant agency-subgroup and agency-tme fxed effects m at = θ+θ a +θ at +θw t + υ t for all,a,t (4) where as before θ a and θ at allows for agency-subgroup and agency-tme fxed effects. The parameter of nterest s θ, whch our model predcts s negatve. As wth (3), we assume s normal, mean zero, and dstrbuted ndependently across tranng agences and we cluster the errors at the tranng agency level. In all specfcatons reported to test H1 and H2, we weght each subgroup-agency-year observaton by the subgroup-agency share of the entre termnee populaton. 19 We have also consdered two varant specfcatons and the results were not affected (not reported): one wth equal weghts and another wth weghts proportonal to the subgroup s share relatve to ts agency populaton. In addton, we have consdered specfcatons where, constructng the subgroups, we exclude those enrollees who are termnated n the frst four months of each two year cycle. Our reasonng s that enrollees enterng a new two year cycle may have been υ t n 19 at That s, we weght each observaton by t n agency a n year t. a k t n kat where n at s the number of enrollees n subgroup 21

23 enrolled to optmze the prevous cycle s weghts (see footnote 16) Results 5-1 Tests of H1 and H2 for the ER Adustment Weghts Table 5 reports the results from our estmaton of the enrolment decson model, equaton (3). In all specfcatons the dependent varable s the subgroup s termnaton share. The rght-hand sde of the regresson ncludes the subgroup s weght for the employment standard (ER) n addton to the α a and α at. Model 1 produces a postve and statstcally sgnfcant estmate of the ER weght coeffcent, a fndng that s consstent wth H1. To gve the reader an dea of the magntude of the mpact of the weght change on enrollee choce, we nclude the standardzed coeffcents. Lterally nterpreted, our result says that a one standard devaton n a subgroup s performance weght relatve to the average ER weght ncreases the subgroup s enrolment share by about.1 percent relatve to the average agency subgroup. Ths response, however, s measured at the subgroup level whch s the correct unt of analyss to understand agency behavor, but s of lmted relevance from an economc or polcy pont of vew. To assess the economc sgnfcance of ths response, consder the followng thought experment. Assume a coeffcent on a demographc characterstc, e.g. female, s ncreased by a standard devaton relatve to the average coeffcent. The enrolment share of all female subgroups wll ncrease by about.1 percent relatve to the average subgroup. Snce there are on average 291 subgroups per agency n our sample (see Appendx 2), the overall ncrease n the share of females wll be 14.7 percent ( *291/2, because half of the subgroups are female on 20 We chose four months, because the average enrolment duraton s between four and fve months long. Fve months nto the new cycle, we reason, enrollees wll be termnatng n the cycle n whch they were ntended to be termnated. 22

24 average). The female PAW alone can have a large mpact on the composton of the enrollee populaton. But there are approxmately a dozen demographc characterstcs n play n the analyss suggestng that changes n PAW do have a sgnfcant nfluence on enrolment. Model 3 ncludes both the subgroup s ER and WE adustment weghts. The coeffcent estmate on the employment weght (n raw and standardzed form) changes lttle from column 1. Ths fndng supports our assumpton that there s lttle nteracton between the dfferent measures of the ncentve system. Table 6, column 1, reports the results of the employment outcome estmaton (equaton 4). We fnd a statstcally sgnfcant and negatve coeffcent estmate on the employment weght, as predcted under H2. The magntude of the estmate suggests that a one standard devaton ncrease n a subgroup s ER weght relatve to the average ER weght decreases the subgroup s relatve employment rate by about 2 ponts. Ths result remans when we add the earnngs weght as an explanatory varable. To assess the economc mplcaton of ths result, consder the thought experment dscussed above. Increasng the female weght by one standard devaton decreases the performance of females relatve to the average subgroup performance by 2 ponts. Ths fgure seems reasonable to us consderng that ths change n weght s assocated wth a 14.7 percent ncrease n the relatve share of females. The qualty of the margnal enrollee wthn a subgroup decreases as more enrollees are drawn from ths subgroup, whch s consstent wth the hypothess that agences cream-skm the best enrollees wthn each subgroup. Stll, ths fgure s small relatve to the varaton n performance across demographc subgroups. Table 3 shows that the standard devaton n the employment outcome across all subgroups and years s 43 ponts. 21 Therefore the potental to 21 What s relevant for cream-skmmng across subgroups n the absence of PAW are the predctable dfferences across subgroups at the agency level. To capture ths, we frst take a year average of the 23

25 cream-skm across subgroups (whch can be curbed wth the PAW) s of an order of magntude greater than wthn subgroups (whch s unaffected by the PAW). The PAW can elmnate the ncentve to cream-skm across-subgroups leavng a resdual ncentve to cream-skm wthn subgroup whch s second order. 5-2 Addtonal Measures and Robustness The prevous analyss s vald under the assumpton that the ER has receved the most emphass n the ncentve scheme, as has been argued n the lterature, and consstent wth the observaton that the falure rate s much larger for the ER measure, or under the alternatve assumpton that the varaton n the dfferent enrolment ncentves assocated wth each set of weghts are orthogonal to one another. Stll, mult-dmensonal ncentves may matter. We address ths ssue n two ways. Frst, we test H1 and H2 for the adustment weghts on the WE measure but the results are nconclusve. Second, we consder the mpact of the weghts on both the ER and WE measures on the sub sample of non-welfare recpents and the logc s that the weghts on the other two measures (the welfare ones) should not nfluence the enrolment ncentves among the non-welfare sub populatons. The results on the WE weghts obtaned from ths sub-sample are consstent wth H1 and H2 and we propose a possble nterpretaton that reconcles these two new sets of results. Adustment Weghts on the Earnngs Performance Measure (WE) Table 5, columns 2 and 3, show the mpact of the WE adustment weghts on the subgroup performance at the agency level (ths elmnates the unpredctable component of performance that s rrelevant n cream-skmmng), then compute the standard devaton n subgroup performance at the agency level, and fnally take the average across all agences. We obtan an average standard devaton of 43.6 whch s very smlar to the above fgure. 24

26 enrolment decson. Both columns show no mpact whch goes aganst H1. Table 7 reports the results of the earnngs outcome specfcaton (model (4) appled to WE). Whether we estmate the model wth ust the earnngs weght or both the earnngs and employment weghts, the estmated coeffcent on the earnngs weght s statstcally nsgnfcant aganst H2. 22 These two results could be because the WE measure plays a lesser role n the ncentve system or because the agences have less dscreton to select enrollees who are lkely to perform well on the WE measure. Interestngly, the coeffcent estmate correspondng to the employment weght n table 7, column 2, s postve and sgnfcant (p value 0.001). Whle there are many potental explanatons for ths fndng, t s consstent wth the exstence of a trade-off under multdmensonal ncentves: the knds of enrollees wthn a subgroup that produce hgher employment outcomes, reduce earnng outcomes. Non-Welfare Recpents The tranng agency s decson to enroll adult non-welfare recpents s less complcated than the decson to enroll welfare ones because non-welfare recpents characterstcs enter nto the determnaton of only the two standards that have been the focus of ths analyss. If the welfare measures play an mportant role, we should obtan a cleaner test of H1 and H2 when we lmt the analyss to the non-welfare adults. Therefore, Table 8 shows the results of the prevous analyses excludng the welfare recpents. Two ponts should be made. Frst, the power of the sgnfcance tests s smaller after we exclude welfare recpents, whch consttute about 40 percent of the adult populaton. Ths s 22 The number of observatons used n ths analyss s smaller than n the employment analyss because, consstent wth the JTPA defnton of the earnngs measure, we use only the enrollees who are employed (by the employment measure defnton) n the calculaton of the earnngs outcome. 25

27 partly responsble for why we observe that the coeffcent estmates on the employment weght n the enrolment share (model 1) and outcome regressons (models 2 and 3) are nsgnfcant. Second, the mpact of the earnngs weght n the regressons s greater when we exclude welfare recpents. In the enrolment share regresson, though the coeffcent estmate on earnngs weght remans nsgnfcant (by conventonal sgnfcance standards) t s postve (as predcted under H1). The coeffcent estmate on earnngs weght n the earnngs outcome regresson (model 3) s now both negatve and sgnfcant as predcted under H2. The standardzed coeffcent correspondng to ths estmate s about -6 suggestng that a one standard devaton ncrease n the WE weght relatve to the average WE weght reduces the relatve subgroup earnngs per week by about $6. Taken together, these two new sets of results suggest that although H2 does not hold for the entre sample, t does hold for the subset of non-welfare recpents. Ths may be because agences have much more dscreton to select applcants who are lkely to perform well on the earnngs measure, when they have to choose among non-welfare recpents, than they do for welfare recpents, who have on average lower levels of human captal. Also the earnngs measure s calculated only off employed termnees. Because welfare recpents are less lkely to be employed ther prospectve earnngs mght not be of such concern n the enrolment decson. 5-Summary and Conclusons The recent ntroducton of performance ncentves n several branches of the publc servce sector, such as n ob tranng, educaton, and health, has rased concerns as to ther mpact on enrolment decsons. In partcular, rewardng publc agences based on 26

28 measurable outcomes such as employment outcomes, test scores, or health outcomes may lead to student-trackng n educaton or the neglect of the hard-to-serve n ob tranng and of the chroncally ll n health care. To retan control over the recpent populaton, some polcy-makers have proposed adustng the measures that are used to assess performance, effectvely settng dfferent shadow prces for dfferent subgroups of clents, but lttle evdence exsts about the effectveness of these methods n practce. In the context of a large government ob tranng program, we nvestgate the nfluence of enrolment ncentves on case workers choce of ntake populaton. Job tranng agences n ths program are rewarded for mprovng the labor market performance of the clents they serve but the reward functon also depends on the enrolment choce. The man obectve of the enrolment ncentves s to level the playng feld, so that a tranng agency enrollng less able applcants has to meet a lower level of performance, effectvely settng a system of shadow prces that correct for the challenge that each demographc subgroup presents. Our emprcal analyss establshes two sets of results. Frst, we measure the mpact of changes n the relatve shadow prces on changes n the relatve fracton of dfferent demographc subgroups. We fnd that changes n the ncentve for enrollng members of a subgroup sgnfcantly change the fracton of enrollees from ths subgroup. Ths s good news for those who wsh to use PAW n ob tranng and n other publc servces to attenuate the negatve dstrbutonal consequences of performance-based ncentve systems. One should keep n mnd that the effectveness of PAW elsewhere, however, wll depend on the nature of the heterogenety among partcpants and n the ablty of the desgner to dentfy dmensons over whch cream-skmmng takes place. 27