Estimating Transition Models with Misclassification

Transcription

1 smang Transon Models wh Msclassfcaon Ncola Torell Dearmen of conomcs and ascs nversy of Trese Adrano Paggaro Dearmen of ascs nversy of Padua Inroducon Longudnal survey daa are wdely used o sudy mcro-level dynamcs of socal and economc henomena. Ineres ofen focuses on he use of longudnal daa o sudy he ranson of uns among a fne se of saes. For nsance when sudyng labor marke dynamcs he esmaon of some descrve measures lke gross flows among labor force saes s crucal. The effecs of msclassfcaon of labor force saes n esmang gross flows have been analyzed by many auhors and s well known ha esmaes may be affeced by severe bas leadng o a oally erroneous cure of labor force dynamcs (for a revew see knner and Torell 993). ascal models have been roosed o esmae gross flows by akng no accoun classfcaon errors hese models can eher use valdaon daa from renervew sudes or auxlary varables (see among ohers Chua and Fuller 987 Pfefferman knner and Humhrey 998). mlar yes of daa may be used o esmae ranson models amed a exlanng how he me sen n a sae affecs he robably of leavng. Analyss of unemloymen duraon s a classc examle of alcaon of hese models n he conex of sudyng labor force dynamcs. A revew of he uses of daa comng from dfferen samlng schemes n esmang ranson models may be found n Lancaser (990). Daa on me sen n each sae are ofen obaned by observng he saes occued by he same un n successve waves of a anel survey. Consderng agan he analyss of labor marke dynamcs f labor force saes are msclassfed eher observed ransons acually refer o suaons n whch uns are sll n he same sae or hose observed n he same sae have changed. Parameer esmaes of ranson models when saes are msclassfed may be execed o be based. The roblem s obvously more general and smlar consequences may be execed whenever longudnal daa from follow-u sudes or anel surveys are avalable for esmang models for survval daa. In hs aer he effec of msclassfcaon n esmang ranson models s assessed and a sraegy s roosed o oban esmaes of model arameers ogeher wh esmaes of msclassfcaon robables. As a movang examle ranson models n he conex of analyss of unemloymen duraon wh daa ycally obaned from labor force surveys are nroduced n secon 2. econ 3 resens a smle verson of a ranson model ollued by

2 msclassfcaon n he desnaon sae and he effec of gnorng msclassfcaon s evaluaed by means of a smulaon sudy. econs 4 and 5 examne he roblem of esmang he model under dfferen assumons by usng sandard sraeges. econ 6 rooses esmaon of a ranson model adjusng for msclassfcaon usng a bayesan aroach. econ 7 consders an alcaon of he roosed model on a real daa se. ome fnal commens and drecons for fuure work are made n secon Transon models and analyss of unemloymen duraon Le us sar by consderng he smles suaon n whch daa are obaned from wo waves of a anel survey. Our neres les n esmang he robably of ranson beween wo saes (hereafer and ) durng me k searang he wo nervews more recsely we focus on esmang he robably of occuyng sae for hose n sae a he frs survey as a funcon of me already sen n sae. I s also assumed ha daa on me T sen n sae before he frs nervew are avalable. The robably of ranson from o deends on me T sen n sae and on a vecor of covaraes X. Le () and f() denoe resecvely he survvor funcon and he densy funcon of T and le = f a ranson o s observed and = 0 oherwse. The deendence of he survvor funcon on X may be modeled n dfferen ways (for nsance by assumng a rooronal hazard secfcaon) so ha he effec of he covaraes s measured by a se of arameers. We assume ha belongs o a gven aramerc famly esmaon of and may be obaned by maxmzng he followng lkelhood funcon (assumng ha a samle of n ndeenden observaons s avalable): ( k x ) ( x ) ( k x ) ( x ) n + + L( ) =. () = nder he assumon of rooronal hazards hazard funcon ( ) f ( ) ( ) on X only hrough arameers whle s shae s defned by baselne hazard h 0 ( ) framework he survvor funcon may be wren as ( ) ex [ H0 ( ) ex ( x' )] H ( ) = h ( z) 0 0 dz 0 h = deends. In hs = where s he negraed hazard. Ths makes lkelhood funcon () formally equvalen o ha obaned n a generalzed lnear model conex for bnary deenden varables he lnk funcon secfcaon deends on assumons made on h 0 ( ). In he followng o smlfy comuaon and o hel undersandng he man resuls we α assume a Webull rooronal hazard model so ha H 0 ( α ) = and a smle secfcaon for () s obaned (noe however ha he resuls do no deend on hs secfc assumon). In hs case nerreaon of shae arameer α s easy and s relaed o how me sen n a sae may nfluence he hazard of ranson o he desnaon sae: he value α = dscrmnaes beween negave duraon deendence (0 < α < ) and osve duraon deendence (α > ). 2

3 The smle suaon consdered here closely corresonds o wha s ycally found when analyzng unemloymen duraon daa from labor force surveys whch ado roang samlng schemes (Trvellao and Torell 989).e. consderng a sae-based samle wh follow-u. These models have frequenly been aled by economercans and oher socal scenss (see Lancaser 990). In he conex oulned above as regards he mac of non-samlng errors comaravely more aenon has been devoed o analyss of he effec of measuremen errors n duraon (e.g. Hol McDonald and knner 99 documened he effec of errors when daa are obaned from a sae-based desgn wh follow-u). 3. ffec of msclassfcaon n desnaon sae n esmang duraon models Le us assume ha: () a he frs nervew he sae s observed whou error () a he second nervew he sae may be msclassfed. Ths smle suaon has n fac some raccal neres as he follow-u nervew s ofen less accurae (e.g. admng a hgher rae of roxy resonses) and/or obaned usng dfferen modes ha may be more error-rone (for nsance elehone nsead of face-o-face nervews). Noe ha he same assumon s made by Poerba and ummers (995) n a smlar conex. Indcaor s hen measured wh error and: (a) P(= )=.e. he robably ha we observe a ranson from o when he rue sae a he second nervew s s no (bu we exec o be very close o ) (b) P(= )=.e. he robably ha we observe a ranson from o bu ha he rue sae a he second nervew s s no 0 (bu we exec o be very close o 0). Hereafer and (- ) are called relables. Ths smle msclassfcaon mechansm may nduce severe bas n arameer esmaes. To arecae hs we analyzed he resuls of a Mone Carlo sudy wh sae-based daa wh follow-u smulaed assumng ha ranson mes are generaed by a rooronal hazard model. Here we only reor some resuls wh a Webull baselne. Noe however ha we obaned he same conclusons even hough dfferen (and more flexble) assumons were made for he baselne hazard funcon. In desgnng he smulaon sudy we mached a real suaon so ha he sze of he samle (999 ndvduals) and daa on covaraes (age gender educaon maral saus) were fxed a he values acually observed n he Ialan labor force survey for hose unemloyed n norhern Ialy n Ocober 997. Moreover he arameers were such ha he average duraon was no far from hose ycally observed for unemloymen duraon n Ialy. Table conans a selecon of he smulaon resuls n whch γ and γ denoe he log ransforms resecvely of and. These robables were fxed resecvely a around 0.97 and 0.05 wh msclassfcaon errors close o hose found n many emrcal works (see among ohers Poerba and ummers 995 whch conans esmaes from valdaon daa) and secfcally n sudes carred ou wh Ialan daa (Bass Torell and Trvellao 998). I s 3

4 neresng o noe ha he average sze of arameers s always reduced showng a sor of aenuaon effec. 4

5 Table. Mone Carlo smulaons for a Webull rooronal hazard model wh msclassfcaon number of relcaons = 00 g = 4 g = -3 Parameer True Average s.dev True Average s.dev. True Average s.dev. Log α Inerce Age Gender (=F) Mar. (=marred) duc. (=hgher) vdence on he srong basng effec due o msclassfcaon has clearly been confrmed by larger smulaon exercses assumng dfferen szes for msclassfcaon robables. Table 2 lss some resuls on a Webull baselne wh negave duraon deendence and symmerc msclassfcaon robables = (- ) rangng from 0.0 o As execed more bas s nduced by a more subsanal amoun of msclassfcaon bu even moderae msclassfcaon leads o a non-neglgble bas n he arameer esmaes. More clear evdence emerges abou he relaon beween bas and rue rae of ransons o : gven a fxed he effec s greaer f here s a revalence of rue ransons whereas s neglgble f such ransons are few he oose s obvously rue for. Table 3 resens he resuls for a rae of ransons close o 5% and msclassfcaon only on one sde (eher = or = 0). A smle exlanaon s ha he effec s sronger whenever here are more canddaes for msclassfcaon. The followng secons show how hs dfferen effec nfluences he ossbly of jonly esmang and boh and. Table 2. Mone Carlo smulaons under dfferen levels of symmerc msclassfcaon robables = (- ) averages over 00 relcaons Parameer True ymmerc msclassfcaon robables = (- ) Log α Inerce Age Gender (=F) Mar. (=marred) duc. (=hgher)

6 Table 3. Mone Carlo smulaons under dfferen levels of asymmerc msclassfcaon robables (eher = 0 or = ) averages over 00 relcaons Parameer True Asymmerc msclassfcaon robables = 0 = = 0.95 = 0.73 = 0.05 = 0.27 Log α Inerce Age Gender (=F) Mar. (=marred) duc. (=hgher) smaon of ranson models wh known relables If and were known would be ossble o oban conssen esmaes of he arameers of he model by maxmzng he followng lkelhood funcon: where L n (2) = ( ) = [ P( = x) ] [ P( = x) ] ( = x) = P( = ) P( x) + P( = ) P( x) = P = + ( k x ) ( x ) + ( ) k x ( x ) + s he robably of observng a ranson. Table 4 lss some smulaons (carred ou n he same condons saed n he revous secon) by usng lkelhood (2) and assumng ha relables of ndcaors of saes are fxed o ceran values. When and are fxed o her rue values he arameers of neres are conssenly esmaed whou bas bu he mos srkng evdence regards he sensvy of he esmaes when and are fxed o values whch are dfferen from he rue ones bu reasonably close o hem. sually n real alcaons s reasonable o have an dea abou he range n whch he robables of error should le and one could be emed o use values obaned from valdaon sudes carred ou n suaons smlar o hose consdered for he relables. Referrng o a smlar suaon.e. esmang models for a dchoomous deenden varable wh msclassfcaon Hausman e al. (998) show ha f conssen esmaes of and are used s ossble o oban conssen esmaes of arameers bu her sandard errors are underesmaed. If and are no esmaed conssenly neher are he esmaes of conssen. 6

7 7 Table 4. Mone Carlo smulaons of a ranson model wh fxed relables averages over 00 relcaons Parameer True ML esmaon wh fxed g = -g (rue g =3) Log α Inerce Age Gender (=F) Mar. (=marred) duc. (=hgher) smaon of ranson models wh unknown relables Lkelhood funcon (2) may be used o oban esmaes of he model arameers even when msclassfcaon robables are unknown. As saed n Abrevaya and Hausman (999) a necessary condon for denfcaon s he obvous resrcon > ndcang ha he robably of beng classfed n a sae say s greaer f he rue sae s han f s. To smlfy he maxmzaon of (2) s convenen o use he M algorhm. To hs end le A denoe a varable ndcang acual ranson o emloymen (.e. whou error). Lkelhood (2) s hen obaned by margnalzaon of he followng jon lkelhood: = + + = n A A x x k x x k L [ ] [ ] = n A A. (3) In he M se we maxmze (3) condonally on he execed value of A for each un n he samle whch s easly calculaed n he se as follows: [ ] [ ] k k k A =. (4) In racce maxmzng he lkelhood funcon for hs model very ofen leads o oally unreasonable values for some arameers and he algorhm converges o ons a he border of he aramerc sace. These roblems may have o do wh weak denfcaon of some arameers of he model nducng flaness of he lkelhood funcon. In fac we have emrcally verfed ha when very large samles are avalable reasonable resuls may be obaned more frequenly. Ths on ceranly deserves more horough nvesgaon.

8 Noneheless some resuls on he erformance of maxmum lkelhood esmaon obaned whn larger Mone Carlo sudes no resened here may shed lgh on he qualy of esmaes of and obaned by usng sandard maxmum lkelhood and on how hey are relaed o ranson raes. If daa nclude few ransons he esmaes of may eher go owards he border of he aramerc sace or show reasonable values. Conversely as oulned n commenng Table 3 here are few ransons whch are canddae for msclassfcaon hus unsable and ofen unreasonable esmaes of are obaned. I s moran o noe however ha hs lack of nformaon seems o have no basng effec on he oher arameers of he model. When he ranson rae s hgh he same resuls are obvously obaned f we nver and. 6. Bayesan esmaon of ranson models wh classfcaon errors In alcaons o real daa s reasonable o have a more or less vague dea abou whch values are credble for msclassfcaon robables. A leas we know ha hey should no be very far from 0. Ths encourages us o formulae he ranson model whn a bayesan framework. In hs case he oseror dsrbuon of he arameers s gven by: ( A X ) L( A X ) g ( A) 0 g = where g 0 denoes ror dsrbuon and L( A X ) has he same form defned n (3). A leas for he mos neresng arameers.e. and he relables s lausble o summarze he nformaon avalable on hem by secfyng nformave rors. Poseror dsrbuon s obvously que comlex bu may be exlored usng MCMC mehods. For he model examned here one suable sraegy s o ado Merools-Hasngs whn Gbbs samler eraons (see Chb and Greenberg 995). The Merools-Hasngs algorhm s used o samle from nracable full condonal dsrbuons arsng whn he Gbbs samler. A he -h eraon of he algorhm condonally on he values drawn a he revous eraon he man ses are he followng:. Draw A from g( A X ) 2. Draw ( ) from g( A X ) 3. Draw from g ( A ) 4. Draw from g ( A ). Drawng a samle from g(a) a se s sraghforward as s a bnomal dsrbuon whose mean s gven by (4). Insead he full condonal dsrbuons of he oher arameers are no n close form so ha a sngle se of he Merools-Hasngs algorhm s used a ses 2 3 and 4. For a generc arameer φ we draw a canddae value φ from a normal dsrbuon cenered a he revous value φ and acce he canddae wh robably 8

9 mn ( g( φ ) g( φ )) f he canddae s rejeced we kee φ = φ. Varances of normal dsrbuons are chosen n order o kee he acceance rae around 20%. Table 5 conans some characerscs of oseror dsrbuons resulng from he use of MCMC echnques as defned above aled o hree dfferen smulaed samles from a rooronal hazard model wh Webull baselne. The ables (frs hree columns) also ls he resuls obaned by esmang arameers by maxmum lkelhood when msclassfcaon s gnored. Once agan smulaed daa were chosen n order o resemble daa from wo successve waves from he Ialan labor force survey amng a analyzng ransons from unemloymen o emloymen. Thus he samle sze s agan ke a 999 n each smulaon and he values of he arameers are fxed close o he ones esmaed for a real samle he only exceon beng he use of dfferen srucures of duraon deendence (dfferen values for α). The msclassfcaon robables chosen for he smulaon sudy are smlar o hose encounered n real suaons as far as classfcaon of labor force saes s concerned (γ = 4 and γ = -3 whch are aroxmaely equvalen o seng = 0.97 and = 0.05). The ror dsrbuons (assumed o be ndeenden) for arameers are gaussan N(04) (wh he only exceon beng he mean of he consan erm whch s fxed a 4 o mach he average unemloymen duraon n he samle n case of exonenal baselne and no effecs of covaraes). For γ and γ he rors are N(4.5) and N(-3.5) hus msclassfcaon robables are a-ror assumed o vary around resecvely % and 3% -.e. less han he rue msclassfcaon robables used o generae he daa Gbbs samler eraons were consdered afer a burn-n erod of eraons and a samle of sze 000 from he oseror s obaned selecng values every 00 eraons. Table 5: ome summares of a samle obaned (by MCMC) from oseror dsrbuon of ranson models wh msclassfcaon (see he man ex for deals) Maxmum lkelhood Poseror dsrbuon summares and ercenles True ML s.d. Mode Mean 5% 25% 50% 75% 95% Log α Inc Age Gender Mar duc γ γ

10 Maxmum lkelhood Poseror dsrbuon summares and ercenles True ML s.d. Mode Mean 5% 25% 50% 75% 95% Log α Inc Age Gender Mar duc γ γ Maxmum lkelhood Poseror dsrbuon summares and ercenles True ML s.d. Mode Mean 5% 25% 50% 75% 95% Log α Inc Age Gender Mar duc γ γ The resuls obaned are encouragng and he bayesan aroach may n hs case overcome some dffcules whch emerge when usng maxmum lkelhood. The avalably of nformave rors for msclassfcaon robables s crucal bu as already noed s que reasonable o use ror dsrbuons whch assgn a subsanal robably (e.g. more han 0.95) o he nerval (00.2) for msclassfcaon robables. As already argued f here are few (or conversely oo many) ransons he daa do no convey enough nformaon o esmae one of he relables and he oseror dsrbuon of γ (γ ) essenally reflecs only nformaon summarzed by ror dsrbuon. Ths nfluences esmaon of he oher arameers very lle as her oseror dsrbuon s mosly concenraed around values whch are close o he rue ones. More recsely s close o s rue value 3 whle arameers α and are clearly moved far aar from her maxmum lkelhood esmaes. To arecae he erformance of he MCMC rocedure Fgure reors one sequence of values from he margnal oseror dsrbuon of he arameer assocaed o educaon smulaed by MCMC echnques usng he second daa se of able 5. Fgure 2 shows he margnal oseror densy of he same arameer (obaned usng a kernel smoohng echnque). The oher arameers show subsanally a smlar behavor. 0

11 Fgure : equence of values smulaed by MCMC for he arameer b assocaed o educaon Parameer mulaed seres True value ML Pror mean

12 Fgure 2: Densy of he margnal oseror dsrbuon for he arameer b assocaed o educaon Densy True value ML Pror mean Parameer 7. smaon of a ranson model wh msclassfcaon usng daa from Ialan labor force survey In Ialy he labor force survey ados a samle roaon scheme whch rovdes longudnal daa. More recsely each household s ncluded n he samle for wo consecuve surveys hen dros ou for wo surveys and re-eners he samle for wo fnal waves. Abou 50% of he samle s common n wo successve surveys. For each household member nformaon on labor force sae s obaned and hose who declare hemselves as unemloyed also answer a queson on how long hey have been unemloyed. These daa have been used o esmae ranson models of he form dscussed n he revous secons. mlar daa are also avalable from oher labor force surveys n develoed counres (n fac he scheme adoed n Ialy closely resembles he Curren Poulaon urvey carred ou n he..) and daa lke hese have acually been used o esmae ranson models o emloymen (see Trvellao and Torell 989). The Ialan daa are robably also ollued by subsanal classfcaon errors (as documened by Bass Torell and Trvellao 998). Alhough he model roosed here ress on some smlfyng assumons whch may easly be consdered 2

13 no comleely realsc we hnk ha s useful o gan some nsghs on he sensvy of esmaes o msclassfcaon. Table 6 conans he resuls of he alcaon of MCMC mehods o a real samle comng from he Ialan labor force survey. A samle of 999 labor force uns from Norhern Ialy who were unemloyed n Ocober 997 s avalable for whom we observe he duraon of he ongong unemloymen sell (and oher covaraes) and he labor sae afer 3 monhs (January 998). arng from hese daa we used he MCMC mehods descrbed n secon 6. Table 6 shows some summares of he samle of 000 values comng from oseror dsrbuon. Table 6: amle obaned (by MCMC) from oseror dsrbuon of ranson models wh msclassfcaon daa from Ialan labor force survey Ocober 997 Maxmum lkelhood Poseror dsrbuon summares and ercenles ML s.d. Mode Mean 5% 25% 50% 75% 95% Log α Inc Age Gender Mar duc γ γ The resuls confrm ha f we ake no accoun he robably of msclassfcaon he absolue values of he arameers of neres end o be hgher han hose obaned by sandard maxmum lkelhood esmaon assumng no msclassfcaon. Ths s arcularly rue for he duraon deendence arameer α and for arameers assocaed wh gender and educaon. Moreover he classfcaon error for he unemloyed s mosly concenraed beween 2% and 5% nerreaon of he esmaed robably of msclassfcaon for he emloyed s more dffcul due o he small fracon of ranson observed n he samle. 8. Concludng remarks Ths aer hghlghs how moran s o ake no accoun msclassfcaon when esmang ranson models. As he use of some sandard mehods s recluded secfyng reasonable models for msclassfcaon may rovde a soluon. Msclassfcaon robables enerng he model are usually unknown bu we can make reasonable guesses abou her robable magnude. When hs s he case classc nferenal mehods are sll ossble bu less naural and less raccal han a bayesan aroach. Ths aroach gves esmaes boh of he arameers of he model and of he msclassfcaon robables. We consdered a smlfed model and he man drecons for fuure research wll exend he framework owards more realsc secfcaons. More secfcally he model may be exended n many drecons: () by examnng daa comng from mul-wave anel surveys nsead of lmng aenon smly o wo waves () by allowng msclassfcaon robables o deend on covaraes () by 3

14 consderng he case of subsanal msclassfcaon also a he frs nervew (v) by exendng he model o analyss of movemens among a se of more han wo saes. xendng he model may be of grea neres when analyzng he general case of modelng even hsory daa obaned from anel surveys. There s a clear connecon of he model examned here wh he more general roblem of analyzng caegorcal daa wh msclassfcaon (for a revew see Kuha and knner 997) whch may be furher exlored o exend he resuls resened here. As already noed more effor s needed o fully undersand denfcaon roblems when usng maxmum lkelhood. RFRNC Abrevaya J. and Hausman J.A. (999). emaramerc smaon wh Msmeasured Deenden Varables: an Alcaon o Duraon Models for nemloymen ells. Annales d conome e de asque Bass F. Torell N. and Trvellao. (998). "Daa and modellng sraeges n esmang labour force gross flows affeced by classfcaon errors". urvey Mehodology Chb. and Greenberg. (995). ndersandng he Merools-Hasngs Algorhm. The Amercan ascan Chua T. and Fuller W.A. (987). A Model for Mulnomal Resonse rrors Aled o Labor Flows. Journal of he Amercan ascal Assocaon Hausman J.A. Abrevaya J. and co-moron F.M. (998). Msclassfcaon of he Deenden Varable n a Dscree-Resonse eng. Journal of conomercs Hol D. McDonald J. and knner C. (99). "The effec of measuremen error on even hsory analyss". In Bemer P.P. Groves R.M. Lyberg L.. Mahowez N.A. and udman. (eds.) Measuremen rror n urveys New York: Wley Kuha J. and knner C. (997). "Caegorcal Daa Analyss and Msclassfcaon". In Lyberg L.. Bemer P.P. De Leeuw. Do C.. chwarz N. and Trewn D (eds.) urvey Measuremen and Process Qualy New York: Wley Lancaser T. (990). The conomerc Analyss of Transon Daa. Cambrdge: Cambrdge nversy Press. Pfefferman D. knner C.J. and Humhrey.K. (998). The smaon of Gross Flows n he Presence of Measuremen rror sng Auxlary Varables. JR eres A

15 Poerba J.M. and ummers L.H. (995). nemloymen Benefs and Labor Marke Transons: a Mulnomal Log Model wh rrors n Classfcaon. Revew of conomcs and ascs knner C.J. and Torell N. (993). Measuremen rrors and he smaon of Gross Flows from Longudnal conomc Daa. asca Trvellao. and Torell N. (989). Analyss of Labor Force Dynamcs from Roang Panel urvey Daa. Bullen of The Inernaonal ascal Insue Vol. LIII Book