Overview. Linear Models Connectionist and Statistical Language Processing. Numeric Prediction. Example

Transcription

1 Overvew Lear Models Coectost ad Statstcal Laguage Processg Frak Keller Computerlgustk Uverstät des Saarlades classfcato vs. umerc predcto lear regresso least square estmato evaluatg a umerc model, correlato selectg a regresso model lear regresso for classfcato regresso trees, model trees Lterature: Wtte ad Frak (2000: ch. 4, 6), Howell (2002: ch. 15). Lear Models p.1/26 Lear Models p.2/26 Numerc Predcto A stace the data set has the followg geeral form: a 1,,a 2,,...,a k,,x where a 1,,...,a k, are attrbute values, ad x s the target value, for the -th stace the data set. So far we have oly see classfcato tasks, where the target value x s categorcal (represets a class). Techques such as decso tree ad Nave Bayes are ot (drectly) applcable f the target s umerc. Istead algorthms for umerc predcto ca be used, e.g., lear models. Lear Models p.3/26 Predct CPU performace from cofgurato data: cycle memory memory cache cha cha perfortme (s) m (kb) max (kb) (kb) m max mace Lear Models p.4/26

2 Lear Regresso Lear regresso s a techque for umerc predctos that s wdely used psychology, medcal research, etc. Key dea: fd a lear equato that predcts the target value x from the attrbute values a 1,...,a k : (1) x = w 0 + w 1 a 1 + w 2 a w k a k Here, w 1,...w k are the regresso coeffcets, w 0 s called the tercept. These are the model parameters that eed to be duced from the data set. Lear Regresso The regresso equato computes the followg predcted value x for the -th stace the data set. (2) x = w 0 + w 1 a 1,,w 2 a 2,,...,w k a k, = w 0 + k w j a j, j=1 Key dea: to determe the coeffcets w 0,...w k, mmze e, the squared dfferece betwee the predcted ad the actual value, summed over all staces the data set: (3) e = =1 (x x ) 2 = =1 ( x w 0 ) 2 k w j a j, j=1 The method for ths s called Least Square Estmato (LSE). Lear Models p.5/26 Lear Models p.6/26 Least Square Estmato Least Square Estmato We demostrate how LSE works wth the smple case of k = 1, droppg the tercept w 0. The error equato (3) smplfes to (abbrevatg w 1 = w ad a 1 = a): (4) e = (x wa ) 2 = (x 2 2wa x + w 2 a 2 ) Now dfferetate the error equato (4) wth respect to w: (5) e w = ( 2a x + 2wa 2 )= 2a x + 2wa 2 The dervatve s the slope of the error fucto. The slope s zero at all pots at whch the fucto has a mmum. To mmze the squared error for the data set, we therefore set the dervatve (5) equal to zero: (6) 2 a x + 2wa 2 = 0 By resolvg ths equato to w, we obta a formula for computg the value of w that mmzes the error: w = a x (7) a 2 Ths formula ca be geeralzed to regresso equatos wth more tha oe coeffcet. Lear Models p.7/26 Lear Models p.8/26

3 Sample data set: a x Use Least Square Estmato to compute w for ths data set: (8) w = x a a 2 = ( 1)( 2) ( 1) = 1.74 Regresso equato: x = wa = 1.74a Lear Models p.9/26 Evaluatg a Numerc Model The ft of a regresso model ca be vsualzed by plottg the predcted data values agast the actual values. target value actual value x predcted value x attrbute value a Lear Models p.10/26 Evaluatg a Numerc Model A sutable umerc measure for the ft of a lear model s the mea squared error: (9) MSE = 1 =1 (x x ) 2 Itutvely, ths represets how much the predcted values dverge from the actual values o average. Note that the MSE s the quatty the LSE algorthm mmzes. Compute the mea squared error for the sample data set ad the regresso equato x = 1.74a: a x x (x x ) MSE = =1 (x x )2 = 1 4 ( )= Lear Models p.11/26 Lear Models p.12/26

4 Correlato Coeffcet The correlato coeffcet r measures the degree of lear assocato betwee predcted ad the actual values: (10) (11) (12) r = S PA S P S A S PA = =1(x x )(x x) 1 S P = =1 (x x ) 2 S A = =1 (x x) Here x ad x are the meas of the actual ad predcted values, S P ad S A ther stadard devatos. S PA s the covarace of the actual ad predcted values. Compute the correlato coeffcet for the example data set: x = 3.25 x = 3.14 S PA = (( )(2 3.25)+( )(5 3.25)+ ( )( )+( )(8 3.25))/3 = SP 2 = (( ) 2 +( ) 2 + ( ) 2 +( ) 2 )/3 = SA 2 = ((2 3.25) 2 +(5 3.25) 2 + ( ) 2 +(8 3.25) 2 )/3 = r = 18.13/( )=0.975 r 2 = Lear Models p.13/26 Lear Models p.14/26 Correlato Coeffcet Partal Correlatos Some mportat propertes: The correlato coeffcet r rages from 1.0 (perfect correlato) to 0 (o correlato) to 1.0 (egatve correlato). Itutvely, r expresses how well the data pots ft o the straght le descrbed by the regresso model. We ca test f r s sgfcat. Null hypothess: there s o lear relatoshp betwee predcted ad actual values. We ca also compute r 2, whch represets the amout of varace accouted for by the regresso model. We ca compute the multple correlato coeffcet that tells us how well the full regresso model (wth all attrbutes) fts the target values. We ca also compute the correlato betwee the values of a sgle attrbute ad the target values. However ths s ot very useful as attrbutes ca be tercorrelated,.e., they correlate wth each other (colearty). We eed to compute the partal correlato coeffcet, whch tells us how much varace s uquely accouted for by a attrbute oce the other attrbutes are partalled out. Lear Models p.15/26 Lear Models p.16/26

5 Selectg a Regresso Model We wat to buld a regresso model that oly cotas the attrbutes that are predctve. Several methods to acheve ths: All subsets: compute models for all subsets of attrbutes ad chose the oe wth the hghest multple r. Backward elmato: compute a model for all attrbutes, ad the elmate the oe wth the lowest partal r. Iterate utl the multple r deterorates. Forward selecto: compute a model cosstg oly of the attrbutes wth the hghest partal r. The add the ext best attrbute. Stop whe the multple r does t mprove. Dfferet model selecto algorthm ca yeld dfferet models. Testg o Usee Data We compute the regresso weghts ad perform the model selecto o the trag data. To evaluate the resultg model, we compute model ft o usee test data usg LSE or the correlato coeffcet. Techques for testg o usee data (see last lecture): Holdout: set asde a radom sample of the data set for testg, tra o the rest. k-fold crossvaldato: splt the data k radom partto ad test o each oe tur. leave-oe-out: set k to the umber of staces the data set,.e., test o each stace separately. Lear Models p.17/26 Lear Models p.18/26 Lear Regresso for Classfcato Regresso ca be appled to classfcato: Perform a separate regresso o each class, set the target value to 1 f a stace s the class, ad 0 f t s ot the class. The regresso equato the approxmates the membershp fucto for the class (1 for members, 0 for o-members). To classfy a ew stace, compute the regresso value for each membershp fucto, ad assg the ew stace the class wth the hghest value. Ths procedure s called multrespose lear regresso. Lear Separablty Lear regresso approxmates a lear fucto. Ths meas thattheclasseshavetobelearly separable. attrbute a2 attrbute a1 attrbute a1 For may terestg problems, ths s ot the case. attrbute a2 Lear Models p.19/26 Lear Models p.20/26

6 Regresso Trees Regresso trees are decso trees for umerc attrbutes. The leaves are ot labeled wth classes, but wth the mea of the target values of the staces classfed by a gve brach. To costruct a regresso tree, chose splttg attrbutes to mmze trasubset varato for each brach. Maxmze stadard devato reducto (stead of formato ga): (13) SDR = σ T T T σ T Where T s the set of staces classfed at a gve ode, T 1,...,T are the subset that T s splt to, ad σ s the stadard devato. Lear Models p.21/26 Lear Models p.22/26 Model Trees Model trees are regresso trees that have lear regresso models at ther leaves, ot just umerc values. Iducto algorthm: Iduce a regresso tree usg stadard devato reducto as the splttg crtero. Prue back the tree startg from the leaves. For each leaf costruct a regresso model that accouts for the staces classfed by ths leaf. Model trees get aroud the problem of lear separablty by combg several regresso models. Lear Models p.23/26 Lear Models p.24/26

7 Summary Refereces Lear regresso models are used for umerc predcto. They ft a lear equato that combes attrbutes values to predct a umerc target attrbute. Howell, Davd C Statstcal Methods for Psychology. Pacfc Grove, CA: Duxbury, 5th ed. Wtte, Ia H., ad Ebe Frak Data Mg: Practcal Mache Learg Tools ad Techques wth Java Implemetatos. Sa Dego, CA: Morga Kaufma. Least square estmato ca be used to determe the coeffcets the regresso equato so that the dfferece betwee predcted ad actual values s mmal. A umerc model ca be evaluates usg the mea squared errororthecorrelato coeffcet. Regresso models ca be used for classfcato ether drectly multrespose regresso or combato wth decso trees: regresso trees, model trees. Lear Models p.25/26 Lear Models p.26/26