CS 2750 Machine Learning. Lecture 7. Linear regression. CS 2750 Machine Learning. Linear regression. is a linear combination of input components x

Transcription

1 CS 75 Mache Learg Lecture 7 Lear regresso Mlos Hauskrecht mlos@cs.ptt.eu 539 Seott Square CS 75 Mache Learg Lear regresso Fucto f : X Y s a lear combato of put compoets f k - parameters eghts Bas term f Iput vector CS 75 Mache Learg

2 Lear regresso. Error. Data: D Fucto: f We oul lke to have f for all.. Error fucto measures ho much our prectos evate from the esre asers Mea-square error.. Learg: We at to f the eghts mmzg the error! f CS 75 Mache Learg Lear regresso. Eample mesoal put CS 75 Mache Learg

3 Lear regresso. Eample. mesoal put CS 75 Mache Learg Lear regresso. Optmzato. We at the eghts mmzg the error f.. For the optmal set of parameters ervatves of the error th respect to each parameter must be Vector of ervatves: gra.. CS 75 Mache Learg 3

4 4 CS 75 Mache Learg Lear regresso. Optmzato. efes a set of equatos gra CS 75 Mache Learg Solvg lear regresso B rearragg the terms e get a sstem of lear equatos th + ukos A b

5 5 CS 75 Mache Learg Solvg lear regresso he optmal set of eghts satsfes: Leas to a sstem of lear equatos SLE th + ukos of the form Soluto to SLE: matr verso A b b A CS 75 Mache Learg Graet escet soluto Goal: the eght optmzato the lear regresso moel A alteratve to SLE soluto: Graet escet Iea: Aust eghts the recto that mproves the Error he graet tells us hat s the rght recto - a learg rate scales the graet chages Error.. f Error

6 Graet escet metho Desce usg the graet formato Error Error * * Drecto of the escet Chage the value of accorg to the graet Error CS 75 Mache Learg Graet escet metho Error Error * * Ne value of the parameter Error * * - a learg rate scales the graet chages CS 75 Mache Learg For all 6

7 Graet escet metho Iteratvel approaches the optmum of the Error fucto Error 3 CS 75 Mache Learg Ole graet algorthm he error fucto s efe for the hole ataset D Error f error for a sample.. ole Error f Ole graet metho: chages eghts after ever sample Error vector form: D Error - Learg rate that epes o the umber of upates CS 75 Mache Learg 7

8 Ole graet metho Lear moel f O-le error ole Error f O-le algorthm: geerates a sequece of ole upates -th upate step th : -th eght: D Error f Fe learg rate: - Use a small costat C Aeale learg rate: - Grauall rescales chages CS 75 Mache Learg Ole regresso algorthm Ole-lear-regresso D umber of teratos Italze eghts for =:: umber of teratos o select a ata pot D from D set learg rate upate eght vector f e for retur eghts Avatages: ver eas to mplemet cotuous ata streams CS 75 Mache Learg 8

9 O-le learg. Eample CS 75 Mache Learg Practcal cocers: Iput ormalzato Iput ormalzato makes the ata var roughl o the same scale. Ca make a huge fferece o-le learg Assume o-le upate elta rule for to eghts k: f = k k f k Chage epes o the magtue of the put For puts th a large magtue the chage the eght s huge: chages to the puts th hgh magtue sproportoal as f the put as more mportat CS 75 Mache Learg 9

10 Iput ormalzato Iput ormalzato: Soluto to the problem of fferet scales Makes all puts var the same rage arou Ne put: ~ More comple ormalzato approach ca be apple he e at to process ata th correlatos Smlarl e ca reormalze outputs CS 75 Mache Learg Etesos of smple lear moel Replace puts to lear uts th feature bass fuctos to moel oleartes f m f - a arbtrar fucto of m m he same techques as before to lear the eghts CS 75 Mache Learg

11 Atve lear moels Moels lear the parameters e at to ft m f k k k... m - parameters... m - feature or bass fuctos Bass fuctos eamples: a hgher orer polomal oe-mesoal put 3 3 Multmesoal quaratc Other tpes of bass fuctos s cos CS 75 Mache Learg Fttg atve lear moels Error fucto /.. f Assume: φ.. Leas to a sstem of m lear equatos f φ m m m Ca be solve eactl lke the lear case CS 75 Mache Learg

12 Eample. Regresso th polomals. Regresso th polomals of egree m Data pots: pars of Feature fuctos: m feature fuctos m Fucto to lear: m f m m m m CS 75 Mache Learg Learg th feature fuctos Fucto to lear: f O le graet upate for the <> par f k f Graet upates are of the same form as the lear a logstc regresso moels CS 75 Mache Learg

13 Eample. Regresso th polomals. Eample: Regresso th polomals of egree m m f O le upate for <> par f m f CS 75 Mache Learg Multmesoal atve moel eample CS 75 Mache Learg 3

14 Multmesoal atve moel eample CS 75 Mache Learg Statstcal moel of regresso A geeratve moel: f f s a etermstc fucto s a raom ose represets thgs e caot capture th f e.g. ~ N Assume f s a lear moel a ~ N he: f E moels the mea of outputs for a the ose moels evatos from the mea he moel efes the cotoal est of gve p ep f CS 75 Mache Learg 4

15 5 CS 75 Mache Learg ML estmato of the parameters lkelhoo of prectos = the probablt of observg outputs D gve Mamum lkelhoo estmato of parameters parameters mamzg the lkelhoo of prectos Log-lkelhoo trck for the ML optmzato Mamzg the log-lkelhoo s equvalet to mamzg the lkelhoo p D L p * ma arg p D L D l log log CS 75 Mache Learg ML estmato of the parameters Usg cotoal est We ca rerte the log-lkelhoo as Mamzg th regar to s equvalet to mmzg square error fuctos p D L D l log log c f p log C f ] ep[ f p

16 ML estmato of parameters Crtera base o mea squares error fucto a the log lkelhoo of the output are relate ole log p c We ko ho to optmze parameters the same approach as use for the least squares ft But hat s the ML estmate of the varace of the ose? Mamze l D th respect to varace ˆ f = mea square precto error for the best prector * CS 75 Mache Learg Regularze lear regresso If the umber of parameters s large relatve to the umber of ata pots use to tra the moel e face the threat of overft geeralzato error of the moel goes up he precto accurac ca be ofte mprove b settg some coeffcets to zero Icreases the bas reuces the varace of estmates Solutos: Subset selecto Rge regresso Lasso regresso Prcpal compoet regresso Net: rge regresso 6

17 Rge regresso Error fucto for the staar least squares estmates:.. * We seek: arg m Rge regresso: Where.... a What oes the e error fucto o? Rge regresso Staar regresso: Rge regresso:.. L.. pealzes o-zero eghts th the cost L proportoal to a shrkage coeffcet If a put attrbute has a small effect o mprovg the error fucto t s shut o b the pealt term Icluso of a shrkage pealt s ofte referre to as regularzato 7

18 Regularze lear regresso Ho to solve the least squares problem f the error fucto s erche b the regularzato term? Aser: he soluto to the optmal set of eghts s obtae aga b solvg a set of lear equato. Staar lear regresso: Soluto: * X X X Regularze lear regresso: here X s a matr th ros correspog to eamples a colums to puts * I X X X Lasso regresso Staar regresso: Lasso regresso:.... pealzes o-zero eghts th the cost proportoal to Lasso regresso s more aggressve tha the rge regresso zerog the eghts Lasso + rge regularzato combe: Elastc et regularzato 8