Density estimation II.

Transcription

1 Lecture 5 esty estmato II. Mlos Hausrecht mlos@cs.tt.eu 539 Seott Square Outle Outle: esty estmato: Bomal strbuto Multomal strbuto ormal strbuto oetal famly

2 ata: esty estmato {.. } a vector of attrbute values Objectve: try to estmate the uerlyg true robablty strbuto over varables X X usg eamles true strbuto samles X.. } { estmate X Staar assumtos: Samles are eeet of each other come from the same etcal strbuto fe X Beroull trals ata: a sequece of outcomes such that hea tal Moel: robablty of a hea robablty of a tal robablty of a outcome of a co fl Beroull strbuto ML Soluto: ML - umber of heas a tals resectvely

3 osteror strbuto osteror esty va Bayes rule - s the ror robablty o ror Lelhoo of ata ormalzg factor Cojugate choce of ror: strbuto b a b a b a b a

4 osteror strbuto * Bayesa framewor he ML estmate cs oe value of the arameter Assume: there are two fferet arameter settgs that are close terms of ther robablty values. Usg oly oe of them may trouce a strog bas f we use them for eamle for rectos. Bayesa arameter estmate Remees the lmtato of oe choce ees all ossble arameter values Where he osteror ca be use to efe A : A A

5 Bayesa framewor amle: A robablty of outcome the et tral quvalet to the eecte value of the arameter eectato s tae wth resect to the osteror strbuto osteror esty ecte value of the arameter How to obta the eecte value? ote:

6 ecte value of the arameter Substtutg the results for the osteror: We get ote that the mea of the osteror s yet aother reasoable arameter choce: Mamum a osteror robablty Mamum a osteror estmate Selects the moe of the osteror strbuto otce that arameters of the ror act le couts of heas a tals sometmes they are also referre to as ror couts MA MA Soluto: ma arg MA

7 Bomal strbuto amle: a base co Outcomes: two ossble values -- hea or tal ata: a set of orer-eeet outcomes We treat as a mult-set - umber of heas see Moel: robablty of a hea robablty of a tal - umber of tals see robablty of a outcome Bomal strbuto Mamum lelhoo ML estmate. Lelhoo of ata: Log-lelhoo log l log log log Costat from the ot of otmzato ML Soluto: ML he same as for Beroull a wth sequece of eamles

8 osteror esty osteror esty ror choce Lelhoo osteror MA estmate ma arg MA va Bayes rule MA ecte value of the arameter he result s the same as for Beroull strbuto ecte value of the arameter rectve robablty of evet

9 Multomal strbuto amle: Mult-way co toss or a roll of a ce ata: a set of trals treate as a mult-set Moel arameters: robablty of ata lelhoo ML estmate: ML s.t. - a umber of tmes a outcome has bee see - robablty of a outcome Multomal strbuto osteror esty a MA estmate Choce of the ror: rchlet strbuto.... r r MA.. MA estmate: osteror esty.. r rchlet s the cojugate choce for the multomal

10 ecte value he result s aalogous to the result for the bomal Reresets the rectve robablty of a evet r ectato base arameter estmate Other strbutos he same eas ca be ale to other strbutos ycally we choose strbutos that behave well so that comutatos lea to ce solutos oetal famly of strbutos Cojugate choces for some of the strbutos from the eoetal famly: Bomal Multomal - rchlet oetal Gamma osso Iverse Gamma Gaussa - Gaussa mea a Wshart covarace

11 Other strbutos Gamma strbuto: a b a b oetal strbuto: A secal case of Gamma for a b e b osso strbuto: b a a e b for [ ] λ e λ λ for { } Gaussa ormal strbuto Gaussa: ~ σ arameters: - mea σ - staar evato esty fucto: σ e[ σ π σ amle: ]

12 arameter estmates Loglelhoo l σ log σ ML estmates of the mea a varace: σ ML varace estmate s base σ σ σ Ubase estmate: σ Multvarate ormal strbuto Multvarate ormal: ~ arameters: - mea - covarace matr esty fucto: e / / π amle:

13 arameter estmates Loglelhoo ML estmates of the mea a covaraces: Covarace estmate s base Ubase estmate: log l osteror of a multvarate ormal Assume a ror o the mea that s ormally strbute: he the osteror of s ormally strbute / / e π e * / / π e / / π

14 osteror of a multvarate ormal he the osteror of s ormally strbute e / / π Sequetal Bayesa arameter estmato Sequetal Bayesa aroach Uer the the estmates of the osteror ca be comute cremetally for a sequece of ata ots If we use a cojugate ror we get bac the same osteror Assume we slt the ata the last elemet a the rest he: A ew ror