LECTURE 5: Quadratic classifiers

Transcription

1 LECURE 5: Quadratc classfers Bayes classfers for Normally dstrbuted classes Case : σ I Case : ( daoal) Case : ( o-daoal) Case : σ I Case 5: j eeral case Numercal example Lear ad quadratc classfers: coclusos Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty

2 Bayes classfers for Normally dstrbuted classes O Lecture we showed that the decso rule (MAP) that mmzed the probablty of error could be formulated terms of a famly of dscrmat fuctos choose f > j where P( x) j As we wll show, for classes that are ormally dstrbuted, ths famly of dscrmat fuctos ca be reduced to very smple expressos Geeral expresso for Gaussa destes he multvarate Normal desty fucto was defed as fx / ( π) / exp (x ) Us Bayes rule, the MAP dscrmat fucto becomes (x ) P(x )P( ) P( x) P / ( π) Elmat costat terms Dscrmat fuctos Features -/ exp (x ) (x ) P( ) We take atural los sce the loarthm s a mootocally creas fucto (x ) (x ) - lo / exp ( ) lo( P( )) (x ) x Class assmet Select max x x (x ) P( ) P C x d Costs hs expresso s called a quadratc dscrmat fucto Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty

3 Case : σ I hs stuato occurs whe the features are statstcally depedet wth the same varace for all classes* I ths case, the quadratc dscrmat fucto becomes Expad ths expresso - ( σ I) (x )- lo( σ I ) lo( P( )) (x ) (x )- Nlo( σ ) lo( P( )) (x ) (x ) (x ) lo( P( ) ) σ x - x - lo P( ) x x σ σ Elmat the term x x, whch s costat for all classes ( ) ( ) ( ) ( x ) lo( P( )) σ secod dropp term the x lo P( ) w x w σ w where σ w lo( P( )) σ Sce the dscrmat s lear, the decso boudares j, wll be hyper-plaes If we assume equal prors σ ( ) ( ) (x ) (x ) x σ C Dstace Dstace Dstace Mmum Selector class hs s called a mmum-dstace or earest mea classfer he loc of costat probablty for each class are hyper-spheres For ut varace (σ ), the dstace becomes the Eucldea dstace From [Schalkoff, 99] Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty

4 Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty Case : σ I, example o llustrate the prevous result, we wll compute the decso boudares for a - class, -dmesoal problem wth the follow class mea vectors ad covarace matrces ad equal prors [ ] [ ] [ ] 5 7

5 Case : ( daoal) he classes stll have the same covarace matrx, but the features are allowed to have dfferet varaces I ths case, the quadratc dscrmat fucto becomes k k ( ) lo( P( )) (x ) (x ) - lo σ σ (x ) (x ) - lo lo( P( )) O O σ σ N N N N (x[k] [k]) lo σk lo( P( ) ) k σk k N N x[k] x[k] [k] [k] lo σk lo( P( ) ) σ k Elmat the term x[k], whch s costat for all classes N N x[k] [k] [k] lo σk lo P( ) σ k k k ( ) hs dscrmat s lear, so the decso boudares j, wll also be hyper-plaes he loc of costat probablty are hyper-ellpses aled wth the feature axes Note that the oly dfferece wth the prevous classfer s that the dstace of each axs s ormalzed by the varace of the axs Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 5

6 Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 6 o llustrate the prevous result, we wll compute the decso boudares for a -class, -dmesoal problem wth the follow class mea vectors ad covarace matrces ad equal prors [ ] [ ] [ ] 5 5 Case : ( daoal), example

7 Case : ( o-daoal) I ths case, all the classes have the same covarace matrx, but ths s o loer daoal he quadratc dscrmat becomes (x ) (x ) (x ) - lo (x ) - lo ( ) lo( P( )) ( ) lo( P( )) Elmat the term lo, whch s costat for all classes (x ) (x ) lo P( ) ( ) he quadratc term s called the Mahalaobs dstace, a very mportat dstace Statstcal PR Mahalaobs Dstace x - y (x y) (x y) x he Mahalaobs dstace s a vector dstace that uses a - orm - ca be thouht of as a stretch factor o the space Note that for a detty covarace matrx (I), the Mahalaobs dstace becomes the famlar Eucldea dstace x x - K x - Κ Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 7

8 Case : ( o-daoal) Expaso of the quadratc term the dscrmat yelds (x ) (x ) lo P( ) ( ) ( x x x ) lo( P( )) Remov the term x - x, whch s costat for all classes ( x ) lo( P( )) Reoraz terms we obta w where x w w w lop( ) x C Dstace Dstace Dstace Mmum Selector class hs dscrmat s lear, so the decso boudares wll also be hyper-plaes he costat probablty loc are hyper-ellpses aled wth the eevectors of If we ca assume equal prors (x ) (x ) he classfer becomes a mmum (Mahalaobs) dstace classfer Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 8

9 Case : ( o-daoal), example o llustrate the prevous result, we wll compute the decso boudares for a - class, -dmesoal problem wth the follow class mea vectors ad covarace matrces ad equal prors [ ] [ 5 ] [ 5] Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 9

10 Case : σ I I ths case, each class has a dfferet covarace matrx, whch s proportoal to the detty matrx he quadratc dscrmat becomes (x ) (x ) σ (x )- lo (x )- Nlo σ ( ) lo( P( )) ( ) lo( P( )) hs expresso caot be reduced further so he decso boudares are quadratc: hyper-ellpses he loc of costat probablty are hyper-spheres aled wth the feature axs Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty

11 Case : σ I, example o llustrate the prevous result, we wll compute the decso boudares for a - class, -dmesoal problem wth the follow class mea vectors ad covarace matrces ad equal prors [ ] [ 5 ] [ 5].5.5 Zoom out Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty

12 Case 5: j eeral case We already derved the expresso for the eeral case at the be of ths dscusso (x ) (x ) - lo ( ) lo( P( )) Reoraz terms a quadratc form yelds x W x w where x w W w w - lo ( ) lo( P( )) he loc of costat probablty for each class are hyper-ellpses, oreted wth the eevectors of for that class he decso boudares are aa quadratc: hyper-ellpses or hyper-parabollods Notce that the quadratc expresso the dscrmat s proportoal to the Mahalaobs dstace us the class-codtoal covarace Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty

13 Case 5: j eeral case, example o llustrate the prevous result, we wll compute the decso boudares for a - class, -dmesoal problem wth the follow class mea vectors ad covarace matrces ad equal prors [ ] [ 5 ] [ 5] Zoom out Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty

14 Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty Numercal example Derve a lear dscrmat fucto for the two-class D classfcato problem defed by the follow Gaussa Lkelhoods Soluto Classfy the test example x u [..7.8] lo z - y - x - z - y - x - lo z - y - x - z - y - x - ; [ ] [ ] ( ) ( ) p p ; / / / ; ; ( ) ( ) ) lop( z - - y - x - z - y - x - lop( ) x - x - σ z y x z y x ( ) ( ) ( ) ( ) ( ) lo z y x - lo z y - x < > < >. lo 6 z y x < > u x < >

15 Coclusos From the prevous examples we ca extract the follow coclusos he Bayes classfer for ormally dstrbuted classes (eeral case) s a quadratc classfer he Bayes classfer for ormally dstrbuted classes wth equal covarace matrces s a lear classfer he mmum Mahalaobs dstace classfer s Bayes-optmal for ormally dstrbuted classes ad equal covarace matrces ad equal prors he mmum Eucldea dstace classfer s Bayes-optmal for ormally dstrbuted classes ad equal covarace matrces proportoal to the detty matrx ad equal prors Both Eucldea ad Mahalaobs dstace classfers are lear classfers he oal of ths dscusso was to show that some of the most popular classfers ca be derved from decso-theoretc prcples ad some smplfy assumptos It s mportat to realze that us a specfc (Eucldea or Mahalaobs) mmum dstace classfer mplctly correspods to certa statstcal assumptos he questo whether these assumptos hold or do t ca rarely be aswered practce; most cases we are lmted to pos ad aswer the questo does ths classfer solve our problem or ot? Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 5