Normal Random Varable and s dscrmnan funcons
Oulne Normal Random Varable Properes Dscrmnan funcons
Why Normal Random Varables? Analycally racable Works well when observaon comes form a corruped sngle prooype 3
The Unvarae Normal Densy s a scalar has dmenson p ep π σ σ, Where: mean or epeced value of σ varance 4
5
Several Feaures Wha f we have several feaures,,, d each normally dsrbued may have dfferen means may have dfferen varances may be dependen or ndependen of each oher How do we model her jon dsrbuon? 6
The Mulvarae Normal Densy Mulvarae normal densy n d dmensons s: p π σ L M O σ d L d / / deermnan of σ d M σ d covarance of and d Each s N,σ ep nverse of [,,, d ] [,,, d ] 7
More on σ M σ d L O L σ M σ d d plays role smlar o he role ha σ plays n one dmenson From we can fnd ou. The ndvdual varances of feaures,,, d. If feaures and j are ndependen σ j 0 have posve correlaon σ j >0 have negave correlaon σ j <0 8
The Mulvarae Normal Densy If s dagonal hen he feaures,, j are ndependen, and σ 0 0 0σ 0 0 0σ3 p d ep σ π σ 9
The Mulvarae Normal Densy p π d / / ep p c ep normalzng consan σ [ ] 3 3 σ σ3 σ σ σ σ3 σ 3 σ3 3 3 scalar s sngle number, he closer s o 0 he larger s p 3 Thus P s larger for smaller 0
s posve sem defne >0 If 0 for nonzero hen de0. Ths case s no neresng, p s no defned. one feaure vecor s a consan has zero varance. or wo componens are mulples of each oher so we wll assume s posve defne >0 If s posve defne hen so s
Egenvalues/egenvecors from Wk Gven a lnear ransformaon A, a non-zero vecor s defned o be an egenvecor of he ransformaon f sasfes he egenvalue equaon A λ for some scalar λ. where λ s called an egenvalue of A, correspondng o he egenvecor.
Egenvalues/egenvecors from Wk Geomercally, means ha under he ransformaon A, egenvecors only change n magnude and sgn he drecon of A s he same as ha of. The egenvalue λ s smply he amoun of "srech" or "shrnk" o whch a vecor s subjeced when ransformed by A. For eample, an egenvalue of means ha he egenvecor s doubled n lengh and pons n he same drecon. An egenvalue of means ha he egenvecor s unchanged, whle an egenvalue of means ha he egenvecor s reversed n sense. 3
Egenvalues/egenvecors from Wk In hs shear mappng he red arrow changes drecon bu he blue arrow does no. Therefore he blue arrow s an egenvecor, wh egenvalue as s lengh s unchanged. 4
Posve defne mar of sze d by d has d dsnc real egenvalues and s d egenvecors are orhogonal Thus f Φ s a mar whose columns are normalzed egenvecors of, hen Φ - Φ Φ ΦΛ where Λ s a dagonal mar wh correspondng egenvalues on he dagonal Thus ΦΛΦ and ΦΛ Φ Thus f Λ / denoes mar s.. Λ / Λ / Λ ΦΛ ΦΛ ΦΛ ΦΛ ΜΜ
Thus Thus MM M M M M Thus where M M Φ Λ Pons whch sasfy le on an ellpse cons M roaon mar scalng mar
usual Eucledan dsance beween and Mahalanobs dsance beween and egenvecors of pons a equal Eucledan dsance from le on a crcle pons a equal Mahalanobs dsance from le on an ellpse: sreches crcles o ellpses
-d Mulvarae Normal Densy Level curves graph p s consan along each conour opologcal map of 3-d surface and are ndependen σ and σ are equal 8
-d Mulvarae Normal Densy 0 0 [ 0,0] 0 0 [,] 0 0 4 [ 0,0] 0 4 0 [ 0,0] 9
-d Mulvarae Normal Densy 0.5 [ ] 0,0 0.9 0.9 4 0.9 0.9 0.5 0.5 0.9 4 0.9 0.5 0.5 0.9 0.9 4 0.9 0.9
The Mulvarae Normal Densy If X has densy N, hen AX has densy NΑ,Α Α Thus X can be ransformed no a sphercal normal varable covarance of sphercal densy s he deny mar I wh whenng ransform X AX A w ΦΛ 0.9 0.9 whose rows are egenvecors of dagonal mar wh egenvalues of 0 0
Dscrmnan Funcons Classfer can be vewed as nework whch compues m dscrmnan funcons and selecs caegory correspondng o he larges dscrmnan selec class gvng mamm dscrmnan funcons g g g m feaures 3 d g can be replaced wh any monooncally ncreasng funcon of g, he resuls wll be unchanged
Dscrmnan Funcons The mnmum error-rae classfcaon s acheved by he dscrmnan funcon g Pc P c Pc /P Snce he observaon s ndependen of he class, he equvalen dscrmnan funcon s g P c Pc For normal densy, convnen o ake logarhms. Snce logarhm s a monooncally ncreasng funcon, he equvalen dscrmnan funcon s g ln P c ln Pc 3
Dscrmnan Funcons for he Normal Densy ep / / d c p π Dscrmnan funcon g ln P c ln Pc Suppose for class c s class condonal densy p c s N, ln ln ln c P d g π Plug n p c and Pc ge ln ln c P g consan for all
Tha s Case σ I 0 0 0 0 0 0 0 0 0 0 0 0 σ σ σ σ In hs case, feaures,.,, d are ndependen wh dfferen means and equal varances σ 5
Case σ I Dscrmnan funcon g ln lnp c De σ d and - /σ I Can smplfy dscrmnan funcon 0 σ 0 σ 0 0 I g lnσ σ 0 0 σ d lnp c consan for all g ln P c σ lnp c σ 6
Case σ I Geomerc Inerpreaon If ln Pc lnpc g j, hen g If ln Pc ln Pc σ ln Pc j, hen decson regon decson regon for c for c 3 decson regon for c 3 vorono dagram: pons n each cell are closer o he mean n ha cell han o any oher mean decson regon for c n c 3 decson regon for c 3 decson regon for c 3
Case σ I ln c P g σ ln Pc σ consan for all classes ln Pc g σ ln P c σ σ 8 σ 0 w w g σ σ dscrmnan funcon s lnear
Case σ I consan n g w w 0 w lnear n : d Thus dscrmnan funcon s lnear, Therefore he decson boundares g g j are lnear lnes f has dmenson planes f has dmenson 3 hyper-planes f has dmenson larger han 3 w
Case σ I: Eample 3 classes, each -dmensonal Gaussan wh 4 6 3 4 Prors c P c and 4 c P Dscrmnan funcon s g 3 P 3 Plug n parameers for each class [ ] [ ] g 3 5.38 6 g 3 [ ] 4 3 0 3 0 3 ln Pc σ σ g 4 6 3 0 0.69 6 5.38 6 30
Case σ I: Eample Need o fnd ou when g < g j for,j,,3 Can be done by solvng g g j for,j,,3 Le s ake g g frs [ ] [ ] 3 5.38 6 3 4 3 Smplfyng, [ ] 4 6 3 5.38 6 47 6 4 3 lne equaon 47 6 3
Case σ I: Eample Ne solve g g 3 3 6.0 Almos fnally solve g g 3 3.8 And fnally solve g g g 3.4 and 4.8 3
Case σ I: Eample Prors c P c and P 4 c P 3 c 3 c lnes connecng means are perpendcular o decson boundares c 33
Case Covarance marces are equal bu arbrary In hs case, feaures,.,, d are no necessarly ndependen 0.5 0.5 34
Case Dscrmnan funcon g ln lnp c Dscrmnan funcon becomes consan for all classes g lnpc squared Mahalanobs Dsance Mahalanobs Dsance y y y If I, Mahalanobs Dsance becomes usual Eucledan dsance y y I y
Eucledan vs. Mahalanobs Dsances egenvecors of pons a equal Eucledan dsance from le on a crcle pons a equal Mahalanobs dsance from le on an ellpse: sreches crles o ellpses
Case Geomerc Inerpreaon If ln Pc g lnpc decson regon j, hen g If ln Pc ln Pc ln Pc, decson regon for c for c decson regon for c 3 j hen decson regon for c 3 decson regon for c 3 pons n each cell are closer o he mean n ha cell han o any oher mean under Mahalanobs dsance decson regon for c 3
Case Can smplfy dscrmnan funcon: ln c P g ln c P ln c P consan for all classes ln c P Thus n hs case dscrmnan s also lnear 0 w w lnpc
Case : Eample 3 classes, each -dmensonal Gaussan wh 5 3 4 4 3 c P c c P P 3.5.5 4 Agan can be done by solvng g g j for,j,,3
Case : Eample j j j j ln Pc ln Pc Le s solve n general frs g g j Le s regroup he erms scalar row vecor j j j j lnpc lnpc j j j j Pc Pc ln Le s regroup he erms We ge he lne where g g j
Case : Eample ln j Now subsue for,j, [ 0] 0 Now subsue for,j,3 Pc Pc j 0 [ 3.4.4]. 4 3.4.4 Now subsue for,j,3 [ 5.4.43]. 4 5.4.43 j.4.4 j
Case : Eample c Prors c P c and P 4 P 3 c c 3 c lnes connecng means are no n general perpendcular o decson boundares 4
General Case are arbrary Covarance marces for each class are arbrary In hs case, feaures,.,, d are no necessarly ndependen 0.5 0.5 j 0.9 0.9 4 43
From prevous dscusson, General Case are arbrary Ths can be smplfed, bu we can rearrange : ln ln c P g ln ln c P g 44 ln ln c P g ln ln c P g 0 w w W g
General Case are arbrary lnear n g W w w 0 consan n quadrac n snce W d d w j j d j, j w j j Thus he dscrmnan funcon s quadrac Therefore he decson boundares are quadrac ellpses and parabollods 45
3 classes, each -dmensonal Gaussan wh General Case are arbrary: Eample 3 0 6 3 4 0.5 0.5 7 3.5.5 3 P c 3 4 c P c Prors: and P Agan can be done by solvng g g j for,j,,3 g ln lnp c Need o solve a bunch of quadrac nequales of varables
General Case are arbrary: Eample 3 0 6 3 4 c P c c P 4 P 3 0.5 0.5 7 3.5.5 3 c c c 3 c
Imporan Pons The Bayes classfer when classes are normally dsrbued s n general quadrac If covarance marces are equal and proporonal o deny mar, he Bayes classfer s lnear If, n addon he prors on classes are equal, he Bayes classfer s he mnmum Eucledan dsance classfer If covarance marces are equal, he Bayes classfer s lnear If, n addon he prors on classes are equal, he Bayes classfer s he mnmum Mahalanobs dsance classfer Popular classfers Eucldean and Mahalanobs dsance are opmal only f dsrbuon of daa s approprae normal