Normal Random Variable and its discriminant functions

Transcription

1 Normal Random Varable and s dscrmnan funcons

2 Oulne Normal Random Varable Properes Dscrmnan funcons

3 Why Normal Random Varables? Analycally racable Works well when observaon comes form a corruped sngle prooype 3

4 The Unvarae Normal Densy s a scalar has dmenson p ep π σ σ, Where: mean or epeced value of σ varance 4

5 5

6 Several Feaures Wha f we have several feaures,,, d each normally dsrbued may have dfferen means may have dfferen varances may be dependen or ndependen of each oher How do we model her jon dsrbuon? 6

7 The Mulvarae Normal Densy Mulvarae normal densy n d dmensons s: p π σ L M O σ d L d / / deermnan of σ d M σ d covarance of and d Each s N,σ ep nverse of [,,, d ] [,,, d ] 7

8 More on σ M σ d L O L σ M σ d d plays role smlar o he role ha σ plays n one dmenson From we can fnd ou. The ndvdual varances of feaures,,, d. If feaures and j are ndependen σ j 0 have posve correlaon σ j >0 have negave correlaon σ j <0 8

9 The Mulvarae Normal Densy If s dagonal hen he feaures,, j are ndependen, and σ 0 0 0σ 0 0 0σ3 p d ep σ π σ 9

10 The Mulvarae Normal Densy p π d / / ep p c ep normalzng consan σ [ ] 3 3 σ σ3 σ σ σ σ3 σ 3 σ3 3 3 scalar s sngle number, he closer s o 0 he larger s p 3 Thus P s larger for smaller 0

11 s posve sem defne >0 If 0 for nonzero hen de0. Ths case s no neresng, p s no defned. one feaure vecor s a consan has zero varance. or wo componens are mulples of each oher so we wll assume s posve defne >0 If s posve defne hen so s

12 Egenvalues/egenvecors from Wk Gven a lnear ransformaon A, a non-zero vecor s defned o be an egenvecor of he ransformaon f sasfes he egenvalue equaon A λ for some scalar λ. where λ s called an egenvalue of A, correspondng o he egenvecor.

13 Egenvalues/egenvecors from Wk Geomercally, means ha under he ransformaon A, egenvecors only change n magnude and sgn he drecon of A s he same as ha of. The egenvalue λ s smply he amoun of "srech" or "shrnk" o whch a vecor s subjeced when ransformed by A. For eample, an egenvalue of means ha he egenvecor s doubled n lengh and pons n he same drecon. An egenvalue of means ha he egenvecor s unchanged, whle an egenvalue of means ha he egenvecor s reversed n sense. 3

14 Egenvalues/egenvecors from Wk In hs shear mappng he red arrow changes drecon bu he blue arrow does no. Therefore he blue arrow s an egenvecor, wh egenvalue as s lengh s unchanged. 4

15 Posve defne mar of sze d by d has d dsnc real egenvalues and s d egenvecors are orhogonal Thus f Φ s a mar whose columns are normalzed egenvecors of, hen Φ - Φ Φ ΦΛ where Λ s a dagonal mar wh correspondng egenvalues on he dagonal Thus ΦΛΦ and ΦΛ Φ Thus f Λ / denoes mar s.. Λ / Λ / Λ ΦΛ ΦΛ ΦΛ ΦΛ ΜΜ

16 Thus Thus MM M M M M Thus where M M Φ Λ Pons whch sasfy le on an ellpse cons M roaon mar scalng mar

17 usual Eucledan dsance beween and Mahalanobs dsance beween and egenvecors of pons a equal Eucledan dsance from le on a crcle pons a equal Mahalanobs dsance from le on an ellpse: sreches crcles o ellpses

18 -d Mulvarae Normal Densy Level curves graph p s consan along each conour opologcal map of 3-d surface and are ndependen σ and σ are equal 8

19 -d Mulvarae Normal Densy 0 0 [ 0,0] 0 0 [,] [ 0,0] [ 0,0] 9

20 -d Mulvarae Normal Densy 0.5 [ ] 0,

21 The Mulvarae Normal Densy If X has densy N, hen AX has densy NΑ,Α Α Thus X can be ransformed no a sphercal normal varable covarance of sphercal densy s he deny mar I wh whenng ransform X AX A w ΦΛ whose rows are egenvecors of dagonal mar wh egenvalues of 0 0

22 Dscrmnan Funcons Classfer can be vewed as nework whch compues m dscrmnan funcons and selecs caegory correspondng o he larges dscrmnan selec class gvng mamm dscrmnan funcons g g g m feaures 3 d g can be replaced wh any monooncally ncreasng funcon of g, he resuls wll be unchanged

23 Dscrmnan Funcons The mnmum error-rae classfcaon s acheved by he dscrmnan funcon g Pc P c Pc /P Snce he observaon s ndependen of he class, he equvalen dscrmnan funcon s g P c Pc For normal densy, convnen o ake logarhms. Snce logarhm s a monooncally ncreasng funcon, he equvalen dscrmnan funcon s g ln P c ln Pc 3

24 Dscrmnan Funcons for he Normal Densy ep / / d c p π Dscrmnan funcon g ln P c ln Pc Suppose for class c s class condonal densy p c s N, ln ln ln c P d g π Plug n p c and Pc ge ln ln c P g consan for all

25 Tha s Case σ I σ σ σ σ In hs case, feaures,.,, d are ndependen wh dfferen means and equal varances σ 5

26 Case σ I Dscrmnan funcon g ln lnp c De σ d and - /σ I Can smplfy dscrmnan funcon 0 σ 0 σ 0 0 I g lnσ σ 0 0 σ d lnp c consan for all g ln P c σ lnp c σ 6

27 Case σ I Geomerc Inerpreaon If ln Pc lnpc g j, hen g If ln Pc ln Pc σ ln Pc j, hen decson regon decson regon for c for c 3 decson regon for c 3 vorono dagram: pons n each cell are closer o he mean n ha cell han o any oher mean decson regon for c n c 3 decson regon for c 3 decson regon for c 3

28 Case σ I ln c P g σ ln Pc σ consan for all classes ln Pc g σ ln P c σ σ 8 σ 0 w w g σ σ dscrmnan funcon s lnear

29 Case σ I consan n g w w 0 w lnear n : d Thus dscrmnan funcon s lnear, Therefore he decson boundares g g j are lnear lnes f has dmenson planes f has dmenson 3 hyper-planes f has dmenson larger han 3 w

30 Case σ I: Eample 3 classes, each -dmensonal Gaussan wh Prors c P c and 4 c P Dscrmnan funcon s g 3 P 3 Plug n parameers for each class [ ] [ ] g g 3 [ ] ln Pc σ σ g

31 Case σ I: Eample Need o fnd ou when g < g j for,j,,3 Can be done by solvng g g j for,j,,3 Le s ake g g frs [ ] [ ] Smplfyng, [ ] lne equaon

32 Case σ I: Eample Ne solve g g Almos fnally solve g g And fnally solve g g g 3.4 and 4.8 3

33 Case σ I: Eample Prors c P c and P 4 c P 3 c 3 c lnes connecng means are perpendcular o decson boundares c 33

34 Case Covarance marces are equal bu arbrary In hs case, feaures,.,, d are no necessarly ndependen

35 Case Dscrmnan funcon g ln lnp c Dscrmnan funcon becomes consan for all classes g lnpc squared Mahalanobs Dsance Mahalanobs Dsance y y y If I, Mahalanobs Dsance becomes usual Eucledan dsance y y I y

36 Eucledan vs. Mahalanobs Dsances egenvecors of pons a equal Eucledan dsance from le on a crcle pons a equal Mahalanobs dsance from le on an ellpse: sreches crles o ellpses

37 Case Geomerc Inerpreaon If ln Pc g lnpc decson regon j, hen g If ln Pc ln Pc ln Pc, decson regon for c for c decson regon for c 3 j hen decson regon for c 3 decson regon for c 3 pons n each cell are closer o he mean n ha cell han o any oher mean under Mahalanobs dsance decson regon for c 3

38 Case Can smplfy dscrmnan funcon: ln c P g ln c P ln c P consan for all classes ln c P Thus n hs case dscrmnan s also lnear 0 w w lnpc

39 Case : Eample 3 classes, each -dmensonal Gaussan wh c P c c P P Agan can be done by solvng g g j for,j,,3

40 Case : Eample j j j j ln Pc ln Pc Le s solve n general frs g g j Le s regroup he erms scalar row vecor j j j j lnpc lnpc j j j j Pc Pc ln Le s regroup he erms We ge he lne where g g j

41 Case : Eample ln j Now subsue for,j, [ 0] 0 Now subsue for,j,3 Pc Pc j 0 [ 3.4.4] Now subsue for,j,3 [ ] j.4.4 j

42 Case : Eample c Prors c P c and P 4 P 3 c c 3 c lnes connecng means are no n general perpendcular o decson boundares 4

43 General Case are arbrary Covarance marces for each class are arbrary In hs case, feaures,.,, d are no necessarly ndependen j

44 From prevous dscusson, General Case are arbrary Ths can be smplfed, bu we can rearrange : ln ln c P g ln ln c P g 44 ln ln c P g ln ln c P g 0 w w W g

45 General Case are arbrary lnear n g W w w 0 consan n quadrac n snce W d d w j j d j, j w j j Thus he dscrmnan funcon s quadrac Therefore he decson boundares are quadrac ellpses and parabollods 45

46 3 classes, each -dmensonal Gaussan wh General Case are arbrary: Eample P c 3 4 c P c Prors: and P Agan can be done by solvng g g j for,j,,3 g ln lnp c Need o solve a bunch of quadrac nequales of varables

47 General Case are arbrary: Eample c P c c P 4 P c c c 3 c

48 Imporan Pons The Bayes classfer when classes are normally dsrbued s n general quadrac If covarance marces are equal and proporonal o deny mar, he Bayes classfer s lnear If, n addon he prors on classes are equal, he Bayes classfer s he mnmum Eucledan dsance classfer If covarance marces are equal, he Bayes classfer s lnear If, n addon he prors on classes are equal, he Bayes classfer s he mnmum Mahalanobs dsance classfer Popular classfers Eucldean and Mahalanobs dsance are opmal only f dsrbuon of daa s approprae normal