2/20/2013. of Manchester. The University COMP Building a yes / no classifier

Transcription

1 COMP4 Lecture 6 Building a yes / no classifier

2 Buildinga feature-basedclassifier Whatis a classifier? What is an information feature? Building a classifier from one feature Probability densities and the normal distribution Training the classifier Building a classifier from multile features Data oints in multile dimensions The naive Bayes classifier

3 What is a classifier? A classifier is an agent the redicts the class of some data Yes or No? 3 3

4 Probabilistic classification It is unusual for a classifier to work erfectly Main objective is to make as few errors as ossible Also very useful to return the belief in the classification Probabilities rovide a consistent reresentation of belief Very useful in seech recognition: allows romting and integration of other knowledge, e.g. From a language model 4 4

5 What is a useful feature for a classifier? Last time we discussed how to extractfeatures from seech signals Each segment is associated with a vector of 3 MFCCs 5 5

6 What is a useful feature for a classifier? For short words we could average the features over time Time averaged MFCC for 65 samles is on the left Sequence lengths on the right much less useful. 6 6

7 Building a classifier from one feature Bayes theorem ( x) C = ( x C) ( C) ( ) ( ) + ( ) ( ) x C C x C C C means the class is C x means the feature is observed to be x (C ) the rior belief in C (x C ) the robability of the feature taking the value x if the class is C The classifier s outut is (C x) the robability of the class being C if the feature has value x A simle classification rule: choose class C if (C x) >

8 Modelling the distribution of features To work out (C x) we need to know (x C ) and (x C ) We can use normal distributions fittedto a training set of data 8 8

9 Probability density functions To assign robability to a continuous valued variable xwe use a robability distribution function (x) (x)dxis the robability of xlying in the interval [x, x + dx] Sums are relaced by integrals so the mean µand variance σ are µ = ( x) xdx σ ( x)( x µ ) The normalisation condition is = dx ( x) dx = 9 9

10 The normal distribution Has two arameters: mean µand variance σ ( x) = ( µ ) x ex π σ σ 0 0

11 Fitting a normal to data We can fit a normal distribution to data {x (), x (),..., x (N) } Set µand σ equal to the emirical mean m and variance s m = N N N ( n) ( n) n= Note that x (n) is the n th item, not x n x s = N ( x m) n= The constructor method for the Normalclass in Lab can do this

12 Building the classifier After fitting (x C ) and (x C ) to normal distributions, we work out: ( x) C = ( x C) ( C) ( ) ( ) + ( ) ( ) x C C x C C Imlemented in the Classifier class in Lab

13 Building a classifier from multile features 3 3

14 Building a classifier from multile features If we choose dfeatures, data can be reresented by a vector x= [x, x,..., x d ] A vector reresents a oint in d-dimensional sace So they re often referred to as data oints The robability of a data oint is written (x) = (x Λx Λ... Λx d ) = (x, x,..., x d ) 4 4

15 Building a classifier from multile features The classification rule is as before ( x) C = ( x C) ( C) ( ) ( ) + ( ) ( ) + x C C x C C But what are (x C ) and (x C )? 5 5

16 Naïve Bayes classifier The simlification used by a naive Bayes classifier is that features are conditionally indeendent given the class: d ( x C ) = ( x C ) ( x C ) L ( x C ) = ( x i C ) d ( x C ) = ( x C ) ( x C ) L ( x C ) = ( x i C ) Is this true? Can again aroximate each (x i C j )by a normal distribution fitted to training data i= i= 6 6

17 The multivariate normal distribution 7 /0/03 7

18 Building the classifier We fit the normal distributions to training data and build the classifier 8 8

19 Evaluating a classifier A simle way to evaluate a classifier is to count the number or errors it makes, i.e. where (true class x) < 0.5 Training error rate Percentage of errors on the training data We can use this to decide on the best classification method This will be done in Lab to comare two different aroaches Test error rate Percentage of errors on the test data It is imortant to evaluate erformance on unseen data This can be reorted as an unbiased estimate of erformance 9 9

20 Summary Data can be reresented by one or more features x= [x, x,..., x d ] A robabilistic classifier works out (class x) A simle classification rule is to choose the class with the highest osterior robability. For two classes, (class x) > 0.5 (x i class) can be aroximated by a normal distribution for continuous valued features We can fit the normal distribution to training data The naive Bayes assumtion rovides a simle aroximation for multivariate data We can evaluate a classifier by calculating the training error and test error rates 0 0

21 What now? The naive Bayesclassifier is also oular for discrete-valuedfeatures, e.g. number of times a articular word aearsin an is a feature used by sam classifiers In examles class 6 you will build a naive Bayesclassifier for a simle examle of a roblem with discrete valued (binary) features