A Bayesian Classifier for Uncertain Data

Transcription

1 A Bayesan Classfer for Uncertan Data Bao Qn, Yun Xa Department of Computer Scence Indana Unversty - Purdue Unversty Indanapols, USA {baoqn, yxa}@cs.upu.edu Fang L Department of Mathematcal Scences Indana Unversty - Purdue Unversty Indanapols, USA fl@math.upu.edu ABSTRACT Data uncertanty s wdespread n a varety of applcatons. Ths paper proposes a new Bayesan classfcaton algorthm for classfyng uncertan data. In the paper, we apply probablty and statstcs theory on uncertan data model, and provde solutons for model parameter estmaton for both uncertan numercal data and uncertan categorcal data. We also prove the correctness of the solutons. The expermental results demonstrate the proposed uncertan Bayesan classfer can be effcently constructed, and t sgnfcantly outperforms the tradtonal Bayesan classfer n predcton accuracy when data s hghly uncertan. 1. INTRODUCTION In many applcatons, data contans nherent uncertanty. A number of factors contrbute to the uncertanty, such as the random nature of the physcal data generaton and collecton process, measurement and decson errors, unrelable data transmsson and data stalng. In ths paper, we focus on Bayesan classfcaton for uncertan data. Data classfcaton s one of the most mportant data mnng problem. Bayesan classfcaton algorthm s tremendously appealng because of ts smplcty, elegance, and robustness. It s one of the oldest formal classfcaton algorthms, and t s often surprsngly effectve. A large number of modfcatons have been ntroduced by the statstcal, data mnng, machne learnng, and pattern recognton communtes n an attempt to make t more flexble []. It s wdely used n varous areas ncludng text classfcaton and spam flterng. In the paper, we extend the Bayesan algorthm to classfy and predct uncertan data. We ntegrate the uncertan data model wth Bayesan classfer and extend the tradtonal Bayesan classfcaton algorthm so that t can process hghly uncertan data. For both uncertan numercal data and uncertan categorcal data, we present solutons for classfcaton model parameter estmaton. We also extend the tradtonal Bayesan predcton algorthm to predct data class based on uncertan attrbutes. We show through experments that the proposed NBU classfer can be effcently generated, and t has a dstnctly hgher predcton accuracy than the tradtonal Bayesan classfer on uncertan data. Classfcaton s a well-studed area n data mnng. Many classfcaton algorthms have been proposed n the lterature, such as decson tree classfers [9], rule-based classfers [3], Bayesan classfers, support vector machnes (SVM), artfcal neural networks and ensemble methods. In spte of numerous classfcaton algorthms, buldng classfcaton based on uncertan data remans a great challenge. There s early work performed on developng decson trees when data contans mssng or nosy values [8, 7, 5]. Varous strateges have been developed to predct or fll mssng attrbute values. However, the problem studed n ths paper s dfferent from before - nstead of assumng part of the data has mssng or nosy values, we allow the whole dataset to be uncertan, and the uncertanty s not shown as mssng or erroneous values, but represented as uncertan ntervals wth probablty dstrbuton functons [] or x-tuples [10].. THE UNCERTAIN DATA MODELS In ths secton, we wll dscuss the uncertan data models for both numercal and categorcal attrbute, whch are the most common types of data encountered n data mnng applcatons. When the value of a numercal attrbute s uncertan, the attrbute s called an uncertan numercal attrbute (UNA), denoted by. Further, we use j to denote the jth nstance of. The concept of UNA has been ntroduced n []. The value of s represented as a range or nterval and an optonal probablty dstrbuton functon (PDF) over ths range. Note that s treated as a contnuous random varable. The PDF f(x) can be related to an attrbute f all nstances have the same dstrbuton, or related to each nstance f each nstance has a dfferent dstrbuton. An uncertan nterval of, denoted by.u, s an n-.l,.r R and.r terval [.l,.r] where.l. An uncertan probablty dstrbuton functon (PDF) of j j.r j.l j j.f(x), s a PDF of Aun j, denoted by Aun j Aun.f(x)dx = 1 and j.r.u. [] j.l j, such that.f(x)dx = 0 f x The uncertan x-tuple model concept has been proposed n

2 database systems such as [10]. Each x-tuple T j ncludes a number of tems as ts alternatves whch are assocated wth probabltes, representng a dscrete probablty dstrbuton of these alternatves beng selected. Independence s assumed among the x-tuples. Gven a categorcal doman Dom = {v 1,..., v n}, an uncertan categorcal attrbute (UCA) A uc s characterzed by probablty dstrbuton over Dom. It can be represented by the probablty vector P = {p j1,..., p jn} such that P(A uc j = v k) = p jk and n k=1 p jk = 1 (1 k n). 3. BACKGROUND Our classfer s developed based on the nave Bayes classfer.we wll frst brefly explan the nave Bayes classfer. The probablty model for a classfer s a condtonal model p(c A 1,..., A n) over a dependent class varable C wth a small number of outcomes or classes, condtonal on several feature varables A 1 through A n. Usng Bayes theorem, p(c A 1,..., A n) = p(c) p(a 1,...,A n C) p(a 1,...,A n). In another word, the above equaton can be wrtten as posteror = pror lkelhood. evdence In practce we are only nterested n the numerator of that fracton, snce the denomnator does not depend on C and the values of the features A are gven, so that the denomnator s effectvely constant. The numerator s equvalent to the jont probablty model p(c, A 1,..., A n) whch can be rewrtten as follows, usng repeated applcatons of the defnton of condtonal probablty: p(c, A 1,..., A n) = p(c) p(a 1,..., A n C) = p(c) p(a 1 C) p(a,..., A n C, A 1) = p(c) p(a 1 C) p(a C, A 1) p(a 3,..., A n C, A 1, A ) and so forth. The nave Bayes classfer assumes that each feature A s condtonally ndependent of every other feature A j for j. Ths means that p(a C, A j) = p(a C). Therefore, the jont model can be expressed as p(c, A 1,..., A n) = p(c) p(a 1 C) p(a C) p(a 3 C) = p(c) n =1 p(a C). All model parameters (such as class prors and feature probablty dstrbutons) can be approxmated wth relatve frequences from the tranng set. For example, n order to estmate p(a C), we often assume p(a C) follows a Gaussan dstrbuton N(µ, σ ), and we can compute the mean(µ) and varance(σ) of p(a C) based on the tranng data. Therefore, for a new test nstance, t s easy to estmate p(a C) accordng to the probablty densty functon. Data uncertanty brng unque challenges to the parameter estmaton. Tradtonal maxmum lkelhood based parameter estmaton needs to be extended to handle data uncertanty. In the next secton, we wll present our approaches for parameter estmaton for both uncertan numercal data and uncertan categorcal data.. PARAMETER ESTIMATION.1 Uncertan numercal Attrbutes As descrbed earler, the value of an uncertan numercal attrbute s an nterval wth an optonal assocated PDF. Each uncertan value has a maxmal value and a mnmal value. For each uncertan numercal data, the observed value s n the form of [L, R ]. Suppose the true value s X, then L X R, as shown n Fgure 1. Fgure 1: An example uncertan data nterval We denote the left error ( negatve error ) wth ǫ L, the rght error(postve error) wth ǫ R, and the overall error wth ǫ. We assume the left error s a postve dstrbuton wth mean µ and varance σ L, that s, ǫ L (µ, σ L), and the rght error s a postve dstrbuton wth mean µ and varance σ R, ǫ R (µ, σ R), and the overall error s a Gaussan dstrbuton wth mean 0 and varance σ, ǫ N(0, σ ). Here we focus on statstcally random error, whch s the most common type of error n practce; therefore we assume that the left error and the rght error have the same mean µ, and the overall error has a mean 0. If the error s based, the assumptons and computatons can be easly adjusted. Below, we gve a theorem for Gaussan dstrbuton parameters estmaton based on uncertan numercal data. Theorem 1. Assume an uncertan numercal attrbute X satsfes Gaussan dstrbuton. Let L and R be the random varable denotng the mnmal and maxmal values of the sample nterval; ǫ L, ǫ R and ǫ denote the left error (negatve error), the rght error(postve error) and the overall error, respectvely. Assume ǫ L (µ, σ L). ǫ R (µ, σ R), ǫ N(0, σ ), ǫ L and ǫ R are ndependent, then X satsfes the dstrbuton of P(X (L, R)) N(, σ (L+R) σ (R L) ). Proof. Assume X = µ +ǫ, hereby, we need to estmate µ and ǫ. As shown n Fgure 1, L R = X ǫ L = X + ǫ R From the above equatons, we know L R R + L R L = µ + ǫ ǫ L = µ + ǫ + ǫ R = µ + ǫ + ǫ R ǫ L = ǫ L + ǫ R Accordng to the assumpton, ǫ L and ǫ R are both random varables. ǫ L (µ, σ L), ǫ R (µ, σ R), ǫ L and ǫ R are

3 ndependent. Snce R L = ǫ L + ǫ R, the varance of R L, denoted as σ (R L), should be σ (R L) = σ ǫ L +ǫ R = σ ǫ L + σ ǫ R = σ L + σ R Snce L + R = µ + ǫ + ǫ R ǫ L, here µ s a constant whose varance s 0, the varance of R + L, denoted as σ (R+L), should be σ (R+L) = σ ǫ +ǫ R ǫ L = σ ǫ + σ ǫ R + ( 1) σ ǫ L = (σ) + σ R + σ L = σ + σ R + σ L Therefore, σ (R+L) σ (R L) = σ, and σ = σ (L+R) σ (R L) Further, snce L + R = µ + ǫ + ǫ R ǫ L, therefore, E(L + R ) = E(µ + ǫ + ǫ R ǫ L) = E(µ ) + E(ǫ ) + E(ǫ R) E(ǫ L) = µ µ µ = µ, from whch we obtan: µ = E(L + R) = µl+r = Therefore the dstrbuton of X can be estmated as P(X (L, R)) N( µ L+µ R, σ (L+R) σ (R L) ). Ths shows that for uncertan numercal data whch are represented as ntervals [L, R], the mean can be estmated as µ L +µ R and the varance can be estmated as σ (L+R) σ (R L). Please note that ths approach also apples to certan data. When a data nstance s certan, t s a pont nstead of an nterval; therefore, the mnmal boundary s equal to ts maxmal boundary, that s, L = R = X. Therefore, the mean of the whole dataset s µ L + µ R µx + µx = = µ X. The varance of the dataset s σ (L+R) σ (R L) = σ (L) σ (0) = σ (L) = σ (L) = σ (L) = σ (X). That shows that for certan data, the mean s estmated to be µ X and the varance s estmated to be σ (X), whch s consstent wth the Nave Bayes classfcaton algorthm. Therefore, P(X (L, R)) N(, σ (L+R) σ (R L) ) (1) s a general form for mean and varance estmaton. It apples to both uncertan and certan numercal data. Actually, certan data can be treated as a specal case of uncertan data wth zero uncertanty. When data has zero uncertanty, ths process automatcally evolves to the tradtonal Bayesan classfcaton algorthm.. Uncertan categorcal attrbutes An uncertan categorcal attrbute (UCA) A uc s characterzed by probablty dstrbuton over ts doman Dom. As mentoned earler, t can be represented by the probablty vector P = {p j1,..., p jn} such that P(A uc j = v k ) = p jk (1 n). Before dscussng parameter estmaton for uncertan categorcal data, we frst ntroduce the concept of probablstc cardnalty. The probablstc cardnalty for class C k of the dataset s the sum of the probablty of each nstance T j belongng to class C k. That s, PC(C k ) = D j=1 P(CT = C j k), where C Tj denotes the class label of nstance T j. Smlarly, the probablstc cardnalty of the dataset over v k of attrbute A uc s the sum of the probablty of each nstance whose correspondng UCA equals v k. That s, PC(v k ) = D j=1 P(v k T j) = D j=1 p jk. The probablstc cardnalty for class C of the dataset over v k of attrbute A uc l s the sum of the probablty of each nstance n class C whose correspondng UCA equals to v k. That s, PC(v k, C ) = D j=1 P(v k T j C Tj = C ). The class dstrbuton of each value of uncertan categorcal attrbutes can be denoted by a vector, whch we call the Class Dstrbuton Vector (CDV). CDV (v j, C) s (PC(v j, C 1), PC(v j, C ),..., PC(v j, C n)) T., n whch PC(v j, C ) s the probablstc cardnalty of nstances n class C wth attrbute value v j. For an uncertan categorcal attrbute nstance A uc j, the condtonal probablty P(A uc j = v k C l ) s estmated accordng to the fracton of the probablstc cardnalty of nstances n class C l that takes on a partcular attrbute value v k over the total probablstc cardnalty of nstances n class C l, that s: P(A uc j = v k C l ) = PC(v k, C l ). PC(C l ) P(A uc j ) gves the uncertan categorcal data dstrbuton for each class, whch wll be used for Bayesan predcton. 5. ALGORITHM DESCRIPTION In ths secton, we wll present our uncertan Bayesan classfcaton algorthm, whch s shown n Algorthm 1. The prncple procedure s as follows: 1. For each uncertan numercal attrbute nstance j, we update the Gaussan dstrbuton parameters µ + A, σ + A, µ A and σ A accordng to A j.r, A j.l by Functon updategaussan() as shown n step3-5.. For each uncertan categorcal attrbute nstance A uc j, we update ts CDV by the weght of the nstance T j.w and p jk usng Functon updatehstogram() (steps 6-9). 3. For each numercal attrbute nstance j, we update the Gaussan dstrbuton parameter (µ, σ) wth Functon updategaussan() (steps 10-11).. For each categorcal attrbute nstance A uc j, we update

4 ts CDV by the weght of the nstance T j.w usng Functon updatehstogram() (steps 1-13). 5. For each nstance T j, we update the Probablstc Cardnalty of ts class PC(T j.class) by the weght of the nstance T j.w usng Functon updateprobablstccardnalty() (steps 16). 6. Fnally, we compute the mean µ and standard devaton σ for each uncertan numercal attrbute usng Theorem 1 (steps 18-1). Algorthm 1 NBU(Dataset D) begn 1: for (Each nstance T j D do) do : for (each attrbute A do) do 3: f (A s uncertan numercal) then : (µ + A, σ + A ) = updategaussan(a j.r + A j.l, T j.w); 5: (µ A, σ A ) = updategaussan(a j.r A j.l, T j.w); 6: else f (A s uncertan categorcal) then 7: for (each v k A ) do 8: PC(v k, T j.class) = updatehstogram(t j.w p jk ); 9: end for; 10: else f (A s numercal) then 11: (µ, σ) = updategaussan(a j, T j.w); 1: else f (A s categorcal) then 13: PC(v k, T j.class) = updatehstogram(t j.w); 1: end f; 15: end for; 16: PC(T j.class) = updateprobablstccardnalty(t j.w); 17: end for; 18: for (each uncertan numercal attrbute A ) do 19: µ A = (µ + A + µ A )/; 0: σ A = (σ + A ) (σ A ) /; 1: end for; end An mportant beneft of Bayesan classfcaton s that t s ncremental, whch means that model can evolve gradually when more tranng data become avalable. Many other classfcaton methods, on the contrary, requre the whole classfcaton model to be rebult from scratch wth newly added tranng data. For example, the decson tree s essentally non-ncremental, wth more tranng data, the splttng pont and tree structure can be completely dfferent and t s better to rebult t. Please note that our NBU algorthm preserve the ncremental feature. For uncertan data, µ A, σ A and PC(v k, T j.class) can all be ncrementally updated. Ths s very mportant n data stream applcaton where new data constantly become avalable and the classfcaton model should be contnuously adjusted. 6. EXPERIMENTS In ths secton, we present the expermental results of the proposed Uncertan Bayesan Classfer algorthm. We mplemented Uncertan Bayesan Classfer classfcaton and predcaton algorthm. All the experments presented n ths secton are executed on a PC wth an Intel Pentum IV 3. GHz CPU and.0 GB man memory. A collecton contanng 10 real-world benchmark datasets were assembled from the UCI Repostory [1]. We try to cover the spectrum of propertes such as sze, attrbute numbers and types, number of classes and class dstrbutons. Among these 10 datasets, 5 of them, namely Dabetes, Glass, Irs, Segment and Sonar, contan manly numercal attrbutes. The other 5 datasets, namely Balance, Brdge, Mushroom, Promote and Soybean, have mostly categorcal attrbutes. Data Uncertanty. Data are made uncertan n the followng way: 1. To make numercal attrbutes uncertan, we convert each numercal value to an uncertan nterval. For each numercal attrbute, we scan all of ts value and get ts maxmum value X max and mnmum value X mn, respectvely. For each attrbute nstance x, ts uncertan nterval s [x (x X mn) rand1, x + (X max x) rand], where rand1 and rand denote two the random numbers. If they range between 0 to X, we denote such dataset as UX. For example, U0.5 stands for the dataset generated wth rand1 and rand between 0 to We make categorcal attrbutes uncertan by convertng them nto probablty vectors. For example, a categorcal attrbute L may have k possble values v m(1 m k). For each attrbute A j, we frst convert t nto a probablty vector P = (p j1, p j,..., p j,..., p jk ), whle p jl s the probablty for A uc j to be equal to v l, that s, P(A uc j = v l) = p jl. If the orgnal value of A j s equal to v l, we set p jl to be a value less than 1, and evenly dstrbute the rest probablty 1 p jl to all other values, that s, n k=1 k j p kl = 1 p jl. We randomly select part of nstances, for example 5% of them, and convert them uncertan, then we call such dataset U0.5. If all nstances n a categorcal dataset are made uncertan, t s represented as U1.00. We use U0 to denote accurate or certan orgnal datasets. When Uncertan Bayesan Classfer works on certan datasets U0, t s the same as the tradtonal Bayesan classfer. We study the accuracy of Uncertan Bayesan Classfer algorthm. In our experments, all attrbutes n the datasets are made uncertan except the ID attrbute. We compare the proposed Uncertan Bayesan Classfer wth the tradtonal Bayesan algorthm. Snce tradtonal Bayesan classfcaton algorthm cannot work uncertan data ntervals drectly, we convert uncertan datasets to certan. For uncertan numercal data, we use the center pont of the uncertan nterval; for uncertan categorcal data, we choose the value wth the hghest probablty. After ths converson, an uncertan dataset becomes a regular certan dataset and the tradtonal Bayesan classfcaton algorthm can be appled on t. We examne the Predcton Accuracy Rato(PAR), whch s the accuracy of the Uncertan Bayesan Classfer classfer on uncertan data to the accuracy of the tradtonal Bayesan classfer on the correspondng converted certan data. Fgure shows the expermental results on uncertan numercal datasets. For each dataset, we generate four uncertan datasets wth dfferent degree of uncertanty. The uncertan datasets are represented as U0.5, U0.5, U0.75 and U1.00, wth uncertanty ncreasng. From ths fgure, we found that the Uncertan Bayesan Classfer classfer almost always has the same or hgher accuracy than the regular Bayesan clas-

5 sfer when data s uncertan. On the Irs, Segment and Sonar dataset, Uncertan Bayesan Classfer clearly outperforms tradtonal Bayesan classfer. Furthermore, the dfference n accuracy between Uncertan Bayesan Classfer and tradtonal Bayesan grows as data become more uncertan. For the U0.5 datasets, the performance gan of Uncertan Bayesan Classfer s mostly wthn 5%, for the U0.75 and U1.00 datasets, the accuracy gan of Uncertan Bayesan Classfer reaches over 10% or even 15%. The reason s that tradtonal Bayesan classfer works wth the center ponts of uncertan ntervals and compute the data dstrbuton parameters solely based on the center ponts, and gnore other valuable nformaton. Uncertan Bayesan Classfer, on the other hand, use a more sophstcated model whch consders not only the centers, but also the left and rght boundares and the nterval length. It enables more precse data dstrbuton parameter estmaton, whch lead to more systematc modelng and more accurate predcton. Predcton Accuracy Rato dabetes glass rs segment sonar 0.8 U0 U0.5 U0.5 U0.75 U1.00 Extent of Uncertanty Fgure : PAR on numercal datasets We also studed the Uncertan Bayesan Classfer classfer accuracy on uncertan categorcal data. The result s shown n Fgure 3. Here, we stll nvestgate the predcton accuracy rato of Uncertan Bayesan Classfer over tradtonal Bayesan as uncertanty vares. The trend s smlar to the results on uncertan numercal data. The beneft of Uncertan Bayesan Classfer s more dstnct n ths experment. For the U0.50 datasets, the accuracy mprovements are mostly wthn 15%. When uncertanty ncreases to U0.75, for 3 out of the 5 datasets (balance, brdge and promote datasets), the performance gan reaches 0-50%. For all the fve U1.00 datasets, Uncertan Bayesan Classfer s 1.9 to 5 tmes better than tradtonal Bayesan. On the soybean dataset, Uncertan Bayesan Classfer has a slghtly worse performance than Bayesan when the uncertanty s low, as uncertanty ncrease t starts to show ts advantage. For the U0.75 dataset, t outperforms the tradtonal Bayesan, and on the U1.00 dataset, ts performance s 5 tmes better. Ths s because the tradtonal Bayesan classfer work on certan datasets, whch only keep the value of the hghest probablty for categorcal attrbute, whle probablty dstrbuton n other values are neglected. Uncertan Bayesan Classfer utlzes all the probablty dstrbuton nformaton avalable and bulds more accurate classfer model. Predcton Accuracay Rato balance brdge mushroom promote soybean 0.5 U0 U0.5 U0.5 U0.75 U1.00 Extent of Uncertanty Fgure 3: PAR on uncertan categorcal datasets In ths paper, we propose an uncertan Bayesan algorthm for classfyng and predctng uncertan datasets. Uncertan data are extensvely present n modern applcatons ncludng sensor network, movng object databases and bologcal databases. Instead of tryng to elmnate uncertanty and nose from datasets, ths paper follows the new paradgm of drectly mnng uncertan data. We ntegrate the uncertan data model wth Bayesan theorem and propose new methods for model parameter estmaton. The new methods allow us to derve more precse model based on uncertan data and attan hgher predcton accuracy. Our expermental evaluaton demonstrates that Uncertan Bayesan Classfer acheves hgher predcton accuracy comparng to the tradtonal Bayesan classfer when workng on uncertan data. 8. REFERENCES [1] [] R. Cheng, D. Kalashnkov, and S. Prabhakar. Evaluatng probablstc queres over mprecse data. In SIGMOD 003, pages [3] W. W. Cohen. Fast effectve rule nducton. In Proceedngs of the 1th Internatonal conference on machne learnng, pages [] X. W. et al. Top 10 algorthms n data mnng. Journal Knowledge and Informaton Systems, 1(1):1 37, Jan [5] L. Hawarah, A. Smonet, and M. Smonet. Dealng wth mssng values n a probablstc decson tree durng classfcaton. In The Second Internatonal Workshop on Mnng Complex Data, pages 35 39, 006. [6] O. Lobo and M. Numao. Ordered estmaton of mssng values. In Pacfc-Asa Conference on Knowledge Dscovery and Data Mnng, pages , [7] J. R. Qunlan. Probablstc decson trees, n Machne Learnng: an Artfcal Intellgence Approach. Morgan Kaufmann Publshers Inc. San Francsco, [8] J. R. Qunlan. C.5: Programs for Machne Learnng. Morgan Kaufman Publshers, [9] J. Wdom. Tro: A system for ntegrated management of data, accuracy, and lneage. In ICDR 005, pages CONCLUSIONS