UNN: A Neural Network for uncertain data classification

Transcription

1 UNN: A Neural Nework for unceran daa classfcaon Jaq Ge, and Yun Xa, Deparmen of Compuer and Informaon Scence, Indana Unversy Purdue Unversy, Indanapols, USA {jaqge, yxa }@cs.upu.edu Absrac. Ths paper proposes a new neural nework mehod for classfyng unceran daa (UNN). Uncerany s wdely spread n real-world daa. Numerous facors lead o daa uncerany ncludng daa acquson devce error, approxmae measuremen, samplng faul, ransmsson laency, daa negraon error and so on. The performance and qualy of daa mnng resuls are largely dependen on wheher daa uncerany are properly modeled and processed. In hs paper, we focus on one commonly encounered ype of daa uncerany - he exac daa value s unavalable and we only know he probably dsrbuon of he daa. An nuve mehod of handlng hs ype of uncerany s o represen he unceran range by s expecaon value, and hen process as ceran daa. Ths mehod, alhough smple and sraghforward, may cause valuable nformaon loss. In hs paper, we exend he convenonal neural neworks classfer so ha can ake no only ceran daa bu also unceran probably dsrbuon as he npu. We sar wh desgnng unceran percepron n lnear classfcaon, and analyze how neurons use he new acvaon funcon o process daa dsrbuon as npus. We hen llusrae how percepron generaes classfcaon prncples upon he knowledge learned from unceran ranng daa. We also consruc a mullayer neural nework as a general classfer, and propose an opmzaon echnque o accelerae he ranng process. Expermen shows ha UNN performs well even for hghly unceran daa and sgnfcanly ouperformed he naïve neural nework algorhm. Furhermore, he opmzaon approach we proposed can grealy mprove he ranng effcency. Keywords: Uncerany, classfcaon, neural nework 1 Inroducon Daa ends o be unceran n many applcaons [1], [], [3], [4], [5]. Uncerany can orgnae from dverse sources such as daa collecon error, measuremen precson lmaon, daa samplng error, obsolee source, nework laency and ransmsson error. I s mporan o cauously handle he uncerany n varous daa mnng applcaons o acheve sasfacory resuls. The error or uncerany n daa s commonly reaed as a random varable wh probably dsrbuon. Thus, unceran arbue value s ofen represened by an nerval wh a probably dsrbuon

2 funcon over he nerval [6], [7]. I s mporan ha daa uncerany models are negraed wh daa mnng algorhms o acheve beer performance n varous daa mnng applcaons. Classfcaon s one of he key processes n machne learnng and daa mnng. Classfcaon s he process of buldng a model ha can descrbe and predc he class label of daa based on he feaure vecor [8]. An nuve way of handlng uncerany n classfcaon s o represen he unceran value by s expecaon value and rea as a ceran daa. Thus, convenonal classfcaon algorhms can be drecly appled. However, hs approach does no effecvely ulze mporan nformaon such as probably funcon and dsrbuon nervals. We exend daa mnng echnques so ha hey can ake unceran daa such as daa nerval and probably dsrbuon as he npu. In hs paper, we desgn and develop a new classfer named unceran neural nework (UNN), whch employs new acvaon funcon n neurons o handle unceran values. We also propose a new approach o mprove he ranng effcency of UNN. We prove hrough expermens ha he new algorhm has sasfacory classfcaon performance even when he ranng daa s hghly unceran. Comparng wh he radonal algorhm, he classfcaon accuracy of UNN s sgnfcanly hgher. Furhermore, wh he new opmzaon mehod, he ranng effcency can be largely mproved. The paper s organzed as follows. In secon, we dscuss relaed work. Secon 3 defnes he classfcaon problem for unceran daa. In secon 4, we frs analyze he prncple of unceran percepron n lnear classfcaon, and hen consruc he mullayer unceran neural nework, and dscuss he ranng approach. Secon 5 nroduces an opmzed acvaon funcon o mprove he effcency. The expermens resuls are shown n secon 6, and secon 7 makes a concluson for he paper. Relaed Works There has been a growng neres n unceran daa mnng. A number of daa mnng algorhms have been exended o process unceran daase. For example, UK- Means [9], unceran suppor vecor machne [10], and unceran decson ree [11].Arfcal neural nework has been used n model-based cluserng wh a probably ganed from expecaon-maxmzaon algorhm for classfcaonlkelhood learnng [1]. We adop he concep o esmae he probably of membershp when he unceran daa are covered by mulples classes. However, probably esmaon presened here s unprecedened. In fuzzy neural nework models for classfcaon, eher arbues or class labels can be fuzzy and are presened n fuzzy erms [13]. Gven a fuzzy arbue of a daa uple, a degree (called membershp) s assgned o each possble class, showng he exen o whch he uple belongs o a parcular class. Our work dffers from prevous work n ha we revse he acvaon funcons o compue he membershp based on unceran daa dsrbuon nformaon, nsead of usng Fuzzy logc for unng neural nework ranng parameers. Our approach can work on boh ceran and unceran daa.

3 3 Problem Defnon In our model, a daase D consss of d ranng uples, { 1,,, d }, and k numercal arbues, A 1,, A k. Each uple s assocaed wh a feaure vecor V = (f,1, f,,, f,k ), and a class label c C. Here, each f,j s a pdf modelng he unceran value of arbue A j n uple. Table. 1 shows an example of an unceran daase. The frs arbue s unceran. The exac value of hs arbue s unavalable, and we only know he expecaon and varance of each daa uple. Ths ype of daa uncerany wdely exss n pracce [1], [], [5], [6], [7]. Table. 1. An example of unceran daase ID Class Type Arbue #1 (expecaon, sandard varance) 1 Yes (105, 5) NO (110,10) 3 No (70,10) 4 Yes (10,18) 5 No (105,10) 6 No (60,0) 7 Yes (10,0) 8 No (90,10) 9 No (85,5) 10 No (10,15) The classfcaon problem s o consruc a relaonshp M ha maps each feaure vecor (f x,1, f x,,, f x,k ) o he membershp P x on class label C, so ha gven a es uple 0 =(f 0,1, f 0,,, f 0,k ), M(f 0,1, f 0,,, f 0,k ) predc he membershp o each class. If he es nsance has posve probably o be n dfferen classes, hen wll be predced o be n he class whch has he hghes probably. The work n hs paper s o buld a neural nework when only unceran ranng daa uples are avalable, and he goal s o fnd he model wh he hghes accuracy despe of he uncerany. 4 Algorhm 4.1 Unceran Percepron We sar wh percepron, whch s a smple ype of arfcal neural nework. Percepron s a classcal model whch consrucs lnear classfer as:

4 n y = F( xω θ), = 1 1, s 0 Fs () =. 1, s < 0 (4.1) Where x = (x 1,,x n ) s he npu vecor, ω = (ω 1,,ω n ) s he wegh vecor, F s he acvaon funcon, and y s he percepron s oupu. For daa ses wh unceran arbues, we need revse he funcons and develop an unceran percepron for lnear classfcaon. We wll llusrae our approach hrough a smple -dmensonal daase. Assume daase has wo arbues X = (x 1, x ) and one class ype y, and assume each unceran arbue has a dsrbuon as x ~N (μ, σ ), and he class ype can be 1 or -1, Fg. 1 s a geomerc represenaon of lnear classfcaon for a -dmensonal unceran daase. In hs fgure, each daa nsance s represened by an area nsead of a sngle pon because each dmenson/arbue s an unceran dsrbuon, no an accurae value. x Q S P R Class:1 L Class:-1 Fg. 1. Geomerc represenaon of unceran Percepron The sragh lne L n Fg 1 represens he equaon: ω1x1 ωx θ = 0. (4.) where x 1, x are unceran arbues. We defne a parameer as = ω x ω x θ (4.3) 1 1. x 1

5 As menoned earler, arbues (x 1, x ) follow he dsrbuon x ~ N (μ, σ ). Snce hese arbues are ndependen, wll have a dsrbuon as: f( ) ~ N( ωµ ωµ θ, ωσ ωσ ). (4.4) Le s = P(>0) represen he probably of larger han 0. If P(>0) = 1, s defnely larger han 0, whch means hs uple s n class 1, and locaes above he lne L n Fg. 1, for example, lke Pon P. If P(>0) = 0, s less han or equal o 0, whch means hs uple s n class -1, and s below lne L such as Pon R. For unceran daa, s possble ha he unceran range of a daa nsance may cover he lnear classfcaon lne L, for example, Pon Q s one such nsance. In hs case, Q has posve probably o belong o boh classes, and he membershp of class wll be deermned by whch class has a hgh probably. Therefore, we consruc an acvaon funcon as equaon (4.5). 1, s 0.5 Fs () =. 1, s < 0.5 (4.5) Where, s = P(>0). Fg. s srucure of he unceran percepron model. In Fg., (μ, σ ) s he expecaon and sandard devaon of unceran arbues, as npus. When he dsrbuon s Gaussan, s can be calculaed as: 1 ( u ) s = exp( ) d. (4.6) 0 πσ σ (μ 1,σ 1 ) ω 1 ω P(>0) F (μ,σ ) θ (1,0) Fg.. unceran percepron srucure Based on he sngle unceran neurons, we can develop a mullayer neural nework. 4. Unceran neural nework An unceran mullayer feed-forward neural nework s consruced by addng a hdden layer whch conans he unceran neurons beween npu and oupu layers.

6 We call hs algorhm as UNN (for unceran neural nework). Fg. 3 s an nsance of he layer srucure of neural nework. Here, he hdden layer has a ransfer funcon as Where, F( µσ, ) = P ( > 0). (4.7) = ω x θ. = 1 x ~ N ( µ, ω ). P(>0) wll be compued based on unceran daa dsrbuon funcon, For example, f he daa follows Gaussan dsrbuon, hen 1 ( u ) P( > 0) = exp( ) d. 0 πσ σ The oupu layer can have an acvaon funcon as Sgmod, snce he oupu values fall n he range (0,1), o represen he membershp of every class. (μ 1,σ 1 ) INPU T HIDDE N OUTPUT ω IH F H ω HO F O y o1 (μ,σ ) F H F O y o Fg. 3. Mullayer neural nework srucure 4.3 Algorhm Analyss A sragh-forward way o deal wh he unceran nformaon s o replace he probably dsrbuon funcon wh s expeced value. Then he unceran daa can be reaed as ceran daa and he radonal neural nework can be used for classfcaon. We call hs approach AVG (for Averagng). Ths approach, as menoned earler, does no ulze valuable unceran nformaon and may resul n loss of accuracy. We llusrae he reason wh he followng example. Fg. 4 s an

7 example of classfyng an unceran daase. Lne L1 and L reflec he ranng resul of he hdden layers of a neural nework. Suppose P s a es daa nsance and we need predc he class ype of P. Because he expecaon of P locaes n area II, wll be assgned o class II f usng AVG algorhm. However, from Fg. 4, s obvous ha f we consder he dsrbuon of P, has a larger probably o be n area I han n area II. Therefore, should be classfed o class I. UNN wll perform he classfcaon correcly snce compues he probably of P belongng o boh classes I and II accordng o he probably dsrbuon nformaon and predcs o be n he class whch has a larger probably. In hs sense, he unceran neural nework can acheve hgher classfcaon accuracy. II x I I P L II x 1 L1 Fg. 4. classfyng a es uple P 4.4 Nework ranng We adop a Levenberg-Marquard back propagaon algorhm [14], o ran hs supervsed feed-forward neural nework. I requres all he acvaon funcon has a dervave. Suppose Equaon (4.7) s he hdden layer acvaon funcon of he unceran neural nework, hen s dervave s lke: µ µ 1 µ σ = = 3 df F F σ (, ) ( e, e ). d( µσ, ) µ σ πσ πσ (4.8) And, dµ dσ = µ, = σ * ω. (4.9) dω dω Therefore, by subsung Equaon (4.8) (4.9) no Equaon (4.10), we can ge he acvaon funcon s dervaves.

8 df F dµ F dσ =. dω µ dω σ dω (4.10) When we have he dervaves of hese acvaon funcons, s nuve o ran he nework based on radonal mehod such as graden decen. Afer ranng, we can hen use he model for predcon for unceran daa. 5 Improve on acvae funcon The hdden layer s acvae funcon, n Equaon (4.7), has an oupu rangng beween 0 and 1. When we consder wo dfferen daa nsances ha are absoluely n he same class, her funcon oupu wll boh be 1. Ths may cause he nework ranng o be me consumng n some scenaros. In order o mprove he ranng effcency, we can desgn new hdden layer acvae funcons. For example, when he uncerany s represen by Gaussan dsrbuon, we devse a new hdden layer acvae funcon, as Equaon (5.1) o accelerae he ranng process. u * P ( > 0), f u > 0 ; F( µσ, ) = 0, f u = 0 ; u * P ( < 0), f u < 0 ; (5.1) I I ( ) ~ N ( ωµ θ, ωσ ). f Here F (μ, σ) s connuous a u = 0, snce lm F ( µσ, ) = lm F ( µσ, ) = F (0, σ) = 0. µ 0 µ 0 F (μ, σ) also has a dervave: df F F = (, ). (5.) d ( µσ, ) µ σ µ µ σ η e, f µ >0 πσ F = 1/, f µ =0. µ µ µ σ 1-η e, f µ <0 πσ (5.3)

9 µ F µ e σ =. 3 σ πσ (5.4) Thus, subsue Equaon (5.) (5.3) (5.4) no Equaon (5.5), we ge he dervave of F. df F dµ F dσ = (5.5) dω µ dω σ dω Equaon (5.5) hen can be used n Levenberg-Marquard back propagaon ranng algorhm. 6 Expermens 6.1 Expermen on accuracy We have mplemened he UNN approach usng Malab6.5[15], and appled hem o 5 real daa ses aken from he UCI Machne Learnng Reposory [16]. The resuls are shown n Table.. For he daases excep Japanese Vowel, he daa uncerany s modeled wh a Gaussan dsrbuon wh a conrollable parameer ω, whch s a percenage of he sandard devaon o he value of expecaon. In our expermens, we vary he ω value o be 0.1, 0.3 and 0.5. For Japanese Vowel daa se, we use he uncerany gven by he orgnal daa o esmae s Gaussan dsrbuon. Table. Accuracy expermen resuls Japanese Vowel Uncerany Tran Tes UNN Dsrbuon based raw daa 98.50% 94.95% AVG 99.17% 94.31% Irs Uncerany Tran Tes UNN ω= % 99.93% ω= % 99.93% ω= % 99.38% AVG 99.17% 98.89% Ionosphere Uncerany Tran Tes UNN ω= % 93.71% ω= % 90.73% ω= % 9.05% AVG 97.17% 87.86% Magc Telescope Uncerany Tran Tes

10 UNN ω= % 80.01% ω= % 76.58% ω= % 80.56% AVG 99.67% 73.17% Glass Uncerany Tran Tes UNN ω= % 65.75% ω= % 69.59% ω= % 65.57% AVG 74.0% 65.% predcon accuracy w = 0.1 w = 0. w = 0.5 AVG Japanese Vowel Irs Ionosphere Magc Telescope Glass Fg. 5. Accuracy Comparson of UNN and AVG In our expermens, we compare UNN wh he AVG (Averagng) approach, whch process unceran daa by smply usng he expeced value. The resuls are shown n Fg 5. From he fgure, we can see ha UNN ouperforms AVG n accuracy almos all he me. For some daases, for example, Ionosphere and Magc Telescope daases, UNN mproves he classfcaon accuracy by over 6% o 7%. The reason s ha UNN ulzes he unceran daa dsrbuon nformaon and compues he probably of daa beng n all dfferen classes. Therefore, he classfcaon and predcon process s more sophscaed and comprehensve han AVG, and has he poenal o acheve hgher accuracy. 6. Expermen on effcency In secon 5, we have dscussed an alernave acvae funcon for mprovng he effcency of nework ranng process. Here, we presen an expermen whch compares he effcency of wo neworks wh dfferen hdden layer acvae funcons. In hs expermen, we name he nework usng he orgnal funcon (Equaon 4.7) as UNN-O, and he nework usng acvae funcon (5.1) as UNN-M. The ranng me of UNN-O and UNN-M s shown n Fg. 6 (a) and he ranng epochs of UNN-O and UNN-M s shown n Fg 6. (b). Because of he more complex calculaons n handlng uncerany, UNNs generally requre more ranng me and epochs han AVG. However, he fgures also ndcae ha effcency of UNN-M s

11 hghly mproved, compared wh UNN-O. The ranng of UNN-M requres much fewer epochs han UNN-O, and s sgnfcanly faser. ranng me(s) AVG UNN-M UNN-O 10 0 Japanese Vowel Irs Inosphere Magc Telescope Glass (a)tranng me ranng epochs AVG UNN-M UNN-O 10 0 Japanese Vowel Irs Inosphere Magc Telescope Glass (b) Tranng epochs Fg. 6. Performance comparson 6 Concluson In hs paper, we propose a new neural nework (UNN) model for classfyng and predcng unceran daa. We employ he probably dsrbuon whch represen he unceran daa arbue, and redesgn he neural nework funcons so ha hey can drecly work on unceran daa dsrbuons.. Expermens show ha UNN has hgher classfcaon accuracy han he radonal approach. The usage of probably dsrbuon can ncreases he compuaonal complexy, and we propose new acvaon funcon for mproved effcency. We plan o explore more classfcaon approaches for varous uncerany models and fnd more effcen ranng algorhms n he fuure.

12 7 REFERENCE 1. C. C. Aggarwal and P. Yu. A framework for cluserng unceran daa sreams. In ICDE, 008. G. Cormode and A. McGregor. Approxmaon algorhms for cluserng unceran daa. In Prncple of Daa base Sysem (PODS), H. Kregel and M. Pfefle. Densy-based cluserng of unceran daa. In KDD 005, pp S.Sngh, C. Mayfeld, S. Prabhakar, R. Shah, and S. Hambrusch. Indexng caegorcal daa wh uncerany. In ICDE 007, pp H. Kregel and M.Pfefle. Herarchcal densy-based cluserng of unceran daa. In ICDM 005, pp C. C. Aggarwal. On Densy Based Transforms for unceran Daa Mnng. In ICDE Conference Proceedngs, C. C. Aggarwal. A Survey of Unceran Daa Algorhms and Applcaons. In IEEE Transacons on Knowledge and Daa Engneerng, Vol. 1, No. 5, R. Agrawal, T. Imelnsk, and A. N. Swam, Daabase mnng: A performance perspecve, IEEE Trans. Knowl. Daa Eng., M. Chau, R. Cheng, B. Kao, and J. Ng. Unceran daa mnng: An example n cluserng locaon daa. In PAKDD, 006, pp J. B and T. Zhang. Suppor Vecor Machnes wh Inpu Daa Uncerany. In Proc. Advances n Neural Informaon Processng Sysems, Bao Qn, Yun Xa, Fang L. DTU: A Decson Tree for Classfyng Unceran Daa. In he Pacfc-Asa Conference on Knowledge Dscovery and Daa Mnng (PAKDD), Shh-San Cheng, Hsn-Cha Fu, and Hsn-Mn Wang. Model-Based Cluserng by Probablsc Self-Organzng Maps. In IEEE Transacons on Neural Neworks, VOL. 0, NO. 5, A. D. Kulkarn and V. K. Mungan. Fuzzy Neural Nework Models For Cluserng. In ACM Symposum on Appled Compung, Ocavan San, Edward W. Kamen. New block recursve MLP ranng algorhms usng he Levenberg-Marquard algorhm. In Neural Neworks, VOL.3, Malab: hp:// 16. A. Asuncon and D. Newman, UCI machne learnng reposory, 007. hp://