A Novel Approach to Model Generation for Heterogeneous Data Classification

Transcription

1 A Novel Approach o Model Generaon for Heerogeneous Daa Classfcaon Rong Jn*, Huan Lu *Dep. of Compuer Scence and Engneerng, Mchgan Sae Unversy, Eas Lansng, MI rongn@cse.msu.edu Deparmen of Compuer Scence and Engneerng, Arzona Sae Unversy, Tempe, AZ hlu@asu.edu Absrac Ensemble mehods such as baggng and boosng have been successfully appled o classfcaon problems. Two mporan ssues assocaed wh an ensemble approach are: how o generae models o consruc an ensemble, and how o combne hem for classfcaon. In hs paper, we focus on he problem of model generaon for heerogeneous daa classfcaon. If we could paron heerogeneous daa no a number of homogeneous parons, we wll lkely generae relable and accurae classfcaon models over he homogeneous parons. We examne dfferen ways of formng homogeneous subses and propose a novel mehod ha allows a daa pon o be assgned mulple mes n order o generae homogeneous parons for ensemble learnng. We presen he deals of he new algorhm and emprcal sudes over he UCI benchmark daases and daases of mage classfcaon, and show ha he proposed approach s effecve for heerogeneous daa classfcaon. 1 Inroducon Ensemble approaches such as baggng and boosng have been successfully appled o many classfcaon problems [Deerch, 2000; Bauer and ohav, 1999]. The basc dea of ensemble mehods s o consruc a number of classfers over ranng daa and hen classfy new daa pons by akng a (weghed) voe of her predcons. Thus, wo mporan ssues assocaed wh an ensemble approach are: 1) how o generae accurae ye dverse classfcaon models, and 2) how o combne he models for ensemble classfcaon. Dverse classfers ensure good ensembles [Qunlan, 1996]. In hs paper, we focus on he frs ssue wh an emphass on heerogeneous daa classfcaon. Heerogeneous daa classfcaon refers o he problem when npu daa of a sngle class are wdely dsrbued no mulple modes. I arses when ranng daa are colleced under dfferen envronmens or hrough dfferen sources. An example of heerogeneous daa classfcaon s mage classfcaon, n whch labeled mages are acqured from mulple resources and exhb dsparae characerscs. For nsance, some mages are black and whe, and ohers are colorful. A wdely used approach for consrucng an ensemble of models s o sample dfferen subses from he ranng daa and creae a classfcaon model for each subse. Baggng [Bremann, 1996] and AdaBoos [Schapre and Snger, 1999] are wo represenave mehods n hs caegory. Baggng randomly draws samples from he ranng daa wh replacemen and AdaBoos samples ranng daa accordng o a dynamcally changed dsrbuon, whch s updaed by pung more wegh on he msclassfed examples and smaller weghs on he correcly classfed examples. Clearly, boh mehods do no rea homogeneous daa and heerogeneous daa dfferenly. For ensemble mehods o work effecvely on heerogeneous daa, one nuve soluon s o frs dvde he heerogeneous daa no a se of homogeneous parons and hen o creae a model for each paron of daa. Member classfers bul wh dfferen homogeneous parons wll lkely resul n good dversy of an ensemble. One way o realze hs homogeney-based paron s o employ sandard cluserng algorhms, such as -means [Hargan and Wong, 1979] and he EM cluserng algorhm [Celeux and Govaer, 1992]. An example s he Gaussan Mxure Model (GMM). Bu, n general, here are wo problems wh hs smple cluserng approach: Sngle cluser membershp. Mos cluserng algorhms assume ha cluser membershp s muually exclusve and each daa pon can only belong o a sngle cluser. Even hough he EM cluserng algorhm allows sof membershp for a daa pon, n he resulng clusers, each daa pon sll only belongs o a sngle cluser [Wen and Frank, 2000]. Therefore, when we use hese cluserng algorhms o paron daa, f he number of clusers s large and he subses of ranng daa formed by a cluserng algorhm are muually dson, some clusers may have a very small number of daa pons, whch can lead o unrelable classfcaon models. Ths s smlar o he daa fragmenaon problem occurred n decson ree nducon [Qunlan, 1993]. In conras, he subses of ranng daa produced by Baggng and AdaBoos are no muually dson. For example, n boosrap samplng, each subse conans around 63.2% of he orgnal ranng daa. Unbalanced cluser szes. Snce mos cluserng algorhms do no have conrol over cluser szes, unbalanced cluser szes resulng from cluserng canno be easly cor-

2 reced. When he resulng clusers have very dfferen szes, a classfer bul over a small cluser can be unrelable and hus degrade he performance of he ensemble n formng fnal ensemble classfcaon. On he conrary, boh Baggng and AdaBoos have daa samples of smlar szes when learnng dfferen models. Noe ha here have been prevous effors on balancng he szes of dfferen clusers, parcularly for specral cluserng algorhms (e.g., he normalzed cu algorhm [Mela & Sh, 2001]). Bu, snce he conrol of cluser sze comes ndrecly from he obecve funcon, he resulng clusers can sll have unbalanced szes. In sum, a cluserng approach may produce homogeneous daa parons, bu canno ensure smlar szes of dfferen parons; mehods lke Baggng can produce equally szed parons, bu parons are no homogeneous. Therefore, we need a novel approach o paronng daa no homogeneous subses of smlar szes n ensemble learnng for heerogeneous daa classfcaon. The goal of hs work s o dvde heerogeneous daa no homogeneous subses of smlar szes n order o generae relable and accurae classfcaon models. By focusng on homogeneous subses, we do no requre ha each daa pon belong o one subse; by ensurng smlar szes of daa subses, each classfcaon model can be bul wh a smlar number of daa pons. In hs paper, we propose a HISS (Homogeneous daa In Smlar Sze) algorhm specally desgned for he above purposes for heerogeneous daa classfcaon. Specfcally, HISS allows he user o specfy he sze of a subse. For example, he user can ask he algorhm o creae 20 subses wh each conanng 40% of he orgnal daa. Ths algorhm s smlar o he boosrap samplng procedure n ha boh he number of subses and he percenage of ranng daa covered by each cluser can be specfed and vared. However, dffers from he smple boosrap samplng procedure n ha pus he smlar daa pons no a sngle subse whle boosrap samplng randomly selecs daa o form a subse. Ths propery s mporan n ensemble learnng for classfyng heerogeneous daa. We wll use sraa for he homogeneous daa parons, and subses for daa parons resulng from random samplng. 2 Relaed Work There have been many prevous sudes on how o creae an ensemble of models. The mehods for consrucng an ensemble of models can be caegorzed no fve groups [Deerch, 2000]: 1) Bayesan mehods, whch creaes an ensemble of model by samplng hem from a esmaed poseror model dsrbuon; 2) Samplng ranng examples, whch creaes mulple subses of ranng examples and rans a classfer for each of he subses; 3) Samplng npu feaures, whch creaes a number of subses of he npu feaures and a classfer s bul for each subse of npu feaures; 4) Error correc oupu code (ECOC), whch conver a mulple class problem no a se of bnary class problems; 5) Inecng randomness, ha generaes ensembles of classfers by necng randomness no he learnng algorhm. Among he fve caegores, our work s closely relaed o he second one, whch creaes mulple classfers by samplng ranng examples. Imporan mehods n hs group nclude Baggng [Breman, 1996] and AdaBoos [Schapre and Snger, 1999]. Alhough hese mehods have been shown o be effecve for classfcaon, hey are no desgned o ake no accoun characerscs of heerogeneous daa. In hs paper, we propose HISS an algorhm ha consrucs homogeneous sraa from heerogeneous daa whle manans he nce propery of boosrap samplng procedure - each sraum conans a smlar number of daa pons. Anoher lne of research closely relaed o hs work s he sudy of cluserng algorhms. In general, cluserng algorhms can be caegorzed no paramerc approaches and non-paramerc approaches. The paramerc approach s o fnd a paramerc model ha mnmzes a cos funcon assocaed wh nsance-cluser assgnmens. Such mehods nclude he Mxure Model [Celeux and Govaer, 1992] and -means algorhm. For he non-paramerc approaches, a cos funcon s mnmzed by eher mergng wo separae clusers no a larger one or dvdng a cluser no wo smaller ones. The represenave examples of hs caegory are he agglomerave approach and he dvsve approach. Mos cluserng approaches assume ha each daa pon only belongs o a sngle cluser. Ths assumpon may no be approprae snce he ulmae goal of cluserng s o group smlar daa pons ogeher. When s unceran o assgn a daa pon o a sngle cluser, s beer off assgnng o mulple clusers. Alhough he radonal probablsc model and he fuzzy cluserng algorhm allow for mul- or sof-membershps, he uncerany of cluser membershp s only exploed durng he process of esmaon. In he resulng clusers, each daa pon s assgned o only a sngle cluser. Furhermore, mos cluserng algorhms do no have any conrol over he sze of clusers. Hence, he resulng clusers can be very unbalanced n sze and he clusers of oo small szes could be useless n learnng. 3 The HISS Algorhm for Model Generaon 3.1 From Probablsc Cluserng o HISS We frs descrbe he radonal probablsc cluserng algorhm, and hen nroduce algorhm HISS. The general dea of probablsc cluserng s o descrbe daa wh a mxure of generave models. Opmal parameers are usually obaned by maxmzng he lkelhood of daa usng he mxure model. Le n be he number of npu daa pons, be he number clusers, { x1, x2,..., x n } be he npu daa, and { m1, m2,..., m } be he underlyng models ha generae he daa. By assumng ha each daa pon s generaed from a mxure of models{ m1, m2,..., m }, we have he lkelhood of he daa wren as: n l({ m} 1, = ) = log p( x m) = 1 = 1 (1)

3 where px ( m ) s he lkelhood of generang x from he model m, and s he lkelhood for daa pon x o be n he -h cluser. Based on he assumpon ha each daa pon can only belong o a sngle cluser, we have consran = 1. An example of probablsc cluserng s he = 1 Gaussan Mxure Model (GMM), n whch boh px ( m ) are parameerzed as: = θ, and T 1 1 ( x µ ) Σ ( x µ ) px ( m) = exp d /2 1/ 2 ( 2 π ) Σ 2 where θ denoes he pror for he -h cluser, and and (2) µ and Σ are he mean and varance marx for he -h cluser, respecvely. Expecaon and Maxmzaon algorhm (EM) (Dempser e al, 1977) can be used o search for he opmal parameers. By removng he consran 1 = 1 =, we allow each daa pon o belong o mulple homogeneous clusers, or n shor, sraa. Hence, he opmzaon problem becomes n max l({ m} 1, ) log ( ) = = p x m m, = 1 = 1 subec o 0 1 for = 1,..., n, and, = 1,..., where all are consraned o beween 0 and 1 o manan he probably nerpreaon. I s easy o see ha he opmal soluon s o se all o be 1, whch means ha each daa pon s ncluded n every sraum. To avod he rval soluon for, we choose o enforce he percenage of ranng daa ha are covered by each cluser o be a predefned consanγ,.e., (3) 1 n, 0 1, for 1,..., 1 = γ γ = (4) = n Wh he above consran, we guaranee ha he number of daa pons ha suppor each sraum s around γ n. Compared o he sngle membershp consran, hs new consran has he followng wo advanages: 1) I does no assume ha each daa pon has o belong o one sraum. For hs new srafyng mehod, on average each daa pon can belong o γ number of sraa. Therefore, when γ s larger han one, each daa pon s allowed o be n more han one sraum smulaneously. 2) I ensures ha dfferen sraa have balanced numbers of daa pons. In conras o mos cluserng algorhms, he new algorhm ensures almos he same sze for each sraum. Ths s parcularly mporan o he research goal of hs paper - generang a relable and accurae ensemble for heerogeneous daa. By seng γ o be a reasonably large value (0.4 n hs work), we ensure ha each sraum has a suffcenly large number of examples for buldng a sascal learnng model. For laer reference, we refer hs new cluserng approach as HISS, whch sands for Homogeneous daa In Smlar Sze. 3.2 Opmzaon for HISS Pung Equaons (3) and (4) ogeher, we have: n max l({ m} 1, ) log ( ) = = p x m m, = 1 = 1 subec o 1 = γ, 0 γ 1, for = 1,..., n 0 1 for = 1,..., n, and, = 1,..., n = 1 Le us assume he Gaussan dsrbuon for px ( m ),.e., px ( m )~ Nµ (, σ ). Followng he dea of he EM algorhm, he dfference n he lkelhood of daa beween wo consecuve eraons s bound by: l({ m( + 1)} = 1, ( + 1)) l({ m( )} = 1, ( )) n ( + 1) = 1 = 1υ ()log + () (6) n px ( m( + 1)) = 1 = 1υ ()log px ( m( )) where υ s defned as (5) () p( x m()) υ () = k (7) () p( x m ()) = 1 k Thus, he opmal soluons for he mean and varance of Gaussan dsrbuon can be obaned as follows: n n 2 υ () () 1 x υ 2 1 x = = 2 n n υ () () 1 υ 1 = = µ ( + 1) =, σ ( + 1) = µ ( + 1) However, he opmal soluon for s raher dffcul o oban because of he nequaly consrans 0 1. Drecly opmzng he Equaon (6) wh only he equaly consran wll resul n he followng soluon for ( + 1) : ( 1) n υ = 1 υ () γ n + = (8) () Apparenly, he above soluon wll always be nonnegave f () s nonnegave. However, does no guaranee ha ( + 1) s no greaer han 1.

4 Fndng Opmal ( + 1) Inpus: υ () for = 1,..., n and = 1,..., Oupus: ( + 1) ha maxmzes Equaon (7). Inalzaon: ( + 1) = 0 for = 1,..., n and = 1,..., for each cluser do For all examples, se ( + 1) = 1 f ( + 1) > 1 Compue he probably mass s = γn { ( + 1) = 1} Re-compue υ () ( + 1) = s s.. ( + 1) < 1 υ () { ( + 1) < 1} whle ( s.. ( + 1) > 1 ) end Fgure 1: Algorhm for fndng opmal ( + 1) In order o sasfy he nequaly consrans 0 1, we use he T condons [Flecher, 1987] o effcenly adus he value of ( + 1). The basc dea s o rese o be 1 whenever he oupu from Equaon (10) volaes he consran 0 1. Afer he adusmen, we wll recompue ( + 1) ha are less han 1 usng Equaon (8). The procedure of adusng and recompung ( + 1) wll connue unl no ( + 1) volaes he consran. Fgure 1 shows he dealed seps for fndng he opmal soluon for ( + 1). Due o he space lm, he proof for he opmaly of he algorhm n Fgure 1 s no provded here. 3.3 Classfyng Heerogeneous Daa For classfcaon problems, heerogeneous daa can be found n many applcaons and n expermens: 1) Daa acqured from mulple sources. In many cases, ranng daa are acqured from mulple sources. Because each source has s own daa dsrbuon ha may be dfferen from ohers, he daa merged from mulple sources are herefore heerogeneous. For example, consder buldng a classfcaon model for oudoor scenes. The ranng mages are colleced from several dfferen ypes of vdeos. Some of he vdeos are news sores and some of hem are of adversemen. Some of hem are of hgh qualy and some of hem are no. Thus, he wdely dsparae characerscs n vdeos cause he merged daa o be heerogeneous. 2) Daa by converng a mulple class problem no a se of bnary class problems. In order o apply he bnary class classfcaon algorhm o mulple class case, we need o Daa Se # Examples #Class # Feaures Ecol Pendg Glass Yeas Vehcle Image/Indoor Image/Oudoor Table 1: Descrpon of daases for he expermen for heerogeneous daa classfcaon. conver he classfcaon problem of mulple classes no a se of bnary class problems. The represenave examples nclude he one-agans-all approach and error correc oupu codng (ECOC) mehod [Deerch, 1995]. Durng hs process, mulple classes are grouped no wo subses of classes. Daa pons from one subse of classes are used as posve examples and he remanng are used as negave examples. Because boh he posve and negave pools can be comprsed of examples from mulple classes, wll creae daa heerogeney for each of he bnary classes. As dscussed, an nuve soluon o classfyng heerogeneous daa s o creae a se of classfcaon models wh each classfer bul on a homogeneous paron (sraum) of he daa, and hen combne classfers for he fnal predcon. The radonal cluserng algorhms are no desgned for hs ask because of he poenal unbalanced cluserszes and he daa fragmenaon problem. Wh he proposed algorhm HISS, we can avod hese wo problems by seng he parameer o be large (0.4 n he expermen). In sum, o classfy heerogeneous daa, we frs apply HISS o oban homogeneous sraa and hen creae a classfcaon model for each sraum o form an ensemble. We wll refer o hs model generaon mehod as HISS-based Model Generaon n our emprcal sudy nex. Fnally, a sackng approach [Wolper, 1992] s used o combne models ha are generaed by he HISS-based model generaon mehod for he fnal predcon of he ensemble. 4. Expermenal Sudy The expermenal sudy s desgned o answer he followng quesons: 1) Is he proposed model generaon mehod effecve for classfyng heerogeneous daa? To hs end, we compare he proposed model generaon mehod o Baggng and AdaBoos n classfyng heerogeneous daases. 2) Is he proposed HISS algorhm effecve for generang relable models? To address hs queson, we wll apply boh he proposed HISS algorhm and he probablsc cluserng algorhm o paron he ranng daa and buld a classfcaon model for each paron. 4.1 Expermenal Desgn Seven dfferen daases are used n he expermens: fve mulple class daases from he UCI Machne Learnng reposory [Blake and Merz, 1998] and wo bnary class daa-

5 Daa Se Baselne AdaBoos Baggng HISS-based Baggng AdaBoos (Sandard) (Sandard) Ensemble (Sackng) (Sackng) Ecol (0.012) (0.006) (0.014) (0.006) (0.006) (0.006) Pendg (0.003) (0.003) (0.002) (0.002) (0.001) (0.003) Glass (0.027) (0.081) (0.046) (0.044) (0.027) (0.027) Yeas (0.012) (0.023) (0.013) (0.013) (0.012) (0.008) Vehcle (0.020) (0.048) (0.024) (0.012) (0.017) (0.033) Image/Indoor (0.008) (0.007) (0.014) (0.013) (0.011) (0.007) Image/Oudoor (0.008) (0.017) (0.011) (0.005) (0.006) (0.007) Table 2: Classfcaon errors for he baselne model (SVM), AdaBoos, Baggng and he propose model generaon mehod ( HISS-based Ensemble ). The column Baggng (Sackng) refers o he case when he ensemble of models s creaed by he Baggng algorhm bu combned hrough he sackng approach usng an SVM. The same s for he column AdaBoos (Sackng). The varance of classfcaon error s lsed n parenhess. ses for mage classfcaon. The characerscs of hese seven daases are lsed n Table 1. For he mulple class daases, we nroduce he heerogeney no he daa by converng he orgnal mulpleclass problem no a bnary one. Smlar o he one-agansall approach, examples from he mos popular class are used as he posve nsances and examples from he remanng classes are assgned o he negave class. Because daa of he negave class are from mulple classes, we would expec some degree of heerogeney nsde he negave class. For he wo daases of mage classfcaon, hey boh are bnary classfcaon problems. The heerogeney of daa s due o he fac ha mages are from seven dfferen vdeo clps and each vdeo clp provdes 500 mages. Snce each vdeo clp s of dfferen ype (e.g., vared qualy n mages), we would expec ceran amoun of heerogeney whn he daa. The baselne algorhm used n hs expermen s suppor vecor machne [Burger, 1998]. In all he expermens, each ensemble mehod generaes 20 dfferen SVMs; a sackng approach [Wolper, 1992] ha also uses a SVM s employed o combne he oupus from all 20 models o form he fnal predcon of he ensemble. For each expermen, we randomly selec 70% of he daa as ranng and he remanng 30% as esng. The expermen s repeaed 10 mes and he average classfcaon error of he en runs s used as he fnal resul wh he varance of classfcaon errors. 4.2 Heerogeneous Daa Classfcaon Table 2 shows classfcaon errors for he baselne suppor Daa Se HISS EM EM (3 Clusers) (10 Clusers) Ecol 0.037(0.006) (0.021) (0.021) Pendg 0.008(0.002) (0.043) (0.023) Glass 0.161(0.044) (0.101) (0.017) Yeas (0.013) (0.013) (0.019) Vehcle (0.012) (0.068) (0.026) Image/Indoor 0.140(0.013) (0.022) (0.014) Image/Oudoor 0.088(0.005) (0.031) (0.036) Table 3: Classfcaon error for usng dfferen cluserng algorhms for model generaon. EM refers o usng Expecaon-Maxmzaon algorhm o cluser daa. vecor machne, he proposed HISS-based ensemble learnng approach, sandard Baggng and sandard AdaBoos. Frs, we can see ha he baselne model performs well comparng wh boh sandard Baggng and AdaBoos. Ths observaon ndcaes ha hese seven heerogeneous daases are raher dffcul for he sandard ensemble approaches o learn. In conras, he proposed HISS-based ensemble mehod performs beer han he baselne model and he wo sandard ensemble mehods. For he daases Glass, Vehcle, and Image/Oudoor, he mprovemen s subsanal, from 38.2% o 16.1% for Galss, 10.3% o 4.8% for Vehcle, and from 11.6% o 8.8% for Image/Oudoor. Snce he HISS-based ensemble mehod uses he sackng approach for combnng dfferen models, s dfferen from he combnaon mehod ha s used by AdaBoos and Baggng. To address hs dfference, we conduc he expermens ha apply a sackng mehod o combne he models generaed by boh Baggng and AdaBoos. The resuls are lsed n Table 2 on he rgh sde of he HISS-based approach, led as Baggng (Sackng) and AdaBoos (Sackng), respecvely. Compared hese resuls o he resuls of Baggng (Sandard) and AdaBoos (Sandard), we see ha here s no subsanal change n classfcaon errors when usng a sackng approach o combne models n ensemble learnng. For all he seven daases, he ensemble of models generaed by HISS performs he bes. The reason why a sackng approach s useful for he HISS-based model generaon mehod bu no o he oher wo s ha models generaed by he HISS-based algorhm are much more dverse han he ones generaed by boh Baggng and AdaBoos. As a resul, applyng anoher layer of classfcaon model o combne he oupus from he dsngushable models (or sackng) wll be able o ake full advanage of all he models and oban he bes performance. Based on he above dscusson, we conclude ha he HISS-based ensemble model s more effecve for classfyng heerogeneous daa han exsng ensemble approaches. 4.3 Comparson wh Oher Cluserng-based Ensemble Mehods The advanage of HISS versus he radonal cluserng algorhms s ha HISS allows each daa pon o be n mulple dfferen sraa. Thus can ensure ha he number of

6 daa pons dsrbued over each sraum s of smlar sze and suffcenly large. In hs expermen, we use boh he radonal cluserng algorhm and he proposed HISS algorhm for model generaon and see how dfferen hey are n classfyng he heerogeneous daases. To observe he effec due o he radeoff beween he number of sraa and he number of daa pons n each sraum, we consder wo dfferen numbers of sraa (or clusers) for he radonal cluserng algorhm: 10 and 3( 1/γ ). We dd no use 20 clusers n he comparson because for some daases he radonal cluserng algorhm s unable o produce he full weny clusers. The radonal cluserng algorhm used n he expermen s he probablsc EM cluserng algorhm. Smlar o he HISSbased ensemble approach, a sackng mehod s used o combne models generaed by he EM cluserng algorhm. The resuls for usng EM cluserng algorhms for model consrucon are lsed n Table 3, led EM (3 clusers) and EM (10 clusers). As suggesed by Table 3, he ncreasng number of clusers can lead o degraded performance. Ths s because a large number of clusers wll form clusers wh a small number of daa pons, whch can be nsuffcen for buldng a relable classfcaon model. On he oher hand, as already ndcaed n he prevous sudy [Derch, 2000], beng able o generae a relavely large number of models s crcal o he success of he ensemble approach. The proposed HISS algorhm can sasfy boh needs by nroducng he subsanal overlappng beween dfferen clusers. As shown n Table 3, he HISS-based mehod ouperforms he EM_cluserng-based ensemble approaches subsanally for almos all daases excep for Yeas (smlar). The mos noceable cases are Ecol and Pendg, for whch he classfcaon errors of EM-based cluserng approaches are one order more han ha of he HISS-based ensemble algorhm. Based on he above expermens and analyss, we conclude ha he HISS-based model generaon s an effecve mehod for model generaon n ensemble learnng for heerogeneous daa classfcaon. 5. Concluson and Fuure Work In hs paper, we propose and examne a new mehod for generang an ensemble of models, whch s o frs paron daa no homogeneous subses and hen creae a model for each subse. A radonal cluserng algorhm lke EM s no suable for he ask of paronng daa due o poenal sze-unbalanced clusers and he daa fragmenaon problem. To address hese wo problems, we propose a novel algorhm HISS, whch allows for daa overlappng beween dfferen clusers (sraa) and promses sze-balanced clusers. Emprcal sudes over seven dfferen heerogeneous daases have shown ha hs new HISS-based model generaon mehod performs very well for heerogeneous daa classfcaon. Currenly, he proposed HISS algorhm assumes equal sze for each sraum (cluser). One possble exenson s o examne alernaves o balance szes of clusers. For example, nsead of enforcng all he clusers o have one sze, we can consran he szes of he clusers no a specfed range o allow some flexbly n mananng hgh homogeney of clusers. References [Hargan and Wong, 1979] Hargan, J.A. and Wong, M.A., A -means Cluserng Algorhm, Appled Sascs 28: [Bremann, 1996] Bremann, L., Baggng Predcaor, Machne Learnng 26, , 1996 [Schapre and Snger, 1999]Schapre, R.E, and Snger, Y., Improved boosng algorhms usng confdence-raed predcons, Machne Learnng 37 (3): , 1999 [Celeux and Govaer, 1992] Celeux, G. and Govaer, G., A Classfcaon EM Algorhm for Cluserng and Two Sochasc Versons, Compuaonal Sascs & Daa Analyss, vol. 14, pp , 1992 [Deerch, 2000] Deerch, T.G., Ensemble Mehods n Machne Learnng. In Mulple Classer Sysems, Caglar, Ialy, 2000 [Dempser e al., 1977] Dempser, A.P., Lard, N.M., and Rubn, D.B., Maxmum Lkelhood from Incomplee Daa va he EM Algorhm, Journal of he Royal sascal Socey, Seres B, 39(1): 1-38, 1977 [Flecher, 1987] Flecher, R., Praccal Mehods of Opmzaon. John Wley and Sons, Inc., 2nd edon, [Deerch and Bakr, 1995] Deerch, T.G. and Bakr, G., Solvng Mulclass Learnng Problems va Error- Correcng Oupu Codes. Journal of Arfcal Inellgence Research, (2): , [Wolper, 1992] Wolper, D.H., Sacked Generalzaon. Neural Neworks, 5: , Pergamon Press, [Burges, 1998] Burges, C.J.C., A Tuoral on Suppor Vecor Machnes for Paern Recognon. Daa Mnng and nowledge Dscovery, 2(2): , [Wen and Frank, 2000] Wen, I.H. and Frank, E., Daa Mnng: Praccal Machne Learnng Tools wh Java Implemenaons. Morgan aufmann, [Blake and Merz, 1998] Blake, C. and Merz, C., UCI reposory of ma-chne learnng daabases. hp:// [Qunlan, 1993] Qunlan, R.J., C4.5: Programs for Machne Learnng, Morgan aufmann, San Maeo, [Qunlan, 1996] Qunlan R.J., Baggng, boosng, and C4.5. In Proceedngs of he Threenh Naonal Conference on Arfcal Inellgence (AAAI 96), [Sh and Malk, 2000] Sh, J., and Malk, J., Normalzed Cu and Image Segmenaon. IEEE Transacons on Paern Analyss and Machne Inellgence, 22(8): , 2000 [Bauer and ohav, 1999] Bauer, E. and ohav, R., An Emprcal Comparson of Vong Classfcaon Algorhms: Baggng, Boosng, and Varans. Machne Learnng, 36(1): , 1999.