Data Quality Inference

Transcription

1 Daa Qualy Inference Raymond K. Pon and Alfonso F. Cárdenas UCLA Compuer Scence Boeler Hall 4829 Los Angeles, CA (310) {rpon, ABSTRACT In he feld of sensor neworks, daa negraon and collaboraon, and nellgence gaherng effors, nformaon on he qualy of daa sources are mporan bu are ofen no avalable. We descrbe a echnque o rank daa sources by observng and comparng her behavor (.e., he daa produced by daa sources) o rank. Inuvely, our measure characerzes daa sources ha agree wh accurae or hgh-qualy daa sources as lkely accurae. Furhermore, our measure ncludes a emporal componen ha akes no accoun a daa source s pas accuracy n evaluang s curren accuracy. Inal expermenal resuls based on smulaon daa o suppor our hypohess demonsrae hgh precson and recall on denfyng he mos accurae daa sources. 1. INTRODUCTION A maor aspec of daa provenance s he ably o rack he qualy of daa as s processed by varous ransformaons, each wh an assocaed compuaonal or nrnsc daa collecon error. I s mporan o choose rusworhy daa sources when queryng over mulple daa sources [1]. Users of daa warehouses regard he qualy of nformaon as mporan and as a facor n measurng he uly of a daa warehouse [2]. Furhermore, he ncluson of daa qualy nformaon has an mpac on decsonmakng and decson-suppor sysems as well [3, 4]. Also daa conflcs, occurrng when heerogeneous daa sources are negraed, can be resolved by consderng he qualy of he daa sources nvolved [5]. The accuracy of daa can also be used o rank query resuls as well (as opposed o he relevance of query resuls o he query) [6]. Clearly, hs rusworhness and qualy nformaon should be sored as par of a daa em s provenance. The followng s a general query n whch daa qualy can be used o answer: Query 1: Gven many genomc daabases where daa has been colleced by varous means and nsuons, fnd a DNA sequence ha sasfes a condon C. In hs query, collecons of daa ses (.e., DNA sequences) have been colleced by dfferen nsrumens and/or possbly derved by varous and possbly mulple ransformaons. Each of hese nsrumens and ransformaons has a dfferen degree of relably and error. Addonally, he daa ses ha are relevan o he query may be numerous, so s necessary o rank daa ses by her qualy or Permsson o make dgal or hard copes of all or par of hs work for personal or classroom use s graned whou fee provded ha copes are no made or dsrbued for prof or commercal advanage and ha copes bear hs noce and he full caon on he frs page. To copy oherwse, or republsh, o pos on servers or o redsrbue o lss, requres pror specfc permsson and/or a fee. IQIS 2005, June 17, 2005, Balmore, MD, USA. Copyrgh 2005 ACM /05/06 $5.00. rusworhness (.e., how relable he daa ses are). However, s unclear as o where meadaa regardng daa qualy comes from. User-provded rangs of daa ses or sources can be used o rae he qualy of daa sources [1], bu s clearly a subecve measure and would requre large samples o ge any meanngful resuls. Error measures can be provded by daa sources provders along wh daa ses, bu may be naccurae, dffcul o use, ncomplee, or unrusworhy [7]. Thus, we descrbe a echnque o rank daa sources by observng and comparng he behavor (.e., he daa produced by daa sources) o rank hem n erms of her qualy. Inuvely, n our measure, daa sources ha agree wh accurae daa sources are lkely o be accurae. Furhermore, n our measure, daa sources ha have been accurae n he pas are also lkely o be accurae n he fuure. We provde some nal expermenal resuls based on smulaon daa o suppor our hypohess. The followng subsecons dscuss movaons for hs work and he relaed works. In secon 2, we descrbe our echnque for daa source rankng. In secon 3 and 4, we presen our nal expermenal resuls and possble roads of fuure research, respecvely. 1.1 Movaon We descrbe hree possble applcaon areas n whch he modelng of daa accuracy and rusworhness are mporan. Applcaon 1: Sensor neworks are becomng ncreasngly prevalen n observng wldlfe, monorng envronmenal condons, monorng of solders n he feld, and he deecon of harmful bologcal and chemcal agens, wh praccal applcaons n homeland secury [8]. The effecveness of hese sensor neworks s hghly dependen on he accuracy of he neworks, whch s a funcon of he curren baery level of he devce, nerference, and nrnsc error n daa collecon. For example, n he near fuure, solders may be equpped wh daa capurng devces, makng each solder a sensor [9], o gve feld commanders curren balefeld saus repors. Daa capured by solders may be conflcng and/or erroneous because of he human elemen nvolved n he daa capurng process. I would be advanageous o be able o deermne he more rusworhy sensors n capurng he curren suaon o fler ou nosy daa and o reduce he consumpon of resources (e.g., manpower, me, and baery-lfe). Applcaon 2: In bomedcal research, research facles frequenly collaborae wh each oher, sharng expermenal daa and resuls. In parcular, comparng genome sequences from dfferen speces has become an mporan ool for denfyng funcons of genes [10]. Ths necessaes dynamcally negrang dfferen daabases or warehousng hem no a sngle reposory. Scenss need o know how relable he daa s f hey are o base her research on. Pursung ncorrec heores cos me and 105

2 money. The obvous soluon o ensure daa qualy s curaon, bu daa sources are auonomous and as a resul sources may provde excellen relably n one area, bu no n all daa provded, and curaon slows he ncorporaon of daa. Daa provders wll no drecly suppor daa qualy evaluaons o he same degree snce here s no equal movaon for hem o and here are no sandards n place for evaluang and comparng daa qualy [7]. Thus, auomac, mparal, and ndependen daa qualy evaluaon would be needed. Applcaon 3: In nellgence gaherng effors, daa s ofen colleced from many heerogeneous daa sources, such as saelles, human asses, ranscrps, wreaps, ec. I s obvous ha each of hese daa sources have dfferen degrees of qualy and rus. And wh he mulude of daa sources o ncorporae, s currenly me-consumng o sf hrough each of hese daa sources o deermne whch he mos accurae sources are. To make he correc decsons based on he nellgence avalable n a mely manner, we wll need an auomac means o deermne accurae daa sources and o be able o deec malcous or compromsed daa sources o preven hem from nfluencng decson-makng processes. 1.2 Relaed Works There has been a sgnfcan amoun of work n he area of nformaon qualy, rangng from echnques n assessng nformaon qualy and accuracy o buldng large-scale daa negraon sysems over heerogeneous daa sources. For example, he DaQunCIS sysem [11-13] s a cooperave nformaon sysem where daa source provders are evaluaed by daa source users n a peer-o-peer sysem. Unforunaely, such a sysem reles heavly on he parcpaon of users n he revew of he qualy of daa n he sysem, whch may no be praccal n real-lfe envronmens. Users may no relably or conssen n evaluang daa sources. Oher works have aken oher approaches n modelng and capurng daa qualy. Some have developed daa models o model daa qualy bu rely on daa qualy meadaa beng avalable, such as daa sources publshng such nformaon [14-19]. Unforunaely, hese approaches rely on precse and accurae meadaa. However, such meadaa are no always avalable [7] and here s no sngle agreed upon sandard n descrbng daa lneage. Addonally, may be possble for malcous processes o corrup or spam query resuls by provdng false meadaa. There has also been work n mehodologes n assessng of he qualy of daa n daabases [20-23]. However, such mehodologes rely on human assessmen of he daa, whch s ofen me-consumng and possbly error-prone. Prevous works have assumed ha he meadaa regardng he qualy of daa s avalable, accurae, and unbased, eher publshed by he daa provders hemselves or provded by userrankngs of he daa sources. Our conrbuon s ha we do no assume ha such meadaa s avalable and relable. Raher, our auomaed approach examnes how well he daa ses produced by daa sources agree wh one anoher, and nfer he rankngs of he daa sources n erms of her accuracy. We ake an approach smlar o Google s PageRank [24]. Insead of evaluang he populary of web ses by measurng how many oher popular webses lnk o hem as n PageRank s approach, we evaluae he accuracy of daa sources by measurng how well oher daa sources agree wh he daa hey produce. Ths approach s auomaed, does no rely on possbly fauly and lmed meadaa, and does no requre human assessmen. 2. DATA SOURCE RANKING Tradonally, he rankng of query resuls was based on he relevance of a user s query. However, he qualy of he resuls could be mproved f we ncorporaed a daa qualy measure n addon o her relevance o he user s query. We wsh o do followng n general: 1. Rank he daa ses or daa sources n order of her accuraces. 2. Deermne he op-k accurae daa ses or daa sources. Ths orderng s mporan parcularly n daa negraon sysems, where here are numerous daa sources avalable of varyng accuracy ha changes dynamcally across me. Ideally, gven a query, we would lke o conac each daa source; however, hs may be prohbvely expensve f here are budge consrans such as me and nework resources. Such applcaons can be found n sensor neworks, where baery-lfe s lmed, and nellgence-gaherng effors, where manpower and me are lmed. Thus, would be advanageous o deermne he mos accurae se of daa sources, so ha hey can be conaced n answerng a query. Addonally, hs mehodology could be used for denfyng malcous or compromsed daa sources ha are aempng o feed false nformaon no he daa negraon sysem. We provde he followng framework for rankng he accuracy or rusworhness of daa sources based on observng and comparng daa source behavor whou any a pror knowledge of her relave accuraces o help solve he problem. In our model, we assume ha schema and daa heerogeney have been reconcled, whch s beyond he scope of hs work. 2.1 General Framework Le D be a se of daa sources. A daa source d! D generaes a able T ( k, v ) for a query Q where s he me ndex, k s he key column, and v s he value column of he able. We wan o derve a merc A![0,1] ha measures he relave accuracy of daa source d a me such ha A < A f d s less accurae han d a me. We defne such a merc as he weghed average of he prevous accuracy esmae a me ndex! 1 and he accuracy esmae derved by observng he daa generaed by daa sources n D: A = h()a!1 + (1! h())c(,) (1) The nuon behnd A s ha a daa source s accuracy should be a funcon of s pas accuracy (.e., repuaon) and s curren behavor. The funcon h(), where 0! h( )! 1, s he hsorcal wegh funcon ha deermnes he conrbuon of he accuracy esmae a he prevous me ndex. The nuon behnd he hsorcal componen of he accuracy measure s ha a daa source ha has been accurae (or naccurae) n he pas should also be accurae (or naccurae) n he near fuure. For smplcy, he above hsorcal componen assumes a Markovan behavor n he evoluon of he daa sources, where he accuracy a me s only dependen on he value a -1. However, wll be neresng o see f we can mprove he qualy of our esmaon by akng no 106

3 accoun a sldng wndow of sze w: [A, A!1,, A!w ]. Thus, he hsorcal componen would conss of a weghed sum of all accuracy esmaes whn he las w me ndexes, where each esmae s weghed wh a decayng wegh funcon. The decayng wegh funcon would assgn a hgher wegh o more recen esmaes han older esmaes. A sldng wndow verson of he equaon (1) would be of he followng form, where w(, 1) s he decayng wegh funcon: A!1 = h() " w(,!1) A + (1! h())c(,) (2) =!w However, we leave hs ssue o fuure research, where we wll sudy he mos approprae decayng wegh funcon and he opmal sldng wndow sze. The coheson funcon c(,) deermnes he new accuracy esmae by observng daa generaed by daa sources n D a he curren me ndex and how well each daa source agrees wh one anoher. The coheson funcon c(,) ha we propose s he followng: c(,) = f (,) + (1! f (,)) D!1 # a(,,)c(,) (3) d "D!{d } The funcon a(,,) s he agreemen funcon, whch oupus 0 when daa sources d and d are n srong dsagreemen regardng he daa n T and T, and oupus 1 when d and d srongly agree, and values beween 0 and 1 for oher levels of agreemen. The nuon behnd c(,) s: If a daa source agrees wh an accurae daa source, should also be accurae. If a daa source agrees wh an naccurae daa source, should also be naccurae. A daa source has a probably f(,) of beng absoluely accurae ndependen of any agreemen/dsagreemen wh he oher daa sources. Thus, gven a sysem of equaons of D equaons and D varables, s possble o deermne c(,) for all d! D. The funcon f(,) s he dampenng facor funcon (smlar o ha defned n Google s PageRank algorhm [24]). In addon o beng he probably ha a daa source d s absoluely accurae ndependen of s agreemen wh he oher daa sources, he funcon f(,) wll preven he soluon o he sysem of equaons from conssng of enrely zeroes for all c(,). 2.2 Agreemen Funcons There are several possble defnons for a(,, ), such as he upleoverlap funcon, whch measures he proporon of uples n approxmae agreemen (whn some allowable dfference! ) n he se of uples whose key values are generaed by boh daa sources d and d : T!" T T. k = T. k T. v! T. v upleoverlap(,, ) = T!" T T. k = T. k Anoher possble defnon for a(,,) s he cosneoverlap funcon, whch measures he complemen of he cosne dsance (4) of wo ses of daa over he same key values generaed by d and d : T V (,, ) V (,, ) cosneoverlap(,, ) = V (,, ) V (,, ) (5) The vecor V(,,) can be roughly defned as V (,, ) ( =! T T )!", excep s an ordered vecor n T. v T. k = T. k whch he values sored n he vecor are ordered by her correspondng key values n T. There s also a Eucldan-based funcon for a(,, ), whch we wll dscuss n furher deal laer. Gven hs sysem of D equaons and D varables, we can arrange he equaons o he followng form: A() s defned as he followng marx: A( ) * C( ) = F( ) (6) " D! 1 # $ a(1,2, )! a(1, D, ) f ( )! 1 $ $ D! 1 f ( )! 1$ a(2,1, ) a(2, D, ) A( ) = f ( )! 1 D! 1 $ $ " " # " $ $ D! 1 a( D,1, ) a( D,2, )! $ & f ( )! 1 ' C() and F() are also defned as he followng marces:! c(1, ) "! 1" # c(2, ) $ # 1 $ C( ) = # $, F( ) = f ( ) # $ #! $ #! $ # $ # $ c( D, ) & 1& The soluon o equaon (6), C(), s a vecor where each enry C() esmaes he accuracy of daa source d. The marx A() can also be normalzed wh respec o maxmum or sum of he enres n each of he rows (horzonal normalzaon) or n each of he columns (vercal normalzaon). We can horzonally normalze he marx A() by performng he followng dvson on every enry A(), n row, column, excep for enres where = : A'( ), A( ) =, Hor(, ) Hor(,) can eher be he sum or he maxmum value of all he enres n row excludng he enry A(),. We can also smlarly defne a funcon Ver(,) for vercal normalzaon A( ), ( A( ), = ) o be eher he sum or he maxmum Ver(, ) value of all enres n column excludng he enry A(),. Gven our normalzaon echnques, we can now dscuss n furher deal he Eucldan-based funcon for a(,, ) menoned brefly before. Because Eucldan-dsance s unbounded, normalzaon would be requred o descrbe he amoun of overlap or agreemen. We defne he Eucldan-based funcon eoverlap for a(,, ): eoverlap(,, ) = 1! eds '( V (,, ), V (,, )) (7) The funcon eds ' s smply he Eucldan dsance of he vecors V(,,) and V(,, ), normalzed n a smlar manner as descrbed above. 107

4 (a) (b) (c) Fgure 1: The precson and recall for denfyng he op 10 mos accurae daa sources wh (a) he Eucldan-based agreemen funcons, (b) he Cosne-based agreemen funcons, and (c) he Overlap-based agreemen funcons wh = 0.1. Gaussan dsrbuon wh a sandard devaon equal o ha of he daa source s error value and an average equal o ha of he acual daa em s value, essenally perurbng each daa em s value wh daa source s error value. Daa sources wh large error values wll generally generae values farher away from he acual value han daa sources wh smaller error values. We ran a oal of fve runs, conssng of fve eraons, and averaged he resuls. 3. EXPERIMENTAL RESULTS We hypohesze ha he above framework can be used as a sprngboard n solvng he general problem of denfyng accurae daa sources. To do so, we wll need o denfy adequae h(), a(,,), and f(,) funcons hrough expermenaon. Our nal expermens examne he coheson funcon c(,) wh a dampenng facor f ndependen of me (.e., he probably of a daa source beng absoluely accurae ndependen of all oher daa sources s consan), and excludng ncorporaon of he hsorcal componen. As a resul, he combnaon of equaons (1) and (3) reduces o he followng: A = c(,) = f + 1 f a(,,)c(,) n 1 d D {d } Fgure 1 shows he precson and recall of he varous agreemen funcons and normalzaons as he dampenng facor f s vared. Noe ha he overlap-based funcons are usng a dfference margn = 0.1. The fgure clearly shows ha he dampenng facor has very lle effec n denfyng he op 10 mos accurae daa sources. However, he fgure does show ha he vercal normalzaon wh respec o he maxmal value of he column yelds he bes performance. Addonally, he fgure shows ha he overlap-based funcons perform he worse, wh he cosnebased funcons performng well and he Eucldan-based funcons performng even beer wh a precson and recall of over 90. The overlap-based funcons suffer from havng a fxed allowable dfference margn ha s dffcul o esmae whou knowng he naure of he daa and he daa sources a pror. The cosne-based funcons perform beer han he overlap-based funcons because no such assumpon s needed bu does no accuraely capure he amoun of dsance/overlap as Eucldanbased funcons do. (8) We mplemened a Java prooype, usng JAMA (Java Marx Package) [25] for solvng he sysem of equaons, and expermened on smulaon daa conssng of 100 daa sources, each producng 20 dfferen uples, each conssng of a key (of ype neger) and a value (of ype double). In each run, a daa se, conssng of 20 keys and values randomly assgned o each key wh a unform dsrbuon, represen he acual daa ha each daa source wll aemp o repor. Addonally, n each run, each daa source was randomly assgned posve error values accordng o a Gaussan dsrbuon wh an average of 0 and a sandard devaon of 1.0. For each run, we ran fve eraons, where each daa source produced a daa se, conssng of values for each key, where each value s randomly generaed wh a To summarze, Fgure 2 shows he performance of he hree agreemen funcons wh varous normalzaons usng a fxed 108

5 Fgure 2: The precson and recall for denfyng he op 10, 15, 20, 25, and 50 mos accurae daa sources wh a dampenng facor of 0.5 dampenng facor of 0.5 (snce curren resuls do no defnvely ndcae he bes value for f, we seleced a md-range value for f). The fgure clearly shows ha he Eucldan-based agreemen funcon wh vercal max and sum normalzaons performs he bes wh a precson and recall of over FUTURE WORK One of he caveas of he curren echnque s ha reles on daa sources reporng on he same se of daa ems. Ofen, may be he case where daa sources wll repor abou dfferen daa ems. Fuure sudy wll have o be done o evaluae he curren echnque s effecveness over ncomplee and heerogeneous daa sources. Addonally, he curren echnque may suffer from possbly expensve pollng of all daa sources. In fuure work, we wll need o devse an effcen and nellgen samplng echnque o allevae such a problem whle sll prevenng he saleness of esmaes. One obvous possbly s o use he daa gahered durng a query (whch s essenally free from he pon of vew of he qualy esmaor snce such a cos wll need o be ncurred anyway o answer he query) o esmae a new relave accuracy measure han can be used for he nex query. However, only daa sources wh hgh accuracy esmaes wll have her esmaes updaed and he esmaes of daa sources of low accuracy wll become sale, snce accurae daa sources are he only daa sources conssenly beng probed snce hey are seleced o he answer he query. Thus, we wll need o explore addonal samplng echnques [26], such as pollng for only small subses of daa from a maory of daa sources, o solve hs problem and o be able o assocae a confdence merc n he rankng generaed by our mehodology. Addonally, compung he soluon o a se of n c(,) equaons wh n varables may be compuaonally expensve f n s very large. Thus, we wll also explore echnques o speed up hs compuaon wh an accepable margn of error, such as usng an erave approach, usng old c(,-1) values for compung he new c(,) value n equaon (3). Fgure 3 shows promsng prelmnary resuls regardng he performance of he erave soluon, ndcang ha we can arrve o a reasonably good esmaon n very few eraons and ha he dampenng facor has some effec on how fas we can arrve o a soluon. We use an nal esmae of c(,-1) = 1 for all daa sources and use he Eucldan-based agreemen funcon wh vercal sum normalzaon whle varyng he dampenng facor. Fuure work wll furher explore he effec of he dampenng facor. In hs prelmnary sudy, we randomly assgn error values o he Fgure 3: Performance of aanng an erave soluon usng c(,-1) = 1 and he Eucldan-based agreemen funcon wh vercal sum normalzaon. 109

6 daa sources wh a Gaussan dsrbuon. Addonal research wll nclude furher sudy on how well our cohesve funcon performs wh oher probably dsrbuons, such as unform dsrbuons. We also hypohesze ha such a echnque can be used o auomacally denfy fauly or falng daa sources dynamcally, such as a sensor or an nellgence asse. We wll need o expermen wh he hsorcal componen of our accuracy measure. We wll sudy how robus and reacve our accuracy measure wll be when he accuracy of daa sources becomes dynamc, as opposed o beng sac as n he case of hs prelmnary sudy. Alhough we have expermened wh an overlap-based funcon usng a dfference margn ε = 0.1 and could have used oher values for ε o see he effec on he precson and recall of denfyng he op-k mos accurae daa sources, he resuls ndcae ha ha he overlap-based funcon performs poorly compared o he Eucldan and cosne-based funcons wh hs value for ε. Anoher value for ε would have probably been beer, bu we hypohesze ha he opmal ε s dependen upon he doman applcaon of he daa. In laer work, we wll examne he effec of ε when real-lfe daa (e.g., sensor daa) becomes readly avalable. Currenly, our accuracy measure evaluaes he accuracy of daa sources based a sngle doman of daa (.e., a sngle opc). However, daa sources may provde daa for mulple domans (.e., mulple opcs) and may be more accurae n one doman han anoher. There are wo possble audes n approachng hs problem. A suspcous aude would suspec all daa (regardless of opc) provded by a daa source f a daa source conradcs a more rusworhy daa source. A rusng aude would only suspec a mnmal se of daa (.e., daa from he conradcng opc) ha conradcs a more rusworhy daa source, whch s a smlar aude aken n [27]. Fuure research wll examne how hese audes can be ncorporaed no he overall accuracy measure. Fgure 4: Nework of agreeng daa sources We also envson ha hs echnque can be used o denfy communes of daa sources n whch members of he communy share common belefs. In Fgure 4, a graph generaed wh JUNG (Java Unversal Nework/Graph Framework) [28] conssng of 50 nodes, each represenng a daa source, are conneced by edges, whose lenghs are he Eucldan-dsance of he daa ses generaed by he connecng nodes. I s clear from he graph ha nodes ha are n hgh agreemen wh one anoher are clusered very closely wh each oher; whereas, oulers n he graph dsagree wh he cluser and can be consdered as naccurae. Fuure work wll nclude sudes how cluserng echnques can be used o denfy communes of daa sources, such as ha from socal nework analyss [29]. 5. CONCLUSION We have presened an auomaed echnque for nferrng he qualy of daa sources whou he luxury of meadaa. Our man conrbuon s a framework o capure he hsorcal accuracy of daa sources and he relaonshp of daa sources n how well hey agree wh one anoher (.e., he cohesve funcon). Our second conrbuon s a prelmnary sudy of he cohesve funcon, examnng he precson and recall of denfyng he op-k mos accurae daa sources wh varous agreemen funcons and normalzaons. We have shown ha he Eucldan-based agreemen funcon vercally normalzed performs he bes. We have also denfed several sgnfcan challenges and fuure roads of research, ncludng performance opmzaons, explorng varous samplng echnques, developng robus ye reacve accuracy esmaons, and denfyng communes of daa sources. 6. ACKNOWLEDGMENT The auhors would lke o hank Roderck Son, from he UCLA Medcal Imagng Informacs Group, Terence Crchlow and Davd Buler from he Lawrence Lvermore Naonal Laboraory, and he anonymous revewers for her nvaluable nspraon and npu for hs work. Ths work s parally funded by he Naonal Foundaon Gran # IIS REFERENCE [1] D. Buler, M. Coleman, T. Crchlow, R. Fleo, W. Han, C. Pu, D. Rocco, and L. Xong, "Queryng mulple bonformacs nformaon sources: can semanc web research help?" SIGMOD Record, vol. 31, pp , [2] A. Rudra and E. Yeo, "Issues n user percepons of daa qualy and sasfacon n usng a daa warehouse-an Ausralan experence," presened a 33rd Annual Hawa Inernaonal Conference on Sysem Scences, [3] I. N. Chengular-Smh, D. P. Ballou, and H. L. Pazer, "The mpac of daa qualy nformaon on decson makng: an exploraory analyss," IEEE Transacons on Knowledge and Daa Engneerng, vol. 11, pp , [4] R. A. Dllard, "Usng daa qualy measures n decsonmakng algorhms," IEEE Exper, vol. 7, pp , [5] F. Naumann, "From daabases o nformaon sysems - nformaon qualy makes he dfference," presened a he Inernaonal Conference on Informaon Qualy (IQ 2001), Cambrdge, MA, [6] M. Gerz, M. T. Ozsu, G. Saake, and K. U. Saler, "Repor on he Dagsuhl semnar: 'daa qualy on he web'," SIGMOD Record, vol. 33, pp ,

7 [7] T. Crchlow, L. Lu, D. Buler, D. Rocco, and C. Pu, "Towards Auomac Dscovery and Idenfcaon of Bonformacs Web Inerfaces," [Onlne] Avalable: hp://srus.cs.ucdavs.edu/dagsuhl03/presenaons/ CrchlowTerence.Sldes.pp, [8] V. Kumar (edor), "Specal Issue on Sensor Nework Technology and Sensor Daa Managemen," SIGMOD Record, vol. 32, [9] F. Donovan, "Army o deploy hand-held devces o make every solder no a sensor," [Onlne] Avalable: hp:// se_sory.sp?d=news/arm04294.xml, [10] F. S. Collns, E. D. Green, A. E. Gumacher, and M. S. Guyer, "A vson for he fuure of genomcs research," Naure, vol. 422, pp , [11] M. Mecella, M. Scannapeco, A. Vrgllo, R. Baldon, T. Caarc, and C. Ban, "Managng daa qualy n cooperave nformaon sysems," n Lecure Noes n Compuer Scence 2519, 2002, pp [12] M. Scannapeco, A. Vrgllo, C. Marche, M. Mecella, and R. Baldon, "The DaQunCIS archecure: a plaform for exchangng and mprovng daa qualy n cooperave nformaon sysems," Informaon Sysems, vol. 29, pp , [13] L. D. Sans, M. Scannapeco, and T. Caarc, "Trusng daa qualy n cooperave nformaon sysems," presened a CoopIS 2003, [14] J. Wdom, "Tro: a sysem for negraed managemen of daa, accuracy, and lneage," presened a CIDR 2005, Pacfc Grove, Calforna, [15] G. A. Mhala, L. Raschd, and M.-E. Vdal, "Usng qualy of daa meadaa for source selecon and rankng," presened a Thrd Inernaonal Workshop on he Web and Daabases, WebDB'2000, Dallax, TX, [16] G. A. Mhala, L. Raschd, and M.-E. Vdal, "Source selecon and rankng n he websemancs archecure usng qualy of daa meadaa," Advances n Compuers, vol. 55, pp , [17] M. Gerz, "Managng daa qualy and negry n federaed daabases," presened a IFIP TC11 Workng Group 11.5, Second Workng Conference on Inegry and Inernal Conrol n Informaon Sysems: Brdgng Busness Requremens and Research Resuls, [18] F. Naumann, J. C. Freyag, and U. Leser, "Compleeness of negraed nformaon sources," Informaon Sysems, vol. 29, pp , [19] F. Naumann, "Qualy-Drven Query Answerng for Inegraed Informaon Sysems," n Lecure Noes n Compuer Scence, G. Goos, J. Harmans, and J. v. Leeuwen, Eds. Berln, Germany: Sprnger-Verlag, 2002, pp [20] A. Moro and I. Rakov, "Esmang he qualy of daabases," presened a 1996 Conference on Informaon Qualy, Cambrdge, MA, [21] M. Bobrowsk, M. Marre, and D. Yankelevch, "A homogeneous framework o measure daa qualy," presened a IQ 1999, Cambrdge, MA, [22] B. Pernc and M. Scannapeco, "Daa qualy n web nformaon sysems," presened a ER 2002, [23] Y. W. Lee, D. M. Srong, B. K. Kahn, and R. Y. Wang, "AIMQ: a mehodology for nformaon qualy assessmen," Informaon Sysems, vol. 29, pp , [24] S. Brn and L. Page, "The anaomy of a large-scale hyperexual web search engne," presened a 7h World Wde Web Conference (WWW7), [25] J. Hckln, C. Moler, P. Webb, R. F. Bosver, B. Mller, R. Pozo, and K. Remngon, "JAMA: Java Marx Package," [Onlne] Avalable: hp://mah.ns.gov/avanumercs/ama/, [26] J. Cho and A. Noulas, "Effecve Change Deecon usng Samplng," presened a VLDB Conference, Hong Kong, Chna, [27] L. Cholvy and C. Garon, "Queryng several conflcng daabases," presened a ECSQARU-03 Workshop Uncerany, Incompleeness, Imprecson, and Conflc n Mulple Daa Sources, Aalborg, [28] Jung Framework Developmen Team, "JUNG: Java Unversal Nework/Graph Framework," [Onlne] Avalable: hp://ung.sourceforge.ne/ndex.hml, [29] S. Saab, P. Domngos, P. Mka, J. Golbeck, L. Dng, T. Fnn, A. Josh, A. Nowak, and R. R. Vallacher, "Socal Neworks Appled," Inellgen Sysems, IEEE [see also IEEE Exper], vol. 20, pp ,