CS54701: Information Retrieval

Transcription

1 CS54701: Informaton Retreval Federated Search 22 March 2016 Prof. Chrs Clfton Federated Search Outlne Introducton to federated search Man research problems Resource Representaton Resource Selecton Results Mergng Jan Chrstopher W. Clfton 1

2 Research on Resource Selecton (Resource Selecton) Goal of Resource Selecton of Informaton Source Recommendaton Hgh-Recall: Select the (few) nformaton sources that have the most relevant documents Resource selecton algorthms that need tranng data - Decson-Theoretc Framework (DTF) (Nottelmann & Fuhr, 1999, 2003) DTF causes large human judgment costs - Lghtweght probes (Hawkng & Thstlewate, 1999) Acqure tranng data n an onlne manner, large communcaton costs (Resource Selecton) Research on Resource Representaton Bg document resource selecton approach: Treat nformaton sources as bg documents, rank them by smlarty of user query - Cue Valdty Varance (CVV) (Yuwono & Lee, 1997) - CORI (Bayesan Inference Network) (Callan,1995) - KL-dvergence (Xu & Croft, 1999)(S & Callan, 2002), Calculate KL dvergence between dstrbuton of nformaton sources and user query CORI and KL were the state-of-the-art (French et al., 1999)(Craswell et al,, 2000) But Bg document approach loses doc boundares and does not optmze the goal of Hgh-Recall Jan Chrstopher W. Clfton 2

3 Language Model Resource Selecton P db Q P( Q db) P( db ) PQ ( ) DB ndependent constant 1 P Q db P q db P q G qq Calculate on Sample Docs In Language Model Framework, P(C ) s set accordng to DB Sze P C j N ^ C ^ N C j (Resource Selecton) Research on Resource Representaton But Bg document approach loses doc boundares and does not optmze the goal of Hgh-Recall Relevant document dstrbuton estmaton (ReDDE) (S & Callan, 2003) Estmate the percentage of relevant docs among sources and rank sources wth no need for relevance data, much more effcent Jan Chrstopher W. Clfton 3

4 (Resource Selecton) Relevant Doc Dstrbuton Estmaton (ReDDE) Algorthm Rel_Q() = P(rel d) P(d db ) N P(rel d) C 0 ddb ddb _samp P(rel d) SF db db Rank on Centralzed Complete DB Q f Rank otherwse CCDB (Q,d) rato Source Scale Factor SF db ^ N = N db db _samp Everythng at the top s (equally) relevant Problem: To estmate doc rankng on Centralzed Complete DB N db Estmated Source Sze Number of Sampled Docs ReDDE Algorthm (Cont) In resource representaton: Buld representatons by QBS, collapse sampled docs nto centralzed sample DB In resource selecton: Construct rankng on CCDB wth rankng on CSDB (Resource Selecton) CSDB Rankng Centralzed Sample DB.. CCDB Rankng Threshold Resource Selecton... Resource Representaton Engne 1 Engne Engne N Jan Chrstopher W. Clfton 4

5 (Resource Selecton) Experments On testbeds wth unform or moderately skewed source szes Evaluated Rankng R = k k =1 k =1 E B Desred Rankng (Resource Selecton) Experments On testbeds wth skewed source szes Jan Chrstopher W. Clfton 5

6 Federated Search Outlne Introducton to federated search Man research problems Resource Representaton Results Selecton Resource Mergng Goal of Results Mergng (Results Mergng) Make dfferent result lsts comparable and merge them nto a sngle lst Dffcultes: - Informaton sources may use dfferent retreval algorthms - Informaton sources have dfferent corpus statstcs Prevous Research on Results Mergng Most accurate methods drectly calculate comparable scores - Use same retreval algorthm and same corpus statstcs (Vles & French, 1997)(Xu and Callan, 1998), need source cooperaton - Download retreved docs and recalculate scores (Krsch, 1997), large communcaton and computaton costs Jan Chrstopher W. Clfton 6

7 Research on Results Mergng (Results Mergng) Methods approxmate comparable scores - Round Robn (Voorhees et al., 1997), only use source rank nformaton and doc rank nformaton, fast but less effectve - CORI mergng formula (Callan et al., 1995), lnear combnaton of doc scores and source scores Use lnear transformaton, a hnt for other method Work n uncooperatve envronment, effectve but need mprovement Thought (Results Mergng) Prevous algorthms ether try to calculate or to mmc the effect of the centralzed scores Can we estmate the centralzed scores effectvely and effcently? Sem-Supervsed Learnng (SSL) Mergng (S & Callan, 2002, 2003) - Some docs exst n both centralzed sample DB and retreved docs From Centralzed sampled DB and ndvdual ranked lsts when long ranked lsts are avalable Download mnmum number of docs wth only short ranked lsts - Lnear transformaton maps source specfc doc scores to source ndependent scores on centralzed sample DB Jan Chrstopher W. Clfton 7

8 SSL Results Mergng (cont) In resource representaton: (Results Mergng) Buld representatons by QBS, collapse CSDB sampled docs nto centralzed sample DB Rankng In resource selecton: Overlap Rank sources, calculate centralzed Docs scores for docs n centralzed sample DB In results mergng: Fnd overlap docs, buld lnear models, estmate centralzed scores for all docs Fnal Results Centralzed Sample DB..... Resource Selecton Resource Representaton Engne Engne N Engne 2 (Results Mergng) Experments Trec123 Trec4-kmeans 3 Sources Selected 10 Sources Selected 50 docs retreved from each source SSL downloads mnmum docs for tranng Jan Chrstopher W. Clfton 8

9 Fnal Project Self-drected fnal project You must decde what to do Frst step: Proposal What s the problem? How s t solved today? What s your approach? Why should t work? How long wll t take? (Mlestones) What s your measure for success? Delverables 34 Fnal Project: Ideas Identfy an unsolved (or poorly solved) problem Try a new soluton Take an exstng approach and try to mprove t Compare exstng approaches Reproducblty : Valdate exstng work Does t hold n dfferent condtons / data? 35 Jan Chrstopher W. Clfton 9

10 Fnal Project: Delverables Dependent on the project Wrtten report descrbng outcomes / experments (Taped) oral presentaton descrbng outcomes Include system demonstraton? System that can be tred out Runs on SSLab machnes Web accessble Other deas? 36 More on Federated Search Search Result Dversfcaton (Hong&S SIGIR 13) Problem: Lack of dversty n results E.g., several copes of the same document Key contrbuton: Metrc Need to be able to measure dversty Bulds on ReDDE and others 37 Jan Chrstopher W. Clfton 10

11 Base: R-Metrc Rankng algorthm ndependent metrc Based on top, or ranked lst, of documents R k = =1 k k E =1 B E s relevant documents n source accordng to algorthm E B s true relevant documents n source Basc dea: Replace Relevant wth a dversty metrc 38 Dversty Query has multple aspects Evaluate each aspect separately Remember somethng lke ths? Macro vs. Mcro F1 What s an aspect? Topc 39 Jan Chrstopher W. Clfton 11