There s a hole in my case-base!

Size: px

Start display at page:

Download "There s a hole in my case-base!"

Elisabeth Ross
5 years ago
Views:

1 There s a hole in my case-base! Barry Smyth Smart Media Institute University College Dublin Elizabeth McKenna Paul Cotter Lorraine McGinty Rachael Rafter Maria Angela Ferrario Keith Bradley : : Padraig Cunningham Mark Keane David McSherry David Aha David Leake David Wilson Derek Bridge : : 28 October

2 Performance models can be constructed Competence Efficiency & Quality A basis for new CBR solutions New insights into the CBR process new solutions 28 October A Case Competence Model for CBR Local Global Competence Related Applications Case-Base Editing, Case Retrieval, Authoring Support The Case Discovery Problem Competence-Guided (Active) Case Discovery Conclusions 28 October

3 Performance(CBR) f (CB,? ) Efficiency Competence Quality 28 October Cases as a primary source of competence Case-Base as a representative of the target problem space A Formal Model of Competence Intuition Cases make differing competence contributions Holes in the case-base can represent lost competence How can we identify competence rich and poor cases? 28 October

4 The Top-Level / Coverage Problem Space Cases & Target Problems Competence Model The Representativeness Assumption Coverage & reachability sets Local Global Competence 28 October Coverage Set of a Case, c The set of cases that c can solve CS(c) = { c CB Solves(c,c )} c CS(c) Competence Implications Cases with large coverage sets make a significant competence contribution 28 October

5 Reachability Set of c The set of cases that can solve c RS(c) = { c CB Solves(c,c)} Competence Implications Cases with small reachability sets make a significant competence contribution c RS(c) 28 October Related Set of c The set of cases that can solve or be solved by c Union of coverage and reachability sets RL(c) = CS(c) RS (c) c RL(c) 28 October

6 Local vs Global Competence Global competence as the sum of local contributions C1 C1 makes a small but unique competence contributions Overlapping competence contributions Compare the coverage sets of C!, C2, C3, C4 C3 C4 Compare competence contributions How to Model Competence Interactions? C2 C2,C4 make no unique contributions 28 October Category-Based Model Smyth & Keane (IJCAI 95) Look for important and common patterns of coverage C1 Pivotal Competence Categories Pivotal Spanning Critical Spanning C3 C4 Support Auxiliary Irrelevant C2 Developments Zhu & Yang (IJCAI 99) Auxiliary 28 October

7 Shared Coverage Related cases exhibit overlapping competence contributions (related sets) SharedCoverage(c,c ) iff RL(c) RL(c ) {} 28 October Competence Groups Maximal groups of cases exhibiting shared coverage The Fundamental Unit of Competence By definition, each group makes a unique contribution to competence Group competence is additive 28 October

8 !"# Footprint Cases Subset of group cases with equivalent combined competence contributions Footprint Set Minimal set of group footprint cases Intuitively, the competence contributions of a group are based on the competence characteristics of its footprint set 28 October !$ Local vs Global Coverage CS, RS ignore the competence characteristics of nearby cases Relative Coverage Competence contribution of a case relative to its neighbours Σ RC(c) = 1/ RS(c ) c CS(c) c RC(c) = 1/4 + 1/2 + 1/2 + 1/2 = 7/4 28 October

9 Group Coverage The coverage of a competence group is the sum of the relative coverage values of its footpring cases Case-Base Coverage The coverage of the case-base is then the sum of the group coverage values CB Coverage = October " Group Evolution Groups Pivots Footprints Evolutionary Phases Creation Competence Inflation Coalescence Saturation Cases Creation Inflation Coalescence Saturation 28 October

10 Prediction & Evaluation Local vs Global Comparing case-bases Learning & Maintenance Case-Base editing Redundancy detection Retrieval Competence-Guided Retrieval Case Authoring Support Competence visualisation Identification of competence rich or poor regions Redundancy detection Case Discovery Competence holes Discovering competent cases 28 October " Predicting Case-Base Competence Correlation studies (> 0.8) Comparing Case-Bases Growth phases Case ratios, etc Predidcted Competence True Predicted True Competence Case-Base Size 28 October

App Intel, 2001 Group Coverage 28 October 2003 21 hot-spots and gaps Redundancy detection Redundancy

11 %"& Authoring Support Current tools Competence-Guided Visualization Identification of competence hot-spots and gaps Group Size Redundancy detection J. App Intel, 2001 Group Coverage 28 October %"& Authoring Support Current tools Competence-Guided Visualization Identification of competence hot-spots and gaps Redundancy detection Redundancy Group Size Saturation Competence Gaps Group Coverage 28 October

12 '( Traditional Editing CNN, ENN, RENN IBL (Aha et al) DROPx (Wilson & Martinez) Competence-Guided Footprint Deletion (IJCAI 95) ICF (Brighton & Mellish 99) Zhu & Yang (IJCAI 99) Leake & Wilson (EWCBR2k) 28 October '( Footprint Selection Methods McKenna & Smyth (ECAI 00, EWCBR2k) A family of hybrid, competence guided editing methods Results UCI ML classification data sets Superior edited case-base sizes for all family members without compromising case-base competence (classification accuracy) 28 October

13 ) Footprint-Based Retrieval Stage 1: Search CB footprint for reference case Stage 2: Search related set of reference case for best case Reference Locate reference case Target Results Comparable competence, quality, and optimality characteristics to brute-force retrieval Superior efficiency Access related set of reference case Locate nearest case from related set 28 October ) Incremental Footprint-Based Retrieval Stage 1: Search CB footprint for k nearest reference cases Stage 2: Search related set of reference cases for best case Results Comparable efficiency characteristics to Footprint-Based Retrieval Optimal competence and quality for suitably large values of k ( 2-5) 28 October

14 * $ The Problem & Related Work The traditional stance What is a Competence Hole? Interestingness Properties Types of Competence Holes Active Case Discovery Identifying and Filling Interesting Holes a competence-guided solution Evaluation 28 October $ The Traditional Stance Machine learning (data mining) focus: discovering rules and regularities within data CBR focus: discovering competent and redundant cases with casebases (case-base editing) What s Missing? What about discovering empty areas or holes within a database or case-base? How can we recognise and propose cases to fill these holes? Case discovery is NOT case selection 28 October

15 ) + Maximal Hyper-Rectangle Liu et al (IJCAI 1997) Discovery of maximal hyper-rectangles CaseMaker David McSherry (EWCBR 2000) The case recognition problem 28 October ,- What is a competence hole? Any uncovered region of the target space What makes a competence hole interesting? Size of the hole? Relevance to target problems 28 October

16 *, Type 1 Insufficient cases within the case-base. Lost coverage. Type 2 Due to domain constraints impossible value combinations. No lost coverage 28 October '!" " Discovering Interesting Holes The use of the competence model as a guide for discovering interesting holes Generating Covering Cases Using the related sets of cases as a basis for new and uncovered cases The Result Generation of cases with higher unique competence contributions 28 October

17 !.(/ Methodology Need a basis from which to derive a new case description Generate a new case from the feature values of the related sets of paired cases Case Pair Nominal Features Most frequent value Continuous Features Mean value Related Sets 28 October !.(/ Random Min Generate new case description based on two randomly chosen cases. Generate new case description based on the two closest cases in the case-base. Max Generate new case description based on the two least similar cases in the case-base. 28 October

18 *, Methodology Competence groups that are close to each other may ultimately merge into a single group The missing cases are competence rich spanning cases Search for new spanning cases in the regions between adjacent competence groups G spanning cases H 28 October *, Boundary Cases Each pair of groups G, H has a corresponding pair of boundary cases, g H, h G ( ) with maximal similarity Each group has a set of n-1 boundary cases corresponding to the n-1 other groups in the case-base G g I g H h G H h I I i G i H 28 October

19 *, Nearest Neighbour Groups Each group, G, has a nearest neighbour group, H, such that the boundary pair of G, (g H, h G ), displays maximal similarity Interesting Holes For each group we can search for new spanning cases between it and its nearest neighbour group in the region of the boundary pair G h G g H g I H h I I i G i H 28 October !.(/ Boundary Method Generate a new case from the feature values of the related sets of the boundary pair cases G Generate spanning case for the parent groups of the boundary pair Related Sets H 28 October

20 0" Test Data Travel Case-Base (1400+) from AI-CBR Property Case-Base (500+) from ML Repository Split cases into case-base (50%) and unseen test cases (50%) (x 100) Comparisons Boundary, Random, Min, Max generation methods 28 October " Evaluation Methodology For each case-base generate new candidate spanning cases for each pair of nearest neighbour groups Locate the test case nearest to the new candidate case Evaluate the coverage characteristics of this test case wrt other test cases (overall coverage vs new coverage) G candidate spanning case H coverage set nearest test case 28 October

21 1"*&. */ Method For each case-base measure the number and size / diameter (similarity between boundary pairs) of each hole Results Hole diameter reducing with increasing case-base size Quantity of Holes Quantity Diameter Hole Diameter Case-Base Size 28 October "*&./ Method For each case-base measure the number and size / diameter (similarity between boundary pairs) of each hole Results Hole diameter reducing with increasing case-base size Quantity of Holes Quantity Diameter Hole Diameter Case-Base Size 28 October

22 ,. */ Method For each case-base measure the coverage (number of cases covered) of the test case nearest to the proposed spanning case Note cumulative coverage Results Boundary method generates poor case coverage Redundancy? Overall Coverage Boundary Random Min Max Case-Base Size 28 October ,./ Method For each case-base measure the coverage (number of cases covered) of the test case nearest to the proposed spanning case Note cumulative coverage Results Boundary method generates poor case coverage Redundancy? Overall Coverage Boundary Random Min Max Case-Base Size 28 October

23 2. */ Method For each case-base measure the new coverage (number of targets covered) of the test case nearest to the proposed spanning case total coverage existing coverage Note cumulative coverage New Coverage Boundary Random Min Max Results Boundary method generates superior target coverage Case-Base Size 28 October / Method For each case-base measure the new coverage (number of targets covered) of the test case nearest to the proposed spanning case total coverage existing coverage Note cumulative coverage Results Boundary method generates superior target coverage New Coverage Boundary Random Min Max Case-Base Size 28 October

24 !. */ What is the gain in new coverage due to the Boundary method? Unique Coverage (Min/Max/Random) Unique Coverage (Boundary) Results Boundary method leads to cases that have up to 15 times the new coverage characteristics of cases generated by min/max/random methods New Coverage Gain Random Min Max Case-Base Size 28 October !./ What is the gain in new coverage due to the Boundary method? Unique Coverage (Min/Max/Random) Unique Coverage (Boundary) Results Boundary method leads to cases that have up to 15 times the new coverage characteristics of cases generated by min/max/random methods New Coverage Gain Random Min Max Case-Base Size 28 October

25 " CBR Competence Models Motivations Category-Based vs. Group-Based Models Applications Case-Base Editing, Visualisation, Case Retrieval The Case Discovery Problem A Competence-Guided Case Discovery Strategy 28 October #""+ Beyond Competence Unified Performance Models Competence vs Quality vs Efficiency (Leake & Wilson, EWCBR2k) Beyond Cases CBR Knowledge Containers Case Knowledge vs Adaptation Knowledge vs Similarity Knowledge 28 October

Reinforcement Learning (1): Discrete MDP, Value Iteration, Policy Iteration

Reinforcement Learning (1): Discrete MDP, Value Iteration, Policy Iteration Piyush Rai CS5350/6350: Machine Learning November 29, 2011 Reinforcement Learning Supervised Learning: Uses explicit supervision