DECISION TREE INDUCTION

Transcription

1 CSc-215 (Gordon) Week 12A notes DECISION TREE INDUCTION A decision tree is a graphic way of representing certain types of Boolean decision processes. Here is a simple example of a decision tree for determining whether or not to serve white wine with a meal: This tree has Boolean leaves and corresponds to the expression: Type of food being served? ((MEAT CHICKEN) FISH) MEAT FISH VEGGIES Decision Tree Induction refers to algorithms that derive a decision tree from sample data. BEEF Type of meat? CHICKEN ID3 (Quinlan, 1983) This is a very simple decision tree induction algorithm. Many other, more sophisticated algorithms are based on it. It is based on the following: The tree is built from the top (root) down to the leaves. At each stage, it selects the feature that provides the most information gain. Information gain reduction in entropy. The entropy H in a data set s is defined as: H(s) = -P 1 log 2(P 1) - P 0 log 2(P 0) where P 1 is the proportion of positive (true) examples, and P 0 is the proportion of negative () examples. Note that Entropy = 0 when all examples are positive (or negative); Entropy = 1 when 50% pos. and 50% neg. EXAMPLE: Consider the following training data set: FILM COUNTRY BIGSTAR GENRE SUCCESS 1 SciFi TRUE 2 Comedy FALSE 3 Comedy TRUE 4 Italy Comedy TRUE 5 Italy SciFi FALSE 6 Italy Romance FALSE 7 England Comedy FALSE 8 England SciFi FALSE 9 Italy Comedy TRUE 10 Comedy TRUE attributes (Of course this data set is rather silly, and much too small to draw meaningful inferences. But it will allow us to demonstrate the steps in the ID3 algorithm)

2 Let s apply ID3 to the data set above, to build a decision tree for determining whether a movie will be a success. First, note that in this example, the original entropy = 1 (50% are TRUE [a success], and 50% are FALSE [not a success]). For ID3, we try to identify which attribute provides the greatest reduction in entropy. First, try COUNTRY: H() = -¾ log 2(¾) - ¼ log 2(¼) = H(Italy) = 1 (by observation, because half of the Italy cases are TRUE and half are FALSE) H(England) = 0 (all are FALSE) Information gain = 1 - (W H), where W is the proportion of data in that category. Thus, for COUNTRY, information gain = 1 [(.4 x.811) + (.4 x 1) + (.2 x 0)] = Now, try BIGSTAR: H() = H() = 0.86 Information gain = 1 [(.7 x.9852) + (.3 x.86)] = Now, try GENRE: note, in these cases we allow (0) log2(0) = 0 H(SciFi) = 0.86 H(Comedy) = 0.86 H(Romance) = 0 Information gain = 1 [(.3 x.86) + (.6 x.86) + (.1 x 0)] = So, COUNTRY provides the most entropy reduction (information gain), and is placed at the top of the tree. Thus far, our tree looks like this: COUNTRY? 3 true 1 2 true 2 2 this one is done! How does the algorithm now proceed? For each leaf, do the following: If all examples in the leaf have the same value (such as, above), return that value. It s done! If there are no examples in the leaf, return some pre-determined default value for that leaf. If there is a mixture of values, and no remaining attributes to consider, return the majority answer. Otherwise, choose an attribute that hasn t yet been used, and split it using ID3 again. For example, we could continue by splitting the leaf node, as follows:

3 Remaining (unused) attributes are BIGSTAR and GENRE. For BIGSTAR: For GENRE: H() = -⅓ log 2(⅓) - ⅔ log 2(⅔) = 0.86 H() = 0 information gain = 1 [(.75 x.86) + (.25 x 0)] = H(Comedy) = 0 H(Romance) = 0 H(SciFi) = 0 information gain = 1 [(.5 x 0) + (.25 x 0) + (.25 x 0)] = 1 So, we would choose GENRE, and the tree now looks like this: COUNTRY? 3 true 1 GENRE? Comedy Romance SciFi true Our tree is almost done. We still have to finish this node. If we continue the process, we eventually derive the complete decision tree, which is as follows: COUNT RY? BIGSTAR? GENRE? true Comedy Romance SciFi true

4 Avoiding overfitting Concept of overfitting : Green line represents the actual model. Yellow samples are noisy data points. Red line is an example of resulting overfitting. This can happen building decision trees, when training data is noisy. In Decision Tree Induction, one approach for avoiding overfitting is: Don t split if information gain is very small (such as when data is noisy or conflicting). Another approach is: Prune when testing cases reveal low accuracy in particular cases. Example (pruning): Suppose training had made this split, based on 3 cases: 1 red/success & 2 blue/failure. Then, suppose that testing include cases: 3 red/failure & 1 blue/success. red color? blue Now, the accuracy over all data is: 3 correct, 4 incorrect. Changing the entire subtree to failure, the overall accuracy is: 5 correct, 2 incorrect. success failure Pruning is done in ID3 variant algorithm C-4-5 (called JSP in WEKA).

5 Weka Notes The latest stable version 3.8 is available for free download at: Preparing the training data (in class country star genre success IND,,COM, IND,,SCI, IND,,ROM, REST,,COM, REST,,SCI, IND,,COM,,,COM, 1. Save the data file with extension.arff 2. Run Weka. 3. Click on Explorer. 4. In the explorer window, click Open File at the upper left. 5. Select the file to load. 6. Click on the classify tab. 7. Under the classifier frame at the top, click Choose. 8. Under the trees folder, click J Specify a test set, if any (format must be identical to training set). Alternatively, if you have a large amount of training data, you can specify a percentage of the training data to be used as testing cases. 10. If you want the results to match ID3, you ll need to change a few settings. First, click in the command line options box to the right of the choose button. In the pop-up: Set minnumobj to 1 Set collapsetree to False Set unpruned to True 11. Click the Start button at the left. Weka then generates a decision tree for the data, including the percentage of correctly classified test cases.