Tree Diagram. Splitting Criterion. Splitting Criterion. Introduction. Building a Decision Tree. MS4424 Data Mining & Modelling Decision Tree

Transcription

1 Introduction MS4424 Data Mining & Modelling Decision Tree Lecturer : Dr Iris Yeung Room No : P7509 Tel No : msiris@cityu.edu.hk decision tree is a set of rules represented in a tree structure that can be used for prediction. Example: When a customer is not a permanent resident of HK ND he/she has been residing in HK for less than 6 months ND he/she does not have a permanent job, a bank rejects a credit card application. 1 2 Tree Diagram Root node -- top node in the tree that contains all observations. Internal nodes -- non terminal nodes (also includes the root node) that contains the splitting rule. Leaf nodes -- terminal nodes that contain the final classification for a set of observations. There is exactly one path from the root to any leaf. given data record starts at the root and then, based on the values in the record, it winds its way to a leaf. The path is an expression of the rule. 3 uilding a Decision Tree uilding a decision tree is a repetitive process (known as recursive partitioning). We start with the root node which contains all the records in the training set. Then find the children of the root. The algorithm considers every split in every column. For each possible split, it measures how good it is using some measure of purity. Eventually, it stops, either because there is no best split or because the leaves are entirely pure. It may have memorized (overfit) the training set. We need to prune back the tree using the validation set in order to get the optimal tree. 4 Splitting Criterion Splitting Criterion 5 6 1

2 Gini Index Gini Index Left Gini 2(4/5)(1/5) =0.32 Right Gini 2*(4/5)(1/5) =0.32 Start Gini 2*(5/10)(5/10) =0.5 Weighted Entropy 5/10*0.32+5/10*0.32 =0.32 Gain (5/9)(4/9) =0.49 2*(1/1)*(0/1) =0 9/10*0.49+1/10*0 = Entropy Entropy Left Entropy -4/5log(4/5)-1/5log(1/5) =0.72-5/9log(5/9)-4/9log(4/9) =0.99 Right Entropy -4/5log(4/5)-1/5log(1/5) =0.72-1/1log(1/1) =0 Start Entropy -5/10log(5/10) -5/10log(5/10) =1 Weighted Entropy 5/10*0.72+5/10*0.72 =0.72 9/10*0.99+1/10*0 =0.89 Gain Stopping Criteria Effect of Pruning Pre-pruning Stop growing if the node is pure; if the number of cases in a node falls below a specified limit or when the split is not statistically significant at a specified level. Post-pruning 11 Figure 1 Error rate on training set and test set as tree complexity increases. 12 2

3 Model ssessment for Pruning dvantages and Disadvantages of Decision Trees Easy to understand results (can explain any single prediction) pplies to both categorical and ordered inputs. This is one way that they differ fundamentally from neural networks and regressions that depends on the magnitude of values as well. Finds attributes that have the biggest effect on the output Expensive to calculate decision tree. Finding the best split requires trying all possible splits and then choosing the best one. Tools, such as EM, that finds the splits on a sample of the data, ameliorate this (by default 2000 rows). Danger of over-fitting data Works best for binary outputs (but algorithm can be modified to estimate continuous outputs or predict multiple categories). However, this is often the case (binary output) for CRM problems, so, in practice, this is not a serious limitation lgorithms CHID (Hartigan, 1975) -- Chi-squared automatic interaction detection CRT (reiman, Friedman, Olsen and Stone1984) -- Classification and Regression Trees C4.5 (Quinland, 1993, earlier version ID3) C5.0 HMEQ Data Set Name Model Measurement Description Role Level D target binary 1=Client defaulted on loan, 0=loan repaid CLGE input interval ge of oldest trade line in months CLNO input interval Number of trade lines DETINC input interval Debt to income ratio DELINQ input interval Number of trade lines DEROG input interval Number of major derogatory reports JO input nominal Job category LON input interval mount of the loan request MORTDUE input interval mount due on existing mortgage NINQ input interval Number of recent credit inquires RESON input binary DebtCon=debt consolidation HomeImp=home improvement VLUE input interval Value of current property YOJ input interval Years at present job 15 Open a new workspace from the vailable Projects window and assemble the diagram shown below. Set up the Input Data Source node 1. Open the Input Data source node. 2. Select the HMEQ data set from the NEW library. 3. Set the Model Role for D to target. 4. Verify the measurement level and model role for the remaining input variables. The Enterprise Miner assumes numeric variables with less than 10 levels in the metadata sample represent ordinal data. Check to make sure that variables such as DEROG or DELINQ have interval for their measurement level. 5. Close the Input Data Source node, saving changes when prompted

4 Partition the HMEQ Data and uild a Default Tree 1. Open the Data Partition node. 2. Set Train, Validation, and Test to 67, 33, and 0 respectively. 3. Close the Data Partition node, saving changes when prompted. 4. For this first analysis pass, use the default settings for the Decision Tree node. Stepwise Regression 1. Open the Regression node and specify Stepwise as the method in the Selection Method tab. 2. Close the Regression node, saving changes when prompted. Note: When modifying a modeling node for the first time, you are prompted to rename the model upon exiting Stepwise Regression 3. Enter StepReg in the model name field Run the diagram from the ssessment node and select Yes when prompted to view the results. Generate a lift chart to compare the two models and inspect the resulting plot. 4. Select OK The chart shows the decision tree model dominating the regression model. The decision tree s first decile contains more than 80% defaulters. y comparison, the regression s first decile contains only about 64% defaulters. Select the %Captured Response radio button. Neural Network Model dd a Neural Network node the workspace and connect it after the Data Replacement node. Connect the Neural Network node to the ssessment node. The completed diagram should appear as pictured below. y rejecting the worst 30% of all applications using the decision tree, you eliminate about 78% of the bad loans from your portfolio. The regression model would required rejecting almost half of the applications to identify this percentage of bad loans

5 Generate a lift chart for all three models and compare performance. Open the Decision Tree node results window by right clicking on the node and selecting Results. The ll tab is active and displays a summary of several of the subtabs. The neural network model does not perform appreciably better than the stepwise regression model, and the decision tree model is still the best. Nonlinear association does not explain the difference between the decision tree and the stepwise regression model. Close the From the table in the lower-left corner of the tab, you see that a seven-leaf model is selected that produces a model with an accuracy of 88.92% on the validation data set ssessment node. View the Tree by selecting View->Tree from the menu bar. portion of the tree appears below. To modify the levels that are visible, proceed as follows: 1. Select View -> Tree Options. 2. Type 6 in the Tree depth down field. 3. Select OK. 4. Verify that all eight leaves are visible. lthough the selected tree was supposed to have eight leaves, not all eight leaves are visible. y default, the decision tree viewer Observe that the colors in the tree ring diagram and the decision tree itself indicate node purity by default. If the node contains all ones or all zeros, the node is colored red. If the node contains an equal mix of ones and zeros, it is colored yellow. displays three levels deep You can change the coloring scheme as follows: 1. Select Tools -> Define Colors. 2. Select the Proportion of a target value radio button Select 0 in the Select a target value table. 4. Select OK. 30 5

6 Inspect the tree diagram to identify the terminal nodes with a high percentage f bad loans (colored red) and those with a high percentage of good loans (colored green). Close the tree diagram when you are done. You can also see splitting information using the Tree Ring tab in the Results-Tree window. Using the View Info tool, you can click on the partitions in the tree ring plot to see the variable and cutoff value used for each split. The sizes of the resulting nodes are proportional to the size of the segments in the tree ring plot. You can see the split statistics by selecting View->Probe tree ring statistics. You can view a path to any node by selecting it and then selecting View -> Path. 31 Understanding and Using Tree Options Consider a competing tree that allows up to 4-way splits. 1. dd another Tree node to the workspace. 2. Connect the Data Partition node to the Tree node. 3. Connect the Tree node o the ssessment node. 32 Understanding and Using Tree Options 4. Open the Decision Tree node. 5. Select the asic tab. 6. Enter 4 for the Maximum number of branches form a node field. This option will allow binary, 3-way, and 4-way splits to be considered. Understanding and Using Tree Options The number of leaves in the selected tree has increased from 8 to 33. It is a matter of personal taste as to whether this tree is more comprehensible than the binary split tree. The increased number of leaves suggests to some a lower degree of comprehensibility. The accuracy on the validation set is only 0.2% higher than the default model in split of greatly increased complexity. 7. Close the tree node, saving changes when prompted. 8. Enter the name DT4way in the model name field when prompted to remind youthat you specified up to 4-way splits. 9. Select OK. 10.Run the flow from this Tree node and view the results. 33 If you inspect the tree diagram, there are may nodes containing only a few applicants. You can employ additional cultivation options to limit this phenomenon. 34 Modify the most recently created Tree node and employ the stopping rules to keep the tree from generating so many small terminal nodes. 1. Open the Tree node. 2. Select the asic tab. 3. Type 25 in the Minimum number of observations in a leaf field and then press the Enter key. 4. Type 50 in the Observations required for a split search field and then press the Enter key. (Note: The Decision Tree node requires that (Observations required for a split search) 2*(Minimum number of observations in a leaf). In this example, the observations required for a split search must be greater than 2*25=50. node with less than 50 observations cannot be split into two nodes, each having at least 25 observations. 5. Close and save your changes to the Tree node. 6. Rerun the Tree node and view the results as before

7 Inspect the results. The tree diagram opens The optimal tree now has 8 leaves. The validation accuracy has dropped slightly to 88.56%. Select View- >Tree to see the tree diagram Note that the initial split on DETINC has produced four branches,. You may wonder which branch contains the missing values. To find out, proceed as follows: 1. Position the tip of the cursor above the variable name DETINC directly below the root node in the tree diagram. 2. Right-click and select View competing splits. The Input Selection window opens. The table lists the top five inputs considered for splitting as ranked by a measure of worth. 3. Select the row for the variable DETINC. 4. Select rowse rule. 39 The Interval Variable Splitting Rule window opens. The table presents the selected ranges for each of the four branches as well as the branch number that contains the missing values (in this case it is the fourth branch). Close the Interval Variable Splitting Rule window, the Input Select window, the Tree Diagram, and the Tree-Results window. 40 7