9/21/2017 MIST.6060 Business Intelligence and Data Mining 1. Decision Trees. Decision trees generate models represented by trees and rules.

Transcription

1 9/1/017 MIST.6060 Business Inelligence and Daa Mining 1 Basic Ideas Decision Trees Decision rees generae models represened y rees and rules. Decision rees are used for oh classificaion (classificaion rees) and numeric predicion (regression rees) prolems. The wo es-known and mos widely used decision ree sysems are: CART (Classificaion and Regression Trees) y Breiman e al. (1984): Breiman L, Friedman JH, Olshen RA, Sone, CJ (1984) Classificaion and Regression Trees (Wadsworh, Belmon, CA). C4.5 y Quinlan (1993): Quinlan JR (1993) C4.5: Programs for Machine Learning (Morgan Kaufmann, San Maeo, CA). Decision ree algorihms egin wih he enire raining daase, spli he daa ino wo or more suses according o some spliing crieria, and hen repeaedly spli each suse ino smaller suses unil a sopping crierion is me. This process is called recursive pariioning. Srengh: Inerpreailiy Ailiy o ackle noisy daa Few assumpions aou daa Weakness For large daases, rees can e very complex

2 9/1/017 MIST.6060 Business Inelligence and Daa Mining A Credi Evaluaion Example Daa sored in a ank daaase: Name Age Income Educaion Occupaion Paymen Defaul Johnson 4 45,600 college manager no Davis 6 9,000 high-school self-employed yes Miller 58 36,800 high-school clerk no Young 35 37,300 college engineer no A decision ree uil ased on he ank daa: Income $31,000 < $31,000 Educaion Age elow college < 8 8 college or higher Income no defaul Occupaion < $4,000 $4,000 self- no selfemployed employed defaul Rules generaed from his decision ree: If an applican has Income $31,000 AND has a college or higher educaion, hen accep he applicaion. Wha is anoher rule ha can e derived from his decision ree?

3 9/1/017 MIST.6060 Business Inelligence and Daa Mining 3 Recursive Pariioning and Decision Trees: An Example Consider a daase ploed elow, which has wo predicor ariues, X 1 and X, and wo classes, circle and riangle. Recursive pariioning works as follows. X X 1 = X = X = X 1 X 1 <? Yes No X < 1? X < 1.5? Yes No Yes No To egin wih, for each predicor ariue, a lengh of ieraion sep (say, 0.01) and a range of he predicor values are specified (say, [0, 4] for X 1 and [0, 3] for X ; u in acual implemenaion, all numeric ariue values are normalized o range [0, 1]). Firs, he algorihm selecs an ariue, say, X 1, makes he firs rial-spli a X 1 = 0.01, and calculaes he misclassificaion rae corresponding o his rial-spli. Nex, he second rialspli is made a X 1 = 0.0 and he corresponding misclassificaion rae is calculaed. The process coninues unil i reaches he upper ound of he X 1 range. The rial-spli ha has he

4 9/1/017 MIST.6060 Business Inelligence and Daa Mining 4 smalles misclassificaion rae is hen recorded. In our case, his occurs a X 1 =. Decision rees use he majoriy rule o assign he class lael for a suse. In he suse/region specified y X 1 <, here are 15 riangles and 9 circles; while for X 1, here are 15 circles and 8 riangles. So he decision rule ased on he spli a X 1 = is if X 1 <, hen classify o riangle; oherwise classify o circle. The numer of misclassified records wih his decision is = 17. I did no really check every rial-spli o see if his is indeed he minimum-error spli for X 1, u le s assume i is. Nex, he algorihm selecs anoher ariue, X, perform he similar sep-y-sep search, and find he es rial-spli a, say, X = 0.8. The algorihm hen compares he error rae eween spliing a X 1 = and spliing a X = 0.8. Suppose spliing a X 1 = has smaller error rae. Then, he firs formal spli is deermined a X 1 =. Susequen splis follow almos he same procedure as he firs spli excep ha error raes are calculaed ased on he daa ha has een divided y he previous splis. For example, he second spli, X = 1, is deermined from he suse of he daa poins ha fall on he lef side of he verical line, X 1 = ; while he spli X = 1.5 comes from he righ side of he daa. Suppose he algorihm sops he pariioning process afer hese hree splis. The whole process can hen e represened y a ree-like srucure on he oom par, which serves as a classificaion model. In acual implemenaion, more efficien procedures for pariioning daa are used. Typically, values are firs sored for each numeric ariue; hen, rial-splis are made only a he midpoins of adjacen values. So, for a numeric ariue wih n values, only (n 1) rialsplis are examined. For a caegorical ariue where soring does no make sense, he spliing procedure ofen involves enumeraing caegorical values. Anoher simplificaion in he aove descripion is he use of misclassificaion error as he spliing crierion. In acual decision ree sysems, more sophisicaed measures are used o deermine he es spli. We will discuss some popular spliing crieria shorly. Terminology A node represens a se of daa. The enire daase is represened as a roo node. When a spli is made, several child nodes are formed. If a node is no o e spli any furher, i is called a leaf (or erminal node); oherwise, i is an inernal node (or non-erminal node). In a inary ree, each spli produces exacly wo child nodes. CART produces inary rees. C4.5 allows more han wo child nodes on each spli.

5 9/1/017 MIST.6060 Business Inelligence and Daa Mining 5 Impuriy Measures An impuriy measure, also known as a spliing crierion, is a goodness measure of a spli. I is consruced such ha afer each spli, he daa in he child nodes are purer (more homogeneous in erms of he class lael) han he daa in he paren node. Impuriy measures: Enropy (Informaion gain): firs used in C4.5. This is argualy he mos widely used impuriy measure. Gini index: used in CART Twoing rule: used in CART Chi-square: used in CHAID (chi-squared auomaic ineracion deecion) Error-ased measures. We have used an error-ased measure in he aove example o show how splis are deermined. Error-ased measures are he easies o undersand u perhaps he leas effecive one. Enropy (Informaion Gain) Consider a class ariue wih m classes. The class of each record is an oucome of a discree random variale, Y, which akes values from a se of m class laels, c,..., c }. { 1 m The proailiy disriuion of Y is denoed as P Y = c k ) = p, for k = 1,..., m, and m k = 1 k 1 pm p = p + p + + = 1. The enropy of a random variale Y is defined as ( k m H ( Y ) = pk log pk = ( p1 log p1 + p log p + + pm log k = 1 p m ). And noe ha 0log 0 = 0. The value of he enropy aains is minimum, 0, when any p k = 1 ( k = 1,..., m) (which implies all oher p k s are 0); and he value reaches is maximum, log m, when all p k s are equal o 1/m. This propery is consisen wih wha is desired y an impuriy measure when applied o pariioning daa, a spli ha reduces he enropy of he daa also reduces he impuriy of he daa. The compuaional formulas for he enropy a a given node wih n records:

6 9/1/017 MIST.6060 Business Inelligence and Daa Mining 6 Before he node is spli m nk nk H ( ) = ( )(log ) (0 log 0 = 0 ) (1) n k = 1 n where n k is he numer (coun) of records in node ha elongs o class c k. Afer node is spli ino s child nodes (suses), he enropy can e calculaed as he weighed sum over he s child nodes, as elow: H a s n j ( ) = H ( j ) () n j= 1 where j represens he jh child node, and n j is he numer of records in j (we use suscrip for efore and a for afer ). I is well known ha a spli reduces enropy; i.e., H ( ) H a ( ). When applied o daa pariioning, a spliing crierion selecs he spli ha maximizes he amoun of reducion in an impuriy measure. When he enropy is used as he impuriy measure, he ariue wih he maximum value of H ( ) H a ( ) is seleced as he spliing ariue. An Example The Weaher Daa Weaher is an imporan facor o some spors, such as aseall, ennis and golf. Tournamen organizers (and players, fans, and eors) will e ineresed in predicing wheher a mach will e played on schedule or no, ased on he weaher condiions. The following example, aken from Quinlan (1993) and also used in he WFHP ook, provides such weaher daa. I is a hypoheical daase relaed o playing golf. Oulook Temperaure Humidiy Windy Play (Class) sunny FALSE no sunny TRUE no overcas FALSE yes rainy FALSE yes rainy FALSE yes rainy TRUE no overcas TRUE yes sunny 7 95 FALSE no sunny FALSE yes rainy FALSE yes sunny TRUE yes overcas 7 90 TRUE yes overcas FALSE yes rainy TRUE no

7 9/1/017 MIST.6060 Business Inelligence and Daa Mining 7 Enropy Compuaion for he Weaher Daa A he roo node (i.e., efore any spli), 9 of he 14 records have class yes and 5 have no. Based on equaion (1), we have: H ( roo) = ( )log ( ) ( )log ( ) = (To calculae he value of log x, ener funcion =LOG(x,) in a cell in Excel.) We firs ry o spli on Oulook, which divides he daa ino hree suses: 5 records wih sunny, 4 wih overcas and 5 wih rainy. Ou of he 5 sunny records, have yes, while 3 have no. All of he 4 overcas records are yes. Three of he 5 rainy records have yes, while have no. Using equaion (), afer his spli we have: H a (roo) = = ( )[ ( )log ( ) ( )log ( )] ( )[ ( )log ( ) ( )log ( )] ( )[ ( )log ( ) ( )log ( )] (0.3571)(0.9710) (0.3571)(0.9710) = Similarly, he enropy afer he firs spli ased on Windy, Humidiy and Temperaure is 0.89, and , respecively (see calculaions in he ale elow). For numeric ariues Humidiy and Temperaure, he enropies are calculaed wih a series of rial splis as descried earlier. The spli ased on Oulook has he smalles enropy value [i.e., minimum H a (roo) ], which implies he larges gain in informaion [i.e., maximum ( H ( roo) H a (roo) )], since H (roo) is he same for any spli. Therefore, Oulook should e seleced as he ariue for he firs spli. Ariues Values #yes #no N log (#yes/n) log (#no/n) H Ni/N (Ni/N)*H No spli Oulook sunny overcas rainy Windy FALSE TRUE Humidiy <= > Temp. <= >

8 9/1/017 MIST.6060 Business Inelligence and Daa Mining 8 Missing Daa Treamen Missing daa presens wo prolems for decision rees: In he raining (daa pariioning) sage, how o assign he daa wih missing values o a suse. In he validaion (esing) and applicaion sage, how o classify an unseen record wih missing values. A decision ree-specific mehod for handling missing daa is o use weighs. Wih his mehod, a record wih missing values is assigned o each pariioned se or classified as a class wih a weigh ha is proporional o he numer of records wih known values in each suse. An example of he weighing mehod (he weaher daa wih a missing value) Oulook Temperaure Humidiy Windy Play (Class) sunny FALSE no sunny TRUE no overcas FALSE yes rainy FALSE yes rainy FALSE yes rainy TRUE no overcas TRUE yes sunny 7 95 FALSE no sunny FALSE yes rainy FALSE yes sunny TRUE yes? 7 90 TRUE yes overcas FALSE yes rainy TRUE no The following is he coun ale of he Oulook values wih respec o he class: Oulook Yes No Sum sunny 3 5 overcas rainy 3 5 Sum In he raining (model uilding) sage, when he 14 records are pariioned y Oulook, he 13 records wih known value of Oulook presen no prolem. The record wih missing Oulook value is assigned o all hree ranches, i.e., Sunny, Overcas and Rainy, wih weighs 5/13, 3/13, and 5/13, respecively. In he esing (acual classificaion) sage, if we have o classify his record a his poin ased on oulook only, hen i will e assigned o class yes wih weigh (proailiy) 8/13, and o no wih weigh 5/13.

9 9/1/017 MIST.6060 Business Inelligence and Daa Mining 9 Issue of Tree Size One can always consruc a sufficienly large ree o drive raining error rae down o zero. For example, a ree having one record on each of is leaves has zero error. Bu his ree overfis he raining daa and is no likely o work well when used for unseen daa. In general, you will oserve a relaionship eween error and ree size as shown on he char elow. Therefore, i is imporan o find an appropriae size for a decision ree. Error Bes Tree Validaion Daa Training Daa Tree Size Direc sopping rules: Sop furher spli if he curren node conains less han a specified numer of records (e.g., 10 records). Sop if furher splis do no produce a significan decrease in he overall error rae or in he value of he impuriy measure (CHAID uses his rule). Direc sopping rules are heurisics in naure and are ofen ineffecive. Find a righ sized ree y pruning. Main mehods include: Cos-complexiy pruning, which requires validaion se used in CART. Reduced-error pruning, which also requires validaion se used in C4.5. Pessimisic-error pruning, which does no require validaion se used in C4.5 (defaul). Single Sample Tes The daase is divided ino wo ses: raining (e.g., 80%) and validaion (e.g., 0%) ses. Build an overly large decision ree ased on he raining se and find he es ree size using he validaion se.

10 9/1/017 MIST.6060 Business Inelligence and Daa Mining Fold Cross-Validaion The enire daase is randomly divided ino 10 equal sized locks. Consruc a decision ree ased on 90% of he daa (raining se) and find he es ree size using remaining 10% of daa (validaion se). Repea he process for 10 imes, each using differen raining and validaion ses. Average he 10 ree sizes o ge he final ree size and hen reuild decision rees ased on he enire daase and he final ree size. An Example for Pruning The following example shows a complee and unpruned ree uil on a daase of 30 records. The class ariue has wo values: Y and N. In each node, he firs row shows he node ID, followed y he class disriuion a his node, expressed as {numer of Y s, numer of N s}. The second row indicaes he classificaion decision, and he numer of errors (i.e., misclassified records) wih he decision. For example, a node #6, which is a leaf (yellow highlighed), here are records wih Y and 4 records wih N. The classificaion decision wih his leaf, which is he majoriy class a he leaf, is N, wih errors (numer of minoriy records). The spliing informaion is irrelevan o pruning and hus is no shown in he diagram. #1: {15, 15} Y or N, 15 #: {5, 4} Y, 4 #3: {, 3} N, #4: {8, 8} Y or N, 8 #5: {3, 0} Y, 0 #6: {, 4} N, #7: {, 1} Y, 1 #8: {0, } N, 0 #9: {5, 8} N, 5 #10: {3, 0} Y, 0 #11: {5, 4} Y, 4 #1: {0, 4} N, 0 #13: {4, 1} Y, 1 #14: {1, 3} N, 1

11 9/1/017 MIST.6060 Business Inelligence and Daa Mining 11 Suree and Branch A suree refers o a pruned ree, which has he same roo as he unpruned ree; a ranch refers o a segmen of he ree, rooed a an inernal node, ha can e a candidae for pruning. In oher words, a suree is oained y pruning off one or more ranches from he unpruned ree. For example, a ranch rooed a node 9 includes nodes 9, 11, 1, 13 and 14. If his ranch is pruned off (o conain node 9 only) from he unpruned ree, he pruned ree, which is a suree of he unpruned ree, consiss of nodes 1 hrough 10. Cos-Complexiy Pruning In cos-complexiy pruning (CCP), he cos refers o he misclassificaion errors and he complexiy refers o he ree size. The cos-complexiy is a measure of radeoff eween error and ree size, which can e wrien as: cos-complexiy raio = increase in errors due o pruning decrease in ree size due o pruning. In general, a smaller cos-complexiy raio suggess eiher a smaller increase in errors and/or a larger decrease in ree size, which are desirale. Le B e a ranch rooed a an inernal node. For example, B 9 is he ranch rooed a node 9, which includes nodes 9, 11, 1, 13 and 14. Le E() e he numer of errors, i.e., misclassified records (minoriy records), if a classificaion decision is made a node. For example, E ( 9) = 5. Le E ( B ) e he numer of errors if classificaion decision is ased on ranch B. This implies ha E ( B ) is he sum of errors made y all leaves in B. For example, E ( B 9 ) = =. Finally, le L e he numer of leaves in B. For example, L = 9 3. The cos-complexiy raio for evaluaing pruning of ranch B ino node can e formally defined as: E( ) E( B ) α =. L 1 The cos-complexiy raio α can e inerpreed as he cos per exra leaf. For example, E(9) E( B9 ) 5 ( ) α 9 = = = 1.5. L The cos-complexiy pruning (CCP) procedure firs calculaes he value of α for each ranch, rooed a each differen inernal node, of he unpruned ree. The ranch ha has smalles value of α is hen pruned (why smalles?), yielding he firs pruned suree. Nex, he values of α are compued again, u ased on he las pruned ree; his will prune away anoher ranch. Repeaing his process will progressively produce a series of surees, each nesed wihin he previous one. Each suree produced in his procedure is opimal wih respec o size; ha is, no oher suree of he same size would have lower error rae han he one oained y he cos-complexiy pruning procedure.

12 9/1/017 MIST.6060 Business Inelligence and Daa Mining 1 To illusrae he CCP procedure wih he aove example, we firs calculae α s ased on he unpruned ree: 15 ( ) α 1 = = Similarly, α =, α = 3 1, α 4 =, α = , and α 11 =. Since α 3 is he smalles, we prune he ranch B 3 ino leaf node 3. This resuls in he suree elow: #1: {15, 15} Y or N, 15 #: {5, 4} Y, 4 #3: {, 3} N, #4: {8, 8} Y or N, 8 #5: {3, 0} Y, 0 #6: {, 4} N, #9: {5, 8} N, 5 #10: {3, 0} Y, 0 #11: {5, 4} Y, 4 #1: {0, 4} N, 0 #13: {4, 1} Y, 1 #14: {1, 3} N, 1 Nex, we calculae α s again, u ased on his pruned ree, as elow: 15 ( ) α 1 = = Similarly, α =, α =, α = 1. 5, and α =. Now, α 1 and α 9 are ied for he smalles. We can randomly selec eiher node 1 or node 9. However, pruning a node 1 desroys he whole ree srucure. Suppose we don wan o go ha far a his ime. So we selec and prune off ranch B 9 ino leaf node 9. This resuls in a suree conaining nodes 1,, 3, 4, 5, 6, 9 and 10. This process coninues unil i reaches he roo node. The CCP procedure generaes a series of pruned rees. However, which one should e seleced as he final ree for use in classificaion of unseen records? To answer his quesion,

13 9/1/017 MIST.6060 Business Inelligence and Daa Mining 13 we need validaion daa (which are no used in growing he unpruned ree). We use each suree generaed y CCP o classify records in he validaion se, and record he corresponding validaion errors. We will see a paern like he char shown on page 9. The final pruned ree will e he one ha has he lowes validaion error. Noe ha a 10-fold (or any fold) cross-validaion can e used for he same purpose, where he final ree size is he average ree size of he 10 validaion resuls.