Exercise: Support Vector Machines

Transcription

1 SMO using Weka Follow these instructions to explore the concept of Sequential Minimal Optimization, or SMO, using the Weka software tool. Write answers to the questions below on a separate sheet or type them in a separated document. 1. Start Weka and the Explorer. 2. In the Preprocess tab, open the file credit-r.arff, which is in the installed Weka data file. This is a set of cases for deciding whether someone has good or bad credit. 3. In the Preprocess tab, with credit-r open: 3a.What is the relation being studied? 3b.How many instances are there? 3c.Which attribute number are we trying to predict? In other words, which is the class? 3d.How many good credit cases are there? How many bad credit? 3e.Looking just at the information in the Preprocess tab, what is the accuracy for the majority classifier? 4. Choose the Classify tab. Look first at the defaults. 4a. What is the default Classifier? 4b. What is the default test protocol? What is the default class? 5. Run the ZeroR classifier by clicking the start button. 5a. How long did it take to run and test the classifier? 5b. What is the accuracy? 5c. How many good credit applications are called bad? 5d. How many bad credit applications are called good? 5c: Is this useful? 6. Change the classifier to SMO. Right-click on the SMO bar, to the right of the Choose button. This brings up the Parameters choice page. 6.a. Whet kernel is the default for SMO (polynomial) 6b. What is the default SMO value for the slack or cost parameter? 7. Look at the Classifier Evaluation Options by Clicking on the More options button. Turn off all options except Output confusion matrix. 8. Run the classifier by clicking Start. Look at the classifier results. 8a. How long did it take to run and test the classifier? 8b. What is the accuracy? 8c. How many good credit applications are called bad? 8d. How many bad credit applications are called good? 8e. Is this useful?

2 SMO using Weka - Answer Key Note: Weka provides a lot of information. This module does not go into detail about the more complex data. If students want to explore more on their own, the Weka documentation page provides more details and can be found by searching for weka online or navigating directly to: This exercise is similar to the one in the Evaluating Classifiers module. It can be used as a follow-on activitiy by adding question 6. This exercise assumes that the students have access to Weka. The questions can actually be addressed without running Weka by giving the students copies of the figures. 1. Start Weka and the Explorer. 2. In the Preprocess tab, open the file credit-r.arff, which is in the installed Weka data file. This is a set of cases for deciding whether someone has good or bad credit.

3 3. In the Preprocess tab, with credit-r open: 3a.What is the relation being studied? (german_credit)? 3b.How many instances are there? (1000) 3c.Which attribute number are we trying to predict? In other words, which is the class? (#21) 3d.How many good cases are there? (700) How many bad? (300) 3e.Looking just at the information in the Preprocess tab, what is the accuracy for the majority classifier? (70%. We will predict good for all thousand cases, and be right for the 700 good cases, so 700/1000, or 70%.

4 4. Choose the Classify tab. Look first at the defaults. 4a. What is the default Classifier? (ZeroR, which is what Weka calls the majority classifier). 4b. What is the default test protocol? (Cross-validation with a 10 fold split). What is the default class? (class, #21. Weka defaults to the last attribute)

5 5. Run the ZeroR classifier by clicking the start button. 5a. How long did it take to run and test the classifier? Answer: no noticeable time, unless your computer is incredibly slow.) 5b. What is the accuracy? (70%)

6 5c. How many good credit applications are called bad? (0) 5d. How many bad credit applications are called good? (300) 5c: Is this useful? (no) 6. Change the classifier to SMO. (This is the Weka Support Vector Machine method. It s slower and not as good as SVM, but does not involve installing additional software). Right-click on the SMO bar, to the right of the Choose button. This brings up the Parameters choice page. 6.a. Whet kernel is the default for SMO (polynomial) 6b. What is the default SMO value for the slack or cost parameter? (c = 1.0. Note that c is an indication of how much it costs to make an error; the larger c is, more expensive it is to make an error, and the harder and narrower the margin. )

7 7. Look at the Classifier Evaluation Options by Clicking on the More options button. Turn off all options except Output confusion matrix.

8 8. Run the classifier by clicking Start. Look at the classifier results. 8a. How long did it take to run and test the classifier? (Typically, a few seconds; much longer than ZeroR, even for a few cases.)). 8b. What is the accuracy? (75.1%) 8c. How many good credit applications are called bad? (90) 8d. How many bad credit applications are called good? (159) 8e. Is this useful? (Note first that although the accuracy hasn t changed much, the confusion matrix is very different. So maybe. It s certainly more informative than the majority classifier, but you re still going to deny credit to some people who would be good credit risks, and give loans to 159 people who are bad risks. Possibly better than your human evaluators do? Comes down to the cost of making a mistake. )