Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
CSE5230/DMS/2004/6 Data Mining - CSE5230 Classifiers 2 Decision Trees CSE5230 - Data Mining, 2004 Lecture 6.1 Lecture Outline Why use Decision Trees? What is a Decision Tree? Examples Use as a data mining technique Popular Models CART CHAID ID3 & C4.5 CSE5230 - Data Mining, 2004 Lecture 6.2 Lecture Objectives By the end of this lecture you should be able to: Describe what a Decision Tree is and how it is used to do classification Explain how and why the explainability of Decision Tree classifications is important in some Data Mining tasks Describe the top-down construction of a Decision Tree by choosing splitting criteria CSE5230 - Data Mining, 2004 Lecture 6.3 Why use Decision Trees? - 1 Last week we talked about classifiers in general, and Bayesian classifiers in particular. Recapping: A classifier assigns items to classes There are many different classification models and algorithms used in Data Mining Naïve Bayes, Decision Trees, Feedforward Neural Networks, etc. Classification models differ in two important ways: the complexity of the classifications they can learn the ease with which humans can understand the reasons why certain items are classifier as they are CSE5230 - Data Mining, 2004 Lecture 6.4 Why use Decision Trees? - 2 Whereas Petal-length 2.6 > 2.6 Iris setosa Bayesian classifiers and neural networks compute a mathematical function of their inputs to generate their outputs, decision trees use logical rules, e.g. Petal-width 1.65 > 1.65 Petal-length 5 Iris virginica >5 Iris versicolor Sepal-length 6.05 Iris versicolor CSE5230 - Data Mining, 2004 > 6.05 Iris virginica IF Petal-length > 2.6 AND Petal-width 1.65 AND Petal-length > 5 AND Sepal-length > 6.05 THEN the flower is Iris virginica NB. This is not the only rule for this species. What is the other? Figure adapted from [SGI2001] Lecture 6.5 Why use Decision Trees? - 3 For some applications accuracy of classification or prediction is sufficient, e.g.: A direct mail firm needing to find a model for identifying customers who will respond to mail Predicting the stock market using past data In other applications it is better (sometimes essential) that the decision be explainable, e.g.: Rejection of a credit card application Medical diagnosis Humans generally require explanations for most decisions Sometimes there are legal and ethical requirements for this CSE5230 - Data Mining, 2004 Lecture 6.6 Why use Decision Trees? - 4 Example: When a bank rejects a credit card application, it is better to explain to the customer that it was due to the fact that: He/she is not a permanent resident of Australia AND He/she has been residing in Australia for < 6 months AND He/she does not have a permanent job. This is better than saying: “We are very sorry, but our neural network thinks that you are not a credit-worthy customer.” (In which case the customer might become angry and move to another bank) CSE5230 - Data Mining, 2004 Lecture 6.7 What is a Decision Tree? root node Built Petal-length 2.6 > 2.6 Iris setosa Petal-width 1.65 Petal-length from root node (top) to leaf test nodes (bottom) child node A record first enters the root node A test is applied to determine to > 1.65 which child node it should go next path 5 Iris virginica >5 Iris versicolor Sepal-length 6.05 Iris versicolor leaf nodes CSE5230 - Data Mining, 2004 A variety of algorithms for choosing the initial test exists. The aim is to discriminate best between the target classes process is repeated until a record arrives at a leaf node Iris virginica The path from the root to a leaf node provides an expression of a rule > 6.05 The Lecture 6.8 Building a Decision Tree - 1 Algorithms for building decision trees (DTs) begin by trying to find the test which does the “best job” of splitting the data into the desired classes The desired classes have to be identified at the start Example: we need to describe the profiles of customers of a telephone company who “churn” (do not renew their contracts). The DT building algorithm examines the customer database to find the best splitting criterion: Phone technology Age of customer Time has been a customer Gender The DT algorithm may discover out that the “Phone technology” variable is best for separating churners from non-churners CSE5230 - Data Mining, 2004 Lecture 6.9 Building a Decision Tree - 2 The process is repeated to discover the best splitting criterion for the records assigned to each node Phone technology old new Time has been a customer 2.3 Churners > 2.3 Once built, the effectiveness of a decision tree can be measured by applying it to a collection of previously unseen records and observing the percentage of correctly classified records CSE5230 - Data Mining, 2004 Lecture 6.10 Phone Technology Example - 1 Classify customers who churn, i.e. do not renew their phone contracts. 50 Churners 50 Non-churners Requirement: (adapted from [BeS1997]) new Time has been a Customer 30 Churners 50 Non-churners <= 2.3 years 5 Churners 40 Non-churners 25 Churners 10 Non-churners 20 Churners 0 Non-churners CSE5230 - Data Mining, 2004 20 Churners 0 Non-churners > 2.3 years Age <= 35 old > 35 5 Churners 10 Non-churners Lecture 6.11 Example - 2 The number of records in a given parent node equals the sum of the records contained in the child nodes Quite easy to understand how the model is being built (unlike NNs, as we will see next week) Easy to use the model e.g. for a targeted marketing campaign aimed at customers likely to churn Provides intuitive ideas about the customer base e.g: “Customers who have been with the company for a couple of years and have new phones are pretty loyal” CSE5230 - Data Mining, 2004 Lecture 6.12 Use as a data mining technique - 1 Exploration Analyzing the predictors and splitting criteria selected by the algorithm may provide interesting insights which can be acted upon e.g. if the following rule was identified: IF time a customer < 1.1 years AND sales channel = telesales THEN chance of churn is 65% It might be worthwhile conducting a study on the way the telesales operators are making their calls CSE5230 - Data Mining, 2004 Lecture 6.13 Use as a data mining technique - 2 Exploration (continued) Gleaning information from rules that fail e.g. from the phone example we obtained the rule: IF Phone technology = old AND Time has been a customer 2.3 years AND Age > 35 THEN there are only 15 customers (15% of total) Can this rule be useful? » Perhaps we can attempt to build up this small market segment. If this is possible then we have the edge over competitors since we have a head start in this knowledge » We can remove these customers from our direct marketing campaign since there are so few of them CSE5230 - Data Mining, 2004 Lecture 6.14 Use as a data mining technique - 3 Exploration (continued) Again from the phone company example we noticed that: » There was no combination of rules to reliably discriminate between churners and non-churners for the small market segment mentioned on the previous slide (5 churners, 10 non-churners). Do we consider this as an occasion where it was not possible to achieve our objective? From this failure we have learnt that age is not all that important for this category churners (unlike those under 35). Perhaps we were asking the wrong questions all along this warrants further analysis CSE5230 - Data Mining, 2004 Lecture 6.15 Use as a data mining technique - 4 Data Pre-processing Decision trees are very robust at handling different predictor types (number/categorical), and run quickly. Therefore the can be good for a first pass over the data in a data mining operation This will create a subset of the possibly useful predictors which can then be fed into another model, say a neural network Prediction Once the decision tree is built it can be then be used as a prediction tool, by using it on a new set of data CSE5230 - Data Mining, 2004 Lecture 6.16 Popular Decision Tree Models: CART CART: Classification And Regression Trees, developed in 1984 by a team of researchers (Leo Breiman et al.) from Stanford University Used in the DM software Darwin - from Thinking Machines Corporation (recently bought by Oracle) http://www.oracle.com/technology/documentation/darwin.html Also available in SPSS Classification Trees add-on module http://www.spss.com/classification_trees/ Often uses an entropy measure to determine the split point (Shannon’s Information theory). measure of disorder (MOD) = p log 2 ( p) where p is is the probability of that prediction value occurring in a particular node of the tree. Other measures used include Gini and twoing. CART produces a binary tree CSE5230 - Data Mining, 2004 Lecture 6.17 CART - 2 Consider the “Churn” problem from slide 6.11 At the first node there are 100 customers to split, 50 who churn and 50 who don’t churn The MOD of this node is: MOD = -0.5*log2(0.5) + -0.5*log2(0.5) = 1.00 The algorithm will try each predictor variable For each predictor the algorithm will calculate the MOD of the split produced by several values to identify the optimum splitting on “Phone technology” produces two nodes, one with 50 churners and 30 non-churners, the other with 20 churners and 0 non-churners. The first of these has: MOD = -5/8*log2(5/8) + -3/8log2(3/8) = 0.95 and the second has a MOD of 0. CART will select the predictor producing nodes with the lowest MOD as the split point CSE5230 - Data Mining, 2004 Lecture 6.18 Node splitting An ideally good split Name Churned? Name Churned? Jim Yes Bob No Sally Yes Betty No Steve Yes Sue No Joe Yes Alex No An ideally bad split Name Churned? Name Churned? Jim Yes Bob No Sally Yes Betty No Steve No Sue Yes Joe No Alex Yes CSE5230 - Data Mining, 2004 Lecture 6.19 Popular Decision Tree Models: CHAID CHAID: Chi-squared Automatic Interaction Detector, developed by J. A. Hartigan in 1975. Widely used since it is distributed as part of the popular statistical packages SAS and SPSS Differs from CART in the way it identifies the split points. Instead of the information measure, it uses chi-squared test to identify the split points (a statistical measure used for identifying independent variables) All predictors must be categorical or put into categorical form by binning The accuracy of the two methods CHAID and CART have been found to be similar CSE5230 - Data Mining, 2004 Lecture 6.20 Popular Decision Tree Models: ID3 & C4.5 ID3: Iterative Dichtomiser, developed by the Australian researcher Ross Quinlan in 1979 Used in the data mining software Clementine of Integral Solutions Ltd. (taken over by SPSS) ID3 picks predictors and their splitting values on the basis of the information gain provided Gain is the difference between the amount of information that is needed to make a correct prediction before a split has been made, and that required after the split If the amount of information required is lower after the split is made, then the split is said to have decreased the disorder of the original data CSE5230 - Data Mining, 2004 Lecture 6.21 ID3 & C4.5 - 2 A B + ---- ++++ By using the entropy left A ++++- +++++---- - p log( p ) right left entropy right entropy start entropy +---- -4/5log(4/5)+ -1/5log(1/5)=.72 -4/5log(4/5)+ -1/5log(1/5)=.72 -5/10log(5/10)+ -5/10(log(5/10) =1 B +++++ ---- CSE5230 - Data Mining, 2004 -5/9log(5/9)+ -4/9log(4/9)=.99 -1/1log(1/1) =0 Lecture 6.22 ID3 & C4.5 - 3 Split Weighted Entropy Gain A (5/10)*0.72+(5/10)*0.72 = 0.72 0.28 B (9/10)*0.99+(1/10)*0 = 0.89 0.11 A will be selected CSE5230 - Data Mining, 2004 Lecture 6.23 ID3 & C4.5 - 4 C4.5 introduces a number of extensions to ID3: Handles unknown field values in training set Tree pruning method » pruning produces a smaller tree (by “pruning” some of the leaves and branches) that performs better on test data the aim is to avoid over-fitting the training data Automated rule generation » extracts human-readable logical rules from the tree Quinlan now has a commercial version with further improvements, See5/C5.0 http://www.rulequest.com/see5-info.html CSE5230 - Data Mining, 2004 Lecture 6.24 Case Study Now we will look at a case study, available on the web at http://web.maths.unsw.edu.au/~inge/statlearn/kolyshkina.pdf CSE5230 - Data Mining, 2004 Lecture 6.25 Strengths and Weaknesses Strengths of decision trees Able to generate understandable rules Classify with very little computation Some decision tree induction algorithms are able to handle both continuous and categorical data Provides a clear indication of which variables are most important for prediction or classification Weaknesses Most are not appropriate for estimation or prediction tasks (income, interest rates, etc.) » exception is regression trees Problematic with time series data (much pre-processing required), can be computationally expensive CSE5230 - Data Mining, 2004 Lecture 6.26 References [BeL1997] J. A. Berry and G. Linoff, Data Mining Techniques: For Marketing, Sales, and Customer Support, John Wiley & Sons Inc.,1997 [BeS1997] A. Berson and S. J. Smith, Data Warehousing, Data Mining and OLAP, McGraw Hill, 1997 [KPR2002] I. Kolyshkina, P. Petocz and I. Rylande, Modeling insurance risk: A comparison of data mining and logistic regression approaches, in Proceedings of the 16th Australian Statistical Conference, Canberra, ACT, Australia, July 7-11, 2002. Related presentation: http://web.maths.unsw.edu.au/~inge/statlearn/kolyshkina.pdf [SGI2001] Silicon Graphics Inc. MLC++ Utilities Manual, 2001 http://www.sgi.com/tech/mlc/utils.html CSE5230 - Data Mining, 2004 Lecture 6.27