Download Classification

Data Mining:
Concepts and Techniques
(3rd ed.)
— Chapter 8 —
Jiawei Han, Micheline Kamber, and Jian Pei
University of Illinois at Urbana-Champaign &
Simon Fraser University
©2011 Han, Kamber & Pei. All rights reserved.
1
Chapter 8. Classification: Basic Concepts

Classification: Basic Concepts

Decision Tree Induction

Bayes Classification Methods

Rule-Based Classification

Model Evaluation and Selection

Techniques to Improve Classification Accuracy:
Ensemble Methods

Summary
2
Supervised vs. Unsupervised Learning

Supervised learning (classification)

Supervision: The training data (observations,
measurements, etc.) are accompanied by labels indicating
the class of the observations


New data is classified based on the training set
Unsupervised learning (clustering)

The class labels of training data is unknown

Given a set of measurements, observations, etc. with the
aim of establishing the existence of classes or clusters in
the data
3
Prediction Problems: Classification vs.
Numeric Prediction



Classification
 predicts categorical class labels (discrete or nominal)
 classifies data (constructs a model) based on the training
set and the values (class labels) in a classifying attribute
and uses it in classifying new data
Numeric Prediction
 models continuous-valued functions, i.e., predicts
unknown or missing values
Typical applications
 Credit/loan approval:
 Medical diagnosis: if a tumor is cancerous or benign
 Fraud detection: if a transaction is fraudulent
 Web page categorization: which category it is
4
Classification—A Two-Step Process



Model construction: describing a set of predetermined classes
 Each tuple/sample is assumed to belong to a predefined class, as
determined by the class label attribute
 The set of tuples used for model construction is training set
 The model is represented as classification rules, decision trees, or
mathematical formulae
Model usage: for classifying future or unknown objects
 Estimate accuracy of the model
 The known label of test sample is compared with the classified
result from the model
 Accuracy rate is the percentage of test set samples that are
correctly classified by the model
 Test set is independent of training set (otherwise overfitting)
 If the accuracy is acceptable, use the model to classify new data
Note: If the test set is used to select models, it is called validation (test) set
5
Process (1): Model Construction
Training
Data
NAME
M ike
M ary
B ill
Jim
D ave
A nne
RANK
YEARS TENURED
A ssistant P rof
3
no
A ssistant P rof
7
yes
P rofessor
2
yes
A ssociate P rof
7
yes
A ssistant P rof
6
no
A ssociate P rof
3
no
Classification
Algorithms
Classifier
(Model)
IF rank = ‘professor’
OR years > 6
THEN tenured = ‘yes’
6
Process (2): Using the Model in Prediction
Classifier
Testing
Data
Unseen Data
(Jeff, Professor, 4)
NAME
T om
M erlisa
G eorge
Joseph
RANK
YEARS TENURED
A ssistant P rof
2
no
A ssociate P rof
7
no
P rofessor
5
yes
A ssistant P rof
7
yes
Tenured?
7
Classification

In this task, data will be defined in terms of attributes, one of which is the
class. It will find a model for class attribute as a function of the values of
other (predictor) attributes, such that previously unseen records can be
assigned a class as accurately as possible.
8
Classification : Another Example
Figure 8.1
The data classification process: (a) Learning: Training data are analyzed by a classification
algorithm. Here, the class label attribute is loan_decision, and the learned model or
classifier is represented in the form of classification rules. (b) Classification: Test data are
used to estimate the accuracy of the classification rules. If the accuracy is considered
acceptable, the rules can be applied to the classification of new data tuples.
9
Classification Techniques






Decision Tree based Methods
Rule-based Methods
Memory based reasoning
Neural Networks
Naïve Bayes and Bayesian Belief Networks
Support Vector Machines
10
Chapter 8. Classification: Basic Concepts

Classification: Basic Concepts

Decision Tree Induction

Bayes Classification Methods

Rule-Based Classification

Model Evaluation and Selection

Techniques to Improve Classification Accuracy:
Ensemble Methods

Summary
11
Decision Tree

A decision tree is a flowchart-like tree structure,
where each internal node (non leaf node) denotes a
test on an attribute, each branch represents an
outcome of the test, and each leaf node (or terminal
node) holds a class label . The top most node in a tree
is the root node.
12
Decision tree :Example
13
Decision tree :Another Example
14
Apply Model to Test Data
15
Apply Model to Test Data
16
Apply Model to Test Data
17
Algorithm for Decision Tree Induction


Basic algorithm (a greedy algorithm)
 Tree is constructed in a top-down recursive divide-andconquer manner
 At start, all the training examples are at the root
 Attributes are categorical (if continuous-valued, they are
discretized in advance)
 Examples are partitioned recursively based on selected
attributes
 Test attributes are selected on the basis of a heuristic or
statistical measure (e.g., information gain)
Conditions for stopping partitioning
 All samples for a given node belong to the same class
 There are no remaining attributes for further partitioning –
majority voting is employed for classifying the leaf
 There are no samples left
18
Decision Tree Induction: An Example
Training data set: Buys_computer
 The data set follows an example of
Quinlan’s ID3 (Playing Tennis)
 Resulting tree:
age?

<=30
31..40
overcast
student?
no
no
yes
yes
yes
>40
age
<=30
<=30
31…40
>40
>40
>40
31…40
<=30
<=30
>40
<=30
31…40
31…40
>40
income student credit_rating buys_computer
high
no fair
no
high
no excellent
no
high
no fair
yes
medium
no fair
yes
low
yes fair
yes
low
yes excellent
no
low
yes excellent
yes
medium
no fair
no
low
yes fair
yes
medium yes fair
yes
medium yes excellent
yes
medium
no excellent
yes
high
yes fair
yes
medium
no excellent
no
credit rating?
excellent
fair
yes
19
Attribute Selection Measure:
Information Gain (ID3/C4.5)

Select the attribute with the highest information gain

Let pi be the probability that an arbitrary tuple in D belongs to
class Ci, estimated by |Ci, D|/|D|

Expected information (entropy) needed to classify a tuple in D:
m
Info( D)   pi log 2 ( pi )


i 1
Information needed (after using A to split D into v partitions) to
v | D |
classify D:
j
InfoA ( D)  
 Info( D j )
j 1 | D |
Information gained by branching on attribute A
Gain(A)  Info(D)  InfoA(D)
20
Attribute Selection: Information Gain


Class P: buys_computer = “yes”
Class N: buys_computer = “no”
Info( D)  I (9,5)  
age
<=30
31…40
>40
age
<=30
<=30
31…40
>40
>40
>40
31…40
<=30
<=30
>40
<=30
31…40
31…40
>40
Infoage ( D) 
9
9
5
5
log 2 ( )  log 2 ( ) 0.940
14
14 14
14
pi
2
4
3
ni I(pi, ni)
3 0.971
0 0
2 0.971
income student credit_rating
high
no
fair
high
no
excellent
high
no
fair
medium
no
fair
low
yes fair
low
yes excellent
low
yes excellent
medium
no
fair
low
yes fair
medium
yes fair
medium
yes excellent
medium
no
excellent
high
yes fair
medium
no
excellent

5
4
I (2,3) 
I (4,0)
14
14
5
I (3,2)  0.694
14
5
I (2,3) means “age <=30” has 5 out of
14
14 samples, with 2 yes’es and 3
buys_computer
no
no
yes
yes
yes
no
yes
no
yes
yes
yes
yes
yes
no
no’s. Hence
Gain(age)  Info( D)  Infoage ( D)  0.246
Similarly,
Gain(income)  0.029
Gain( student )  0.151
Gain(credit _ rating )  0.048
21
Attribute Selection: Information Gain
22
Attribute Selection: Information Gain
23
24
Comparing Attribute Selection Measures

The three measures, in general, return good results but

Information gain:


Gain ratio:


biased towards multivalued attributes
tends to prefer unbalanced splits in which one partition is
much smaller than the others
Gini index:

biased to multivalued attributes

has difficulty when # of classes is large

tends to favor tests that result in equal-sized partitions
and purity in both partitions
25