Download Classification: Basic Concepts, Decision Trees, and Model

Classification: Basic Concepts, Decision
Trees, and Model Evaluation
Dr. Hui Xiong
Rutgers University
Introduction to Data Mining
1/2/2009
1
Classification: Definition
z
Given a collection of records (training set )
– Each record is by characterized by a tuple
( ) where
(x,y),
h
x is
i th
the attribute
tt ib t sett and
d y is
i th
the
class label
x: attribute, predictor, independent variable, input
‹ y: class, response, dependent variable, output
‹
z
Task:
– Learn a model that maps each attribute set x
into one of the predefined class labels y
Introduction to Data Mining
1/2/2009
2
Examples of Classification Task
Attribute set, x
Task
Class label, y
Categorizing
C
t
i i
F t
Features
extracted
t t d from
f
email
email message header
messages
and content
spam or non-spam
Identifying
tumor cells
Features extracted from
MRI scans
malignant or benign
cells
Cataloging
galaxies
Features extracted from
telescope images
Elliptical, spiral, or
irregular-shaped
galaxies
Introduction to Data Mining
1/2/2009
3
General Approach for Building
Classification Model
Tid
Attrib1
1
Yes
Large
Attrib2
125K
Attrib3
No
Class
2
No
Medium
100K
No
3
No
Small
70K
No
4
Yes
Medium
120K
No
5
No
Large
95K
Yes
6
No
Medium
60K
No
7
Yes
Large
220K
No
8
No
Small
85K
Yes
9
No
Medium
75K
No
10
No
Small
90K
Yes
Tid
Attrib1
11
No
Small
55K
?
12
Yes
Medium
80K
?
13
Yes
Large
110K
?
14
No
Small
95K
?
15
No
Large
67K
?
Learn
Model
10
Attrib2
Attrib3
Class
Apply
Model
10
Introduction to Data Mining
1/2/2009
4
Classification Techniques
z
Base Classifiers
– Decision Tree based Methods
– Rule-based
Rule based Methods
– Nearest-neighbor
– Neural Networks
– Naïve Bayes and Bayesian Belief Networks
– Support Vector Machines
z
Ensemble Classifiers
– Boosting, Bagging, Random Forests
Introduction to Data Mining
1/2/2009
5
Example of a Decision Tree
Splitting Attributes
ID
Home
Owner
Marital
Status
Annual Defaulted
Income Borrower
1
Yes
Single
125K
No
2
No
Married
100K
No
3
No
Single
70K
No
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
7
Yes
Divorced 220K
60K
Home
Owner
Yes
No
NO
MarSt
Income
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
Married
Single, Divorced
< 80K
NO
NO
> 80K
YES
10
Training Data
Introduction to Data Mining
Model: Decision Tree
1/2/2009
6
Another Example of Decision Tree
MarSt
ID
Home
Owner
1
Yes
Marital
Status
Single
Annual Defaulted
Income Borrower
125K
NO
No
2
No
Married
100K
No
3
No
Single
70K
No
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
7
Yes
Divorced 220K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
60K
Married
Yes
Single,
Divorced
Home
Owner
NO
No
Income
< 80K
> 80K
YES
NO
There could be more than one tree that
fits the same data!
10
Introduction to Data Mining
1/2/2009
7
Decision Tree Classification Task
Tid
Attrib1
1
Yes
Large
Attrib2
125K
Attrib3
No
Class
2
No
Medium
100K
No
3
No
Small
70K
No
4
Yes
Medium
120K
No
5
No
Large
95K
Yes
6
No
Medium
60K
No
7
Yes
Large
220K
No
8
No
Small
85K
Yes
9
No
Medium
75K
No
10
No
Small
90K
Yes
Learn
Model
10
Tid
Att ib1
Attrib1
11
No
Small
Att ib2
Attrib2
55K
Att ib3
Attrib3
?
Cl
Class
12
Yes
Medium
80K
?
13
Yes
Large
110K
?
14
No
Small
95K
?
15
No
Large
67K
?
Apply
Model
Decision
Tree
10
Introduction to Data Mining
1/2/2009
8
Apply Model to Test Data
Test Data
Start from the root of tree.
Home
Owner
N
No
Home
Owner
Yes
NO
Marital
Status
M i d
Married
Annual Defaulted
Income Borrower
80K
?
10
No
MarSt
Single, Divorced
Income
< 80K
Married
NO
> 80K
YES
NO
Introduction to Data Mining
1/2/2009
9
Apply Model to Test Data
Test Data
Home
Owner
N
No
Home
Owner
Yes
NO
Marital
Status
M i d
Married
Annual
Income
Defaulted
Borrower
80K
?
10
No
MarSt
Single, Divorced
Income
< 80K
NO
Married
NO
> 80K
YES
Introduction to Data Mining
1/2/2009
10
Apply Model to Test Data
Test Data
Home
Owner
N
No
Home
Owner
Yes
NO
Marital
Status
M i d
Married
Annual Defaulted
Income Borrower
80K
?
10
No
MarSt
Single, Divorced
Income
< 80K
Married
NO
> 80K
YES
NO
Introduction to Data Mining
1/2/2009
11
Apply Model to Test Data
Test Data
Home
Owner
N
No
Home
Owner
Yes
NO
Marital
Status
M i d
Married
Annual Defaulted
Income Borrower
80K
?
10
No
MarSt
Single, Divorced
Income
< 80K
NO
Married
NO
> 80K
YES
Introduction to Data Mining
1/2/2009
12
Apply Model to Test Data
Test Data
Home
Owner
N
No
Home
Owner
Yes
NO
Marital
Status
M i d
Married
Annual
Income
Defaulted
Borrower
80K
?
10
No
MarSt
Single, Divorced
Income
< 80K
Married
NO
> 80K
YES
NO
Introduction to Data Mining
1/2/2009
13
Apply Model to Test Data
Test Data
Home
Owner
N
No
Home
Owner
Yes
NO
Marital
Status
M i d
Married
Annual Defaulted
Income Borrower
80K
?
10
No
MarSt
Single, Divorced
Income
< 80K
NO
Married
Assign Defaulted to
“No”
NO
> 80K
YES
Introduction to Data Mining
1/2/2009
14
Decision Tree Classification Task
Tid
Attrib1
1
Yes
Large
Attrib2
125K
Attrib3
No
Class
2
No
Medium
100K
No
3
No
Small
70K
No
4
Yes
Medium
120K
No
5
No
Large
95K
Yes
6
No
Medium
60K
No
7
Yes
Large
220K
No
8
No
Small
85K
Yes
9
No
Medium
75K
No
10
No
Small
90K
Yes
Tid
Attrib1
11
No
Small
55K
?
12
Yes
Medium
80K
?
13
Yes
Large
110K
?
14
No
Small
95K
?
15
No
Large
67K
?
Learn
Model
10
Attrib2
Attrib3
Class
Apply
M d l
Model
Decision
Tree
10
Introduction to Data Mining
1/2/2009
15
Decision Tree Induction
z
Many Algorithms:
– Hunt’s Algorithm (one of the earliest)
– CART
– ID3, C4.5
– SLIQ,SPRINT
Introduction to Data Mining
1/2/2009
16
General Structure of Hunt’s Algorithm
z
z
Let Dt be the set of training
records that reach a node t
General Procedure:
– If Dt contains records that
belong the same class yt,
then t is a leaf node
labeled as yt
– If Dt contains records that
belong to more than one
class,
l
use an attribute
tt ib t ttestt
to split the data into smaller
subsets. Recursively apply
the procedure to each
subset.
Introduction to Data Mining
ID
Home
Owner
Marital
Status
Annual Defaulted
Income Borrower
1
Yes
Single
125K
No
2
No
Married
100K
No
3
No
Single
g
70K
No
o
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
7
Yes
Divorced 220K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
60K
10
Dt
?
1/2/2009
17
Hunt’s Algorithm
ID
Home
Owner
Marital
Status
Annual Defaulted
Income Borrower
1
Yes
Single
125K
No
2
No
Married
100K
No
3
No
Single
70K
No
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
7
Yes
Divorced 220K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
60K
10
Introduction to Data Mining
1/2/2009
18
Design Issues of Decision Tree Induction
z
How should training records be split?
– Method for specifying test condition
‹
depending on attribute types
– Measure for evaluating the goodness of a test
condition
z
How should the splitting procedure stop?
– Stop splitting if all the records belong to the
same class or have identical attribute values
– Early termination
Introduction to Data Mining
1/2/2009
19
Methods for Expressing Test Conditions
z
Depends on attribute types
– Binary
– Nominal
– Ordinal
– Continuous
z
Depends on number of ways to split
– 2-way split
– Multi-way split
Introduction to Data Mining
1/2/2009
20
Test Condition for Nominal Attributes
z
Multi-way split:
– Use as many partitions as
distinct values.
z
Binary split:
– Divides values into two subsets
– Need
eed to
o find
d op
optimal
a
partitioning.
Introduction to Data Mining
1/2/2009
21
Test Condition for Ordinal Attributes
z
Multi-way split:
– Use as many partitions
as distinct values
z
Binary split:
– Divides values into two
subsets
– Need to find optimal
partitioning
– Preserve the order
property among
attribute values
Introduction to Data Mining
This grouping
violates order
property
1/2/2009
22
Test Condition for Continuous Attributes
Introduction to Data Mining
1/2/2009
23
Splitting Based on Continuous Attributes
z
Different ways of handling
– Discretization to form an ordinal categorical
attribute
tt ib t
Static – discretize once at the beginning
‹ Dynamic – ranges can be found by equal interval
bucketing, equal frequency bucketing
(percentiles), or clustering.
‹
– Binary Decision: (A < v) or (A ≥ v)
consider all possible splits and finds the best cut
‹ can be more compute-intensive
‹
Introduction to Data Mining
1/2/2009
24
How to determine the Best Split
Before Splitting: 10 records of class 0,
10 records of class 1
Which test condition is the best?
Introduction to Data Mining
1/2/2009
25
How to determine the Best Split
z
Greedy approach:
– Nodes with purer class distribution are
preferred
f
d
z
Need a measure of node impurity:
High degree of impurity
Introduction to Data Mining
Low degree of impurity
1/2/2009
26
Measures of Node Impurity
z
Gini Index
GINI (t ) = 1 − ∑ [ p ( j | t )]2
j
z
Entropy
Entropy (t ) = − ∑ p ( j | t ) log p ( j | t )
j
z
Misclassification error
Error (t ) = 1 − max P (i | t )
i
Introduction to Data Mining
1/2/2009
27
Finding the Best Split
1.
2.
Compute impurity measure (P) before splitting
Compute impurity measure (M) after splitting
z
z
3.
Compute impurity measure of each child node
Compute the average impurity of the children (M)
Choose the attribute test condition that
produces the highest gain
Gain = P – M
or equivalently, lowest impurity measure after
splitting (M)
Introduction to Data Mining
1/2/2009
28
Finding the Best Split
C0
C1
Before Splitting:
N00
N01
P
A?
B?
Yes
No
Node N1
C0
C1
Yes
Node N2
N10
N11
C0
C1
Node N3
N20
N21
C0
C1
M12
M11
No
Node N4
C0
C1
N30
N31
M21
N40
N41
M22
M1
M2
Gain = P – M1
vs
P – M2
Introduction to Data Mining
1/2/2009
29
Measure of Impurity: GINI
z
Gini Index for a given node t :
GINI (t ) = 1 − ∑ [ p ( j | t )]2
j
(NOTE: p( j | t) is the relative frequency of class j at node t).
– Maximum (1 - 1/nc) when records are equally
distributed among all classes, implying least
interesting information
– Minimum (0
(0.0)
0) when all records belong to one class
class,
implying most interesting information
C1
C2
0
6
Gini=0.000
C1
C2
1
5
Gini=0.278
Introduction to Data Mining
C1
C2
2
4
Gini=0.444
C1
C2
3
3
Gini=0.500
1/2/2009
30
Computing Gini Index of a Single Node
GINI (t ) = 1 − ∑ [ p ( j | t )]2
j
C1
C2
0
6
P(C1) = 0/6 = 0
C1
C2
1
5
P(C1) = 1/6
C1
C2
2
4
P(C1) = 2/6
Gini = 1 –
P(C2) = 6/6 = 1
P(C1)2 –
P(C2)2 = 1 – 0 – 1 = 0
P(C2) = 5/6
Gini = 1 – ((1/6))2 – ((5/6))2 = 0.278
P(C2) = 4/6
Gini = 1 – (2/6)2 – (4/6)2 = 0.444
Introduction to Data Mining
1/2/2009
31
Computing Gini Index for a Collection of
Nodes
z
When a node p is split into k partitions (children)
k
GINI splitli = ∑
i =1
where,
ni
GINI (i )
n
ni = number of records at child i,
n = number of records at parent node p.
z
Choose the attribute that minimizes weighted average
Gini index of the children
z
Gini index is used in decision tree algorithms such as
CART, SLIQ, SPRINT
Introduction to Data Mining
1/2/2009
32
Binary Attributes: Computing GINI Index
z
z
Splits into two partitions
Effect of Weighing partitions:
– Larger and Purer Partitions are sought for
for.
Parent
B?
Yes
No
C1
6
C2
6
Gini = 0.500
Gini(N1)
= 1 – (5/6)2 – (1/6)2
= 0.278
Node N1
Node N2
C1
C2
Gini(N2)
= 1 – (2/6)2 – (4/6)2
= 0.444
N1
5
1
N2
2
4
Gini=0.361
Introduction to Data Mining
Gini(Children)
= 6/12 * 0.278 +
6/12 * 0.444
= 0.361
1/2/2009
33
Categorical Attributes: Computing Gini Index
z
z
For each distinct value, gather counts for each class in
the dataset
Use the count matrix to make decisions
Multi-way split
Two-way split
(find best partition of values)
CarType
Family Sports Luxury
C1
C2
Gini
1
3
8
0
0.163
1
7
Introduction to Data Mining
C1
C2
Gini
CarType
{Sports,
{Family}
Luxury}
9
1
7
3
0.468
C1
C2
Gini
1/2/2009
CarType
{Family,
{Sports}
Luxury}
8
2
0
10
0.167
34
Continuous Attributes: Computing Gini Index
z
z
z
z
Use Binary Decisions based on one
value
Several Choices for the splitting value
– Number of possible splitting values
= Number of distinct values
Each splitting value has a count matrix
associated with it
– Class counts in each of the
partitions, A < v and A ≥ v
Simple method to choose best v
– For each vv, scan the database to
gather count matrix and compute
its Gini index
– Computationally Inefficient!
Repetition of work.
ID
Home
Owner
Marital
Status
Annual
Defaulted
Income
1
Yes
Single
125K
No
2
No
Married
100K
No
3
N
No
Si l
Single
70K
N
No
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
7
Yes
Divorced 220K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
60K
10
Introduction to Data Mining
1/2/2009
35
Continuous Attributes: Computing Gini Index...
z
For efficient computation: for each attribute,
– Sort the attribute on values
– Linearly scan these values, each time updating the count matrix
and
d computing
i gini
i i iindex
d
– Choose the split position that has the least gini index
Cheat
No
No
No
Yes
Yes
Yes
No
No
No
No
Annual Income
Sorted Values
60
Split Positions
70
55
75
65
85
72
90
80
95
87
100
92
97
120
110
125
122
220
172
230
<=
>
<=
>
<=
>
<=
>
<=
>
<=
>
<=
>
<=
>
<=
>
<=
>
<=
>
Yes
0
3
0
3
0
3
0
3
1
2
2
1
3
0
3
0
3
0
3
0
3
0
No
0
7
1
6
2
5
3
4
3
4
3
4
3
4
4
3
5
2
6
1
7
0
Gini
0.420
0.400
0.375
Introduction to Data Mining
0.343
0.417
0.400
0.300
0.343
1/2/2009
0.375
0.400
36
0.420
Measure of Impurity: Entropy
z
Entropy at a given node t:
Entropy (t ) = − ∑ p ( j | t ) log p ( j | t )
j
(NOTE: p( j | t) is the relative frequency of class j at node t).
‹ Maximum
(log nc) when records are equally distributed
among all classes implying least information
‹ Minimum (0.0) when all records belong to one class,
implying most information
– Entropy based computations are quite similar to
the GINI index computations
Introduction to Data Mining
1/2/2009
37
Computing Entropy of a Single Node
Entropy (t ) = − ∑ p ( j | t ) log p ( j | t )
j
C1
C2
0
6
P(C1) = 0/6 = 0
C1
C2
1
5
P(C1) = 1/6
C1
C2
2
4
P(C1) = 2/6
2
P(C2) = 6/6 = 1
Entropy = – 0 log 0 – 1 log 1 = – 0 – 0 = 0
P(C2) = 5/6
Entropy = – (1/6) log2 (1/6) – (5/6) log2 (1/6) = 0.65
P(C2) = 4/6
Entropy = – (2/6) log2 (2/6) – (4/6) log2 (4/6) = 0.92
Introduction to Data Mining
1/2/2009
38
Computing Information Gain After Splitting
z
Information Gain:
GAIN
⎛ n
⎞
= Entropy
py ( p ) − ⎜ ∑ Entropy
py (i ) ⎟
⎝ n
⎠
k
split
lit
i
i =1
Parent Node, p is split into k partitions;
ni is number of records in partition i
– Choose the split that achieves most reduction
(maximizes GAIN)
– Used in the ID3 and C4.5 decision tree algorithms
Introduction to Data Mining
1/2/2009
39
Problems with Information Gain
z
Info Gain tends to prefer splits that result in large
number of partitions, each being small but pure
– Customer ID has highest information gain
because entropy for all the children is zero
Introduction to Data Mining
1/2/2009
40
Gain Ratio
z
Gain Ratio:
GainRATIO
split
=
GAIN
n
n
SplitINFO = − ∑ log
SplitINFO
n
n
k
Split
i
i =1
Parent Node, p is split into k partitions
ni is the number of records in partition i
– Adjusts Information Gain by the entropy of the partitioning
(SplitINFO).
‹
Higher entropy partitioning (large number of small partitions) is
penalized!
– Used in C4.5 algorithm
– Designed to overcome the disadvantage of Information Gain
Introduction to Data Mining
1/2/2009
41
Measure of Impurity: Classification Error
z
Classification error at a node t :
Error (t ) = 1 − max P (i | t )
i
– Maximum (1 - 1/nc) when records are equally
distributed among all classes, implying least
interesting information
– Minimum (0) when all records belong to one class
class,
implying most interesting information
Introduction to Data Mining
1/2/2009
42
i
Computing Error of a Single Node
Error (t ) = 1 − max P (i | t )
i
C1
C2
0
6
P(C1) = 0/6 = 0
C1
C2
1
5
P(C1) = 1/6
C1
C2
2
4
P(C1) = 2/6
P(C2) = 6/6 = 1
Error = 1 – max (0, 1) = 1 – 1 = 0
P(C2) = 5/6
Error = 1 – max (1/6, 5/6) = 1 – 5/6 = 1/6
P(C2) = 4/6
Error = 1 – max (2/6, 4/6) = 1 – 4/6 = 1/3
Introduction to Data Mining
1/2/2009
43
Comparison among Impurity Measures
For a 2-class problem:
Introduction to Data Mining
1/2/2009
44
Misclassification Error vs Gini Index
Parent
A?
Yes
No
Node N1
Gini(N1)
= 1 – (3/3)2 – (0/3)2
=0
Gini(N2)
= 1 – (4/7)2 – (3/7)2
= 0.489
Node N2
C1
C2
N1
3
0
N2
4
3
Gini=0.342
Introduction to Data Mining
C1
7
C2
3
Gini = 0.42
Gini(Children)
= 3/10 * 0
+ 7/10 * 0.489
0 489
= 0.342
Gini improves but
error remains the
same!!
1/2/2009
45
Decision Tree Based Classification
z
Advantages:
– Inexpensive to construct
– Extremely fast at classifying unknown records
– Easy to interpret for small-sized trees
– Accuracy is comparable to other classification
techniques for many simple data sets
Introduction to Data Mining
1/2/2009
46