Download Slides

Data Mining
Classification: Alternative Techniques
Lecture Notes for Chapter 5
Rule-Based Classifier
l
Classify records by using a collection of
“if…then…” rules
l
Rule:
(Condition) → y
– where
Introduction to Data Mining
by
Tan, Steinbach, Kumar
© Tan,Steinbach, Kumar
Introduction to Data Mining
human
python
salmon
whale
frog
komodo
bat
pigeon
cat
leopard shark
turtle
penguin
porcupine
eel
salamander
gila monster
platypus
owl
dolphin
eagle
Blood Type
warm
cold
cold
warm
cold
cold
warm
warm
warm
cold
cold
warm
warm
cold
cold
cold
warm
warm
warm
warm
Give Birth
4/18/2004
yes
no
no
yes
no
no
yes
no
yes
yes
no
no
yes
no
no
no
no
no
yes
no
Can Fly
Live in Water
no
no
no
no
no
no
yes
yes
no
no
no
no
no
no
no
no
no
yes
no
yes
no
no
yes
yes
sometimes
no
no
no
no
yes
sometimes
sometimes
no
yes
sometimes
no
no
no
yes
no
1
(Taxable Income < 50K) ∧ (Refund=Yes) → Evade=No
Introduction to Data Mining
4/18/2004
2
R2: (Give Birth = no) ∧ (Live in Water = yes) → Fishes
R3: (Give Birth = yes) ∧ (Blood Type = warm) → Mammals
R4: (Give Birth = no) ∧ (Can Fly = no) → Reptiles
R5: (Live in Water = sometimes) → Amphibians
4/18/2004
Tid Refund Marital
A rule r covers an instance x if the attributes of
the instance satisfy the condition of the rule
R1: (Give Birth = no) ∧ (Can Fly = yes) → Birds
Name
hawk
grizzly bear
Blood Type
warm
warm
Give Birth
Can Fly
Live in Water
Class
no
yes
yes
no
no
no
?
?
The rule R1 covers a hawk => Bird
The rule R3 covers the grizzly bear => Mammal
3
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
4
How does Rule-based Classifier Work?
Taxable
Coverage of a rule:
Status
Income Class
1
Yes
Single
125K
No
– Fraction of records
2
No
Married 100K
No
that satisfy the
3
No
Single
70K
No
antecedent of a rule
4
Yes
Married 120K
No
5
No
Divorced 95K
Yes
l Accuracy of a rule:
6
No
Married 60K
No
– Fraction of the
7
Yes
Divorced 220K
No
records that satisfy
8
No
85K
Single
Yes
9
No
Married 75K
No
the antecedent, that
10 No
Single
90K
Yes
also satisfy the
(Status=Single)
→
No
consequent of a
rule
Coverage = 40%, Accuracy = 50%
0
1
Introduction to Data Mining
(Blood Type=Warm) ∧ (Lay Eggs=Yes) → Birds
u
© Tan,Steinbach, Kumar
l
Rule Coverage and Accuracy
© Tan,Steinbach, Kumar
u
Application of Rule-Based Classifier
mammals
reptiles
fishes
mammals
amphibians
reptiles
mammals
birds
mammals
fishes
reptiles
birds
mammals
fishes
amphibians
reptiles
mammals
birds
mammals
birds
Introduction to Data Mining
l
y is the class label
Class
R1: (Give Birth = no) ∧ (Can Fly = yes) → Birds
R2: (Give Birth = no) ∧ (Live in Water = yes) → Fishes
R3: (Give Birth = yes) ∧ (Blood Type = warm) → Mammals
R4: (Give Birth = no) ∧ (Can Fly = no) → Reptiles
R5: (Live in Water = sometimes) → Amphibians
© Tan,Steinbach, Kumar
Condition is a conjunction of attribute tests
u
– LHS: rule antecedent or condition
– RHS: rule consequent
– Examples of classification rules:
Rule-based Classifier (Example)
Name
u
4/18/2004
5
R1: (Give Birth = no) ∧ (Can Fly = yes) → Birds
R2: (Give Birth = no) ∧ (Live in Water = yes) → Fishes
R3: (Give Birth = yes) ∧ (Blood Type = warm) → Mammals
R4: (Give Birth = no) ∧ (Can Fly = no) → Reptiles
R5: (Live in Water = sometimes) → Amphibians
Name
lemur
turtle
dogfish shark
Blood Type
warm
cold
cold
Give Birth
Can Fly
Live in Water
Class
yes
no
yes
no
no
no
no
sometimes
yes
?
?
?
A lemur triggers rule R3, so it is classified as a mammal
A turtle triggers both R4 and R5
A dogfish shark triggers none of the rules
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
6
Characteristics of Rule-Based Classifier
l
l
From Decision Trees To Rules
Mutually exclusive rules
– Every record is covered by at most one rule
Classification Rules
(Refund=Yes) ==> No
Refund
Exhaustive rules
– Classifier has exhaustive coverage if it
accounts for every possible combination of
attribute values
– Each record is covered by at least one rule
Yes
No
NO
Marita l
Status
{Single,
Divorced}
(Refund=No, Marital Status={Single,Divorced},
Taxable Income<80K) ==> No
{Married}
(Refund=No, Marital Status={Single,Divorced},
Taxable Income>80K) ==> Yes
(Refund=No, Marital Status={Married}) ==> No
NO
Taxable
Income
< 80K
> 80K
NO
YES
Rules are mutually exclusive and exhaustive
Rule set contains as much information as the
tree
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
7
Rules Can Be Simplified
© Tan,Steinbach, Kumar
No
NO
Marita l
Status
{Single,
Divorced}
{Married}
NO
Taxable
Income
< 80K
> 80K
NO
YES
Tid Refund Marital
Status
Taxable
Income Cheat
1
Yes
Single
125K
No
2
No
Married
100K
No
3
No
Single
70K
No
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
7
Yes
Divorced 220K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
l
60K
u
l
(Refund=No) ∧ (Status=Married) → No
Introduction to Data Mining
4/18/2004
9
Ordered Rule Set
l
Use a default class
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
10
Rule Ordering Schemes
Rules are rank ordered according to their priority
l
Rule-based ordering
l
Class-based ordering
– An ordered rule set is known as a decision list
l
Unordered rule set: use voting schemes
Ordered rule set
Rules are no longer exhaustive
– A record may not trigger any rules
– Solution?
u
Simplified Rule: (Status=Married) → No
© Tan,Steinbach, Kumar
8
Rules are no longer mutually exclusive
– A record may trigger more than one rule
– Solution?
u
1
0
Initial Rule:
4/18/2004
Effect of Rule Simplification
Refund
Yes
Introduction to Data Mining
– Individual rules are ranked based on their quality
When a test record is presented to the classifier
– It is assigned to the class label of the highest ranked rule it has
triggered
– Rules that belong to the same class appear together
– If none of the rules fired, it is assigned to the default class
R1: (Give Birth = no) ∧ (Can Fly = yes) → Birds
R2: (Give Birth = no) ∧ (Live in Water = yes) → Fishes
R3: (Give Birth = yes) ∧ (Blood Type = warm) → Mammals
R4: (Give Birth = no) ∧ (Can Fly = no) → Reptiles
R5: (Live in Water = sometimes) → Amphibians
Name
turtle
© Tan,Steinbach, Kumar
Blood Type
cold
Give Birth
Can Fly
Live in Water
Class
no
no
sometimes
?
Introduction to Data Mining
4/18/2004
11
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
12
Building Classification Rules
l
Direct Method: Sequential Covering
Direct Method:
1.
Extract rules directly from data
u e.g.: RIPPER, CN2, Holte’s 1R
u
2.
3.
4.
l
Indirect Method:
Start from an empty rule-list
Grow a rule using the Learn-One-Rule function
Remove training records covered by the rule
Repeat Step (2) and (3) until stopping criterion
is met
Extract rules from other classification models (e.g.
decision trees, neural networks, etc).
u e.g: C4.5rules
u
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
13
Example of Sequential Covering
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
14
Example of Sequential Covering…
R1
R1
R2
(iii) Step 2
(ii) Step 1
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
15
Aspects of Sequential Covering
l
Rule Growing
l
Instance Elimination
l
Rule Evaluation
l
Stopping Criterion
l
Rule Pruning
© Tan,Steinbach, Kumar
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
16
4/18/2004
18
Rule Growing
l
Introduction to Data Mining
(iv) Step 3
4/18/2004
17
Two common strategies
© Tan,Steinbach, Kumar
Introduction to Data Mining
Rule Growing (Examples)
l
Rule growing: Ripper
CN2 Algorithm:
Suppose R0 covers 60 positive cases and 40 negative.
We compare two possible refinements of R0:
1. R1 covers 50 positive and 10 negative cases.
2. R2 covers 30 positive and 5 negative cases.
– Start from an empty conjunct: {}
– Add conjuncts that minimizes the entropy measure: {A}, {A,B}, …
– Determine the rule consequent by taking majority class of instances
covered by the rule
l
RIPPER Algorithm:
Their information gains are:
– Start from an empty rule: {} => class
– Add conjuncts that maximizes FOIL’s information gain measure:
R0: {} => class (initial rule)
R1: {A} => class (rule after adding conjunct)
u Gain(R0, R1) = p1 [ log (p1/(p1+n1)) – log (p0/(p0 + n0)) ]
u where p0: number of positive instances covered by R0
n0: number of negative instances covered by R0
p1: number of positive instances covered by R1
n1: number of negative instances covered by R1
u
u
© Tan,Steinbach, Kumar
Introduction to Data Mining
Hence we prefer refinement R1 even though it has lower
accuracy.
4/18/2004
19
Instance Elimination
l
© Tan,Steinbach, Kumar
Introduction to Data Mining
Why do we need to
eliminate instances?
If we don’t remove the cases
covered by R1 then R3 covers
8 positive cases and 4 negative
cases. Otherwise it covers 6
positive and 2 negative cases.
Why do we remove
positive instances?
– Ensure that the next rule is
different
l
For R2 it doesn’t make any
difference: it covers 7 positive
and 3 negative cases.
Why do we remove
negative instances?
– Prevent underestimating
accuracy of rule
– Compare rules R2 and R3
in the diagram
© Tan,Steinbach, Kumar
Introduction to Data Mining
So if we don’t remove the covered cases, the accuracy of
R3 is underestimated (8/12 instead of 6/8).
4/18/2004
21
Rule Evaluation
l
Metrics:
– Accuracy
– Laplace
n
= c
n
n +1
= c
n+k
n + kp
– M-estimate = c
n+k
20
Instance Elimination
– Otherwise, the next rule is
identical to previous rule
l
4/18/2004
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
22
Rule evaluation: example
R: condition ⇒ class = c
Suppose we have a 2-class problem with 60 positive and
100 negative examples. Consider
1. R1 covers 50 positive examples and 5 negative.
2. R2 covers 2 positive examples and no negative.
Accuracy of R1 is 50/55 = 0.91. Accuracy of R2 is 2/2 = 1.
n : Number of instances
covered by rule
nc : Number of instances
of class c covered by rule
With Laplace correction this becomes:
R1: (50+1)/(55+2) = 0.89
R2: (2+1)/(2+2) = 0.75
k : Number of classes
p : Prior probability of
class c
With m-estimate (p=60/160=3/8):
R1: (50+2×3/8)/(55+2)=0.89 R2: (2+2×3/8)/(2+2) = 0.69
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
23
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
24
Stopping Criterion and Rule Pruning
l
l
Summary of Direct Method
Stopping criterion
– Compute the gain
– If gain is not significant, stop growing rule
Rule Pruning
– Similar to post-pruning of decision trees
– Reduced Error Pruning:
Remove one of the conjuncts in the rule
Compare error rate on validation set before and
after pruning
u If error improves, prune the conjunct
u
l
Grow a single rule
l
Prune the rule (if necessary)
l
Remove instances of rule
l
Add rule to Current Rule Set
l
Repeat
u
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
25
Direct Method: RIPPER
l
l
Introduction to Data Mining
4/18/2004
27
Pruning in Ripper: Example (cf. exc. 2)
Suppose we have the rule R: A and B ⇒ class=c.
Suppose we have a validation set (not used for growing
the rule) that contains 500 positive examples (i.e. with
class=c) and 500 negative examples. Suppose that among
all cases in the validation set that satisfy condition A,
there are 200 positive examples and 50 negative examples.
Among all cases in the validation set that meet both condition
A and condition B, there are 100 positive examples and 5
negative examples. Should condition B be pruned?
© Tan,Steinbach, Kumar
Introduction to Data Mining
Introduction to Data Mining
4/18/2004
26
Direct Method: RIPPER
For 2-class problem, choose one of the classes as
positive class, and the other as negative class
– Learn rules for positive class
– Negative class will be default class
For multi-class problem
– Order the classes according to increasing class
prevalence (fraction of instances that belong to a
particular class)
– Learn the rule set for smallest class first, treat the rest
as negative class
– Repeat with next smallest class as positive class
© Tan,Steinbach, Kumar
© Tan,Steinbach, Kumar
4/18/2004
29
l
Growing a rule:
– Start from empty rule-body
– Add conjuncts as long as they improve (?) FOIL’s
information gain
– Stop when rule no longer covers negative examples
– Prune the rule immediately using incremental reduced
error pruning
– Measure for pruning: v = (p-n)/(p+n)
p: number of positive examples covered by the rule in
the validation set
u n: number of negative examples covered by the rule in
the validation set
u
– Pruning method: delete any final sequence of
conditions that maximizes v
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
28
Pruning in Ripper
LHS Rule
v=(p-n)/(p+n)
{}
(500-500)/1000=0
{A}
(200-50)/250=0.6
{A,B}
(100-5)/105=0.9
Since {A,B} maximizes v, the rule is not pruned.
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
30
Direct Method: RIPPER
l
Indirect Methods
Building a Rule Set:
– Use sequential covering algorithm
Finds the best rule that covers the current set of
positive examples
u Eliminate both positive and negative examples
covered by the rule
u
– Each time a rule is added to the rule set,
compute the new description length
stop adding new rules when the new description
length is d bits longer than the smallest description
length obtained so far (description length depends on
accuracy and complexity of the rule).
u
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
31
© Tan,Steinbach, Kumar
Introduction to Data Mining
Indirect Method: C4.5rules
Rule Pruning in C4.5rules
Extract rules from an unpruned decision tree
l For each rule, R: A → class=c,
– consider an alternative rule R-: A- → class=c
where A- is obtained by removing one of the
conditions in A
– Compare the pessimistic error rate for R
against all R- s.
– Prune if one of the R- s has lower pessimistic
error rate
– Repeat until we can no longer improve
generalization error
R: if A then class=c
l
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
33
4/18/2004
R-: if A- then class=c
where A- has one condition less than A.
Let X be the condition considered for removal from A.
Make a cross-table:
A- and
X
Not X
Class=c
Class ≠ c
Y1
Y2
E1
E2
R covers Y1 + E1 cases, and makes E1 errors.
Pessimistic estimate UCF(E1,Y1 + E1).
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
Rule Pruning in C4.5rules
Rule Pruning in C4.5rules
Consider the rule
Condition
Deleted
TSH > 6
Y1+Y2
E1+E2
3
1
Pessimistic
Error rate
55%
FTI ≤ 64
6
1
34%
TSH
2
measured=t
1
68%
2
T4U
measured=t
1
68%
Thyroid
surgery=t
59
97%
if TSH > 6
FTI ≤ 64
TSH measured = true
T4U measured = true
thyroid surgery = true
then class negative
Suppose this rule covers 3 training cases, 2 of which have
class negative. The pessimistic error estimate, using CF=0.25
is U25%(1,3) is 68%, because P(E ≤ 1) is approximately 0.25
under a binomial distribution with N=3, and probability of
error is 0.68 on each draw.
32
3
34
Lowest pessimistic error is obtained by deleting FTI ≤ 64
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
35
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
36
Indirect Method: C4.5rules
l
Example
Name
Instead of ordering the rules, order subsets of
rules (class ordering)
– Each subset is a collection of rules with the
same rule consequent (class)
– Compute description length of each subset
human
python
salmon
whale
frog
komodo
bat
pigeon
cat
leopard shark
turtle
penguin
porcupine
eel
salamander
gila monster
platypus
owl
dolphin
eagle
Description length = L(error) + g L(model)
g is a parameter that takes into account the
presence of redundant attributes in a rule set
(default value = 0.5)
u
u
– Order according to increasing description
length
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
37
C4.5 versus C4.5rules versus RIPPER
No
(Give Birth=Yes) → Mammals
(Give Birth=No, Can Fly=No, Live in Water=No) → Reptiles
Live In
Water?
Mammals
Yes
( ) → Amphibians
RIPPER:
No
(Live in Water=Yes) → Fishes
Fishes
Yes
Birds
© Tan,Steinbach, Kumar
(Give Birth=No, Can Fly=No, Live In Water=No)
→ Reptiles
Can
Fly?
Amphibians
(Can Fly=Yes,Give Birth=No) → Birds
No
() → Mammals
no
no
yes
yes
sometimes
no
no
no
no
yes
sometimes
sometimes
no
yes
sometimes
no
no
no
yes
no
yes
no
no
no
yes
yes
yes
yes
yes
no
yes
yes
yes
no
yes
yes
yes
yes
no
yes
Class
mammals
reptiles
fishes
mammals
amphibians
reptiles
mammals
birds
mammals
fishes
reptiles
birds
mammals
fishes
amphibians
reptiles
mammals
birds
mammals
birds
4/18/2004
38
PREDICT ED CLASS
Amphibians Fishes Reptiles Birds
ACT UAL Amphibians
2
0
0
CLASS Fishes
0
2
0
Reptiles
1
0
3
Birds
1
0
0
M ammals
0
0
1
0
0
0
3
0
M ammals
0
1
0
0
6
PREDICT ED CLASS
Amphibians Fishes Reptiles Birds
ACT UAL Amphibians
0
0
0
CLASS Fishes
0
3
0
Reptiles
0
0
3
Birds
0
0
1
M ammals
0
2
1
0
0
0
2
0
M ammals
2
0
1
1
4
Reptiles
Introduction to Data Mining
4/18/2004
39
Advantages of Rule-Based Classifiers
As highly expressive as decision trees
Easy to interpret
l Easy to generate
l Can classify new instances rapidly
l Performance comparable to decision trees
l
l
© Tan,Steinbach, Kumar
Introduction to Data Mining
Live in Water Have Legs
RIPPER:
(Have Legs=No) → Reptiles
Sometimes
Can Fly
no
no
no
no
no
no
yes
yes
no
no
no
no
no
no
no
no
no
yes
no
yes
C4.5 and C4.5rules:
(Give Birth=No, Can Fly=Yes) → Birds
(Give Birth=No, Live in Water=Yes) → Fishes
Yes
© Tan,Steinbach, Kumar
Lay Eggs
no
yes
yes
no
yes
yes
no
yes
no
no
yes
yes
no
yes
yes
yes
yes
yes
no
yes
C4.5 versus C4.5rules versus RIPPER
C4.5rules:
Give
Birth?
Give Birth
yes
no
no
yes
no
no
yes
no
yes
yes
no
no
yes
no
no
no
no
no
yes
no
Introduction to Data Mining
4/18/2004
41
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
40