Download apriori

CSE 711 Seminar on Data Mining:
Apriori Algorithm
By
Sung-Hyuk Cha
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
1
Association Rules
Definition: Rules that state a statistical correlation
between the occurrence of certain attributes in a
database table.
Given a set of transactions, where each transaction is a
set of items, X1 , ... , Xn and Y, an association rule is an
expression X1 , ... , Xn Y.
This means that the attributes X1 , ... , Xn predict Y
Intuitive meaning of such a rule: transactions in the
database which contain the items in X tend also to
contain the items in Y.
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
2
Measures for an Association Rule
Support :
• Given the association rule X1 , ... , Xn Y, the support is
the percentage of records for which X1 , ... , Xn and Y both
hold.
•The statistical significance of the association rule.
Confidence :
• Given the association rule X1 , ... , Xn Y, the confidence
is the percentage of records for which Y holds, within the
group of records for which X1 , ... , Xn hold.
• The degree of correlation in the dataset between X and Y.
• A measure of the rule’s strength.
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
3
Quiz # 2
Problem:
Given a transaction table D, find the support and
confidence for an association rule B,D E.
Database D
TID Items
01
ABE
02 A C D E
03 B C D E
04 A B D E
05
BDE
06
ABC
07
ABD
Answer:
support = 3/7,
2/8/00
confidence = 3/4
CSE 711 data mining: Apriori
Algorithm by S. Cha
4
Apriori Algorithm
An efficient algorithm to find association rules.
 Procedure
 Find all the frequent itemsets :
A frequent itemset is a set of items that
have support greater than a user defined
minimum.
 Use the frequent itemsets to generate the
association rules
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
5
Notation
D
The sample transaction database
F
The set of all frequent items.
k-itemset
Lk
Ck
2/8/00
An itemset having k items.
Set of candidate k-itemsets (those with minimum
support). Each member of this set has two fields:
i) itemset and ii) support count.
Set of candidate k-itemsets (potentially frequent
itemsets). Each member of this set has two fields:
i) itemset and ii) support count.
CSE 711 data mining: Apriori
Algorithm by S. Cha
6
Example
Database D
* Suppose a user
defined minimum
= .49
(k = 1) itemset
C1 Support
{A}
.50
{B}
.75
{C}
.75
{D}
.25
{E}
.75
TID
Items
100 A C D
200
BCE
300 A B C E
400
BE
(k = 2) itemset
L1
Y
Y
Y
N
Y
C2
Support L2
{A,B}
.25
N
{A,C}
.50
Y
{A,E}
.25
N
{B,C}
.50
Y
{B,E}
.75
Y
{C,E}
.50
Y
(k = 3) itemset
C3
Support L3
{B,C,E}
.50
Y
(k = 4) itemset
C4
Support L4
{A,B,C,E}
.25
N
* n items implies O(n2 - 2) computational complexity?
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
7
Procedure
Apriorialgo()
{
F = ;
Lk = {frequent 1-itemsets};
k = 2;
/* k represents the pass number. */
while (Lk-1 != )
{
F = F U Lk ;
Ck = New candidates of size k generated from Lk-1 ;
for all transactions t  D
increment the count of all candidates in Ck that are contained in t ;
Lk = All candidates in Ck with minimum support ;
k++ ;
}
return ( F ) ;
}
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
8
Candidate Generation
Given Lk-1, the set of all frequent (k-1)-itemsets,
generate a superset of the set of all frequent k-itemsets.
Idea : if an itemset X has minimum support, so do all subsets of X.
1. Join Lk-1 with Lk-1
2. Prune: delete all itemsets c  Ck such that some (k-1)-subset of c is
not in Lk-1 .
ex) L2 = { {A,C}, {B,C}, {B,E}, {C,E} }
1. Join : { {A,B,C}, {A,C,E}, {B,C,E} }
2. Prune : { {A,B,C}, {A,C,E}, {B,C,E} }
{A, B}  L2
{A, E}  L2
Instead of 5C3 = 10, we have only 1 candidate.
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
9
Thoughts
Association rules are always defined on binary attributes.
 Need to flatten the tables.
ex)
Phone Company DB.
CID
Gender
CID
M
F
Ethnicity
W
B
H
Call
A
D
I
- Support for Asian ethnicity will never exceed .5.
- No need to consider itemsets {M,F}, {W,B} nor {D,I}.
- M  F or D  I are not of interest at all.
* Considering the original schema before flattening
may be a good idea.
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
10
Finding association rules
with item constraints
When item constraints are considered, the Apriori candidate
generation procedure does not generate all the potential
frequent itemsets as candidates.
 Procedure
1. Find all the frequent itemsets that satisfy the
boolean expression B.
2. Find the support of all subsets of frequent
itemsets that do not satisfy B.
3. Generate the association rules from the
frequent itemsets found in Step 1. by computing
confidences from the frequent itemsets found in
Steps 1 & 2.
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
11
Additional Notation
B
Di
S
Boolean expression with m disjuncts:
B = D1  D2  ...  Dm
N conjuncts in Di,
Di = ai,1  ai,2  ...  ai,n
Set of items such that any itemset that satisfies
B contains an item from S.
Ls(k)
Set of frequent k-itemsets that contain an item in S.
Lb(k)
Set of frequent k-itemsets that satisfy B.
Cs(k)
Set of candidate k-itemsets that contain an item in S.
Cb(k)
Set of candidate k-itemsets that satisfy B.
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
12
Direct Algorithm
Procedure
1. Scan the data and determine L1 and F.
2. Find Lb(1)
3. Generate Cb(k+1) from Lb(k)
3-1. Ck+1 = Lk x F
3-2. Delete all candidates in Ck+1 that do not satisfy B.
3-3. Delete all candidates in Ck+1 below the minimum
support.
3-4. for each Di with exactly k + 1 non-negated elements,
add the itemset to Ck+1 if all the items are frequent.
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
13
Example
Database D
TID
Items
100 A C D
200
BCE
300 A B C E
400
BE
Given B = (A  B)  (C  E)
C1 = { {A}, {B}, {C}, {D}, {E} }
L1 = { {A}, {B}, {C}, {E} }
step
1&2
Lb(1) = { C }
step 3-1
C2 = Lb(1) x F
= { {A,C}, {B,C}, {C,E} }
step 3-2
Cb(2) =
{ {A,C}, {B,C} }
step 3-3
L2 = { {A,C}, {B,C} }
step 3-4
Lb(2) = { {A,B},
{A,C}, {B,C} }
step 3-1
C3 = Lb(2) x F = { {A,B,C},
{A,B,E}, {A,C,E}, {B,C,E} }
step 3-2
Cb(3) = { {A,B,C},
{A,B,E} }
step 3-3
L3 = 
step 3-4
Lb(3) = 
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
14
MultipleJoins and Reorder algorithms
to find association rules
with item constraints will be added.
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
15
Mining Sequential Patterns
Given a database D of customer transactions, the problem of
mining sequential patterns is to find the maximal sequences
among all sequences that have certain user-specified
minimum support.
- Transaction-time field is added.
- Itemset in a sequence is denoted as
<s1, s2, …, sn>
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
16
Sequence Version of DB Conversion
Sequential version of D’
D
CustomerID Transaction Time
1
Jun 25 93
1
Jun 30 93
2
Jun 10 93
2
Jun 15 93
2
Jun 20 93
3
Jun 25 93
4
Jun 25 93
4
Jun 30 93
4
July 25 93
5
Jun 12 93
Items
30
90
10,20
30
40,60,70
30,50,70
30
40,70
90
90
CustomerID
Customer Sequence
1
2
3
4
5
<(30),(90)>
<(10 20),(30),(40 60 70)>
<(30 50 70)>
<(30),(40 70),(90)>
<(90)>
* Customer sequence : all the
transactions of a customer is a
sequence ordered by increasing
transaction time.
Answer set with support > .25 = { <(30),(90)>, <(30),(40 70)> }
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
17
Definitions
Def 1. A sequence <a1, a2, …, an> is contained in another
sequence <b1, b2, …, bm> if there exists integers i1 < i2 < …
< in such that a1  bi1, a2  bi2 , … , an  bin
Yes
ex) <(3), (4 5), (8)> is contained in <(7) ,(3 8),(9),(4 5 6),(8)>.
<(3), (5)> is contained in <(3 5)>.No
Def 2. A sequence s is maximal if s is not contained in any
other sequence.
- Ti is transaction time.
- itemset(Ti) is transaction the set of items in Ti.
- litemset : an item set with minimum support.
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
18
Procedure
Procedure
1. Convert D into a D’ of customer sequences.
2. Litemset mapping
3. Transform each customer sequence into a litemset
representation. <s1, s2, …, sn>  <l1, l3, …, ln>
4. Find the desired sequences using the set of litemsets.
4-1. AprioriAll
4-2. AprioriSome
4-3. DynamicSome
5. Find the maximal sequences among the set of large
sequences.
for(k = n; k > 1; k--)
foreach k-sequence sk
delete from S all subsequences of sk.
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
19
Example
step 2
Large Itemsets
Mapped to
(30)
(40)
(70)
(40 70)
(90)
1
2
3
4
5
step 3
CID
1
2
3
4
5
Customer Sequence
Transformed Sequence
Mapping
<(30),(90)>
<{(30)}{(90)}>
<{1}{5}>
<(10 20),(30),(40 60 70)>
<{(30)}{(40),(70),(40 70)}>
<{1}{2,3,4}>
<(30 50 70)>
<{(30),(70)}>
<{1,3}>
<(30),(40 70),(90)>
<{(30)}{(40),(70),(40 70)}{(90)}> <{1}{2,3,4}{5}>
<(90)>
<{(90)}>
<{5}>
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
20
AprioriAll
Aprioriall()
{
Lk = {frequent 1-itemsets};
k = 2;
/* k represents the pass number. */
while (Lk-1 != )
{
F = F U Lk ;
Ck = New candidates of size k generated from Lk-1 ;
for each customer-sequence c  D
increment the count of all candidates in Ck that are contained in c ;
Lk = All candidates in Ck with minimum support ;
k++ ;
}
return ( F ) ;
}
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
21
Example
Customer Seq’s.
Minimum support = .40
<{1,5}{2}{3}{4}>
<{1}{3}{4}{3,5}>
<{1}{2}{3}{4}>
<{1}{3}{5}>
<{4}{5}>
L1
Supp
<1>
<2>
<3>
<4>
<5>
2/8/00
4
2
4
4
4
L2
Supp
<12>
<13>
<14>
<15>
<23>
<24>
<34>
<35>
<45>
2
2
3
2
2
2
2
3
2
C4
L3
Supp
<123>
<124>
<134>
<135>
<234>
2
2
3
2
2
<1234>
<1243>
<1345>
<1354>
L4
Supp.
<1234>
2
The maximal large sequences are
{<1 2 3 4>, <1 3 5>, <4 5>}.
CSE 711 data mining: Apriori
Algorithm by S. Cha
22
AprioriSome and DynamicSome algorithms
to find association rules
with sequential patterns will be added.
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
23
GSC features
-1
tan
Sy(i,j)
Sx(i,j)
Gradient (local) Structural (intermediate) and Concavity (global)
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
24
GSC feature table
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
25
A sample GSC features
Gradient : 000000000011000000001100001110000000111000000011000000
11000100000000110000000000000111001100011111000011110000000010
01010000010001110011111001111100000100000100000000000000000000
01000001001000
(192)
Structure : 000000000000000000001100001110001000010000100000010000
000000000100101000000000011000010100110000110000000000000100100
011001100000000000000110010100000000000001100000000000000000000
000000010000
(192)
Concavity : 11110110100111110110011000000110111101101001100100000
110000011100000000000000000000000000000000000000000111111100000
000000000000
(128)
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
26
Class A
800 samples
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
27
Class A, B and C
A
2/8/00
B
CSE 711 data mining: Apriori
Algorithm by S. Cha
C
28
Reordered by Frequency
A
2/8/00
B
CSE 711 data mining: Apriori
Algorithm by S. Cha
C
29
Associate Rules in GSC
- G  S, G  C
- F1, F2, F3  “A”
- F1  F2
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
30
References
Agrawal, R.; Imielinski, T.; and Swami, A. “Mining Association Rules
between Sets of Items in Large Databases.” Proc. Of the ACM SIGMOD
Conference on Management of Data, 207-216, 1993
Agrawal, R. and Srikant, R. “Fast Algorithms for Mining Association
Rules in Large Databases” Proc. Of the 20th Int’l Conference on Very
Large Databases, p478-499, Sept. 1994
Agrawal, R. and Srikant, R. “Mining Sequential Patterns”, Research
Report RJ 9910, IBM Almaden Research Center, San Jose, California,
October 1994
Agrawal, R. and Shafer, J. “Parallel Mining of Association Rules.” IEEE
Transactions on Knowledge and Data Engineering. 8(6), 1996
2/8/00
CSE 711 data mining: Apriori
Algorithm by S. Cha
31
MultipleJoins Algorithm
Generate the Selected Item Set()
{
S =;
for each Di, i = 1 to m ;
{
for each ai,j, j = 1 to n ;
cost of conjunct = support(S U ai,j) - support(ai,j) ;
Add
i,j with the minimum cost to S ;
 aProcedure
}
1. Scan the data and determine F.
}
2. Ls(1) = S  F
Database D
...
B = (A  B)  (C  E)
TID
Items
100 A C D
200
BCE
300 A B C E
400
BE
2/8/00
The algorithm gives S = { A, C }.
CSE 711 data mining: Apriori
Algorithm by S. Cha
32