The Apriori Algorithm: The Apriori
t
Algorithm is an influential algori mining frequent
itemsets for boolean associ
Key Concepts :
h
Frequent Itemsets: The sets of item which support
i t
(denoted by L for i h-Itemset). •
e
Apriori Property: Any subset of frequent it frequent. •
k, a set of candidat Join Operation: To find L •
k
is generated by joining L -1 with itself.
The Apriori Algorithm
Nutshell
s
Find the frequent itemsets: the sets of item minimum
support •
s
– A subset of a frequent itemset must al frequent
itemset
t
• i.e., if {AB} is a frequent itemset, bo should be a
frequent itemset
a
– Iteratively find frequent itemsets with c from 1 to
k (k-itemset)
Use the frequent itemsets to generate asso•
P
The Apriori Algorithm : code
k
k is generated by joining L -1with itself
a
Prune Step: Any (k-1)-itemset that is not frequent c subset of a
frequent k-itemset
k;
increment the count of all candidates in
k
C +1 with min_supp
L
as
t in
e
dat
do
ab
• Pseudo-code: Ck: Candidate itemset of size k
k: frequent itemset of size k
∅; k++) do begin
;
1 = {frequent items} for
k
C +1 = candidates generated from
for each transaction
t that are contained in
= candidates in
k
Join k != L L +1 retur
n
Step: k=
C• k;
• ( end L
1; k
L L ∪
The Apriori Algorithm: Exa• Consider a
s n p
databa consisting of 9 tra • Suppose min. sup required is 2 (i.e.
i
m 22 % )
d
Let minimum confi is 70%. •
n i
We have to first fi frequent itemset us algorithm. •
r i
Then, Association generated using m min. confidence. •
s f e T I , 2 I 1
5
L t I m I 1 I , 0
s D T 0
i o t
I 0 1 I4 , 0 1 I5 ,
0 T 0 T I3
2 0 I3 I2 0
1
0 1 0 T
, T , 0
0 1
I2 I1 0
I 1 I1
0 I3
I1 0
4 , , ,
0 0
, T I1
I2
T I3 0 I3
I2 I1 ,I ,
1
1 T T
, 3, I2
I
Step 1: Generating 1-itemset FrequeItemset
[Link] Compare candidate
support count with
minimum support
count
1 a
• The set of frequent 1-itemsets, L , consists of the c itemsets
satisfying minimum support.
e
• In the first iteration of the algorithm, each item is a m of
candidate.
6
Scan t of {I3}
D for
6 {I1} each {I2}
cand
7 idate
coun
2 2
{I4} {I5}
C1
Step 2: Generating 2-itemset FrequeItemset
Compare
candidate
support
count with
minimum
support
count
It Cou I2} 1
e nt 4
{I1, {I2, {I2, {I2, {I3, {I3, {I4,
S
I5} I3} I4} I5} I4} I5} I5}
m 4 {I1,
u
s {I1, I4} 4 2 2 0 1 0
p.
{I1, I3} C2
et 2
D for each
coun cand
Scan t of idate
{I1, {I1, {I1, {I2, {I2, {I2, {I3, {I3, {I4,
{I1,
I3} I4} I5} I3} I4} I5} I4} I5} I5}
C2
I2}
Generate
2
C
candidates
1
from L
Step 2: Generating 2-itemset Freque• To
discover the set of frequent 2-itemsets, L algorithm
1 1 d 2
uses L Join L to generate a can 2-itemsets, C .
d
Next, the transactions in D are scanned an count for
2 c
each candidate itemset in C is a (as shown in the
middle table). •
2
The set of frequent 2-itemsets, L , is then
d
consisting of those candidate 2-itemsets in minimum
support. •
Note: We haven’t used Apriori Property yet.•
Step 3: Generating 3-itemset FrequeItemset
count with
3 i
• The generation of the set of candidate 3-itemsets, C , the
Apriori Property.
3 I
• C = L2 Join L2 = {{I1, I2, I3}, {I1, I2, I5}, {I1, I3, I5}, {I2, I5},
{I2, I4, I5}}.
• Now, Join step is complete and Prune step will be used size of
3
C . Prune step helps to avoid heavy computation
Compare
candidate
Sup. Itemset
support
Count
o
min supp count 2
{I1, I2, I3}
2 {I1, I2, I5}
C3
3 2 2
• In order to find C , we compute L Join L .
Scan D for coun t of
each cand idate
} I5}
{I1, I3 3
C
I2, {I1,
I2,
Scan coun each idate
D for t of cand
Step 3: Generating 3-itemset Freque• Based on
t
the Apriori property that all subsets of a frequen also be frequent, we
a
can determine that four latter candid possibly be frequent. How ?
For example , lets take {I1, I2, I3}. The 2-item subsets of it I3} & {I2, I3}.
e 3
Since all 2-item subsets of {I1, I2, I3} are m will keep {I1, I2, I3} in C .•
Lets take another example of {I2, I3, I5} which shows how performed.
The 2-item subsets are {I2, I3}, {I2, I5} & {I3,I5}.•
2 q
BUT, {I3, I5} is not a member of L and hence it is not fre Apriori
Property. Thus We will have to remove {I2, I3, I5} fr•
3 l
Therefore, C = {{I1, I2, I3}, {I1, I2, I5}} after checking for a result of
Join operation for Pruning. •
i
Now, the transactions in D are scanned in order to determ of those
3
candidates 3-itemsets in C having minimum supp•
Step 4: Generating 4-itemset Freque• The
3 3
algorithm uses L Join L to generate a set of
4 l
4-itemsets, C . Although the join resu I3, I5}}, this
s
itemset is pruned since its sub is not frequent.
4 , a
Thus, C = φ and algorithm terminates, h all of
r
the frequent items. This completes ou Algorithm. •
What’s Next ? •
These frequent itemsets will be used to
n n
ge association rules ( where strong associatio both
minimum support & minimum confiden
o
Step 5: Generating Association Rules fr Itemsets
Procedure: •
n
For each frequent itemset “l”, generate all no of l. •
For every nonempty subset s of l, output the
l _
ru support_count(l) / support_count(s) >= min min_conf is
minimum confidence threshold. •
Back To Example: •
,
We had L = {{I1}, {I2}, {I3}, {I4}, {I5}, {I1,I2}, {I1,I3}, {I1 {I2,I4},
{I2,I5}, {I1,I2,I3}, {I1,I2,I5}}.
Lets take l = {I1,I2,I5}. –
Its all nonempty subsets are {I1,I2}, {I1,I5}, {I2,I5}, {I1}, {–
l
Step 5: Generating Association Ru Frequent
Itemsets
,
• Let minimum confidence threshold is • The
h
resulting association rules are s each listed
with its confidence.
I5 – R1: I1 ^ I2
• Confidence = sc{I1,I2,I5}/sc{I1,I2} = 2/4 • R1 is
Rejected.
I2 – R2: I1 ^ I5
• Confidence = sc{I1,I2,I5}/sc{I1,I5} = 2/2 • R2 is
Selected.
I1 – R3: I2 ^ I5
• Confidence = sc{I1,I2,I5}/sc{I2,I5} = 2/2 • R3 is
Selected.
Step 5: Generating Association
R
Frequent Itemsets
I2 ^ I5 R4: I1 –
• Confidence = sc{I1,I2,I5}/sc{I1} = 2/6 • R4 is
Rejected.
I1 ^ I5 R5: I2 –
2
• Confidence = sc{I1,I2,I5}/{I2} = 2/7 = • R5 is
Rejected.
I1 ^ I2 R6: I5 –
• Confidence = sc{I1,I2,I5}/ {I5} = 2/2 = • R6 is
Selected.
o
In this way, We have found three str association
rules.
k-itemset whose cor
Methods to Improve Apriori’s •
Hash-based itemset counting: A
hashing bucket count is below the threshold cannot
o
Transaction reduction: A transaction that does not c frequent
k-itemset is useless in subsequent scans. •
n
Partitioning: Any itemset that is potentially frequent i frequent in
at least one of the partitions of DB. •
u
Sampling: mining on a subset of given data, lower s + a method
to determine the completeness. •
s
Dynamic itemset counting: add new candidate item all of their
subsets are estimated to be frequent.•
Mining Frequent Patterns Without
Generation
,
Compress a large database into a compact Pattern
tree (FP-tree) structure •
q
– highly condensed, but complete for fre mining
– avoid costly database scans
n
Develop an efficient, FP-tree-based freque mining
method •
e
– A divide-and-conquer methodology: d mining
tasks into smaller ones
b
– Avoid candidate generation: sub-data only!
FP-Growth Method : An Ex• Consider the
s d t s
example of a consisting of 9 • Suppose min. required is 2
.
(i 2/9 = 22 % )
o i e
The first scan same as Aprior the set of 1-it support counts. •
u r
The set of freq sorted in the o descending su•
s
The resulting L = {I2:7, I1:6, I•
t t I , I 0 , 1
D 5 0
o e I
L I4 0 I2
I
f m T I2
i 2
s 0
1 T ,
I 1
s ,
T
I3 1 I3 1 I3 , 0 I3
I1 0
0 I2 T
T T I2 0 T
, 0 ,I
I1 1
0
1 1 , 1
I2 3,
, 0
I3 I5
0 , I1 0 0 0
I2
I1 T
0 I4 0 0
, T
,
1
T ,
I1
FP-Growth Method: Construction of
First, create the root of the tree, labeled with “null”. •
s
Scan the database D a second time. (First time we create
1-itemset and then L). •
d
The items in each transaction are processed in L or order). •
h
A branch is created for each transaction with items support
count separated by colon. •
Whenever the same node is encountered in another just
increment the support count of the common nod•
i
To facilitate tree traversal, an item header table is bu item points
to its occurrences in the tree via a chain •
t
Now, The problem of mining frequent patterns in da transformed
to that of mining the FP-Tree.•
FP-Growth Method: Construction of
I4:
nul
An FP-Tree that registers compressed, frequent patteItem
I2:7 I3:2
I4:1
I5:1
I3:2 I5:1
Node
I1:4
link
7
C I I I I
o
I
u 1 3 4 5
n
t 2 6
2 2
I
6
d
Sup
Mining the FP-Tree by Creating
i
Cond pattern bases
Steps:
Start from each frequent length-1 pattern (as pattern). 1.
c
Construct its conditional pattern base which set of prefix
g
paths in the FP-Tree co-occurrin pattern. 2.
r
Then, Construct its conditional FP-Tree & pe on such a
tree. 3.
t
The pattern growth is achieved by concatena suffix
r
pattern with the frequent patterns gene conditional
FP-Tree. 4.
The union of all frequent patterns (generated gives the
required frequent itemset. 5.
FP-Tree Example Contin
e
Frequ genera
I2 I5:2,
I2 I4: 2
,
I2 I3:4 2
I2 I1: 4
Mining the FP-Tree by creating conditional (sub) patt
Lets start from I5. The I5 is involved in 2 branches namely I1 I3 I5: 1}.
i
Therefore considering I5 as suffix, its 2 corresponding pref {I2 I1: 1}
and {I2 I1 I3: 1}, which forms its conditional patter
Conditional Conditional pattern Item
FP-Tree base
<I2:2 , I1:2> {(I2 I1: 1),(I2 I1 I3: 1)} I5
<I2: 2> {(I2 I1: 1),(I2: 1)} I4
<I2: 4, I1: {(I2 I1: 1),(I2: 2), (I1: 2)} I3
2>,<I1:2>
<I2: 4>
{(I2: 4)} I2
Now, Following the above mentioned steps:
•
•
FP-Tree Example Contin
n
Out of these, Only I1 & I2 is selected in the conditio because I3
u
is not satisfying the minimum support co For I1 , support count
in conditional pattern base = 1 For I2 , support count in
conditional pattern base = 1 For I3, support count in conditional
pattern base = 1 •
s
Thus support count for I3 is less than required min_ here.
Now , We have conditional FP-Tree with us. •
e
All frequent pattern corresponding to suffix I5 are g considering
all possible combinations of I5 and condi•
The same procedure is applied to suffixes I4, I3 and I•
a
Note: I2 is not taken into consideration for suffix bec have any
prefix at all.•
Why Frequent Pattern Growt
Performance study shows •
t
– FP-growth is an order of magnitude faster and is also
faster than tree-projection
Reasoning •
– No candidate generation, no candidate tes
– Use compact data structure
– Eliminate repeated database scan
– Basic operation is counting and FP-tree bu