Download Closed Pattern Mining for the Discovery of User Preferences in a

Closed Pattern Mining for the Discovery of
User Preferences in a Calendar Assistant
Alfred Krzywicki and Wayne Wobcke
School of Computer Science and Engineering
University of New South Wales
Sydney NSW 2052, Australia
{alfredk|wobcke}@cse.unsw.edu.au
Abstract. We use closed pattern mining to discover user preferences in appointments in order to build structured solutions for a calendar assistant. Our choice
of closed patterns as a user preference representation is based on both theoretical
and practical considerations supported by Formal Concept Analysis. We simulated interaction with a calendar application using 16 months of real data from
a user’s calendar to evaluate the accuracy and consistency of suggestions, in order to determine the best data mining and solution generation techniques from a
range of available methods. The best performing data mining method was then
compared with decision tree learning, the best machine learning algorithm in this
domain. The results show that our data mining method based on closed patterns
converges faster than decision tree learning, whilst generating only consistent solutions. Thus closed pattern mining is a better technique for generating appointment attributes in the calendar domain.
Keywords: Data mining, closed patterns, Formal Concept Analysis, calendar assistants.
1 Introduction
We are interested in the problem of providing automated assistance to the user of a calendar system to help in defining appointments. In a calendar application, the user may
initially specify some of an appointment’s attributes (e.g. title and day), and the task of
the system is to suggest any or all of the remaining attributes (e.g. time and location).
What makes this problem difficult is that both the set of attributes given initially by the
user and the set of attributes that can be suggested are not fixed; some appointments
may contain only basic information such as the title, time, date and duration, while
others may have additional attributes specified, such as the location, the attendees, etc.
Furthermore the attributes mutually constrain one another.
A calendar appointment can be regarded as a structured solution. A problem requires a structured solution if the solution has the form of a set of components that are
constrained by other components in the solution. In many practical systems, the challenge of building a consistent solution is solved by defining a set of rules that describe
the constraints between components, McDermott [8]. In the calendar domain, the solution “components” are attributes with their values, and the “constraints” are provided
by a model of the user’s preferences. Additional constraints affecting the solution are
the presence of other appointments, dependencies between attributes and other user
knowledge not directly represented in the calendar system. For example, an appointment time and duration may depend on the availability of attendees and the meeting
location. These dependencies are not given explicitly, but may be represented in the
form of patterns.
In this paper, we investigate the use of closed pattern mining to discover user preferences over calendar appointments and to build structured solutions for a calendar
assistant. Traditionally the aim of data mining is to discover association rules, Agrawal
and Srikant [1]. We found, however, that mining association rules is not the most suitable method for applications with real-time user interaction, due to the potentially large
number of frequent patterns and the number of rules that can be generated from each
pattern. In contrast, the number of closed frequent patterns can be an order of magnitude smaller than the number of frequent patterns. In fact, all frequent patterns can be
generated from a complete set of closed frequent patterns. The data mining algorithm
used in this paper is based on the FP-Growth algorithm introduced by Han, Pei and
Yin [6] and implemented by Coenen, Goulbourne and Leng [3]. In order to find closed
frequent patterns, we filter out all non-closed patterns as they are computed by the FPGrowth method. Details of the pattern mining algorithm can be found in Section 3.
Discovered frequent patterns are treated as possibly inconsistent fragments of different solutions that need to be integrated into consistent suggestions before presenting
them to the user. We found that it is best to create solutions only from non-conflicting
patterns, which makes generated solutions less likely to conflict with user preferences.
The method for pattern selection and the support required for pattern generation
were determined from the results of simulated user sessions. The simulation enabled us
to compare the accuracy of our appointment prediction method with the best machine
learning technique, decision tree learning, on realistic calendar data taken from a user’s
diary for a 16 month period. We present the results of the comparison and discuss some
advantages of our pattern mining approach over decision tree learning
The remainder of this paper is organized as follows. In the next section, we provide the formal framework for the problem of generating structured solutions in the
calendar domain. Section 3 describes our data mining and solution generation method,
which is evaluated and compared with other methods in Section 4. Section 5 contains a
discussion of related research.
2 Formal Problem Statement
This section provides definitions specific to the problem of closed pattern mining for
generating structured solutions in the calendar domain.
Definition 1. Let A = {a1 , a2 , . . . , an } be a set of n attributes used in all appointments.
Let each attribute ai have a set of values Vai specific for the domain of the attribute.
For example, Vday = {Sunday, Monday, . . . , Saturday}. A feature is an attribute-value
pair {ai , vij }, where vij is an element of Vai . The set of all features is denoted I.
Definition 2. A data case or case is a nonempty set of features stored in the database
of cases, e.g. {{ai1 , vi1 j1 }, . . . , {aim , vim jm }}. An attribute may appear in a case at
most once and may not occur at all.
For example, a single appointment stored in the calendar database is a data case.
Definition 3. A solution is a potential data case created by the system. A number of
solutions can be selected by the system from a set of solutions and presented to the user
as suggestions for consideration.
Definition 4. A pattern is any part of a data case, a set of features, containing at least
one feature. Solutions/cases may contain more than one pattern.
Definition 5. Two features are overlapping if they have the same attribute.
Definition 6. Two features are conflicting if they have the same attribute with different
values.
Definition 7. Two patterns are conflicting if they contain at least one pair of conflicting
features.
Definition 8. Two patterns are overlapping if they contain overlapping features.
We also call two conflicting patterns inconsistent. It is worth noting that conflicting features/patterns are always overlapping, therefore the “no overlap” condition is stronger
than the “no conflict” condition in the solution generation algorithms below.
The underlying theory of closed patterns is based on Formal Concept Analysis,
Wille [10]. Pasquier et al. [9] extended the theory and introduced the idea of closed
patterns, applying Formal Concept Analysis to data mining. The key terminology of
this theory is summarized below, slightly adjusted for consistency with the above definitions.
Definition 9. A data mining context is a triple D=hO, I, Ri, where O is a set of objects,
I is a set of features and R ⊆ O × I is a binary relation between objects and features.
The fact that object o has feature i can be expressed as (o, i) ∈ R.
Definition 10. Let D=hO, I, Ri be a data mining context and let O ⊆ O, I ⊆ I. The f
and g functions map powersets 2O →2I and 2I →2O respectively:
f (O) = {i ∈ I|∀o ∈ O, (o, i) ∈ R}
(1)
g(I) = {o ∈ O|∀i ∈ I, (o, i) ∈ R}
(2)
Less formally, f maps a set of objects into a set of features common to those objects.
Similarly, g maps a set of features into a set of objects containing all those features.
Definition 11. The functions h = f ◦ g, i.e. h(I) = f (g(I)), and h′ = g ◦ f , i.e.
h′ (O) = g(f (O)), are Galois closure operators.
Definition 12. Let I ⊆ I be a set of features. I is a closed pattern iff h(I) = I.
It follows from the last two definitions that a closed pattern is a maximal set of features
common to a given set of objects. We regard each mined closed pattern as an implicit
user preference. This mapping between closed patterns and user preferences proved to
be very useful in data mining for supporting appointment attribute suggestion in the
calendar domain.
3 Pattern Mining for Generating Structured Solutions
This section provides a summary of the closed pattern mining method based on the
FP-Tree algorithm of Han, Pei and Yin [6], and our approach to generating structured
solutions.
3.1
Mining Closed Frequent Patterns
Closed frequent patterns are mined in two steps: 1) build an FP-Tree from the database,
and 2) retrieve frequent patterns from the FP-Tree, filtering out all non-closed patterns.
In the method implemented by Coenen, Goulbourne and Leng [3], frequent patterns
are mined using the FP-Growth algorithm and then stored in a T-Tree structure (Total
Support Tree), which also stores the support calculated for all frequent patterns. In our
implementation, we store only closed frequent patterns in the T-Tree, which provides
fast access to the set of closed frequent patterns. In the first step, an FP-Tree is constructed from the database of past cases using the original FP-Growth method. In the
second step, all closed frequent patterns are extracted from the FP-Tree and stored in a
T-Tree.
In order to filter out non-closed patterns we use the following property, due to
Pasquier et al. [9]: if I is any pattern, then support(I) = support(h(I)). Thus the
support of any pattern is the same as the support of the smallest closed pattern containing it. Therefore any frequent pattern properly contained in the smallest closed pattern
containing it is not a closed pattern. This means we can use the following simple algorithm to filter out non-closed patterns.
Algorithm 1 (Finding closed patterns)
1
T-Tree = {}
2
while not last frequent pattern
3
FrPat = GetFrPatFromFP-Tree()
4
SmallestClosedFrPat = FindSmallestClosedPatContaining(FrPat, T-Tree)
5
if (SmallestClosedFrPat does not exist)
6
or (SmallestClosedFrPat.Support 6= FrPat.Support)
7
T-Tree = Add(FrPat, T-Tree)
8
end
9
end
10
Output(T-Tree)
The algorithm searches the T-Tree for a smallest closed pattern (line 4) containing the
pattern collected from FP-Tree (line 3). If such a pattern is found and it has the same
support as the original FP-Tree pattern, it is discarded, otherwise the pattern is stored in
the T-Tree (line 7). The original FP-Tree mining algorithm has been modified in such a
way that larger patterns are always mined before smaller ones, which enables the above
algorithm to discover all closed frequent patterns.
3.2
Generating Solutions
Patterns found in the data mining process are used as building blocks to construct calendar appointment solutions. Individual patterns may complement one another, conflict or
overlap (as defined in Section 2). In order to generate useful suggestions, we aim to efficiently find solutions that make use of as many patterns as possible. The algorithm presented below uses the “no conflict” method for pattern selection (for the “no-overlap”
method, lines 8 and 17 need to be modified).
The following algorithm is not guaranteed to find all possible solutions, though it
has been experimentally verified to provide sufficient time performance and solution
quality. The algorithm first computes the set of all patterns that do not conflict with, but
have at least one common feature with, the initial user features. The algorithm heuristically finds subsets of these patterns jointly consistent with the initial user features; each
such set is heuristically extended to one maximal set of non-conflicting features.
Algorithm 2 (Generating user suggestions)
1 Input: InitFeatures, ClosedPatterns
2 Output: Solns
3
Solns = {}
4
InitSoln = MakeInitSoln(InitFeatures)
5
InitSoln.Patterns = {}
6
PatternList = {}
7
for each Pattern in ClosedPatterns
8
if not Conflicting(Pattern, FeatureList)
9
and HasCommonFeature(Pattern, FeatureList)
10
Add(Pattern, PatternList)
11
end
12
end
13
UnusedPatterns = PatternList
14
while UnusedPatterns.Size > 0
15
Soln = InitSoln
16
for each Pattern in UnusedPatterns
17
if not Conflicting(Pattern,Soln.Patterns)
18
Soln = Update(Soln, Pattern)
19
Soln.Patterns = Add(Pattern,Soln.Patterns)
20
end
21
end
22
for each Pattern in PatternList
23
if not Conflicting (Pattern,Soln.Patterns)
24
Soln = Update(Soln, Pattern)
25
Soln.Patterns = Add(Pattern,Soln.Patterns)
26
end
27
end
28
for each Pattern in Soln.Patterns
29
UnusedPatterns = Delete(Pattern,UnusedPatterns)
30
end
31
Solns = Add(Soln, Solns)
32
end
As an example, suppose the initial features are as follows:
Title=”Project Meeting”, Category=”Team Meeting”, Period=”Semester”
Suppose the existing closed frequent patterns are as follows:
P1. Category=”Team Meeting”, Period=”Semester”, AmPm=”am”, Time=1030
P2. Category=”Team Meeting”, Period=”Break”, AmPm=”pm”
P3. Category=”AI Lecture”, Period=”Semester”, AmPm=”pm”, Time=1500
P4. AmPm=”pm”, Day=”Wednesday”, Attendees=”Anna, Alfred, Rita, Wayne”
P5. Period=”Semester”, AmPm=”am”, Time=1030, Day=”Monday”,
Attendees=”Anna, Alfred, Wayne”
P6. Category=”Team Meeting”, Day=”Wednesday”
The initial solution (line 4) is just the initial set of features entered by the user. Since
patterns P2 and P3 are conflicting and P4 has no common features with the initial solution, the PatternList and UnusedPatterns sets (line 13) contain only patterns P1, P5 and
P6. Solutions always start with an initial user solution. A new solution is generated in
lines 16–27. Since the initial solution has no associated patterns, P1 is added and the
solution becomes:
Title=”Project Meeting”, Category=”Team Meeting”, Period=”Semester”, AmPm=”am”,
Time=1030
In the next iteration (lines 16–21), P5 is evaluated and, since it is not conflicting, is also
added to the solution, which becomes:
Title=”Project Meeting”, Category=”Team Meeting”, Period=”Semester”, AmPm=”am”,
Time=1030, Day=”Monday”, Attendees=”Anna, Alfred, Wayne”
Next P6 is evaluated and rejected as conflicting with this solution. The procedure then
continues to add patterns from the PatternsList set (lines 22–27), but there is nothing
new to add at this stage. Therefore it becomes the first solution and UnusedPatterns
is updated. UnusedPatterns is still not empty, so the solution generation iterates again,
this time adding P6 to the initial solution, generating the second solution, as follows:
Title=”Project Meeting”, Category=”Team Meeting”, Period=”Semester”,
Day=”Wednesday”
4 Experimental Evaluation
In this section, we describe our experimental framework for evaluating the closed pattern mining approach for solution generation using simulations over data extracted from
a single user’s calendar. This gave us around 1000 cases of real calendar data (about 16
months of appointments).
4.1
Method
The simulation was conducted in two stages. In the first stage, we compared two methods of appointment solution generation used in conjunction with closed pattern mining:
the “no conflict” and the “no overlap” methods. This confirmed the superiority of the
“no conflict” approach, which is the algorithm presented above. In the second stage,
we compared these results with those generated using decision tree learning, the best
performing machine learning method.
The simulator runs real calendar data through the solution generation system in a
manner resembling interaction with the real user of a calendar system. The calendar
data used for the simulation had 8 attributes: Category, Period, Attendees, Location,
Duration, AmPm, Day and Time. The simulation was conducted as follows. The “user”
(which means “simulated user”) enters case n, which is stored in the database and data
mining on all past cases is performed. Then the “user” enters the first three attributes
of case n + 1 as initial attributes, which are always assumed to be the Category, Period
and Attendees (this is the typical behaviour of actual users based on our informal observation). The system produces a number of suggestions out of which the “user” selects
one closest to case n + 1. Differences are then calculated between the best suggestion
and the real case n + 1. These differences reflect the number of modifications the “user”
needs to make to turn the suggestion into case n + 1. The “user” needs to either add a
missing feature or delete one which is not required, therefore each difference is counted
as 1. These differences are then averaged over a number of data cases. For compatibility with the decision tree learning method, as explained further in this section, the
simulator produces 32 suggestions.
The machine learning part of the simulation was done using the C4.5 decision tree
algorithm implemented in the Weka toolkit [11], called J48. The method was selected
by testing the performance of a range of machine learning algorithms on the calendar
data for various combinations of attributes. The tested algorithms were rule induction
(OneR), decision tree learning (J48), Bayesian methods (Naive Bayes, BayesNet), knearest neighbour (IBk) and case based reasoning (KStar). The best performing, J48,
was then used on five calendar data sets, each to predict one of the five attributes of case
n + 1 not specified by the “user” (i.e. the Location, Duration, AmPm, Day and Time).
The predicted values were put together to make a set of complete calendar appointments
as in the data mining method. So that the decision tree learning methods could generate
a number of alternative solutions, we had to combine a number of suggested values for
each attribute. This was achieved by modifying the Weka code so that each prediction
consisted of two different values rather than one. For the five attributes to be predicted
this was equivalent to 25 = 32 possible solutions for each appointment.
4.2
Results
We first present the results comparing the “no conflict” and “no overlap” methods used
with closed pattern mining, shown in Figure 1.
The difference between the two methods shows that the “no conflict” method produces significantly better results. This can be explained by the way solutions are created.
It is generally easier to find overlapping patterns in a solution than non-overlapping,
hence the former method creates a higher number and variety of solutions.
One of our objectives in evaluating machine learning methods for generating solutions was to compare our system with CAP, Dent et al. [4]. Although a direct comparison was not possible, the method (ID3 for CAP) and overall results are similar. The
accuracy achieved by decision tree learning on our data set is shown in Figure 2. These
results broadly confirm those reported in the original experiments with CAP, e.g. accuracy for location is close to 70% after around 150 cases. Note that our experiments
compensate for a deficiency in the experimental setup with CAP in that a parameter
Fig. 1. Average accuracy of appointment prediction for two methods: “no conflict” (thick line)
and “no overlap” (normal line).
Fig. 2. Decision tree learning prediction results of calendar appointments. The thick line shows the
overall average accuracy of appointment prediction, the continuous line is the average appointment date prediction and the dotted line reflects the number of inconsistent values in predicted
data (AmPm and Time).
specific to the time period (e.g. semester, break) is included, which means the average
accuracy fluctuates much less than in the original CAP evaluation. However, our results also show a greater fluctuation of average accuracy for decision tree learning than
with closed pattern mining. We suspect that this could be because at certain points in
the simulation, the decision tree is restructured, resulting in a loss of accuracy, whereas
the pattern mining approach produces smoother behaviour over time. Additionally, unlike the pattern mining method, where values within one case are created from nonconflicting patterns, the decision tree learning method predicts values separately for
each attribute. In consequence, it is possible that some associated attributes may conflict, e.g. AmPm=am and Time=1500. In effect, the system has to choose randomly
between the two options to resolve the conflict, meaning that the date suggestion is often wrong (our simulation arbitrarily chooses the value of Time to resolve the conflict,
and the date is determined from the next free time slot satisfying the chosen solution).
The chart in Figure 2 provides some confirmation of this explanation, where a low average date prediction accuracy corresponds roughly to a high number of inconsistencies
between AmPm and Time.
Comparison of Figure 2 with Figure 1 shows that, although the average accuracy
of prediction is similar for the two methods (closed pattern mining 69%, decision tree
learning 68%), closed pattern mining gives significantly better prediction in the first 200
cases. More specifically, the closed pattern mining method reaches its average accuracy
after only 62 cases, whereas the decision tree learning method reaches its average after
224 cases, staying about 10 percent lower in the first 200 cases. This is an important
difference for interactive calendar users, who would clearly prefer useful suggestions in
a shorter period of time (this corresponds to roughly 1 month for closed pattern mining
vs. 3 months for decision tree learning). Moreover, decision tree learning prediction is
less stable, showing greater sensitivity to user preference changes in transition periods.
5 Related Work
As far as we know, there are no calendar applications supported by pattern mining,
however there are examples of research where some kind of machine learning has been
applied. As described above, the CAP system, Dent et al. [4], provides suggestions for
various appointment attributes. Two methods for predicting attributes were compared:
backpropagation neural networks and ID3. Their results showed that for the Location
attribute, around 70% accuracy was achieved by both learning methods after sufficient
training. As described above, our experiments broadly confirm this result in the case
of decision tree learning, though over the whole set of predicted attributes (not only
Location). As also mentioned above, CAP predicts each attribute of the appointment
separately, which may result in inconsistent appointment solutions when these predictions are combined.
Another preference learning calendar assistant is described by Berry et al. [2]. Their
system, PCalM, is a framework designed to schedule meetings in the open calendar environment. Instead of learning to predict individual appointment attributes, as in CAP,
PCalM learns to rank candidate appointments from pairwise selections provided by
the user. Unlike our calendar system, designed to build and present suggestions unobtrusively, PCalM forces the user to choose amongst suggestions in order to generate
training data for learning the preference function. Furthermore, similar to our method,
PCalM has been evaluated using simulated user interactions, however the data used in
the PCalM evaluation is synthetically generated, while we have used appointments from
a user’s real calendar, providing a more realistic data set for experimentation.
6 Conclusion
We have designed and evaluated a structured solution builder with data mining support
for generating suggestions for calendar appointments. Closed patterns proved to be a
suitable alternative to association rules due to their compactness and flexibility. Moreover, pattern mining has an advantage over single class machine learning methods in
that it better supports creating multiple solutions with consistent structures. We simulated user interaction with real calendar data to configure and tune data mining and
appointment solution generation methods. Our results show the superiority of closed
pattern mining to decision tree learning, the best performing machine learning algorithm in this domain.
We believe that concept based data mining for building structured solution can be
applied to other configuration domains. Due to the fact that cases are added to the
system incrementally, it might be possible to use incremental data mining methods in
conjunction with the FP-Growth algorithm, similar to the approaches of Ezeife and
Su [5] and Koh and Shiehr [7].
Acknowledgments. This work was funded by the CRC for Smart Internet Technology. We would like to thank Paul Compton for making his calendar data available
for research and Frans Coenen for his open source implementation of the FP-Growth
algorithm.
References
1. Agrawal, R. and Srikant, R. Fast Algorithms for Mining Association Rules. In Proceedings
of the 20th Conference on Very Large Data Bases, pp. 478–499, 1994.
2. Berry, P. M., Gervasio, M., Uribe, T., Myers, K. and Nitz, K. A Personalized Calendar
Assistant. In Proceedings of the AAAI Spring Symposium on Interaction between Humans
and Autonomous Systems over Extended Operation, 2004.
3. Coenen, F., Goulbourne, G. and Leng, P. Tree Structures for Mining Association Rules. Data
Mining and Knowledge Discovery, vol. 8, pp. 25–51, 2004.
4. Dent, L., Boticario, J., Mitchell, T. M. and Zabowski, D. A. A Personal Learning Apprentice.
In Proceedings of the Tenth National Conference on Artificial Intelligence (AAAI-92), pp.
96–103, 1992.
5. Ezeife, C. I. and Su, Y. Mining Incremental Association Rules with Generalized FP-Tree.
In Cohen, R. and Spencer, B., editors, Advances in Artificial Intelligence, pp. 147–160,
Springer-Verlag, Berlin, 2002.
6. Han, J., Pei, J. and Yin, Y. Mining Frequent Patterns without Candidate Generation. In
Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data,
pp. 1–12, 2000.
7. Koh, J.-L. and Shiehr, S.-F. An Efficient Approach for Maintaining Association Rules Based
on Adjusting FP-Tree Structures. In Lee, Y., Li, J., Whang, K.-Y. and Lee, D., editors,
Database Systems for Advances Applications, pp. 417–424, Springer-Verlag, Berlin, 2004.
8. McDermott, J. R1: A Computer-Based Configurer of Computer Systems. Artificial Intelligence, vol. 19, pp. 39–88, 1982.
9. Pasquier, N., Bastide, Y., Taouil, R. and Lakhal, L. Efficient Mining of Association Rules
Using Closed Itemset Lattices. Information Systems, vol. 24, pp. 25–46, 1999.
10. Wille, R. Formal Concept Analysis as Mathematical Theory of Concepts and Concept Hierarchies. In Ganter, B., Stumme, G. and Wille, R., editors, Formal Concept Analysis, pp.
1–23, Springer-Verlag, Berlin, 2005.
11. Witten, I. H. and Frank, E. Data Mining. Morgan Kaufmann, San Francisco, CA, 2005.