Download Higher order mining - HHL Leipzig Graduate School of Management

Higher order mining: modelling and mining the
results of knowledge discovery
Myra Spiliopoulou
1
2
and John F. Roddick
1
Institut fur Wirschaftsinformatik, Humboldt-Universitat, Berlin, Germany.
School of Informatics and Engineering, Flinders University of South Australia,
Adelaide, Australia.
2
Abstract
To date, most data mining algorithms and frameworks have concentrated on
the extraction of interesting rules directly from collected data. This paper
investigates the generic modelling of these rules and the utility of deriving
rules from the results of other data mining routines, that is, mining from
rulesets (or meta-mining). It is argued that this approach has three signicant
advantages. Firstly, with the expansion of dataset size, the tractability of
mining from the complete dataset may be diÆcult on a regular basis, secondly,
changes in observations (and therefore in the observed system) can be more
easily discovered by inspecting changes in extracted rules over time (or over
any other sequential progression), and nally, the nature of the rules extracted
by this process are that they contain dierent higher order semantics from
that exhibited by rst order discovery process. We argue that, in many cases,
such rules are closer to the sorts of rule frequently used to describe everyday
phenomena.
1
Introduction
The body of research in data mining and knowledge discovery has grown
rapidly over the past few years. Moreover, research into the mining of data
with a time component has also received increased attention1 [1]. However,
1
Available data on the number of research publications shows over 1,800 research
papers published in data mining and knowledge discovery in a variety of journals,
with a few notable exceptions, research has largely focussed on the extraction
of knowledge directly from the source data.
This paper discusses the issues surrounding higher order mining, ie. mining from the results of previous mining runs. We argue that this approach
would be particularly useful as it has a number of desirable characteristics.
1. The technique would facilitate the combining of data mining strategies
through the modular combination of components. For example, the analysis of clusters of association rules or the characterisation of proles resulting from a classication algorithm over time.
2. It would provide, in a natural way, for the development of higher order
explanations in describing facts about data, particularly those describing
changes over time or location. For example, the cluster of items with
characteristic X is increasing or the association between X and Y is
weaker during the summer months or farther East. We argue that many
useful descriptions of everyday phenomena have a higher order component
and such rules are often more useful than simple rst order rules.
3. Higher order mining would allow for the development of maintenance
algorithms for mined sets of rules. That is, a higher order rule mining
algorithm could be used to monitor and describe changes in rulesets.
The growing body of research in incremental mining methods provides a
useful point of comparison in this regard.
4. The technique would provide a way of describing the output from dierent knowledge discovery routines. Dierent association rule mining algorithms may dier in the features or properties they give to the rules that
they produce. For example, Rule Miner A generally produces rules that
possess a shorter antecedent at the expense of the length of the consequent.
5. In general, higher order algorithms would run comparatively faster as
they would need to deal with much reduced volumes of data. Indeed, for
some datasets, higher order methods may be the only tractable method of
analysing trend data. Note that higher order knowledge discovery would
still require at least one rst order pass over each dataset, but this may be
done either over time or in a distributed manner. Moreover, since much
of the pruning will be done by the higher order pass, the time taken for
the rst order routines will be reduced.
This paper provides an investigation of these issues. Previous research is
surveyed, the benets and limitations are discussed and a strategy for future
research is proposed.
The paper is structured as follows. In the next section we provide a motivating example based on trend analysis of association rules. The next section
discusses previous relevant research in the area, including research in articial
intelligence. In section 4, we provide an innovative framework for modelling
higher order rules as temporal sequences of conventional rules obtained from
conferences and workshops indicating an approximate compounded 90% growth.
This does not include relevant research in articial intelligence or statistics.
dierent mining sessions. We use this framework to propose mechanisms appropriate for higher order mining. The last section concludes with a discussion
on open issues in this new data mining area.
2
The Notion of Higher Order Rules
2.1 A Simple Example
As an introduction to the problem, consider a subset of the rules extracted
from six successive association rule mining runs as shown in Fig. 1.
Time Rule
t1 High-Income, No-Children Car-Owner
s(13%), c(91%)
t1 High-Income, One-Child Car-Owner
s(8%), c(96%)
t1 High-Income, Two-or-More-Children Car-Owner s(18%), c(96%)
t2
t2
t2
t3
t3
t3
t4
t4
t5
t5
t5
t6
!
!
!
High-Income, No-Children ! Car-Owner
s(12%), c(87%)
High-Income, One-Child ! Car-Owner
s(7%), c(95%)
High-Income, Two-or-More-Children ! Car-Owner s(17%), c(97%)
High-Income, No-Children ! Car-Owner
s(13%), c(88%)
High-Income, One-Child ! Car-Owner
s(8%), c(95%)
High-Income, Two-or-More-Children ! Car-Owner s(14%), c(97%)
High-Income, No-Children ! Car-Owner
s(13%), c(85%)
High-Income, Two-or-More-Children ! Car-Owner s(14%), c(96%)
High-Income, No-Children ! Car-Owner
s(11%), c(84%)
High-Income, One-Child ! Car-Owner
s(9%), c(96%)
High-Income, Two-or-More-Children ! Car-Owner s(13%), c(96%)
High-Income ! Car-Owner
s(33%), c(96%)
Fig. 1.
Example Association Rules
An inspection of these rules suggests that inter alia there is a possible
trend away from car ownership for childless, high income respondents and
that the number of high income respondents with two or more children as a
proportion of the total dataset is falling. However, it is interesting to note
that these sorts of rules cannot generally be found by current knowledge
discovery tools.
2.2 Inferring Higher Order Rules from First Order Rules
The semantics of higher order rules must be carefully determined. Informally,
the use of dierent combinations of rst and higher order knowledge discovery
algorithms will produce dierent interpretations as shown by the examples
in Fig. 2. In the gure, Trend Analysis refers to an analysis of the rules
discovered over time, Association refers to a modied association rule mining
algorithm that looks for rules that are changing in parallel and Rule Structure
Analysis looks at changes to the structure of discovered rules over time.
Higher Order
Rule Algorithm
Trend Analysis
Association
Rule Structure
Analysis
Association
First Order Rule Algorithm
Clustering
Characterisation
The purchase of hard- The number of people Readers of murder
cover books is becoming living close to cities is mysteries are increasmore popular amongst increasing.
ingly coming from the
the 18-25 year old age
35-45 year age group.
group.
The increase in pop- It is becoming more dif- Groups who buy home
ularity of hard-cover cult to classify the improvement books are
books in some stores types of books mid- increasingly likely to be
is associated with an dle income readers will either home owners or
increasing auence in purchase.
under 25.
those demographic areas.
In order to reach the The number of clusters The characterisations
required condence, as- being produced by the being produced are besociation rules with c- mining algorithm is in- coming more accurate.
tion in the antecedent creasing.
are becoming more rened.
Fig. 2.
Examples of Interpreted Higher Order Rules
It is important to note that these sorts of rules occur frequently in the description of everyday phenomena. While the determination of interestingness
is a diÆcult and largely unsolved problem (qv. [2{6]) we believe that many
rules in this category would qualify as useful.
The problem of extracting higher order rules by analyzing mining results
and observing their evolution is thus two-fold. It involves:
1. The development of algorithms and processes that allow the automated
extraction of useful knowledge from such rules. These would need to cater
for missing data (which may be the result of rules not considered signicant enough to be retained in the ruleset) and rules in multiple formats
(such as the more general rule derived at t6 in the example above).
2. The development of interpretative methods that permit the semantics of
any derived rules to be reliably understood. This would include ensuring
that such rules adhere to accepted statistical condence conventions.
2.3 Permissible Rule Inferences
The problem of inferring knowledge from changes in rst order rules encompasses the following issues:
1. The nature of the inference (see Fig. 3 which provides a table of the ways
in which some inferences might be made).
2. Its statistical validity according to agreed condence measures. Ie. the
magnitude of the change will play a role in determining level of signicance.
Fig. 3 reects the crucial problem of explicitly dening the notion of rule
\identity". Researchers on rule maintenance assume that the statistics of a
Antecedent
Unchanged
Consequent
Unchanged
Support
Unchanged
Condence
Unchanged
Unchanged
Unchanged
Rising
(Falling)
Unchanged
Unchanged
Unchanged
Unchanged
Rising
(Falling)
Unchanged
Additional
Term
Unchanged
Unchanged
Unchanged
Missing Term Unchanged
Unchanged
Additional
Term
Unchanged
Unchanged
Unchanged
Fig. 3.
Inference
No change in rule. May increase condence in
any hypothesis that predicts the rule.
While there are an increasing (descreasing)
number of transactions (or observations) exhibiting the antecedent condition, the proportion exhibiting the behaviour characterised by
the rule remains the same. This may be because of a change to the collection mechanism
for the sample or a deeper reason.
An increasing (descreasing) number of transactions are exhibiting the behaviour characterised
by the rule although the total proportion exhibiting the antecedent condition remains the
same. This inference has a high possibility of
being interesting.
An additional, hitherto unreported, item has
been discovered. This may be due to a change
in the collected data or a signicant change in
behaviour. Some changes in the support and
condence values may be tolerated.
An item that was previously reported has
ceased to play a signicant role in the discovered rule. Again, this may be due to a change
in the collected data or a change in behaviour.
A rening of the rule has been necessary to determine the behaviour. More commonly, a rening of the rule leads to a decrease in support
and an increase in condence.
Examples of inferences in rules.
rule may change over time, i.e. that the identity of a rule remains the same,
even if its statistics change. The orthogonal issue of whether the rule's identity remains the same if its contents change is left open. As can be seen from
the last three rows of Fig. 3, changes in a rule's contents can be interpreted as
a remarkable evolution of this rule. Conventional data miners observe rules
with dierent contents as distinct entities. This problem can be resolved by
introducing a notion of ruleset \closure" or by specifying the conditions under which the identity of a rule is deemed to remain unaected by changes
in both its statistics and its contents.
3
Previous Related Research
There has been relatively little direct previous research in this area. Schoenauer and Sebag [7, 8] discuss a reduction operator which is applied to examples extracted from learning by discovery algorithms to produce behavioural
rules. Chakrabarti et al [9] consider the evolution of the statistics of pairs of
association rules by combining research in association rule mining and time
series analysis. Chen and Petrounias investigate methods for determining the
longest temporal interval, in which a discovered association rule is valid [10].
Apart from these approaches that focus on rule evolution, there is also a
growing volume of work in incremental knowledge discovery, which acknowl-
edges the changing nature of collected data and attempts to verify the validity
of rules over time (see for example [11{17]). These routines operate by either
providing an explicit incremental algorithm (time windowing is a commonly
adopted way of accomplishing this) or by providing routines that can easily
operate in an incremental manner.
A variety of temporal mining routines have also been developed which
attempt to develop rules that process either an explicit temporal or spatial
component or are able to handle temporal or spatial data. These including
temporal series and sequence miners and temporal and spatial extensions to
association, clustering and characterisation rule algorithms [19, 18].
Our approach diers from these works in two aspects. First, we are primarily interested in providing a generic framework for the modelling of evolving rules and the acquisition of new knowledge from their evolution. We show
that once a valid framework is dened, these existing techniques can nicely be
applied, although many open issues emerge that call for further research. Second, we focus on issues related to the notion and semantics of conventional
and higher-order rules. A crucial investigation issue in our work concerns
determining when two rules found in dierent mining sessions are actually
impressions of the same rule, an issue made explicit by means of a rule
invariance statement (qv. subsection 4.2).
4
General Framework
In this section, we introduce a framework, in which higher order rules are
dened on the basis of conventional rules discovered during mining sessions.
The overarching structure is depicted in Fig. 4, which shows that higher order
rules can be dened as rules derived from a combination of rst order and/or
other higher order rules.
Higher
Order Rules
First Order
Rules
Data
t1
t2
t3
t4
t5
Time or Sequence
Fig. 4.
Overall Higher Order Mining Structure
We rst model rst order rules in a generic way, placing particular emphasis on the question on how a rule can mutate between mining sessions.
This forms the basis for the next step, in which we dene a group of discrete
time series that model the evolution of a rule. With this model we return to
the aspects of rule analysis depicted in Fig. 2, identify appropriate existing
methodologies for them, as well as open issues requiring new approaches.
4.1 Modelling First Order Rules
In general, we consider mining results to be in the form of clausal form rules
(ie. of the form A1 ; A2; : : :Am ! C1; C2; : : :Cn) [20]. We aim to include in
our analysis rules output by association rules algorithms as well as rules
developed as a result of classication, characterisation and sequence mining.
Denition 1. A \rst order rule" is a tuple r = (ruleContent; statistics)
where the two parameters describe the rule's contents and its statistics, as
derived during the mining session.
The ruleContent and the statistics are themselves vectors of elements.
In particular, the ruleContent is comprised of the rule's antecedent lhs and
the rule's consequent, rhs. In market basket analysis, the antecedent and
the consequent are sets of items. In sequence mining, they are ordered lists
(sequences) of items. In classication, they are predicates. The statistics
include the support, condence, lift and 2-test values for association rules
and sequences. In classication, the accuracy is retained instead.
The statistics of a rule should not be observed in isolation. For example,
the 2 -test and the accuracy are computed at a given condence level. For
the discovery of association rules and sequences, a support threshold is usually specied. These settings are common to all rules discovered in the same
mining session. We therefore model them as part of the mining session.
Denition 2. A mining session is a tuple
M = (sessionT ype; timestamp; data settings; query settings; statistic settings; R)
where:
sessionType is one of \association rules", \sequence mining", \classication",
timestamp denotes the time at which the dataset was analysed,
R is the set of rst order rules discovered,
and the settings describe the specications made by the expert to guide the
mining process.
In this denition, data settings determines the transactions inspected
in the mining session. The query settings describe the predicates issued
against the original dataset to extract the transactions inspected in the mining session; in many cases, the whole dataset is inspected, but occasionally,
the expert prefers to use only a sample. Finally, the statistic settings describe the statistical thresholds, like support or condence level, specied by
the expert to restrict the search space of the miner.
We assume that no mining sessions with dierent settings are performed
in parallel. Thus, a mining session is uniquely identied by its timestamp,
which can be the start or end of the session. We can thus model rst order
rules as temporal objects, annotated with the timestamp of their discovery.
Lemma 1. Let M be a mining session with timestamp t, in which the set of
rst order rules R has been discovered. For each r 2 R, it holds that r r(t).
The new notation forms the basis for building temporal sequences and then
time series for rule objects.
4.2 Modelling a Higher Order Rule as a Temporal Sequence
According to the denition of a rst order rule and a mining session, a rule
is uniquely associated with the mining session in which it was found. For
our analysis of rule evolution, we are interested in the changes in the rules'
statistics across mining sessions. We therefore dene a second order rule as a
temporal object over rst rule objects.
Denition 3. Let T = ft1; : : :; tng be the set of timestamps, at which the
mining sessions M1; : : :; Mn have been issued. We dene a second order rule
R as a temporal sequence of rst order rules (r1 (tR1 ); : : :; rn(tRn ) such that:
1. tR1 < tR2 < : : : < tRn and ftR1 ; : : :; tRn g T .
2. For each tRi ; tRj , the corresponding mining sessions have the same query settings
and statistic settings.
3. For each i; j it holds that ri (tRi ) and rj (tRj ) have the same ruleContent.
\Rule invariance statement" RIS
4. There is no timestamp tk 2 T and rst order rule rk (tk ) that fullls the
2nd and 3rd condition above and does not belong to the temporal sequence.
This denition of a second order rule as a temporal sequence of rst order
rules contains some fundamental assertions. First, a second order rule is well
dened only if the rst order rules comprising it have been discovered in
a comparable way, i.e. in a dataset fragment built in the same way and
processed under the same statistic threshold specications. The dataset size
need not be the same.
The second fundamental assertion is the rule invariance statement
(RIS) determining when two rst order rules are components of the same
second order rule. It says that two rst order rules are considered mutations
of the same object across dierent sessions, when only their statistics dier
but the content remains the same. The RIS is in compliance with research
on rule maintenance, where one observes the new statistics of a previously
discovered rule. The RIS and the last condition in Def. 3 imply a second order
rule is comprised of all rst order rules with the same content and discovered
under the same settings. Therefore:
Lemma 2. Let R = (r1 (tR1 ); : : :; rn(tRn ) be a second order rule and c be the
common ruleContent of all rst order rules comprising it.
R = (c; (s(tR1 ); : : :; s(tRn )))
where s(tRi ) are the statistics of ri(tRi ).
The lemma holds because the rule's content also determines the statistics
at each timepoint, since we have assumed that there are no parallel mining
sessions at the same timepoint. This lemma sets the basis for time series
analysis among second order rules.
4.3 Time Series Analysis for Second Order Rules
We have dened a second order rule as a temporal sequence of timestamps
at which a rst order rule content has been annotated with statistics as the
result of a mining session. These statistics are conventional numeric values,
over which we can dene discrete time series. However, two issues must be addressed. First, the statistics recorded at each timestamp must be mapped into
comparable values. Second, the series must be dened over all timestamps,
i.e. the absence of statistics at some timestamps must be dealt with.
The rst issue arises from the dierent data settings of the mining sessions, from which the statistical values for each rst order rule are obtained.
We deal with this problem by maintaining the number of transactions processed in each mining session (cf. Def. 2). We can use this value to normalize
the statistics recorded at each session into comparable numbers.
The second issue concerns the fact that the rst order rules comprising a
second order rule are not necessarily recorded over all timestamps in T . This
can have two causes:
1. Not all mining sessions are comparable, because of discrepancies in the
query and statistic settings. In this case, the missing value at a given
timestamp can be approximated by e.g. taking the average of the neighbour values in the temporal sequence.
2. The rst order rule may have statistics below the thresholds specied in
the statistic settings. In this case, we can still compute the statistics of
the rule by querying the dataset. If the overhead of this operation is large,
a dummy value, e.g. zero or the threshold value, can be used instead.
Once the time series for the statistics of a second order rule have been
specied properly, time series analysis can be performed on them. This analysis includes:
(i) Observation of the curve(s) of the rule's statistics ; even by simple observation, an expert can identify trends, as depicted in the rst row of Fig. 2.
(ii) Prediction of the future shape of each curve , i.e. trend forecasting.
(iii) Discovery of similar patterns among the time series of the same statistic
values of dierent rules ; these patterns reect associations pertaining over
time, as depicted in the second row of Fig. 2.
(iv) Derivation of new time series by combining the values of existing ones
and analysis of the derived time series ; the analysis of derived time series
corresponds to the analysis of rules derived from second order rules, thus
justifying our concept of higher order rules and higher order mining.
In Fig. 2, the last row depicts issues related to the evolution of a rule's
structure. This evolution is reected in changes in the content of rst order
rules. According to the rule invariance statement , changes in a rule's content are not permitted in the context of second order rules. We are currently
investigating whether the issues related to the evolution of a rule's structure
can be captured by analysing the time series of structurally similar rules, or
whether the above statement should be replaced by a less restrictive one.
5
Higher Order Rules in Current and Future Research
In this study, we have introduced what we consider a new research area for
data mining, namely the analysis of the mining results. While much work has
been devoted to knowledge management and to the incremental discovery of
mining results, there was no generic framework describing how the mining
results themselves change and how such changes should be studied methodically. We have introduced such a framework, based on the concept of \higher
order rule" as a temporal sequence of comparable mining results.
The model establishes a basis for studying the evolution of mining results,
by specifying which results are comparable, how they should be combined,
and importantly, how existing methodologies can be applied in this context.
In particular, we have mapped higher order rules to constructs, upon which
time series analysis for prediction and pattern discovery can be applied.
This work also provides a framework in which existing research on the
processing of mining results can be placed. In particular, research on incremental discovery of rules provides a basis upon which second order rules can
be drawn as temporal objects. At the same time, the conditions specifying
when higher order rules are well-dened also indicates when incremental rule
discovery yields comparable results upon which rule evolution can be studied.
The work of Chakrabarti et al on the evolution of relationships among
association rules [9] can be described as the observation of trends in time
series derived by combining the statistics of pairs of association rules. Our
framework shows how existing techniques can be applied to the same purpose
in a less sophisticated way.
Our framework for higher order mining also opens new perspectives on
the analysis of mining results and their evolution. First of all, mining results
are patterns of many dierent types, discovered by methodologies based on
various parametric settings. Our framework is still at a rather abstract level
and requires much detailed work to capture the particularities inherent to
each methodology and accommodate them into a common model.
Moreover, mining results are patterns annotated with a lot of statistical
information, which is not always reproducible. Thus, the challenge emerges
of how a minimal set of statistical data can be determined, upon which the
statistics of potential interest can be reproduced. Our framework includes
some initial steps in this direction.
Finally, existing research is focussing on the evolution of the statistics of
the same rule. However, changes in the underlying data set may cause a rule
to become ner or coarser. Our model for higher order rules does not observe
such changes as part of a single rule's evolution. We are thus investigating
whether the evolution of structural changes requires a still more generic model
of higher order rules, or whether it can be captured by applying methods of
pattern discovery among structurally similar rules.
Acknowledgement: We would like to thank Stean Baron, Institute of Information Systems, HU Berlin, for his fruitful suggestions on rst order rule
modelling.
References
1. J. Roddick and M. Spiliopoulou, \A bibliography of temporal, spatial and
spatio-temporal data mining research," SIGKDD Explorations, vol. 1, no. 1,
pp. 34{38., 1999.
2. G. Dong and J. Li, \Interestingness of discovered association rules in terms of
neighbourhood-based unexpectedness," in Second Pacic-Asia Conference on
Knowledge Discovery and Data Mining: Research and Development (X. Wu,
R. Kotagiri, and K. Korb, eds.), (Melbourne, Australia), pp. 72{86, SpringerVerlag, 1998.
3. A. Freitas, \A multi-criteria approach for the evaluation of rule interestingness,"
in International Conference on Data Mining, (Rio de Janeiro), pp. 7{20, WIT
Press, 1998.
4. A. Freitas, \On rule interestingness measures," Knowledge Based Systems,
vol. 12, no. 5-6, pp. 309{315, 1999.
5. S. Sahar, \Interestingness via what is not interesting," in Fifth International
Conference on Knowledge Discovery and Data Mining (S. Chaudhuri and
D. Madigan, eds.), (San Diego, CA, USA), pp. 332{336, ACM Press, 1999.
6. A. Silberschatz and A. Tuzhilin, \What makes patterns interesting in knowledge
discovery systems," IEEE Transactions on Knowledge and Data Engineering,
vol. 8, no. 6, pp. 970{974, 1996.
7. M. Schoenauer and M. Sebag, \Incremental learning of rules and meta-rules,"
in Seventh International Conference on Machine Learning, pp. 49{57, Morgan
Kaufmann, Palo Alto, CA, USA, 1990.
8. M. Sebag and M. Schoenauer, \Discovering meta-rules from rules and examples," in Eleventh International Conference on Expert Systems and their Applications, vol. 1, (Nanterre, France), pp. 205{216, 1991.
9. S. Chakrabarti, S. Sarawagi, and B. Dom, \Mining surprising patterns using temporal description length," in VLDB'98 (A. Gupta, O. Shmueli, and
J. Widom, eds.), (New York City, NY), pp. 606{617, Morgan Kaufmann, Aug.
1998.
10. X. Chen and I. Petrounias, \Mining temporal features in association rules," in
3rd European Conference on Principles and Practice of Knowledge Discovery
in Databases (PKDD'99) (J. Zytkow and J. Rauch, eds.), vol. 1704 of Lecture
Notes in Articial Intelligence, (Prague), pp. 295{300, Springer, 1999.
11. D. Cheung, J. Han, V. Ng, and C. Wong, \Maintenance of discovered association rules in large databases: An incremental updating technique.," in International Conference on Data Engineering (ICDE'96), (New Orleans, Louisiana,
USA), 1996.
12. D. Cheung, S. Lee, and B. Kao, \A general incremental technique for maintaining discovered association rules," in Fifth International Conference on Database
Systems for Advanced Applications, (Melbourne, Australia), 1997.
13. D. Cheung, V. Ng, and B. Tam, \Maintenance of discovered knowledge : A
case in multi-level association rules," in Second International Conference on
Knowledge Discovery and Data Mining (KDD-96) (E. Simoudis, J. Han, and
U. Fayyad, eds.), (Portland, Oregon), pp. 307{310, AAAI Press, Menlo Park,
CA, USA, 1996.
14. J. Feng and Y. Feng, \Incremental updating algorithms for mining association
rules," in Eight International Conference on the Management of Data (COMAD'97), (Madras, India), 1997.
15. V. Guralnik and J. Srivastava, \Event detection from time series data," in Fifth
International Conference on Knowledge Discovery and Data Mining (S. Chaudhuri and D. Madigan, eds.), (San Diego, CA, USA), pp. 33{42, ACM Press,
1999.
16. C. Rainsford, M. Mohania, and J. Roddick, \A temporal windowing approach
to the incremental maintenance of association rules," in Eighth International
Database Workshop, Data Mining, Data Warehousing and Client/Server Databases (IDW'97) (J. Fong, ed.), (Hong Kong), pp. 78{94., Springer Verlag, 1997.
17. K. Wang and J. Tan, \Incremental discovery of sequential patterns," in ACM
SIGMOD Workshop on Research Issues on Data Mining and Knowledge Discovery, (Montreal, Canada), 1996.
18. H. Miller and J. Han, eds., Geographic Data Mining and Knowledge Discovery.
Research Monographs in Geographic Information Systems, Taylor and Francis,
2000.
19. J. Roddick and M. Spiliopoulou, \Temporal data mining: Survey and issues,"
Research Report ACRC-99-007, School of Computer and Information Science,
University of South Australia, May 1999.
20. T. Richards, Clausal form logic: an introduction to the logic of computer reasoning. Sydney: Addison Wesley, 1989.