Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Neural Networks as Artificial Memories for Association Rule Mining Vicente Oswaldo Baez Monroy Submitted for the degree of Doctor of Philosophy Department of Computer Science December, 2006 Abstract The collection of data is a high priority task in our daily life because humans are interested in understanding more about the variables of the diverse events happening around us. This analysis or understanding is derived from the usage of techniques which aim to produce predictive or descriptive models from data. While the former generates models to predict new status of data, the latter finds important patterns to describe it. In Computer Science, Data Mining is a multidisciplinary area responsible for producing these data-analysis techniques by developing algorithms which aim to form novel understandings of data. Among the descriptive data-mining techniques, the task of association rule mining stands out because of its simple, but powerful knowledge rule-format to represent how the attributes or items, which form events or patterns in an environment, associate amongst each other and how strong these associations are. Based on the support-confident framework proposed by Agrawal for association rule mining, the generation of these rules is typically achieved by first identifying the group of interesting or frequent itemsets in data and then generating rules from the discovered itemsets. To determine whether an itemset is frequent or not, the calculation of its corresponding support property must be performed since it defines its frequency of occurrence in the mined environment. i Although the number of approaches and strategies for association rule mining have been growing since 1993, there are very few proposals based on biologicallyinspired technology. In particular, there are barely any neural-network based approaches for the generation of this type of rules. To further the research in this field, we explore how neural networks can be used for association rule mining in this thesis. Since it has been assumed that association rules are a type of knowledge that humans can generate mechanically, and considering that neural networks imitate human behavior, we have stated that an implicit neural-based framework may exist for this data mining technique. In particular, we have followed the premise that association rules can be derived from the knowledge learnt by a neural network similar to those generated by traditional algorithms like Apriori. In order to perform association rule mining with neural networks, we focus on investigating how to perform the counting of patterns or itemsets, which is normally produced by looking for the patterns by scanning the high dimensional space defined by the original data environment, through decoding the knowledge embedded in an auto-associative memory and a self-organising map. This is, we have worked in the first stage of the neural-based proposed framework which involves the building of artificial memories that are able to learn, store and recall itemset support after they have been trained with data defining associations. Especially, we analyse and decode the training process and the weight matrix of a self-organising map and an auto-associative memory to propose itemsetsupport extraction mechanisms through which they are able to recall itemsetsupport when an itemset is presented as stimulus to the trained networks. ii Since data sources or environments are not static and any knowledge therefore derived from them, like rules or itemsets, tend to outdate as fast as new events occur, we have also investigated how the itemset-support knowledge accumulated by a memory must be maintained throughout time. Particularly, we propose how a self-organising-map based memory can maintain its knowledge of itemset support valid throughout time while it learns from a non-stationary environment. iii Contents 1 Introduction 1 1.1 The Role of Neural Networks in Data Mining . . . . . . . . . . 12 1.2 Linking ANNs and ARM: The Motivations . . . . . . . . . . . 14 1.3 The Neural-Network Candidates . . . . . . . . . . . . . . . . . 23 1.4 The Research Questions . . . . . . . . . . . . . . . . . . . . . 24 1.4.1 Aims and Objectives . . . . . . . . . . . . . . . . . . . 28 Organisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 1.5 2 Association Rule Mining 33 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.2 The Scope of ARM . . . . . . . . . . . . . . . . . . . . . . . . 35 2.2.1 Formal Definition . . . . . . . . . . . . . . . . . . . . . 36 Frequent Itemset Mining . . . . . . . . . . . . . . . . . . . . . 37 2.3.1 The Calculation of Itemset Support . . . . . . . . . . . 38 2.4 Taxonomy of the FIMers . . . . . . . . . . . . . . . . . . . . . 40 2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 2.3 3 Hypothetical Neural Network for Association Rule Mining 46 3.1 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.2 Hypothetical ARM Framework Based on ANNs . . . . . . . . . 52 3.3 A Formal Definition of the Problem . . . . . . . . . . . . . . . 56 iv 3.4 Ideal ANN Characteristics for Building Memories for ARM . . 57 3.5 Reasons for Studying an AAM and a SOM for ARM . . . . . . 58 3.6 Similarities and Differences with Surveyed Approaches . . . . . 67 3.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4 An Auto-Associative Memory for ARM 4.1 70 Correlation Matrix Memory for ARM . . . . . . . . . . . . . . 71 4.1.1 The Learning of Itemset Support by a CMM . . . . . . . 71 4.1.2 Recalling Itesemt Support from The Weight Matrix of a CMM . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 Complexity Analysis: CMM vs. Apriori . . . . . . . . . 79 4.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 4.1.3 5 Itemset Support Generation From a Self-Organising Map 91 5.1 Considering a SOM for ARM: Principles . . . . . . . . . . . . 5.2 A Probabilistic Itemset-support Estimation Mechanism . . . . . 101 5.3 Experiments and Results . . . . . . . . . . . . . . . . . . . . . 111 5.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 6 Incremental Training for Incremental ARM: A SOM Model 92 131 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 6.2 Batch SOM for Non-stationary Environments . . . . . . . . . . 135 6.3 6.2.1 The Problem Definition . . . . . . . . . . . . . . . . . 136 6.2.2 Interpretation by Node Influences of the Batch Training 6.2.3 The Algorithm . . . . . . . . . . . . . . . . . . . . . . 141 6.2.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . 144 137 Itemset Support Maintenance by Incremental SOM Training . . 145 6.3.1 Experiments . . . . . . . . . . . . . . . . . . . . . . . 145 v 6.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 7 Conclusions and Future Work 7.1 Final Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 7.1.1 7.2 168 Contributions . . . . . . . . . . . . . . . . . . . . . . . 185 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 7.2.1 For The Auto-associativity-based Memory . . . . . . . 187 7.2.2 For The Self-organising-map-based Memory . . . . . . 187 7.2.3 For the Quality of the Itemset-support Estimation . . . . 192 7.2.4 ANNs-based Candidate Generation Procedures . . . . . 192 7.2.5 Distributed Association Rule Mining . . . . . . . . . . 193 7.2.6 The Itemset Concept in Dynamic Data . . . . . . . . . . 196 A The Apriori Algorithm 199 B The Neural Network Candidate Algorithms 201 vi List of Figures 1.1 An illustration of ARM applied to the data source defined in (a). The aim is to generate rules with a minimal threshold of 20%. As the first part of ARM, the frequent itemsets have to be discovered, such as in (b), based on their support. Then, association rules, as in (c), are formed from them. . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 A shopping receipt as an example of a data source in which the associativity concept can be exploited for the extraction of knowledge. . . 1.3 7 9 Framework defined by the processes and strategies for the problem of association rule mining. Its conception is based on the support- . . . . . . 18 1.4 Neural-based framework for ARM. . . . . . . . . . . . . . . . . . 23 2.1 Example of an itemset-search-space lattice. In this case, the data space confidence framework of Agrawal (Agrawal et al., 1993). is formed by 4 items. Indexes represent the lexicographic order of the itemsets in the space. . . . . . . . . . . . . . . . . . . . . . . . . 3.1 38 Hypothetical Neural-based framework for ARM. In particular, this thesis focuses on developing an artificial memory for its purposes (colored area). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 53 General structure of a mapping neural network. This appears in (Ham and Kostanic, 2001). . . . . . . . . . . . . . . . . . . . . . . . . vii 59 3.3 Outline of the theoretical internal support model defined in (GardnerMedwin and Barlow, 2001) to produce the counting of patterns with distributed representations in a group neurons. . . . . . . . . . . . . 3.4 62 Theoretical projection models defined in (Gardner-Medwin and Barlow, 2001) to produce the counting of patterns with distributed repre- . . . . . . . . . . . . . . . . . . . 66 4.1 Illustration of the accumulation of knowledge by a CMM. . . . . . . 73 4.2 Illustration of the accumulation of knowledge by a weightless CMM sentations in a group neurons. which has been modified to collect frequency information. The dark matrix illustrates the new matrix called the frequency matrix Mf which contains the corresponding pattern frequencies. 4.3 . . . . . . . . . . . 75-by-75 frequency matrix formed by a CMM, from which itemsetsupport recalls about the Chess dataset will be made. . . . . . . . . . 4.4 85 119-by-119 frequency matrix formed by a CMM, from which itemsetsupport recalls about the Mushroom dataset will be made. . . . . . . 5.1 84 129-by-129 frequency matrix formed by a CMM, from which itemsetsupport recalls about the Connect dataset will be made. . . . . . . . . 4.5 75 85 Maps resulting from training a SOM with artificial datasets describing associations. The red hexagons on the gray maps define the hits received from the input patterns during training. Cluster formations are presented with the coloured maps. viii . . . . . . . . . . . . . . . . . 97 5.2 This figure illustrates the importance of the mean in the calculation of the support of an item from a trained SOM. Different number of transactions (n) composing of zeros and ones have been used to form the bottom graphs. These graphs show that different concentrations (densities) of these bistate values captured in an item induce the tendency of the distribution of the curve to approach the densest value (e.g., in the left graph the number of failures (zi =0) is greater than the number of successes, therefore the highest point of the distribution tends to be placed at 0). 5.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 The figure on the left depicts the intersections of an event B with events A1 ,. . . ,A5 of a partition over S. The figure on the right depicts the concept of Voronoi regions which can be formed on the SOM (the dots represent the codewords while the stars represent the data points assigned to each Voronoi region). . . . . . . . . . . . . . . . . . . . 109 5.4 Representation of the Probabilistic Itemset-support Estimation Mechanism (PISM) proposed in this chapter. . . . . . . . . . . . . . . . 111 5.5 Results for the support value of 15 itemsets (top graph), 255 itemsets (centre graph) and 65535 (bottom graph) obtained after using PISM in order to satisfy the query -All- to the map trained with the dataset Bin4, Bin8 and Bin16 respectively. For reference, the values corresponding to the same queries using an Apriori implementation are also plotted. . 115 5.6 Intermediate results (The support values of 15 itemsets) generated from using PISM for the query -All- to the map being trained with dataset Bin4x100. In both cases, the SOM needs five epochs to converge but after the first epoch, good estimations can be formed for the support of itemsets. The small difference in the performance between these two exercises is due to the type of initialisation chosen. . . . . . . . . . . 116 ix 5.7 Results (plots on the right) obtained after using PISM in order to satisfy the queries -1Itemset : 4Itemset- to the map trained with the dataset Chess. For reference, the values corresponding to the same queries (plots on the left) against the dataset Chess using an Apriori implementation are also plotted. . . . . . . . . . . . . . . . . . . . . . . 117 5.8 Results (plots on the right) obtained after using PISM in order to satisfy the queries -1Itemset : 3Itemset- to the map trained with the dataset Mushroom. For reference, the values corresponding to the same queries (plots on the left) against the dataset Mushroom using an Apriori implementation are also plotted. . . . . . . . . . . . . . . . . . . . . 118 5.9 Results (plots on the right) obtained after using PISM in order to satisfy the queries -1Itemset : 3Itemset- to the map trained with the dataset Connect. For reference, the values corresponding to the same queries (plots on the left) against the dataset Connect using an Apriori implementation are also plotted. . . . . . . . . . . . . . . . . . . . . . . 118 5.10 Distribution of the itemset-support estimations made by SOM via our method for the query -90to100Itemsets- for the Chess dataset. The corresponding errors are summarized in Table 5.8. . . . . . . . . . . 123 5.11 Generalising errors for the results given by Table 5.8. While the xaxis represents the different itemset groups, the y-axis determines the calculated error. . . . . . . . . . . . . . . . . . . . . . . . . . . 125 5.12 Distribution of the itemset-support generalisation error made by SOM via our method for the queries: 1Itemsets, 2Itemsets and 3Itemsets for the Chess dataset, when the size of the map, representing an itemsetsupport memory for ARM, increases. . . . . . . . . . . . . . . . . 127 x 5.13 Distribution of the itemset-support generalisation error made by SOM via our method for the query 45to100Itemsets for the Chess dataset, when the size of the map, representing an itemset-support memory for ARM, increases. . . . . . . . . . . . . . . . . . . . . . . . . . . 128 6.1 Algorithm SOM training in batch. . . . . . . . . . . . . . . . . . . 137 6.2 Tendency of the final influence given by the node mj depending on the strength of the influences received from other nodes. . . . . . . . . . 141 6.3 Incremental algorithm proposed for SOM in batch. While the first (top) function triggers the learning at each stage of a non-environment, the second (bottom) function performs the training of the SOM with the current data chunk and the old information coming from the set of best matching units of the latest trained map. . . . . . . . . . . . . . . . 143 6.4 Representation of a training data space describing a non-stationary environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 6.5 Topologies formed by a SOM trained with our incremental batch approach through the six different phases defining the non-stationary environment represented in Figure 6.4. The black dots define the structure of the trained map. The data points used for the training of a SOM at each phase of the environment, according to the order in Table 6.1, are defined by the green and blue dots, which represent respectively the old knowledge (data extracted from the BMUs) and the current data chunk. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 xi 6.6 Differences between estimations (approximations) and calculations (real values) made respectively by a trained SOM and the Apriori algorithm for the group of 1-itemsets throughout the four phases of the environment I. The first column from the left describes the type of estimations that can be produced with a SOM trained with only the data chunk in turn in the environment (Chunk-SOM). The second column represents the estimations that can be made from a SOM trained with our incremental approach (Bincremental-SOM). The last column shows the estimation made by a SOM trained with always all the data chunks available en the environment (Allchunks-SOM). . . . . . . . . . . . 151 6.7 Differences between estimations (approximations) and calculations (real values) made respectively by a trained SOM and the Apriori algorithm for the group of 2-itemsets throughout the four phases of the environment I. The first column from the left describes the type of estimations that can be produced with a SOM trained with only the data chunk in turn in the environment (Chunk-SOM). The second column represents the estimations that can be made from a SOM trained with our incremental approach (Bincremental-SOM). The last column shows the estimation made by a SOM trained with always all the data chunks available en the environment (Allchunks-SOM). . . . . . . . . . . . 152 6.8 The RMS error during the phases of the environment I for the group of the 1-itemsets. . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 6.9 The RMS error during the phases of the environment I for the group of the 2-itemsets. . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 6.10 Error describing the quality of the quantization of the approaches tested for the data chunk describing the changes at each phase of the environment I. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 xii 6.11 Error describing the quality of the quantization of the approaches tested for the data chunks describing the previous phases (history) at each phase of the environment I. . . . . . . . . . . . . . . . . . . . . . 155 6.12 RMS Error during the phases of environment II for the group of the 1-itemsets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 6.13 RMS Error during the phases of environment II for the group of the 2-itemsets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 6.14 Error describing the quality of the quantization of the approaches tested for the data chunk describing the changes at each phase of the environment II. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 6.15 Error describing the quality of the quantization of the approaches tested for the data chunks describing the previous phases (history) at each phase of the environment II. . . . . . . . . . . . . . . . . . . . . . 159 6.16 RMS Error during the phases of environment III for the group of the 1-itemsets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 6.17 RMS Error during the phases of environment III for the group of the 2-itemsets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 6.18 Error describing the quality of the quantization of the approaches tested for the data chunk describing the changes at each phase of the environment III. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 6.19 Error describing the quality of the quantization of the approaches tested for the data chunks describing the previous phases (history) at each phase of the environment III. . . . . . . . . . . . . . . . . . . . . 161 6.20 Runtime for the three approaches evaluated on the environment III. . . 165 xiii 7.1 Example of association rules which can be further generated from the knowledge learnt by a neural network about the Mushroom dataset defined in (D.J. Newman and Merz, 1998). All these rules, describing the associativity among the attributes of a dataset, have the format of: if (list of items or attributes) then (list of items or attributes) with [support=% and confidence=%]. . . . . . . . . . . . . . . . . . . . 171 7.2 Triangular binary formations formed with the itemset search space formed with 3 and 4 items. . . . . . . . . . . . . . . . . . . . . . 189 7.3 Function generated with the support of some groups of itemsets derived from the Chess dataset. . . . . . . . . . . . . . . . . . . . . . . . 194 7.4 Incremental SOM-based approach for distributed ARM. The rules will be generated from the latest trained SOM. . . . . . . . . . . 195 7.5 Local SOM-based approach for distributed ARM. While the local maps are queried remotely in the model on the left, the trained maps are transmitted to be the source from rules will be generated in the model on the right. . . . . . . . . . . . . . . . . . . 196 7.6 A neural-based approach for ARM for data streams. . . . . . . . . . 197 A.1 The Apriori algorithm. This figure was extracted from the original paper of Agrawal (Agrawal and Srikant, 1994). The top pseudocode describes the main steps of Apriori. The bottom SQL query defines the way in which candidates are formed during a mining process. . . . . . 200 B.1 An Associative memory based on a CMM. xiv . . . . . . . . . . . . . 202 List of Tables 4.1 List of real-life binary training datasets used in the testing of an autoassociative memory for ARM. They are part of the datasets normally used for testing FIM algorithms or FIM benchmarks (Jr. et al., 2004; Goethals and Zaki, 2003). . . . . . . . . . . . . . . . . . . . . . . 4.2 Support constraint conditions used to form the group of itemsets on which the memories will be tested. 4.3 81 . . . . . . . . . . . . . . . . . 83 Error results obtained in the experiments for the support recall for the groups of 3-itemsets, constrained as defined in Table 4.2, made by a CMM through our proposals. . . . . . . . . . . . . . . . . . . . . 4.4 86 Error results obtained in the experiments for the support recall for the groups of 4-itemsets, constrained as defined in Table 4.2, made by a CMM through our proposals. . . . . . . . . . . . . . . . . . . . . 4.5 86 Error results obtained in the experiments for the support recall for groups of different k-itemsets of the Chess dataset made by a CMM through our proposals. The elements (itemsets) of the groups used to make the CMM recall were defined by the Apriori algorithm through constraining the itemsets with a minimum support between 90 and 100%. 87 xv 4.6 Error results obtained in the experiments for the support recall for groups of different k-itemsets of the Chess dataset made by a CMM through our proposals. The elements (itemsets) of the groups used to make the CMM recall were defined by the Apriori algorithm by constraining the itemsets with a minimum support between 45 and 100%. 4.7 87 Error results obtained in the experiments for the support recall for groups of different k-itemsets of the Connect dataset made by a CMM through our proposals. The elements (itemsets) of the groups used to make the CMM recall were defined by the Apriori algorithm by constraining the itemsets with a minimum support between 75 and 100%. 4.8 88 Error results obtained in the experiments for the support recall for groups of different k-itemsets of the Mushroom dataset made by a CMM through our proposals. The elements (itemsets) of the groups used to make the CMM recall were defined by the Apriori algorithm by constraining the itemsets with a minimum support between 45 and 100%. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 88 List of binary-artificial training datasets used in the experiments for analysing SOM properties for ARM. Each dataset contains all the possible n itemsets generated by m items. . . . . . . . . . . . . . . . . 5.2 95 List of real-life binary training datasets used in the testing of PISM for SOM. They have been used in FIM-algorithm benchmarks (Jr. et al., 2004; Goethals and Zaki, 2003). . . . . . . . . . . . . . . . . . . . 112 5.3 List of queries used to form the groups of itemsets used for the testing of the itemset-support estimations from SOMs via PISM. In this case, k means the size of the itemsets and σ refers to the support used to form such itemset groups. . . . . . . . . . . . . . . . . . . . . . 113 xvi 5.4 Generalisation errors produced by a trained SOM through PISM for the queries 1Itemsets, 2Itemsets and 3Itemsets derived from the Chess dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 5.5 Generalisation errors produced by a trained SOM through PISM for the queries 1Itemsets, 2Itemsets and 3Itemsets derived from the Mushroom dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 5.6 Generalisation errors produced by a trained SOM through PISM for the queries 1Itemsets, 2Itemsets and 3Itemsets derived from the Connect dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.7 Generalisation errors for the trained SOMs with linear initialisation. . 122 5.8 Generalised errors for trained SOMs with linear initialisation on different ranges of groups of k-itemsets. . . . . . . . . . . . . . . . . . . 124 6.1 Data order followed in the incremental batch training. . . . . . . . . 144 6.2 Definition of the non-stationary environments using the Chess dataset. In the first two environments, each data chunk has the same (fixed) number of transactions, while in the case of the third environment, the number of transactions was chosen randomly (unfixed). . . . . . . . 147 6.3 Results obtained of the approaches tested for Environment I. . . . . . 162 6.4 Results obtained of the approaches tested for Environment II. . . . . . 163 6.5 Results obtained of the approaches tested for Environment III. . . . . 164 7.1 Characteristics of the two itemset-support memories based on the studied neural networks. m defines the total number of items from which the n transactions or itemsets of a training data D are derived. mb corresponds to the number of BMUs formed in an epoch training. xvii . . 179 7.2 Comparison of the generalised errors between our approaches for SOM and CMM for the different number of tested itemsets, which resulted a query from the Chess dataset with a minimal support between 90 and 100 %. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 7.3 Comparison of the generalised errors between our approaches for SOM and CMM for the different number of tested itemsets, which resulted a query from the Chess dataset with a minimal support between 45 and 100 %. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 xviii Acronyms and Symbols xix Acknowledgement I first thank My supervisor Dr. Simon O’Keefe for his time given throughout the development of this research. I must express my deeply gratitude to the CONACYT, which is the national counsel of science and technology in Mexico, and the central bank of Mexico for the support offered, without which this thesis and the experience around it would not be possible. Throughout these years I had the pleasure to meet many nice people and become their friend. Among them, I wish to acknowledge Patricia and Michael Lee for their support and positive words. I also want to express many thanks to the staff of the Computer Science Department of the University of York. Moreover, I want to thank the anonymous conference reviewers who provided some valuable comments which helped me in the conclusion of this work. Most importantly, I would like to express my thanks to Victoria Mueller who has been an important person to me in one way or another. Finally, I must thank my family (Marcela, Vicente and Carlos) for all their love, encouragement and support given to me throughout all these years. In particular, I must express my gratitude to my mother for showing me that everything is possible in this life. My deepest gratitude I must reserve for myself because we never gave up in the difficult times. xx Declaration I declare that work proposed by this thesis is solely my own, except where indicated, attributed or cited to other authors. Some of the material of this thesis have been previously published and presented in conferences. A complete list of publications is provided on the next page. xxi List of Publication The following is a list of publications that has been produced during the course of this research. 2005 • Vicente O. Baez-Monroy and Simon O’Keefe, ”Modelling Incremental Learning With The Batch SOM Training Method”, HIS ’05: Proceedings of the Fifth International Conference on Hybrid Intelligent Systems, IEEE Computer Society, Rio, Brazil, pages 542–544, 2005. • Vicente O. Baez-Monroy and Simon O’Keefe, ”Principles of Employing a Self-organizing Map as a Frequent Itemset Miner”, Artificial Neural Networks: Biological Inspirations - ICANN Proceedings, 15th International Conference, Warsaw, Poland, September 11-15, 2005. 2006 • Vicente O. Baez-Monroy and Simon O’Keefe, ”The Identification and Extraction of Itemset Support Defined by the Weight Matrix of a SelfOrganising Map”, 2006 IEEE World Congress on Computational Intelligence, 2006 Proceedings of the International Joint Conference on Neural Networks IJCNN, July 16-21, Vancouver, BC, Canada, 2006. xxii Chapter 1 Introduction Data is a common term to denote the collection of facts or raw information describing processes or activities, or the status of events occurring in an environment. Data normally exists in a variety of forms; however, it is often categorised into the groups of numbers (numeric or quantitative data) and symbols (categorical or qualitative data), depending on the meaning of the variables or features of the events that want to be modeled. The collection of data is an important activity which has been performed throughout our history and has contributed to the development of our societies in one form or another. For instance, its collection has allowed human beings develop methods for climate forecasting, medical diagnosis and treatments, highway and house planning, and so forth. As we are surrounded by data everywhere and its collection has an important significance for our daily life, some areas in Computer Science (CS) have been particularly dedicated to the development of the necessary technology to collect, store and model large amounts of data from almost every possible recordable source of information. For example: bank operations, video, medical checks, images, shopping activities, vehicles, web transactions, and many others. Among the areas responsible for the management of data, a first example is the 1 database area which focuses on its modeling. Normally, the term database in this area refers to an entity-relationship model environment, in which the data changes periodically due to new operations (inserts, deletes, updates) represented by transactions. A second example is the data warehouse area in which data is seen as a large repository containing integrated information for efficient querying and analysis. The resources of the data modelling are based on a star-schema concept (Kimball, 1996) rather than a transactional one. The main ambition behind the data collection is to provide humans with precious resources of data regarding a specific event which can be converted into the more valuable commodity called knowledge or information. Once data have been collected, the search for interpretations begins, aiming to gain a novel or better understanding of the variables modelled in the data. This search activity, known as data analysis, has been a permanent motivation for researchers in the development of new methods which can provide faster, better and new interpretations from diverse and unlimited data sources. For the last decade, it has gradually been becoming a rule that any project, addressing the extraction of knowledge from a data source, must follow the framework defined by the area of knowledge discovery. In this area, the term of KDD1 , attributed to Piatetsky-Shapiro et al (Piatetsky-Shapiro and Frawley, 1991; Piatetsky-Shapiro, 2000), has been coined to define that: The knowledge discovery in databases is the untrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns2 . 1 KDD stands for Knowledge Discovery in Databases. The term pattern is used here to define any final product (e.g., a series of clusters, a classification model, rules, etc.) resulted of any knowledge discovery process. 2 2 In the KDD process, while the first stages have been dedicated to the management, cleaning and transformation of data, the last ones are devoted to the manipulation of the knowledge which focuses on presenting the results in the most profitable and understandable format for the human experts. Occupying the central activity and representing the core of the process, data-mining algorithms are placed. These algorithms are responsible for conducting the discovery of knowledge by performing a series of operations to form a model which satisfies the definition of the values assigned to the input parameters for the mining activity. DM (Data Mining) (Han and Kamber, 2000; Hand et al., 2001; Cios, 2000) is an area in Computer Science whose efforts are set on developing these new algorithms for data analysis. The central focus of these algorithms is to build either predictive models, in which some features or variables of the original data are used to build a model which will be predicting unknown or future states of the model variables, or descriptive models, which focus on describing the data through finding hidden patterns or relationships in the data to gain some understanding of it. The quality of the models generated as well as the performance of the DM algorithm for a specific problem are often affected by the nature of the data, which can be redundant, imprecise, noisy, high-dimensional or dynamic. Therefore, a successful DM approach is often considered an algorithm which deals reasonably well with the characteristics defined by data. The spectrum of techniques in DM can be classified (according to the type of task to be performed) into clustering, classification, regression and prediction, association rule mining and sequence analysis. Nevertheless, this DM classifica- 3 tion is not unique and specialised extensions to each technique can also be found in literature (e.g., spatial mining, web mining, outlier detection, frequent itemset mining, text mining, bio mining, and so forth.). Data mining is often defined as a multi-disciplinary area which integrates miscellaneous techniques from different fields such as Machine learning, Statistics, Neural networks, etc. Therefore, it is rare to find a unique solution for a data-mining problem. Among the descriptive DM techniques, ARM (Association Rule Mining) stands out because of its comprehensive and powerful format to represent the knowledge discovered from a data source. ARM, introduced by Agrawal et al (Agrawal et al., 1993), is a technique to form rules defining the associativity among the components, attributes or variables of a data source or environment D. In its beginnings, ARM was conceived to analyse shopping-basket databases to identify customer-behavior patterns. Nowadays, it has been applied to problems in which the associativity property of the variables defining a dataset can be exploited (e.g., web mining, text mining, bio mining). In ARM, the data source D (database, dataset), which is represented by n probable different patterns x1 , ..., xk , ..., xn (transactions), can be modeled horizontally mainly in two forms. The first is as a collection of binary vectors of length m (m defines the number of components, elements, or attributes representing the transactions) in which while the absence of value at some pattern element, for instance xki =0, refers to the lack of participation of the element i for the pattern (transaction or association) xk , a value of one (xki =1) defines the contrary event for the element (its participation). The other method of representation is as size-varied-integer vectors whose elements represent the indexes i of elements with participation (xki =1) in the binary representation. Individual 4 elements of the patterns are known as items Ii , and an itemset is defined as a grouping or a combination of some items defining a pattern. Rules with the format of X ⇒ Y [supp, conf ]3 are the aim of ARM for describing the participation (associations) of a total set of items, Γ = {I1 , .., Im }, in the transactions contained in D. To form such descriptions of D, according to Agrawal (Agrawal et al., 1993), ARM can be divided into a two-stage problem. This standpoint of ARM is known as the support-confidence framework which basically defines that association rules can be formed by first finding the frequent itemsets in D and second by generating the rules from such itemsets. The former is a DM task known as FIM (Frequent Itemset Mining) (Goethals, 2003) and involves the formation and counting of the different possible k-itemsets for all 1 ≤ k ≤ m to determine the relevant ones based on a minimal support threshold σ. The latter, which defines the RG (Rule Generation) stage and has been identified as a straightforward process, uses the information resulting from FIM for the generation of rules. To determine which k-itemsets are the important (frequent) ones in a mining exercise, the generation of candidates and the calculation of itemset support are performed. Candidate generation refers to the formation of new k-itemsets (combinations of k items) during the process, while the calculation of the support refers to action taken (counting) by the algorithm to measure the appearance of the candidates in the n transactions. The relevance of the support for ARM, in particular for FIM, is high since it is the main metric by which an algorithm determines the status of an itemset against the threshold. This is, the support of a 3 This is the simplest format of an association rule. X and Y are known as the precedent (the ”if” part) and consequent (the ”then” part) of the rule. These two parts are called itemsets. Two basic metrics, supp and conf , defining the support and confidence of the rule, are used to define the interestingness of the rule. 5 rule, for instance A → B, is equal to the support of its itemset; i.e., supp(A → B) = supp(AB). For the generation of rules, another metric, known as confidence, is used to determine the strength of the rules which are derived from the final set of frequent itemsets along with their corresponding support. To illustrate the concept of ARM, an example is given in Figure 1.1 in which we show the type of knowledge that can be derived from performing ARM on a set of transactions representing a shopping basket dataset. As commented above and shown with our example, the generation of association rules is produced by performing first FIM, and then using the discovered itemsets to produce the target rules. Association rules provide a description of data in the form of ”if-then” statements. Unlike the if-then rules of logic, association rules are probabilistic in nature. The challenge of the generation of association rules from a database is considerable, since its complexity, which is mainly defined by the complexity of FIM, is exponential. That is, the exponential search space, formed by m items conforming n transactions in a database, is defined by 2m . Therefore, according to Goethals (Goethals, 2002), in many applications which involve hundreds of items, it is easy to find a search space with a number of itemsets larger than the number of atoms in the universe (≈ 1079 ). Even though other metrics have been proposed to define rule or itemset interestingness (Hilderman and Hamilton, 1999; Tan et al., 2002; Meo, 2003; Omiecinski, 2003) and some critiques have been made of the support-confidence framework (Brin et al., 1997; Aggarwal and Yu, 1998), the importance of the framework is not in question because its methodology has been adopted in one 6 Figure 1.1: An illustration of ARM applied to the data source defined in (a). The aim is to generate rules with a minimal threshold of 20%. As the first part of ARM, the frequent itemsets have to be discovered, such as in (b), based on their support. Then, association rules, as in (c), are formed from them. form or another by the majority of the successful and current state-of-the-art algorithms. For instance, Apriori (Agrawal and Srikant, 1994; Mannila et al., 1994), Eclat (Zaki et al., 1997a) and FP-growth (Han et al., 2000a). The diversity of the FIM algorithms in terms of the knowledge area which they are based on is not as vast as in other DM techniques. For example, as in pattern classification (Duda et al., 2000), in which approaches based on Decision Trees, Neural Networks, Bayesian-based methods, and many others, can be used 7 for performing the task. The reason of the latter is that the main work on ARM has focused on finding the best data structure to reduce the complexity of FIM (the counting of items and the selection of the frequent ones) rather than trying to develop possible alternatives to tackle the problem. For instance, proposals involving biologically-based technologies, such as genetic algorithms, have rarely been undertaken (Vázquez et al., 2002; Yan et al., 2005b; Alatas and Akin, 2006). In particular, it seems that the concept of approaches inspired by the learning capabilities of biological systems is not acceptable or suitable for generating such associativity descriptivedata models even if it could be possible to relate the problem of ARM to the general notion of inference in the learning activity performed by humans in which data-driven interactions with the environment constantly occur. For instance, in a task of learning from a data source such as the one represented in Figure 1.2, a human is able to perform two classical types of inference, called induction and deduction, in order to obtain and use an abstract representation (model or knowledge) of the data source. With such a model formed (deduction), he is able to categorise (predict) the corresponding group to new items depending on their nature and purpose (e.g., garlic and olive oil can be grouped into the taxonomy of condiments and cooking ingredients). A third inference case known as transduction4 could also be performed if the task would imply to form answers for particular points of interest directly from the data (training dataset) without having to build a global model of it. As stated in (Kantardzic, 2002), an important application of the latter is ARM. Moreover, this human could perform the counting of real items or the formation of new abstract representa4 This term refers to the fact of directly drawing conclusions about data without constructing a model. Transduction has been introduced in (Vapnik, 1998). 8 Figure 1.2: A shopping receipt as an example of a data source in which the associativity concept can be exploited for the extraction of knowledge. tions of items by combining them (e.g., milk and eggs and vanilla = cake). The counting of either real or abstract items will be done by scanning the information defined by either his shopping basket or his receipt. As a result of performing a pass over the data (learning), he would have kept some knowledge in his memory (brain area, neurons) regarding single or collective, real or abstract itemsets, which can be used to answer questions about the item appearance such as - How many trios involving diet coke, crisps and napkins have been bought? Or have toothpaste and aspirin been bought together? - It is very likely that his answers regarding the frequency (support) of either real or abstract items would not be necessarily exact at first, therefore, an error in his item-frequency recalls, defined by Errorrecalling = Realf requency -HumanApproximationf requency , can be stated to exist. The accuracy of his answers will depend on how well he is able to remember (memory issues) the number of appearances (itemset support) for each of the different 2m combinations that can be formed by his list of m individual items. In order to elevate the accuracy of his answers more passes on the data (training epochs) will have to be performed by him. 9 If the current situation involved a large number of sources (multiple transactions) needing to be mined, then the problem would become more complicated for the human. The complexity of this situation can be measured by O(mcn) in which m, c and n define the number of items, the number of possible candidates (the possible itemset search space whose size grows exponentially) and the number of transactions respectively. Even though the complexity grows as a result of adding a new item or transaction, a human will always be able to produce an answer, defining a generalisation of the frequency occurrence of the queried itemsets, through a learning (scanning and counting) of the associations existing in the data. For the time being, we can therefore conclude that ARM is a mechanical activity performed naturally by humans, which involves factors such as pattern association, pattern counting, and memory, in which the better the answer, the larger the memory abilities and the number of passes over the data source. Additionally, it has been stated by A.R. Gardner-Medwin and H.B. Barlow (Gardner-Medwin and Barlow, 2001) that the concepts of learning, counting, frequency, and association are all intimately related in the genetic mechanism of learning as follows: ...Sensory inputs are graded in character and may provide weak or strong evidence for identification of a discrete binary state of the environment such as the presence or absence of a specific object. Such classifications are the data on which much simple inference is built and about which associations must be learned. Learning any association requires a quantitative step in which the frequency of a joint event is observed to be very different from the frequency predicted from the probabilities of its constituents. Without this step, 10 associations cannot be reliably recognized, and inappropriate behavior could result from attaching too much importance to chance conjunctions or too little to genuine causal ones. Estimating a frequency depends in its turn on counting, using that word in the rather general sense of marking when a discrete event occurs and forming a measure of how many times it has occurred during some epoch. Counting is thus a crucial prerequisite for all learning, but the form in which sensory experiences are represented limits how accurately it can be done... The situation explained above is just a basic example of an associative reallife problem in which the computational power of the brain can be used to process a correct answer, which in this case cannot be performed as fast as we wish to. Nevertheless, in this digital era, artificial architectures such as an ANN (Artificial Neural Network) can be used to imitate the human brain since as defined in (Haykin, 1999), an ANN is: a massively parallel distributed processor that has a natural propensity for storing experimental knowledge and making it available for use. It resembles the brain in two respects: 1. Knowledge is acquired by the network through a learning process. 2. Interneuron connection strengths known as a synaptic weights are used to store its knowledge. Hence, an ANN can be used to perform tasks in which the concepts of learning and generalisation are involved. Moreover, its use becomes more attractive when the problem in turn implies making use of its power to recall previous 11 knowledge or their experience of the environment learnt. Therefore, in order to contribute with new biologically-inspired approaches that can be used for problems such as ARM in which the counting of patterns representing associations (events) is significant, we investigate in this thesis the suitability of ANNs for ARM. In particular, we focus on determining if association rules can be generated from the knowledge embedded in the synaptic weights of a neural network. Moreover, since the training data can be representing a dynamic environment, we also conduct research on determining if the synaptic weights of an ANN, which serve as the basis of the rules, can be updated throughout time with the changes occurring in the training environment. 1.1 The Role of Neural Networks in Data Mining As mentioned previously, data mining is constituted by a large number of techniques coming from different CS fields which have been incorporating throughout time. The diversity in techniques opens up the possibility of addressing a data-mining problem with different proposals. For instance, a classification problem is often achieved by using decision trees; nevertheless, neural networks have already been shown to be promising alternatives to conventional methods (Zhang, 2000). Nowadays, among the techniques forming the data-mining toolbox, ANNs have emerged to be considered as one of the most useful tools to the point of having become part of the current-commercial KDD solutions such as Clementine (SPSS, 1968). Their inclusion in the data-mining framework has not been easy since they were not initially considered suitable for these tasks due to the main criticism of neural networks which lies in the following aspects. First, neural networks have been categorised as black-boxes whose re- 12 sults lack for symbolic representation. Therefore their verification, integration and interpretation are difficult. Second, the time used-up for the training of a neural network can be beaten by conventional methods. As a response to these critiques, approaches, such as (Lu et al., 1996; Craven and Shavlik, 1997; Zhang, 2000), have appeared to argue and support for the reconsideration of neural networks for the data-mining framework. To argue for their inclusion, some points are outlined as follows: • To tackle the incomprehensibility of the neural models, some work has been produced for the extraction of rules and the visualization of the knowledge modeled by their weight matrices (Ultsch and Siemon, 1990; Hammer et al., 2002). • Neural networks are data driven self-adaptive methods which adjust to the input data. This adaptation is sometimes done without any explicit specification of functional or distributional form of the underlying model. For instance, the SOM (Self-Organizing Map) (Kohonen, 1996). • They are able to approximate any function with arbitrary accuracy. • They have the capability of remembering past states of the input data within their weight matrices; i.e., they resolve the elasticity-plasticity dilemma such as the ART network (Carpenter and Grossberg, 1989). • Neural networks are nonlinear models so that they can be more flexible in modeling real world complex relationships. • Statistical analysis can be conducted on the basis given by ANNs. • New training algorithms have reduced the training time without sacrificing the accuracy of the results. 13 • They form a more suitable inductive bias than the conventional algorithms. Because of the type of results given by neural networks, they have also been considered an important part of the soft computing5 framework for datamining (Mitra et al., 2002), which carries on with the aim of producing methods of computation that perform a reasonable solution at low cost by seeking an approximate solution to either imprecise or precise problems. Considering the above characteristics of neural networks, we can conclude that they are on the right track of scaling-up to problems involving large datasets and becoming important members of a new generation of algorithms in which human interaction to compute hypotheses will be relevant. Speaking generally, the number of contributions of neural network approaches for DM problems, whose total description goes beyond the limit of this thesis, such as classification, clustering and prediction is already vast and constantly increasing. Nevertheless, for other problems, such as descriptive DM tasks, in particular ARM, the attention of the ANN research community remains passive and uncertain. 1.2 Linking ANNs and ARM: The Motivations The conception of the use of ANNs for ARM may at first appear inappropriate since it can be argued that in the task of ARM there is nothing to be classified, clustered or predicted. Nevertheless, the use of ANNs starts making sense if some factors are well thought out as follows: The Meaning of Association. In ARM, association is a concept defined implic5 Soft computing (Kecman, 2001) is not a clearly defined discipline at present, but it follows the premises that: 1) the real world is pervasively imprecise and uncertain and 2) precision and certainty carry a cost. In other words, they are methodologies which aim to exploit the tolerance for impression, uncertainty, approximate reasoning, and partial truth in order to achieve tractability, robustness, and low-cost solutions. 14 itly by the appearance of m items in the n transactions of a data source D. It is a concept exploited for the formation of knowledge formatted like rules about some D and its discovery is the main purpose of this DM task. Associations are represented physically by patterns formed through the grouping of binary or numeric elements whose strengths of associativity are mainly measured by the frequency (support) of the members. Similarly, in the field of Neural Networks, the concept of associativity has also been employed. In this case, it defines an explicit relationship that exists between a pair of input patterns (input and target) presented to the neural network under supervised training. This concept is mainly exploited by neural architectures which imitate the concept of associative memory presented in our brain. In general terms, the explicit association among the inputs is the target to be learnt by some networks to form a knowledge which can be used for pattern association or recognition tasks. Hence, based on the fact that both technologies know and manage the concept of association for creating a better understanding of some environment D; why should we therefore not assume that one of the best techniques for pattern recognition under association concepts can perform the counting of patterns or sub-patterns (itemsets) defining associativity among their elements. Rules as Symbolic Representation of Knowledge. One of the fundamental considerations for not using ANNs for data analysis tasks has been the comprehensibility of their results. Therefore, since the late 80’s (see beginnings in (Towell and Shavlik, 1993; Craven and Shavlik, 1993; Andrews 15 et al., 1995)), some efforts have been put into gaining interpretations of these nonlinear systems. One approach to tackle this drawback has been to develop methods of rule extraction (Benitez et al., 1997; Tickle et al., 1998; Taha and Ghosh, 1999; Craven and Shavlik, 1999; Tsukimoto, 2000; Browne A., 2003; Malone et al., 2006), which turn the conversion of the real-valued knowledge formed in the weight matrix into symbolic rules, describing the inside of the neural model in a global or local manner (depending on the level of description). Rule-extraction mechanisms already exist for different neural architectures such as multi-layer perceptron (Duch et al., 1996; McGarry et al., 1999), recurrent networks (Jacobsson, 2005), self-organising networks (Hammer et al., 2002; Malone et al., 2006), radial basis function networks (McGarry et al., 1999), and even for hybrid architectures (Eggermont, 1998). They have focused on the formation of if -then or logic, m-of-n (Setiono, 2000) and fuzzy rules, in particular for classification tasks. According to Krishnan (Krishnan et al., 1999), rule extraction is a search problem because it requires exploring a space of candidate rules to test each individual candidate against the network to determine if they are valid. This validity refers to the rule property of describing what happens inside of the neural network. In particular, the behavior of the network for those instances (inputs) that will form the antecedents of such rules. A rule-extraction method searches until all the maximally-general rules have been found. Heuristics have also been employed to limit the combinatorics of the rule-transverse, for instance, the number of elements in the rule antecedent and/or the search has been limited to combinations of elements that only occur in the training dataset. Speaking generally, these approaches have been efficient mechanisms to make neural networks un- 16 derstandable. In ARM, rules are generated to explain the events occurring in a data source D. Their generation is realised by performing a process which resembles the counting of patterns in the high dimensional space formed by D. During rule generation, an algorithm typically uses a data structure (e.g., tries, hash trees, and many others.) to represent the findings (patterns and their properties) and to speed up the mining process. Moreover, heuristics have been also proposed to prune the exponential itemset search space formed by the items of D. Hence, based on the similarity between the extraction of rules from neural networks for classification tasks, and the process for generating associations rules from a database, it may be possible to extract association rules from an ANN through a mechanism which can perform the correct interpretation of the knowledge embedded in the weight matrix. Activities and Strategies in ARM. As a result of all the diversity of approaches in ARM proposed since 1993, a framework, which we have summarised and depicted in Figure 1.3, can be defined. Taking into account the current capabilities of neural networks, we could begin by assuming that they may be seen as alternatives to the following activities constituting such a framework: • Candidate-generation Strategies. Up to this day, researchers do not have a certain idea about what decreases the complexity of ARM (Bodon, 2006). Nevertheless, it has 17 Figure 1.3: Framework defined by the processes and strategies for the problem of as- sociation rule mining. Its conception is based on the support-confidence framework of Agrawal (Agrawal et al., 1993). 18 T1-{1,2,4,6,7} T2-{1,2,3,5,6,7} T3-{1,3,5,6} T4-{1,2,3,5,6,7} T5-{1,3,4,6} T6-{1,3,5,7} T7-{1,2,4,5,7} T8-{1,3,4,5,6} Integer (item index) T1-{1,1,0,1,0,1,1} T2-{1,1,1,0,1,1,1} T3-{1,0,1,0,1,1,0} T4-{1,1,1,0,1,1,1} T5-{1,0,1,1,0,1,0} T6-{1,0,1,0,1,0,1} T7-{1,1,0,1,1,0,1} T8-{1,0,1,1,1,1,0} Binary Data Representation Sampling T1-{1,1,0,1,0,1,1} T3-{1,0,1,0,1,1,0} T4-{1,1,1,0,1,1,1} T6-{1,0,1,0,1,0,1} T8-{1,0,1,1,1,1,0} Vertical (tidlist) i(1)-{T1,T2,T3,T4,T5,T6,T7,T8} i(2)-{T1,T2,T4,T7} Horizontal T1-{1,1,0,1,0,1,1} T2-{1,1,1,0,1,1,1} Data Format Data Definition (DD) PATRICIA Hash-trees Iterative Recursive Frequent, closed, maximal ITEMSETS AB (100%) ABC (10%) ACDF (100%) CDEFGH (70%) DE (10%) Intersecting Counting The counting of support Itemset-support monotonicity property Pruning itemsets Breadth-first search Depth-first search Traversing the search space Candidate Generation Tries RULES A->B (100%, 90%) AB->C (10%, 9%) AC->DF (100%, 12%) BF->GH (50%, 50%) ABCF->H (50%, 50%) CDE->FGH (70%, 90%) D->E (10%, 20%) Frequent itemset and support needed The calculation of confidence Minimal Confidence Threshold Minimal Support Threshold Itemset Representation Rule Generation (RG) Frequent Itemset Mining (FIM) been stated that a possibility may be defined by the total number of itemsets which have to be checked by an algorithm during the process in order to detect those which are interesting. To reduce such a number (c), a phase called candidate generation has been included as part of the definition of some FIM approaches. The phase aims to generate the corresponding group of k-itemset candidates, which may turn out to be frequent after the calculation of their support (e.g., by counting), based on the information provided, for instance, by the k-1th iteration regarding the frequent (k-1)-itemsets. The pruning of the unnecessary itemsets has been possible by using the following heuristic called the downward closure property of itemset support (Agrawal and Srikant, 1994): All subsets of a frequent itemset must also be frequent. Because the main purpose of this strategy is to determine which itemsets will turn out to be unfrequent as soon as possible in the mining process, a neural network, which learns incrementally some information provided from a FIM algorithm during an ARM process, could be taken into account to predict which itemsets are not worth to be considered in the mining process. • Itemset-storage-structure Strategies. Most of the work produced for FIM has focused on this type of strategy since it has been proven that the use of different data structures for representing the itemsets and candidates found during FIM can result in a reduction of time and memory issues (different datastructure-based algorithms are summarized in (Goethals, 2003)). The 19 strategics aim to find the most compact and easy-to-traverse data structure for representing the search space of FIM. Hence, we could think of employing a type of neural network which can organise its neural structure dynamically based on the incoming patterns (itemsets) in order to adapt it to form a representation of the search space lattice in which the mapping of the itemsets can be performed. This may be achieved by an unsupervised neural network. • Input-data-layout Strategies. One of the drawbacks of any DM technique is the dealing with data of high dimensionality. This problem, better known as the curse of dimensionality, is a factor which directly affects the performance of any algorithm. Moreover, the performance tends to get worse when the problem concerned also involves scanning large datasets with dense patterns. In order to tackle this nature of the data, solutions like, for instance, PCA (Principal Components Analysis) (Jolliffe, 1986), have been used to reduce the dimensionality of the original data, before it is mined or visualised. It is important to mention that the representation of the input data plays an important role within the data-mining framework, because it directly affects the result’s accuracy and the performance of the algorithm. Thus, a trade-off always has to be made between performance and accuracy. In ARM, proposals have been employing three different layouts for the input dataset. They involve either the representation (vertical or horizontal) in which the input data will be handled by the algorithm or the quantity (sampling techniques) of transactions that 20 will be employed to form the rules. It is possible that a neural network which has learnt interesting relationships inherent to the input dataset can also provide such a good representation of the associations. That is, input-data-attribute relationships will be encoded in the weight matrix whose size is considerably smaller than the original dataset. This manner of tackling the problem matches with the activities performed in the extraction of knowledge from neural networks explained above. Moreover, this idea would lead to building a form of artificial memory which would learn and remember associations from the input data (imitating what a human would do in the same situation). Nevertheless, in order to form such memories, it is crucial to investigate if counting of patterns can be performed by ANNs. • The Maintenance of Rules. Since data does not only describe stationary environments, researches have proposed the incorporation of the maintenance of rules as part of the general framework. For instance, this data characteristic was first taking into account in (Cheung et al., 1996b). To address the maintenance, the frequent itemset-representation data structure is periodically updated with the changes occurring in the original dataset. The major inconvenience of these approaches are that they are performed on top of, or derived from, the traditional FIM algorithms which brings a new complexity to the entire framework. Moreover, the re-use of knowledge from past mining activities is needed to realise the maintenance which results in keeping certain information of considerable size throughout time which can turn into a new data21 maintenance complexity for ARM. Hence, we believe it would not be mistaken to assume that a neural network with the capability of incremental learning can perform the maintenance of the itemsets by incorporating the changes suffered in the environment of the original data source. This idea can be interpreted to be very closely related to the previous one (artificial memory for ARM) since the ANN may be collecting new knowledge about the current state of the environment infinitely throughout time. Other Factors. One of the advantages of neural networks compared to some typical algorithms is that they can be implemented in hardware, which means that we can think about having an ANN-based piece of hardware in future, in which itemset support can be recalled and therefore FIM can be performed. Moreover, ANNs are considered to be parallel since they simulate the way the biological counterparts work. Employing neural networks for data analysis brings out the advantage of re-using their weight matrices for diverse mining tasks. For instance, it would no longer be necessary for analysts to run different environments in order to create models for clustering and association rules, if good results could be obtained from the same source of knowledge. Moreover, discovering that such re-use happens for different purposes (not only for predictive but also descriptive tasks) in some of the current models of ANNs would reinforce the idea that this biologically-inspired technology models some conditions happening in the human brain, in which different brain areas process or control different biological activities in response to stimuli by exploiting the knowledge allocated by their neurons. 22 D(t) RECALLING ITEMSET SUPPORT ENVIRONMENT TRAINING (COUNTING) A->B (100%, 90%) AB->C (10%, 9%) FIM / ARM AC->DF (100%, 12%) BF->GH (50%, 50%) LOGIC ABCF->H (50%, 50%) CDE->FGH (70%, 90%) D->E (10%, 20%) DECODING COUNTING D(t+k) NEURAL NETWORK (ITEMSET- SUPPORT MEMORY) (FREQUENT- PATTERN MEMORY) QUERYING SUPPORT FOR AN ITEMSET Figure 1.4: Neural-based framework for ARM. As described in this section, there are some links, which can be interpreted as part of the motivations for this research, between ARM and ANNs to believe that the use of an ANN for problems like ARM is possible. Thus, this research carries on with the hypotheses that the ARM framework can be transformed in such a way that neural networks constitute the core of the process. This conception of the idea is illustrated in Figure 1.4. 1.3 The Neural-Network Candidates An introduction of the models considered in this work will herein be given. Moreover, some of the reasons behind their inclusion within this study are described. Details of the justification for research on each neural network will be stated in Section 3.5 in Chapter 3. Auto-Associative Memory (AAM). This neural network, as its name says, is trained to capture associations between incoming pairs of patterns for future recognition. The input patterns can be expressed in form of unipolar 23 vectors. In learning tasks in which labels or targets yi do not exist to be associated with the training data xi , this ANN allows the formation of associations between a pattern xi and itself. The maximum size of the weight matrix formed can be expected to be m-by-m for an auto-associative memory, where m refers to the pattern dimensionality. It has the property to remember the associativity expressed between the input patterns under supervised training. Thus, it has the faculty to retrieve from its memory (weight matrix) the pattern most closely associated with the input pattern normally corrupted. Self-Organizing Map (SOM). This is one of the most commonly used neural networks for data mining. The main qualities of this neural network are: i) it can organise its structure according to the description of the dataset and ii) it learns from data in an entirely unsupervised training environment. iii) the maximum size of the map expected is high likely to be smaller than the size of the original dataset. iv) a self-organising map has also inherited some properties of Vector Quantization, therefore it compresses the distributions of the m-dimensional data into the two-dimensional map. 1.4 The Research Questions Inspired by the critiques on the application of ANNs for DM, the research presented here aims to investigate whether artificial neural networks are suitable for a data-mining technique which concerns neither clustering nor classification; in contrast, it forms descriptions of the content of a database D by providing knowledge in a rule format: Rules which describe not only the associativity among the elements or attributes of the patterns in a database, but also the frequency in which they occur; rules which normally are produced as a consequence of per- 24 forming the counting of patterns registered in a database. Although the course of this research can be addressed towards the different possibilities laid out above, and posteriorly discussed in Chapter 3, the scope of this thesis has been confined to investigating if the itemset property of support (the frequency of patterns) can be reproduced (discovered or calculated) from the knowledge embedded in the weight matrix of our candidate neural networks. In other words, we will be taking the problem of determining if our chosen ANN candidates can be biologically-based solutions for ARM. In particular, we focus on determining if they are suitable (if they have the information resources and properties) for becoming artificial memories for ARM, which will be able to count, store and recall itemset support after they have been trained with some data modelling associations (binary patterns). Moreover, because data is dynamic, we also study the problem of the maintenance of the knowledge in the memories through the concept of the incremental learning with neural networks. The reasons for moving the focus of the research in this direction can be summarized as follows: The Description of Data by ANNs. Data is often described by the results given through methods which form general models or identify local patterns. ANNs have been used for the former, as an alternative of the traditional algorithms, in particular, when predictions of new states of the learnt variables are needed. For the latter, ANNs are able to describe data by forming clusters of the patterns, existing in the data, with the information presented in their learnt knowledge. Nevertheless, the idea of describing data from such knowledge in the form of association rules, as humans do from the knowledge captured by their brains, is still unexplored. 25 The Importance of Itemset Support for ARM. Even though the itemset property of support has been criticised for not being the best metric to measure the interestingness of itemsets and therefore rules, its use and definition has been a fundamental part in the evolution of the approaches for ARM. In particular, it has an important role in the support-confidence framework defined since the problem of ARM was introduced to the DM field. Therefore, in order to determine which patterns, associations or itemsets may be statistically important for the generation of association rules, a neural network should at least be able to know or calculate the occurrence frequency of the itemsets; i.e., their support property in the learnt environment. Reproducing Counting Abilities with ANNs. Counting is an important activity in biological systems since it is a method to summarise what occurs in an environment. It also takes part in a learning process (Gardner-Medwin and Barlow, 2001). Moreover, results of counting are an important part of our daily knowledge which often is utilised for decision making. Reproducing counting abilities on artificial models is a challenging and relevant task for the modelling and conception of what may be happening in real life (in the human brain). Moreover, performing counting with ANNs can also be seen as the development of either interpretations of, or knowledge extraction mechanisms from them (weight matrices) in order to reproduce values which describe the frequency of events (patterns, itemsets) occurring within the data. Building Artificial Memories for ARM. In order to develop the framework illustrated in Figure 1.4, it is important to start defining an ANN architecture which can learn, recall, and store knowledge about its environment (the frequency of patterns) autonomously. This would result in its hypotheses 26 and recalls being formed exclusively from its knowledge, which was previously captured directly from the environment, rather than depending on results from the counting of patterns given by other proposals. In addition, different networks could be incorporated into the system in order to make more complex hypotheses from the knowledge stored in the ANN-based memory for other purposes. In detail, the questions which we are interested in answering with this thesis are as follows: • ANNs have been applied to form data models for classification and clustering for instance, but can they be used for descriptive-data-mining techniques in which the aim is to represent the data in form of association rules? • The interpretation of the knowledge (weight matrix, mapping) generated by ANNs has been a permanent task of research in the ANN field. In particular, mechanisms have been developed to describe hypotheses formed by the nodes of a neural network into symbolic representations for classification problems. Nevertheless, would the knowledge learnt by an ANN be useful for describing associations among the elements of a database? • In the support-confidence framework of ARM, FIM (first stage of ARM) calculates a metric called support to produce the raw material known as the frequent itemsets for the generation of rules. Therefore, could it be possible that our chosen neural networks can have some knowledge describing the frequency of patterns distributed along their nodes? I.e., Could the results of counting be generated from our ANN candidates? • For the last decade, an implicit framework has been built in ARM, illustrated in Figure 1.3, which is replete with processes and strategies in order 27 to break down the complexity of the data-mining task. Therefore, could it be possible to have an itemset-support memory based on an ANN, as a substitute for the original database, to take part in such framework ? • If the implementation of such neural-network itemset-support memory was feasible, could it continue accumulating knowledge of the pattern frequency throughout time while the original data environment changes? In general terms, this thesis focuses on the process of counting patterns and the maintenance of the counters with ANNs in order to propose their inclusion as itemset-support memories in frameworks which involve tasks like FIM. It is important to note that the development of this topic, artificial neural networks for association rule mining, as summarized in Chapter 3, has been rarely undertaken. Additionally, to the best of our knowledge, the topic with which this thesis deals has not yet been studied by other researchers. Therefore, this work primarily focuses on investigating which biologically-based capabilities, in conjunction with the different learning paradigms (supervised, unsupervised and hybrid learning) and architectures of the neural networks, can initially be used for the calculation of the itemset support and consequently for its maintenance. 1.4.1 Aims and Objectives Having defined the problem area domain and focus of the thesis, and having selected two neural networks as candidates, this work aims to achieve the following: • To gain some insight into the use of neural networks for techniques involving neither induction nor deduction (classification, prediction), but transduction (association rule mining) (Kantardzic, 2002). 28 – Normally, ANNs have been used to create global models by learning the dependencies of some given data (induction) in order to use such models for predicting outputs for future values (deduction). Therefore, it will be investigated if our ANN models can reproduce the outputs which can normally be given by the application of a transduction approach, such as the process of association rule mining. • To investigate if the counting of patterns, which normally is realised in a high-dimensional space defined by the training data, can be reproduced with the information defined within the low-dimensional space formed by the weight matrix of a trained neural network. – The exploration of counting with ANNs has been proposed in (GardnerMedwin and Barlow, 2001). However, many assumptions on two theoretical models were made in order to make a proposal. In our case, an analysis of two well-known neural networks, a self-organising map and an auto-associative memory, will be made to determine if these two models perform pattern counting (collect knowledge about frequency) while they learn. • To develop mechanisms of knowledge extraction for the weight matrix of the ANN candidates in order to generate (calculate) itemset support. – Once it has been stated that, as a result of training, pattern frequency is embedded in the knowledge of a neural network, we will focus on giving the right interpretation to the weight matrix in order to recall the frequency (support) of patterns (itemsets), which can be composed by different elements (items) of the learnt environment. 29 • To propose how the frequency pattern knowledge in a neural network can be maintained while the environment is changing. – In particular, a proposal on this topic will be given by exploiting the incremental learning abilities of a self-organising map, since it is an ANN which learns without supervision. • To propose the development of a neural framework for tasks like ARM by proving that itemset support can be taught, stored and recalled from an artificial memory, based on either a self-organising map or an autoassociative memory. – To consider our neural candidates as support-itemset memories, it would be important to evaluate the accuracy of their results. Therefore, the comparison of the neural networks against the values given by the counting process performed by Apriori will be realised. Since ANNs have been successfully applied to problems involving classification, prediction, clustering. This work aims to complement the success of ANNs by serving as an evidence that the knowledge learnt by them on normal training conditions can be reused for data-mining tasks whose performance requires the knowledge or information about the frequency of the training patterns. 1.5 Organisation The remaining chapters of the thesis are organized as follows: • We first provide the formal definition of ARM in Chapter 2. In particular, we focus on describing the sub-task of FIM. Moreover, since the Apriori algorithm, which is the gold standard and historically most important for 30 ARM, will be used to compare our results, its definition is given in Appendix A. • The little literature related to our aims is summarised in Chapter 3. Details regarding the characteristics, which are needed in a neural network to be able to perform FIM, are given here. In this chapter, we justify the study on our neural network candidates. Moreover, conceptions are given about how the ARM problem can be understood from an ANN point of view. • Chapter 4 describes our own interpretation on how the counting process is realized by an auto-associative memory. In particular, a correlation matrix memory is studied, and two methods to estimate itemset support from its weight matrix are proposed. Information about the CMM training is defined in Appendix B. • Chapter 5 explores the idea of using a self-organising map for the counting of patterns. Unlike other approaches of SOM for ARM, our approach called PISM (Probabilistic Itemset-support eStimation Mechanism), which is based on Probability theory, only uses the information coded by the best-matching units of the map for the estimation (recall) of itemset support. Our novel perspective of the use of SOM for ARM alleviates the drawbacks presented in similar approaches, which limit the use of SOM to just clustering the input data before FIM performs. Information about the SOM training is defined in Appendix B. • Chapter 6 develops the idea that the incremental learning capability of neural networks can be exploited for the task of the maintenance of rules in ARM. In particular, a study with the SOM is carried out to update itemsetsupport on the map. Since a batch training has been considered unsuitable for the update of a 31 SOM in non-stationary environments, we have developed BincrementalSOM (Batch incremental training for SOM), which is a mechanism based on using the knowledge captured by the best-matching units, to update the information of a SOM regarding the itemset support occurring in these environments. • Finally, the conclusions drawn by this thesis, its limitations and future directions are all defined in Chapter 7. 32 Chapter 2 Association Rule Mining 2.1 Introduction In the field of DM (Data Mining), the techniques responsible for the analysis of data are often classified into predictive and descriptive. While the former aims to build models from data for producing future inferences about it, the latter performs the discovery of hidden patterns in data. Among the descriptive techniques, ARM (Association Rule Mining) excels because of its easy manner of describing important patterns existing in data. Roughly speaking, it is a technique which searchers for patterns describing relevant associations among attributes, elements, or items of a given data source or environment. In the beginnings of ARM, which was introduced in (Agrawal et al., 1993), this DM task was only proposed for the analysis of market-basket data, but today its application has been extended to a variety of other domains. Taking into account the straight manner in which ARM forms its knowledge from an environment, which does not consider the building of a model, ARM has been classified as a transductive learning process (Kantardzic, 2002). The knowledge or inferences generated by ARM are described by a group of rules 33 with the format of: X → Y, which symbolise associations of patterns occurring in data. As the total number of possible rules, generated from a given data, can be exponentially large, their generation is considered to be infeasible; therefore, ARM proposals have focused on generating only the relevant rules based on calculating metrics which define their interestingness. For instance, state-of-theart algorithms, like Apriori, developed independently in (Agrawal and Srikant, 1994; Mannila et al., 1994), are based on the definition of the support-confidence framework (Agrawal et al., 1993), which states that ARM must be tackled by dividing it into the tasks of: FIM (Frequent Itemset Mining) and RG (Rule Generation). This is, rules are derived from a knowledge known as frequent itemsets discovered by FIM. Overall, ARM aims for the discovery of interesting rules which satisfy minimal constraints respecting, for instance, the properties of itemset support and rule confidence which are respectively involved in FIM and RG. ARM has been addressed by a large variety of algorithms. A description of the most well-known algorithms can be found in (Goethals, 2003; Bodon, 2006; Aaron Ceglar, 2006). The strategies to break down the complexity of the problem, which the current algorithms are based on, have involved: • The usage of different data structures to represent the knowledge or findings during the mining process. • The exploration of different layouts to represent the input data. • The implementation of efficient strategies to traverse the search space. • The definition of optimization methods for the reduction of I/O operations. This has involved the definition of parallel and distributed approaches. 34 • The approximation of the rule properties to reduce the number of data scans. • The type of target knowledge from which rules will be derived. In the history of ARM, the Apriori algorithm has been very important to the extent that its philosophy has been utilised in one form or another in the development of the state-of-the-art algorithms. Although many approaches have claimed reducing the complexity of the problem, the real causes that produce a good performance in the algorithms are still uncertain (Bodon, 2006), as well as the different sources from which these rules can be generated. Hence, it is important to continue research in this area in order to find answers to the current enigmas. 2.2 The Scope of ARM As a technique which makes inferences from data, ARM can be seen as addressing the generation of rules R from an environment, represented by a dataset or database D, through an algorithm A in charge of searching for rules which must satisfy thresholds T set up for a mining exercise. Such a search is typically conducted by the usage of diverse strategies S. In the best theoretical scenario, ARM should be tackled by an algorithm whose performance, represented by a function FA , minimizes Equation 2.1. This is, R, satisfying T , should be produced by the execution of A, which minimizes Ω that defines the complexity and computational resources utilized in a mining exercise over D, based on the use of S. R = min FA (Ω, D, T ) S 35 (2.1) 2.2.1 Formal Definition Let any symbol, literal or element ij from a set I = {i1 , i2 , . . . , im } be called an item, and a grouping or formation of items X such that X ⊆ I be called an itemset. In particular, an itemset X with k = |X| is called a k-itemset. Let D be a set of n transactions, events or patterns grouped in a dataset or database representing an environment. Each transaction ti is defined by a unique identifier id together with an itemset Y, satisfying Y ⊆ I. A ti is said to hold or support an itemset X iff X ⊆ ti .Y. A basic association rule is an implication defined by A → B in which the itemsets A ∧ B ⊂ I, but A ∩ B = ∅. The support of an itemset X with respect to a D is defined by the fraction of transactions supporting it. This can be defined as follows: supp(X) = |{ti | ti ∈ D ∧ ti.Y ⊇ X}| |D| (2.2) In the case of a rule defined by A → B, its support is given by the support of A∪B; therefore, supp(A → B)=supp(A∪B=X). Since the support of X defines the occurrence frequency of X in D, it can also be understood as the probability of X , P(X). The strength of a rule in D is defined by its confidence as follows: conf(A → B) = supp(A ∪ B) supp(A) (2.3) Since it is impractical and not desirable to mine and generate the total space of itemsets and rules, which grows exponentially in function of |I|, the challenge 36 of ARM has been to discover only those which are interesting. The interestingness of an itemset and a rule is determined by evaluating the properties of support and confidence against the thresholds of minsupp and minconf respectively. For instance, an itemset X is frequent iff supp(X) ≥ minsupp. Hence, based on the above defined, which describes the support-confidence framework (Agrawal et al., 1993), the aim of ARM is then: 1) to discover a set F = {Xi ∈ D|supp(Xi ) ≥ minsupp} representing all frequent itemsets out of a space defined by 2m possible ones, and 2) to perform rule generation with the information generated in F . In a simplified manner, association rule mining is the result of performing frequent itemset mining and rule generation. 2.3 Frequent Itemset Mining Frequent itemset mining has become an essential factor in the generation of association rules from data because it is in charge of seeking out the right raw material, known as frequent itemsets, from which rules are derived. Since its complexity mainly defines the complexity of ARM, it has been the focus of attention of researchers who have looked at developing an algorithm which can satisfy Equation 2.1. This is, an algorithm whose performance does not deteriorate abruptly with the possible conditions presented by the data and thresholds. FIM is a combinatorial search problem which aims to form a set of itemsets F through discovering the frequent ones from an itemset search space SI defined by the m items of a data source D. The target set can then be stated as follows: 37 F = {Xi ∈ SI |supp(Xi ) > minsupp} (2.4) In which, the search space SI, which is often represented by a lattice structure as in Figure 2.1, is defined by, SI = {X ∈ C(m, k) |∀k : k > 0 ∧ k ≤ |I| ∧ m = |I|} (2.5) Whose size satisfies |SI| = m X m m =2 −1 k k>0 (2.6) Which refers to the total number of combinations, associations, or itemsets that define SI and which could occur in D. Figure 2.1: Example of an itemset-search-space lattice. In this case, the data space is formed by 4 items. Indexes represent the lexicographic order of the itemsets in the space. 2.3.1 The Calculation of Itemset Support In order to discover the set corresponding to the frequent itemsets from which association rules will be derived, the itemset property of support must be calcu38 lated. This property not only defines the frequency or probability of occurrence of an itemset within the mined data, but also represents a metric to determine the importance or interestingness of an itemset in the mining process. Therefore, to achieve such a calculation, three approaches have been proposed by the state-of-the-art algorithms as follows: 1. By occurrence counting. In this case, each itemset Xi under investigation has associated a counter which is incremented when it is discovered that a ti ⊇ Xi while the scanning of D is carried out. Since it is not feasible to count all the possible itemsets of the defined search space, algorithms in this category often make use of a procedure for CG (Candidate Generation) in order to focus the support calculation just on potential itemsets called candidates. A CG procedure is a function which forms candidates based on the frequent itemsets already discovered. At the beginning of ARM, these potential itemsets were compared to each transaction of D to determine their corresponding support; nevertheless, in order to reduce the runtime of ARM, it has been proposed the projection of the transactions into data structures representing the candidate itemsets. No error is produced in the itemset-support values generated from occurrence counting since they define the real number of appearances of an investigated itemset in the data. Occurrence counting is normally utilised by algorithms which perform a breath-first search and/or make use of the horizontal layout for the input data. Important approaches based on occurrence counting are the Apriori (Agrawal and Srikant, 1994; Mannila et al., 1994) and FPgrowth (Han et al., 2000b) algorithms. 2. By set intersections. In this type of approaches, the typical horizontal layout of the input data D is replaced by a vertical one. Therefore, contrary to traditional transactions, each item ij has associated a list, known as tidlist, 39 containing the identifiers of the transactions that support it. In this case, a candidate C, which represents X∪Y, is formed by an intersection such that C.tidlist = X.tidlist ∩ Y.tidlist. Therefore, the support of an itemset is determined by |C.idlist|. A good representative algorithm based on set intersections is Eclat (Zaki, 2000). Similar to occurrence-counting-based values, the support values generated through set intersections are errorless. 3. By estimation. As the itemset support calculated from the above approaches involves scanning the original data in one way or another, it has been stated that they calculate the real itemset support. Nevertheless, since the number of candidates, whose support needs to be determined, and the number of transactions or lists, representing the input data, can be large in real-life, algorithms based on support estimation aim to infer this itemset property without having to use the original data. This is, they propose using other information to make the estimations; for instance, information defined by the discovered frequent itemsets during the process. The advantage of this type of approach above the traditional one is that the number of candidates generated, as well as the number of data passes, are reduced. Since the itemset-support produced is an estimation and not a calculation, it is important to state that the total number of itemsets found, and consequently rules formed, can be affected as a result of the quality of the estimations. One example of itemset-support estimation is the PASCAL algorithm in (Bastide et al., 2000). 2.4 Taxonomy of the FIMers There is no a unique classification of the current algorithms for FIM. Therefore, in this section, we will form a taxonomy based on the different strategies pro- 40 posed for the benefit of the mining task. Input Data Definition Strategy. This type of strategy based its existence on either the amount of transactions needed to describe the tendency behavior of the items in the original data D satisfactorily, or the type of layout used to define D. For instance, it has been proposed that transforming the natural horizontal layout of D into a vertical one, which is formed by sets describing the behavior of each item, can provide advantages for the mining; cf., (Shenoy et al., 2000). Moreover, since D is often large in real life, it has also been investigated whether the tendency of the items, defined in m transactions of D, can be captured by just k transactions. Therefore, the use of sampling d to define D has been already studied and tested (Toivonen, 1996; Zaki et al., 1996). As FIM is performed over d rather than D, discrepancies in the number of frequent itemsets discovered along with their corresponding support can be found with respect to the traditional approaches. Itemset Support Calculation Strategy. These strategies refer to the manner in which the support of itemset candidates are calculated during a mining process. As described above in Section 2.3.1, there are three main approaches utilised by the current algorithms. Itemset Storage Structure Strategy. While an algorithm searches the space, the knowledge already discovered, which is formed by the frequent itemsets and their respective support and often used for the generation of new candidates, has to be stored or mapped into a data structure. Since the size of this data structure directly influences the performance of a FMI algorithm, a large part of research in the field has been done on finding the most suitable data structure for ARM. This has involved looking for a 41 data structure that allows not only itemsets to be represented compactly, but also fast access within the structure. In other words, this type of strategy aims to produce the most effective itemset storage structure which can be successfully exploited by an algorithm. The data structures proposed have involved: hash trees (Agrawal and Srikant, 1994), enumeration-set trees (Agarwal et al., 2001; Coenen et al., 2004a), adhoc trees (Han et al., 2000c; Coenen et al., 2004b), matrices (El-Haji and Zaiane, 2003), arrays (Grahne and Zhu, 2005; Liu et al., 2002), tries (Woon et al., 2004) and others (Goethals, 2003). Search Itemset Space Strategy. Since FIM seeks the most interesting itemsets within a search space that can be represented by a tree structure, approaches have investigated efficient manners to traverse such a space. For instance, algorithms based on Breadth-First Search (BFS) and Depth-First Search (DFS) have been proposed. The former is also known as a level-wise search since the algorithm moves level by level generating candidates and discovering frequent k-itemsets until no more candidates exist. In contrast, a DFS-based or class-wise algorithm works recursively by checking a family of k-itemsets before it moves towards a new one. In order to perform the best searching, complement techniques, involving heuristics and procedures, have also been proposed. The objective, in this case, is to lead the course of an algorithm to focus on certain areas of the space in which interesting itemsets are likely to be found. The most frequent used heuristic is defined by the downward closure property of itemset support, which establishes that: All subsets of a frequent itemset must be also frequent. The latter is a consequence of the anti-monotonic relationship between the support σ and the k number of items defining an 42 itemset within the space. This is, while k increases during the m levels of a search space, σ tends to decrease. In the case of the procedures, candidate generation has been one of the most used since it prunes fake candidate itemsets as soon as possible during a mining exercise. Hence, the number of elements visited and checked in the space is reduced to just those itemsets which are very certain to become frequent. By considering the properties of the frequent itemsets, for instance, their location in the search space, which often occurs to be at the top of the lattice, approaches, like in (Mannila and Toivonen, 1997) and (Goethals, 2002), have pointed out respectively the existence of borders between the frequent and unfrequent itemsets, and in the generation of candidates. Optimization Strategy. Approaches in this group have focused on finding the best way of administrating the computational resources for an ARM process. That is, to provide the best mining runtime by performing the best management of the hardware is the aim of this category. For instance, some researchers have worked on the parallelization of ARM (Zaki et al., 1997b; Han et al., 1997; Joshi et al., 1999; Jin and Agrawal, 2002; Veloso, 2003). Since the formation of association rules can demand large amounts of memory, proposals, for instance, in (Goethals, 2004), have considered simple techniques to extend the capability of the algorithms. Other approaches have identified that most of the time is spent when the algorithms output their results; therefore, the implementation of fast routines has been developed (Rácz et al., 2005). 43 Target Knowledge Strategy. Although the target knowledge of ARM is defined by a set of rules representing item associations, most of the approaches have focused on undertaking only the discovery of the set of the frequent itemsets F because the generation of rules can be produced from its elements. Based on the fact that set F can still be large in real life, other approaches have investigated to seek for memberless set representations of F from which rules can be generated indistinctly. Therefore, attention has been given to the concepts of frequent maximal (Gunopulos et al., 1997) and closed (Pasquier et al., 1998) itemsets. The MFI (Maximal Frequent Itemsets) form a set M whose elements satisfy that no frequent supersets exist. In other words, M is composed by the itemsets lying down in the frequent boundary space. Even though MFIbased algorithms have helped to reduce the complexity of ARM and to develop new pruning and search strategies such as, for instance, top-down and/or bottom-up searches, the definition of M has been criticised because it is not possible to generate the support of their itemset subsets that are also frequent by definition. Important algorithms with this conception are Maxminer (Roberto J. Bayardo, 1998), Mafia (Burdick et al., 2001), FPMax (Grahne and Zhu, 2003) and GenMax (Gouda and Zaki, 2001). In order to overcome the disadvantage of the set M , the discovery of CFI (Closed Frequent Itemsets) has been proposed because the support for all frequent itemsets can be generated from the closed ones. An itemset Y is closed if no proper superset of Y exists that has the same support. Some 44 CFI-based algorithms are A-Close (Pasquier et al., 1999), CLOSET (Pei et al., 2000) and CHARM (Zaki and Hsiao, 2002). 2.5 Conclusions Since this thesis is investigating the suitability of two neural networks for association rule mining, we have given the relevant background on ARM in this chapter. In particular, we have focused on providing information on the first phase known as frequent itemset mining. Some concepts about this mining task have been described above, because our main interest is to reproduce the itemset support values calculated by the current algorithms, which in one way or another have to perform pattern counting from data. Inspire of many algorithms, based on different strategies, having been proposed for ARM or FIM since 1993, biologically-inspired approaches have been rarely investigated. Therefore, in the next chapters we develop some ideas regarding the creation of ANNs-based approaches for the generation of this type of rules by exploiting the knowledge allocated in their weight matrix after training. 45 Chapter 3 Hypothetical Neural Network for Association Rule Mining Unlike other DM research topics, in which the literature is so vast, the lack of research is a property of the addressed topic of ANNs for ARM. Nevertheless, a summary of the little literature available for the topic will be presented in this chapter. We also justify the study on our neural network candidates for ARM. Additionally, we define the stages of the proposed ANN-based framework for ARM and a list of the properties that we consider as relevant in an ANN for ARM. As part of the conclusions of this chapter, we will point out the differences between our research and the existing work related. 3.1 Literature Review To the best of our knowledge, the use of neural networks for association rule mining was first explored by Sallans (Sallans, 1997). In this work, he made use of the unsupervised techniques of FA (Factor Analysis) and MGM (Mixture 46 of Gaussian Models) for his study. This work linked ANNs and ARM due to the fact that classification rules can be generated from ANNs for classification. The series of transactions were simulated and based on some underlying patterns (seed patterns), which might or might not be correlated, and the addition of some noise. The conclusions of this study have reported that while the FA-based model could never converge with the ARM data, and no clear reason was found for such behavior, the MGM network showed that it is able to learn the seed patterns used to generate the noisy transactions. The worst performance of MGM was shown when the patterns were highly correlated and the data was noisy, while the best performance resulted from uncorrelated patterns and clean data. This work does not state anything regarding the calculation of support, neither on how the MGM model should be understood for the generation of itemsets or rules. Gupta et al. (Gupta et al., 1999) undertook the problem of pruning or grouping final association rules for inspection and analysis. Even if the work focused on proposing a new distance metric to group association rules, a SOM took part in the proposed grouping methodology. The idea developed was initially to calculate the distance values among the rules through their metric. Then, MDS (Multi-Dimensional Scaling) was employed to form a vectorial representation of the distances which served as inputs to the SOM which clustered such vector space in order to visualize the rules. Due to the orientation of the work, it can be categorized as part of the techniques developed for the management and visualization of rules rather than their discovery, in which a neural network was used to cluster the rule space. In 2000, a frequent itemset algorithm based on the Hopfield neural network was presented by Gaber et al. (Gaber et al., 2000b). The work was inspired by the 47 facts that it has been demonstrated to be possible the extraction of classification rules from trained neural networks and that the Hopfield network has been used for combinatorial optimization problems. The authors concluded that a Hopfield network in an arrangement of n-by-p nodes with their proposed energy function should be enough to map the maximal frequent itemsets1 of a given input set of n transactions and p items. Even if the idea of ANNs for ARM makes sense, the absence of experiments and results makes it difficult to consider this work as a formal solution to the problem. Moreover, no indications have been given on how to interpret the neural network after its training. Even though the work of A.R. Gardner-Medwin and H.B. Barlow (GardnerMedwin and Barlow, 2001) is not related to the topic of ANNs for ARM, it is highly relevant for our purposes since it deals with the problem of counting in distributed neural representation. Initially, it was stated that: Learning about a causal or statistical association depends on comparing frequencies of joint occurrence with frequencies expected from separate occurrences, and to do this, events must somehow be counted... Hence, interested in how events can be counted by biological mechanisms, Gardner-Medwin and Barlow defined two theoretical neural models to explore the effects of counting. In this work, in the process of counting, each event E, representing some stimulation, produces an activity representation (a representation or activity pattern is formed by the state activity [0,1] of the neurons) in the network. 1 A maximal itemset is a variant of a frequent itemset. It can be understood as a super-frequent itemset whose derived itemsets are all frequent. Algorithms targeting this type of itemsets were commented in Chapter 2. 48 Two types of representations have been established to model the relation between events and their neuronal representations. First, a direct representation, which would model the ideal state for counting, is one that has at least one of the neurons active exclusively for the presented stimulation. Therefore, those neurons are defined to have a one-to-one relationship between their activity and the occurrence of the event. Second, a representation is known to be distributed when its active neurons participate also in the representation of other events in a counting epoch. Therefore, the relationship between activity nodes, supporting distributed representations, and frequency events can be stated to be many-tomany. They followed the idea that distributed representation exists in the biological models, therefore, interference that results from the overlap of the distributed representations was defined as a problem which can be dealt with in two ways: (1) direct representation must be generated or (2) the frequency of an event must be estimated from the frequency of use of its individual active elements. It was also noticed that although a distributed representation may increase the variance of the estimated counts and impairs the speed and reliability of learning, it is often regarded as a desirable feature of the brain since it brings the capacity to distinguish a large number of events with a finite and relative number of neurons. As a result of this work, two neural models were proposed: (a) the projection model is composed by an arrangement of Z binary neurons which will get activated or deactivated as a response to some events during a counting epoch. The frequency of the occurrence of a particular event Ec is estimated by the usage of 49 all the neurons that respond actively in its representation. In other words, the Z synaptic weights, which are proportional to the usages of individual neurons, are summed up into an accumulator neuron X to produce a frequency estimation. (b) the internal support model employs Z2 (full connectivity) synapses to count all possible parings of activity. Its excitatory synapses acquire strengths proportional to the number of times that pre- and postsynaptic neurons have been activated together during events experienced in a counting epoch. In other words, internal excitatory synapses within the network measure the frequency of cooccurrences of activity in pairs of neurons by a Hebbian mechanism. Therefore, the total internal activation, stabilizing the representation of an event Ec , is estimated by testing the effect of a diffuse inhibitory influence on the number of active neurons. In general, this theoretical work addressed the problem of counting on distributed representations by focusing on forming event frequency estimations taking into account the usage of the neurons and the representation overlaps produced in a counting epoch. A procedure for mining association rules from a database for on-line recommendation is developed by Changchien and Lu (Changchien and Lu, 2001). The implemented system depends on three different technologies: a start schema database, a SOM, and RST (Rough Set Theory). In particular, SOM is just used to cluster the transformed and normalised transaction records. Since the authors determined that SOM cannot explain the resulting clusters itself, they proposed association rules to explain their meaning. Hence, RST is employed to derive rules that explain the characteristics of each cluster and the attribute relationships among the different clusters. This work does neither generate frequent itemsets nor calculate itemset support; instead, it uses a confidence metric based 50 on RST to form rules. It does not discuss the accuracy of the final rules at all, but based on the explanation of the work, it can be classified as a soft-mining solution. Moreover, a strong dependency between the SOM and the database is presented and exploited for the performance of the entire system. Yang and Zhang (Shangming Yang, 2004) have also proposed that a SOM can be used for ARM because of its clustering properties, in particular due to the fact that its use makes clustering be a two-level approach (first, the data is clustered with a SOM and then the SOM is clustered), which gives the benefit that computational load decreases (Vesanto and Alhoniemi, 2000). This work neither provides any type of experimentation nor results. Roughly speaking, the authors only expose the idea that the split of the binary data, representing associations, into similar groups formed by SOM may benefit the task of FIM but no results are shown to support such an assumptions. To continue with the improvement of their approaches on PPI (Protein-Protein Interaction) prediction methods, in which an ANN (Eom and Zhang, 2004) and an adaptation of ARM (Eom et al., 2004) were used separately, Eom et al. decided to explore the idea of combining their past proposals in order to generate association rules directly from the ANN weight matrix to improve accuracy in the protein predictions (Eom and Zhang, 2005; Eom, 2006). Mainly, a supervised ART-1 network is used to classify vectors Xi , modeling PPI attributes, into k different classes. After this network converges, a weight-to-rule decoding procedure is initialized to transform its weight matrix into a form of association rules. To form rules between input attributes and their corresponding class k , the authors stated that a vector which maximizes the final value of the kth output node must be calculated. Taking into account the characteristic of the problem, 51 Eom et al. stated that rule formation from an ANN can be addressed as a nonlinear integer optimization problem. Therefore, a GA (Genetic Algorithm) was used to do the maximization of the objective function. The idea behind using a GA is that it would look for the best chromosome which can maximize the corresponding network output and give the best combination of input features as a result. Once these chromosomes have been discovered, they are decoded into a form of association rules called neural feature association rule for protein prediction. The few results presented show that the combination between an ANN and ARM provides better accuracy in the PPI prediction than its antecedents. 3.2 Hypothetical ARM Framework Based on ANNs Because it has been defined that relevant concepts for ARM such as counting and association take part in the process of learning performed by biological systems (Gardner-Medwin and Barlow, 2001) and knowing that such biological behaviors can be artificially imitated by using ANNs, we can state that an implicit ANN-based framework exists for tasks like association rule mining. This hypothetical framework, which is depicted in Figure 3.1, can be defined to be constituted for the following stages: 1. Environment Definition. Roughly speaking, this stage focuses on performing tasks for transformation, collection, manipulation of the data describing a collection of events of some environment. In this thesis, we use the original representation of the events in which the association among their elements is formatted binary. Nonetheless, finding a more advantageous representation of the input patterns without losing either the hierarchial property among them or the associativity property of their elements is a relevant clue for the improvement of their learning. In particular, since 52 ENVIRONMENT E DG LE NG I OWAR KN SH ARTIFICIAL MEMORY EXTRACTING TRAVERSING 53 NEW FEATURE SPACE OTHER NEURAL NETWORKS TASK LOGIC E TR XTR AV AC E T R SI ING NG Figure 3.1: Hypothetical Neural-based framework for ARM. In particular, this thesis focuses on developing an artificial memory for its purposes (colored area). EXTRACTING QUERYING INCREMENTAL LEARNING (counting) ARM LOGIC A->B (100%, 90%) AB->C (10%, 9%) AC->DF (100%, 12%) BF->GH (50%, 50%) ABCF->H (50%, 50%) CDE->FGH (70%, 90%) D->E (10%, 20%) KNOWLEDGE Candidate Generation Itemset Representation patterns can be spare and/or high dimensional by nature. 2. Learning. This is a task mainly governed by the course of some learning algorithm responsible for modifying the neural-network architecture (nodes or weight matrix) in charge of acting as the artificial memory which accumulates the knowledge presented in the environment throughout time. Therefore, the learning algorithm is principally determined by the type of neural network used to learn the data coming from the environment. In our particular case, the training algorithms of a self-organising map and an auto-associative memory will be evaluated to fulfill this framework task. 3. Artificial Memory. The purpose of this stage is important for the generation of rules because it forms the base knowledge of the framework from which the properties of the rules to be generated and therefore the rules will be derived. The main quality of any ANN to become an artificial memory for ARM is that its embedded knowledge can allow functions or methods F to be defined to describe properties of the learnt associations. Initially, for the generation of rules, itemset support must be calculated (recalled) from the nodes of the chosen neural network. In other words, it is necessary for the neural network to be able of acting as an artificial memory which has the ability of producing the counting of patterns in order to identify their corresponding number of occurrence in the environment. Another way of comprehending the knowledge formed at this stage is as a new mapping or feature space in which the original associations or patterns have been quantified and coded in a compact representation, limited by the space formed with the nodes of the network for future usages. Therefore, to generate rules from the resulting space, it is important to define 54 the correct decoding of it in order for the frequency of the taught patterns (support) to be determined as an estimation of what is normally discovered in the original space through the counting of the patterns. Moreover, this embedded knowledge about the environment can be used to supply the formation (training) of other neural architectures in order to create more complex rules describing the environment. In this thesis, we focus on developing this section. Therefore, our two ANN candidates will be studied to establish whether they can become our desired itemset-support memory by determining if they can reproduce frequency knowledge about the patterns in the training data, which is usually generated by performing the counting of them. 4. Knowledge Sharing. This is a task in charge of supplying or transferring knowledge from the main memory to procedures for the formations of new neural architectures (training of other neural networks), so that the proposed framework can re-utilise the collected knowledge for other tasks, such as, for instance, the prediction of behaviors of associations in the original environment. 5. Extracting-Traversing and Extracting-Querying. Since the arrangement of nodes, forming the artificial memory, can be seen as a data structure in which knowledge of an environment is distributed and accumulated because of training, it is necessary to build techniques which permit the exploration, recovery and exploitation of the information satisfactorily defined in the neural structure. To accomplish these tasks, especially in the representation of the extracted 55 knowledge, techniques produced in the ARM field could be reused; nevertheless, a re-definition of them needs to done to adjust their performance to the new space defined by the weight-matrix structure. 6. Task Logic. This stage groups techniques and methodologies to lead the extraction of knowledge in order to solve a mining process. It concentrates some of the strategies to be followed in the extraction of knowledge from the memory in order to speed up the process. This stage will keep a direct relationship with the strategies defined to carry on with the competition of the finding of association rules. Operation boundaries are not completly defined as the aims of their existence can be supported by results given by other neural structures or other stages existing in the framework. 7. ARM Logic. This is the stage which leads to the generation of rules from the knowledge embedded in the neural network. For instance, it will be responsible for applying the correct strategies for planning the best logistic of the mining exercise. In other words, this stage would administrate the total resources, data and processes, for the generation of association rules from a trained neural network with information about events occurred in some environment. 3.3 A Formal Definition of the Problem Let D be a finite n collection of binary events or discrete patterns X called itemsets which represent different formations of associations among the m elements (items) of a set in an environment. 56 Let Φ be a feature space or knowledge formed (weight matrix) by the m nodes of a neural model mAN N trained with D. Under the original conditions of D, a value f (itemset support), representing the frequency of occurrence of an association or pattern X in D, is calculated by performing the counting process P of that pattern in the environment so that f (X|D) = P(X, D). Since mAN N has already acquired knowledge about D, the aim is to reproduce an estimation of the frequency of occurrence of pattern X known as fb through the application of a decoding procedure Θ on the feature space Φ formed such that f (X|D) ≈ fb(X|Φ) = Θ(X, Φ). Therefore, the problem can be set up as seeking a definition of Θ for a particular Φ in order to produce frequency estimations for patterns in D. 3.4 Ideal ANN Characteristics for Building Memories for ARM Based on the problem defined above, we consider important to state some properties of ANNs which might be relevant for tackling our current problem as follows: • It is important to consider ANNs whose mapping or weight matrix formed is smaller than the size of the environment represented by patterns in D. Therefore, quantization properties can be relevant for tackling the summarisation or compacting of the original environment. 57 • It is required that the neural-based model deals well with large n-dimensional patterns, particularly with data that could be formatted as binary arrays. • The neural-network model will have to be able to learn new patterns without forgetting past knowledge. This characteristic in neural networks is known as the elasticity-plasticity dilemma. The exploit of this characteristic of a neural network would avoid employing tree-party processes for the maintenance of rules. • Initially, neural networks that conduct unsupervised learning attract our attention because no information about the patterns or associations in the environment is known a priori. Although, supervised trained ANNs can also be considered for our aims if they are able to learn the associations of the elements of the input pattern by supporting the definition of autoassociative inputs. 3.5 Reasons for Studying an AAM and a SOM for ARM The world of neural networks is vast in terms of the different types of paradigms of learning algorithms (e.g., error-correction learning, hebbian learning, competitive learning, etc.) and architectures (recurrent and feed-forward networks). It is the combination of learning algorithms and architectures which makes the generation of solutions for learning tasks such as pattern association, pattern recognition (classification), control, function approximation, filtering, and others possible. 58 Since this thesis concerns investigating the suitability of ANNs for association rule mining, we have chosen two different neural networks from the large list of supervised and unsupervised candidates. The selection of an auto-associative memory and a self-organising map to take part in our study is mainly because of their properties, which are summarised as follows: Auto-Associative Memory Among the learning tasks, PA (Pattern Association) has caught our interest as a viable alternative to finding answers to the research questions, stated in Section 1.4 in Chapter 1, because it is a task which can be performed by involving concepts of learning, memory and association. To achieve pattern association, according to Ham and Kostanic (Ham and Kostanic, 2001), any neural network (e.g., feedforward multilayer perceptron networks, counterpropagation networks, radial basis function networks and associative memory networks) from the group of the mapping networks, whose general structure is shown in Figure 3.2, can be employed for this purpose. Figure 3.2: General structure of a mapping neural network. This appears in (Ham and Kostanic, 2001). As a mapping network, in which the input patterns are coded and projected into the synapse weights during training, an associative memory aims to imitate the memory capabilities performed by the human brain, which has the ability to retrieve and store information via the management of the association con59 cept (Kohonen, 1978). In other words, this type of artificial neural network is taught to associate (O’Keefe, 1995). It learns the relationships, established by the pairs of input patterns, through storing them in a content addressable manner. This neural network learns knowledge from its environment by exploiting the explicit associations defined by each of the input-pattern pairs {X, Y } utilized for its training. Moreover, the concept of association is presented and used when information is retrieved from it. In other words, this neural network is able to give a response to the environment when a stimulus is presented to its inputs by using the concept of association between the stimulus and its weight matrix. One of the main abilities of this neural network is to remember information given in its training. This is possible because it forms a mapping (weight matrix) in which the target patterns (memorized patterns) and the associations are organised and stored for future recalls, that can involve queries with corrupted or noisy stimuli. Like its biological counterpart (the human brain), this ANN also handles the concept of memory in two stages: storage and recall of information. While the former phase refers to the training of the network, the latter refers to the extraction of information from the weight matrix in response to a stimulus. These two phases resemble the operations performed by the human in our example, stated in Chapter 1, in which he scans the data defining some shopping baskets (shopping transactions) to memorise the most of the information in order to answer queries about the contents of the baskets, for instance, queries regarding the associations among the items bought. To recall information from its weight matrix, a stimulus is always required. Un- 60 like conventional memories, they have the ability to learn and generalise. Moreover, no exact location of the required information is needed for its recovery; instead, the recall is formed with the distributed information in its nodes. Although a task of pattern association can be confused with a task of PR (Patter Recognition), which can be understood as the process whereby a received pattern is assigned to one of the prescribed classes (Haykin, 1999), a difference in the definition of the targets yi between these two tasks can be pointed out. Whereas an ANN for a pure PR task normally uses yi as a value to define the class ci (ci = yi ) to which an input pattern xi belongs, an ANN for PA uses yi as a vector, known as the memorised pattern, to represent a pattern to which an input pattern ,called the key pattern, will be associated. For our problem, which involves the estimation of itemset support from a weight matrix, we believe that a neural network for PA is practically more appropriate than for PR, mainly because, in the definition of our problem, in particular, in the part concerning the definition of the input data, which can be summarised as a dataset D composed of itemsets (binary patterns or binary transactions), there is no other information apart from the itemsets (transactions) themselves, which can serve to fulfill the targets needed in a pure PR task. Therefore, it can be stated that the target definition of the different learning tasks has been an important factor in leading this research towards the study of neural networks which let data speak for themselves, especially in cases, such as ours, in which there are no labels associated to the input data before they are learnt. Since it has been stated that the target yi (memorized pattern) will be defined by the corresponding key pattern xi of each input pair, our current research could be confined to the study of the suitability of an AM (Auto-Associative Memory) 61 for ARM since this neural network holds the characteristic of yi = xi ∀xi in D in its inputs. We focus on studying an AM, because it also satisfies the properties of the theoretical internal support model depicted in Figure 3.3, which was defined by A.R. Gardner-Medwin and H.B. Barlow (Gardner-Medwin and Barlow, 2001) to study the limits of counting. Being the latter, a very important task needed for the generation of association rules from databases. Figure 3.3: Outline of the theoretical internal support model defined in (GardnerMedwin and Barlow, 2001) to produce the counting of patterns with distributed representations in a group neurons. According to Austin (Austin, 1996) almost any ANN used for pattern classification can be used for building an associative memory. Nevertheless, factors to be considered for real-life problems are the speed of training and the type of inputs they handle (Ham and Kostanic, 2001). The Hopfield neural network (Hopfield, 1982) can be regarded as a good candidate since it is an ANN which bases its operativity of learning and recovering memories (patterns) in a manner similar to the human brain. For instance, a 62 Hopfield network is able to recover (remember) a thought pattern with just partial information given as a stimulus to the network. Hence, it can be stated that a characteristic of this ANN is robustness. A Hopfield network for ARM has already been proposed by Gaber et al. (Gaber et al., 2000a) from which I have concluded the following: This neural network was not used to calculate the support of the itemsets from a dataset, instead the identification of a variant of them, called maximal itemsets, was proposed. Due to the nature of the maximal itemsets, the main drawback of this work is that to calculate support for all itemsets derived from the maximal discovered itemsets an extra pass over the training data is needed. In other words, even if the network has learnt to detect the maximal itemsets from an environment D, it is not capable of providing information about the support of the all possible frequent itemsets (associations) happening in it. Taking Gaber’s work as an antecedent, and considering the options available for building an AM exposed, for instance, by Austin (Austin, 1996) and Ham and Kostanic (Ham and Kostanic, 2001), we have decided to study an AM based on a CMM (Correlation Matrix Memory), because a CMM is purely oriented to PA tasks which deals with binary and non-binary data and has fast training and recall stages. A CMM, whose training is defined in Appendix B, has been chosen because its operativity has been used to build more robust and complex ANN systems; cf., Konohen’s book (Kohonen, 1978), or ADAM (Advanced Distributed Associative Memory) and AURA (Advanced Uncertain Reasoning Architecture) system both 63 developed by Austin et al. (Austin and Stonham, 1987; Austin, 1995; Austin et al., 1995). Moreover, as it will be explained in next chapter, it has a natural ability for ARM due to it accumulates knowledge from its environment through a superposition of the memories representing the input pairs, which allows the estimation of itemset support be possible. Self-Organising Map According to Kohonen (Kohonen, 1996), various human sensory impressions are neurologically mapped into the brain so that spatial or other relations among stimuli correspond to spatial relations among the neurons organised into a twodimensional map. Hence, in order to build a SOM-based memory, our aim is to determine whether the counting of patterns occurred in a high dimensional space can be reproduced with the knowledge embedded in the two-dimensional mapping produced by a SOM. Our study is focused on a SOM in Chapter 5 because it is an astonishing model of the unsupervised neural networks which form a knowledge based on the resources expressed by the training data by letting data speak for itself. Furthermore, the interpretation of its embedded knowledge has become an important activity for data analysis. For instance, the inspection of correlations among the learnt variables has been investigated by considering the similarities existing in the planes formed by such variables (Vesanto and Ahola, 1999). The visualisation of the contribution of each variable in the formation of the map has also been addressed (Kaski et al., 1998a). To facilitate visual inspection of the highdimensional data, visual techniques for the cluster results given by a SOM, based on data projection methods, have been developed (Su and Chang, 2001). Also, the extraction of logical rules from trained self-organising networks, in particular 64 for classification problems, has been explored to create different understandings from them (Hammer et al., 2002; Malone et al., 2006). More importantly, a SOM has the abilities to undertake tasks, such as data clustering (Vesanto and Alhoniemi, 2000; Kiang, 2001) and vector quantization (Heskes, 2001) which make the formation of a compact representation of the training environment possible. Kohonen et al. (Kohonen et al., 2000) have tackled the problem of producing massive maps under environments in which data naturally occur in large amounts to perform their visual exploration. The usage of a SOM for combinatorial problems has already been explored since the individual SOM neurons tend to learn the properties of the underlaying distribution of the space in which they operate (Aras et al., 2003). The inclusion of SOM technology to the process of the generation of association rules has been investigated in (Changchien and Lu, 2001; Shangming Yang, 2004). Nevertheless, as above stated, these proposals have the disadvantage that, neither work on the interpretation of the trained map for ARM was proposed, nor the reproduction of the counting process was investigated. On the contrary, the proposals have based their implementation on limiting the SOM to clustering the input data space and making strong dependencies between the SOM clusters and the original data for the generation of association rules. An indirect consideration of SOMs for ARM can be interpreted from the work of Heskes (Heskes, 2001), where the relationship between SOMs, VQ and Mixture Modeling is explored. In this work, neither itemset support, nor rules 65 are generated. Nevertheless, in one experiment, Heskes uses basket-market data in order to build a map to model the relationships between the items defining the transactions of a dataset, following the assumption that items of similar groups have similar co-occurrence frequencies with other items in the basket. In this case, the training data is defined by a matrix with the relative frequencies (support) of the items, in order to calculate conditional probabilities to be used in the distance metric proposed for the formation of the map. Figure 3.4: Theoretical projection models defined in (Gardner-Medwin and Barlow, 2001) to produce the counting of patterns with distributed representations in a group neurons. Additionally, due to the way in which a SOM learns from its environment, we have categorised it as a practical representation of the theoretical projection model, depicted in Figure 3.4, defined by A.R. Gardner-Medwin and H.B. Barlow (Gardner-Medwin and Barlow, 2001), in which the knowledge about the frequency of the pattern occurrences in an environment, is distributed during learning in the neural components of the networks. Therefore, we have assumed that estimations about the occurrences of patterns, known as itemset support in ARM, can be produced from a SOM by interpreting the local knowledge generated by the nodes of the map. 66 3.6 Similarities and Differences with Surveyed Approaches Taking into account our aims and the current pieces of research for the topic of ANNs for ARM, we strongly believe it is convenient to define that this research differs from the approaches summarized above as follows: Our work will not focus on detecting either seed patterns (Sallans, 1997) or maximal itemsets (Gaber et al., 2000b) because even if an ANN was able to detect or learn them correctly, it would still need to know at least one of their properties, such as, for instance, support, to measure the relevance of such patterns or associations in the environment to generate the desired rules. Therefore, this thesis concentrates more on evaluating if our ANN candidates can learn something about the support of the input associations in order to recall those values when some stimulus is presented to the network. Additionally, we believe that through capturing support with ANNs, the calculation of other metrics, for instance, like rule confidence, which defines the conditional probability among the itemsets forming the body of the rules, can be done straightforward from the same embedded knowledge in the memory. Similar to other approaches, a SOM will be used. Nevertheless, we do not want to restrict its abilities to just clustering data for ARM as proposed in (Changchien and Lu, 2001; Shangming Yang, 2004); instead, it is believed that due to its unsupervised properties, the SOM is able to let data speak for themselves and therefore properties like the frequency of the training patterns, must exist in the resultant mapping and its decoding is the target. 67 Because we believe the dependency between the proposed neural network and its training data shown in (Gaber et al., 2000b; Changchien and Lu, 2001; Shangming Yang, 2004) is a negative property for the future neural framework for ARM, we are interested in breaking it down and alternatively decoding the knowledge distributed within the nodes of the chosen ANNs in order to discover or calculate itemset support from it. Regarding our main interest in finding a pattern counting ability in our candidates, it can be stated that our work will produce applied neural models rather than theoretical ones as in (Gardner-Medwin and Barlow, 2001). However, our selected neural networks, a self-organising map and an auto-associative memory, can theoretically be categorised into the proposed projection and internal support models respectively. In addition, our studies will not ignore the associative statistical factor in the input patterns as it was done in (Gardner-Medwin and Barlow, 2001). Therefore, our proposal will endeavor to estimate support values for any varied item combinations. Probably the work most similar to ours, in the sense of generating association rules from an ANN and employing incremental training of ANNs for maintaining rules throughout time, is the one of (Eom and Zhang, 2005; Eom, 2006); nevertheless, we focus on studying ANNs which can handle unsupervised tasks rather than supervised ones because we do believe that employing a supervised ANN would limit the operativity of the wanted framework to only data which could be classified a priori. The latter is not a feature of the problems often tackled by ARM. It is important to state that our work is about making the baseline for the 68 representation of a system in the format of association rules through the knowledge learnt by a neural network about such a system, and not to do with getting the optimal set of rules for that system, which we have considered to be a further task after having discovered that association rules can be produced from a trained neural network. 3.7 Conclusions In this chapter, the literature involved in our prime objectives has been summarized. The proposed ANN-based framework for ARM has been explained in more detail. We stated some of the characteristics that we think are important for an ANN to have if it is to be used for developing a neural-based framework for ARM or similar task involving the counting of patterns. Not less important, the reasons for studying an auto-associative memory and a self-organising map for association rule mining have been stated here. Additionally, we have specified the differences and affinities with this research and current literature. 69 Chapter 4 An Auto-Associative Memory for ARM To provide answers to the research questions stated in the introduction, in particular, to determine if a trained ANN can have the resources (information) in its weight matrix to answer queries regarding the support of the itemsets drawn from the training dataset, we begin by studying the suitability of an associative memory for association rule mining, because this particular ANN bases its operativity on the concept of association. This neural network exploits the associativity property among the components of the inputs not only for learning its environment, but also for emulating human-memory operations in the recall of information from the knowledge embedded in its weight matrix. In particular, we focus on studying an auto-associative memory based on a Correlation Matrix Memory (CMM). After justifying the research on this type of neural network in Chapter 3, it will be first studied and analysed here if changes need to be applied to the training rule of this neural network in order to learn not only the associations defined 70 by the input patterns, but also information about the appearance frequency of the patterns that is needed for itemset-support calculations. Secondly, an interpretation of the resulting mapping (the weight matrix formed by supervised training) is proposed to perform itemset-support recalls when stimuli (queries about itemsets) are presented to the memory. To evaluate the accuracy of the recalls made by the associative memory through our proposals, we compare its results with the results obtained by the Apriori algorithm, which is an important algorithm for the calculation of support and consequently for the general process of association rule mining. Conclusions about the response of the associative memory for the calculation of itemset support will be given at the end of this chapter. 4.1 Correlation Matrix Memory for ARM Since the operativity of a correlation matrix memory is known to be in two stages: learning and recalling; we will first point out the natural ability of the CMM to learn information about the number of appearances of the components describing the input patterns. Secondly, we will propose how the mapping, resulted from a correlation matrix memory, should be understood in order to retrieve information regarding the support of any itemset, defined in the training patterns, when this memory is queried to recall such information. 4.1.1 The Learning of Itemset Support by a CMM To begin, it is necessary to re-state that a CMM is a one-layer memory structure which basically is defined by a square array whose dimension is the m number of elements in the patterns (this statement on the memory size is only valid in the 71 case of an auto-associative memory). Therefore, the corresponding memory matrix, resulting from supervised training, is represented an M ∈ <m×m or Bm×m for non-binary and binary models respectively. In order to be trained in any D environment, both types of CMM, which have also been identified by O’Keefe (O’Keefe, 1995) as weightless or weighted memories respectively, require pairs of patterns representing associations with a form X → Y in which an input or key pattern is associated with an output or memorised pattern. In our case, the available data is signified by a group of n unipolar vectors (patterns or itemsets) ∈ {0, 1}, in which a value of one assigned to some k elements or items is used to define their existence within the pattern, otherwise zero is assigned. This is, each input is an m-vector defining a particular association among items from the set I = {i1 , i2 , ..., im }. The corresponding associations to be presented to the memory may look as {(X1 , Y1 ), (X2 , Y2 ), ..., (Xn , Yn )} (4.1) In which, both patterns, the key and memorised, will obtain their values from the n transactions in D. In particular, each pattern of each pair (Xk {0, 1}mx1 , Yk {0, 1}mx1 ) will take the same value defined by a transaction tk . Hence, a pair k satisfies: Xk → Yk = (Xk , Yk ) Xk = {x1k , x2k , ..., xmk } ; Yk = {y1k , y2k , ..., ymk } (4.2) Xk = Yk = tk .x To train a weighted CMM, the equation governing this procedure, has been defined in literature (Haykin, 1999; Ham and Kostanic, 2001) as follows: M= m X k=1 72 Yk XkT (4.3) It is the term Yk XkT or outer product which has been determined to be an estimation of the weight matrix W(k) of the neural network functioning as a linear associator (Ham and Kostanic, 2001). This matrix W(k), which associates or maps Yk onto Xk , forms a mapping representing solely the association described by the k th input pair in turn. Therefore, a resulting matrix M must be understood as a grouping or encoding (sum) of the m weighted matrices W. This summing or correlation among matrices can be illustrated as in Figure 4.1. Figure 4.1: Illustration of the accumulation of knowledge by a CMM. Individual weight values of the network, whose update resembles a generalisation of the Hebbian rule learning, can be expressed by wij = m X yik xjk (4.4) k=1 As a consequence of using unipolar elements as inputs, the product yi xj of some k th pattern will be in one of the two following states: yik xjk = 1 existance of association ij in k 0 (4.5) otherwise In particular, it can be noticed that a matrix k, representing a thought association between patterns X and Y, will show existence values in its components which satisfy: 73 wij = 1 ∀ {i, j}|{i, j} ∈ {P ({j|ij = 1 ∈ X}, 2)} (4.6) In which, the pais of indexes {i, j} derived from P, which represents the set of n2 permutations with repetitions among the indexes of items that have an existence in the input pattern X, define the matrix elements that need to be updated to learn the corresponding auto-association of the pattern X. Being conscious of the sum operation performed by the CMM to learn the incoming data, it can then be deduced from Equation 4.3 that, for the benefit of this thesis, the value at each component wij not only means the existence of an association between elements i and j, but also the number of times that such association has occurred in the environment. Therefore, we can asseverate that a weighted CMM naturally builds a frequency matrix when its training involves bipolar inputs. Having discovered the latter characteristic in this ANN has been relevant because its values associated to its nodes can be used, as will be explained further later, to recall itemset support. In the case of the weightless CMM whose training is based on : wij = m _ yik xjk (4.7) k=1 in which the accumulation of knowledge is produced by performing a superposition, through a bitwise-OR operator, among the different W matrices resulting from the different associations described by the training pairs (Austin and Stonham, 1987). This is, this type of CMM has the characteristic of unifying the Ws rather than summing them up. Therefore, the final weight matrix W fires its elements wij in those cases in which the individual matrices Ws have active 74 Figure 4.2: Illustration of the accumulation of knowledge by a weightless CMM which has been modified to collect frequency information. The dark matrix illustrates the new matrix called the frequency matrix Mf which contains the corresponding pattern frequencies. intersections in order to record the associations. This W is defined by m _ W = Xk YkT (4.8) k=1 Unfortunately for the weightless CMM, it does not have the natural ability, as its weighted counterpart, to collect knowledge regarding frequencies of the pattern components. Therefore, to reproduce such an ability, we propose to build a matrix as in Figure 4.2 for the gathering of such information; nevertheless, a proper justification for the construction and maintenance of this data structure needs to be found. In particular, when it has been concluded that the weighted CMM performs this process naturally. So far it has been found that, due to the auto-associative property of a weighted CMM, this ANN is able to learn information about the occurrences of the components of its inputs naturally. This is, no change was made to its traditional training to gather frequency knowledge about the input patterns. Hence, in the next section, we will explain how a trained CMM can be understood in order to 75 estimate itemset support from it. 4.1.2 Recalling Itesemt Support from The Weight Matrix of a CMM One important characteristic of the frequency matrix formed by a CMM is that it is symmetric. This leads our desired itemset-support recall mechanism to focus on the knowledge of only n(n + 1)/2 nodes rather than the n2 elements of the complete matrix. This is, more than half of the nodes, defined by (n2 − (n(n + 1)/2)), can be disposed of for our aims. The new resource matrix, from which itemset-support values will be estimated, is represented by 0 w11 w21 w22 W = . .. .. .. . . wm1 wm1 · · · wmm (4.9) To use the embedded information in this triangular W matrix for our purposes, the right interpretation needs to be drawn in order to produce accurate support values from it. Therefore, to achieve itemset-support recalls from W when a stimulus is presented, we propose the following mechanism of interpretation: Calculating The Support for 1-itemsets In this case, the aim is to generate a recall of support for any of the individual items ij ∈ I. This can be understood as the recall of a value among the elements of the matrix whose indexes satisfy that i = j. Therefore, our recall action is focused on the elements of the main diagonal of the matrix {w00 , w11 , ..., wmm }. Since it has already been stated that the number of occurrences of such elements 76 are defined in the corresponding elements wii , giving a recall for the support of these items only involves carrying out the calculation of P(wii ) which defines the probability of the ith item in the n patterns defining the training dataset. Therefore, the corresponding P(wii ) is defined as follows: supp(i) = P (wii ) = f req(wii ) n (4.10) Calculating The Support for 2-itemsets To calculate support to the group of 2-itemsets, which contain C(m,2) combinations derived from m items, we need to apply the Equation 4.10 to each of the elements located off of the main diagonal. In particular, to recall support of a rule ij → ik defined by supp(ij → ik ) or simply supp(ij ik ), it is only necessary to use the corresponding value of wij from the matrix. Therefore, the recall of support for any itemset, defined by X = {ij} from this particular group, can be defined as follows: supp(X) = supp(ij) = P (wij ) = f req(wij ) n f or all i 6= j (4.11) Estimating Support for The k-itemsets (2 > k ≤ m) The calculation of support for the groups of the k-itemsets when k >2 is not as straightforward as the two previous cases. This is, in this case, we need to define a mechanism which can combine the knowledge in the CMM to estimate itemset support. In order to achieve this new aim, two proposals, based on Definition 1, will be discussed below. Definition 1 (Independent Random Variables) Let k random variables (X1 ,X2 ,...,Xk ) 77 represent k events (i1 ,i2 ,...,ik ). As such events do not have influence in any form among them, it can be stated they are independent from one another; therefore, the probability holding that they occur together can be specified by calculating its joint probability as follows: Pr(X1 = i1 , . . . , Xk = ik ) = k Y Pr(Xi = ii ) (4.12) i=1 In our first proposal, which will be identified as the method A, the support value will be estimated by employing the probabilities given by elements of the diagonal matrix. In other words, it will be assumed that the probability of a k-itemset X is held by the joint probability of the items associated within it. Therefore, a support value can be represented by supp(X) ˆ = P (X) = k Y P (wii ) (4.13) i∈X For the second proposal or method B, instead of using the k individual probabilities, the following is proposed: By using the associative property of multiplication over the elements of Equation 4.13, paired arrangements of products among the k elements can be established as follows: P (X) = (P (w11 ) ∗ P (w22 )) ∗ · · · ∗ (P (wjj ) ∗ P (wkk )) (4.14) In which, each paired product of probabilities can be stated to hold the following property, P (A ∩ B) = P (AB) = P (A)P (B) 78 (4.15) Which establishes that the probability of the intersection between two events is defined by the product of their single probabilities. Hence, P(X) in Equation 4.14 can be re-defined as P (X) = (P (w12 )) ∗ · · · ∗ (P (wjk )) (4.16) Therefore, in order to give a recall of the support of a k-itemset where k >2, the calculation needed is given by supp(X) ˆ = P (X) = k/2 Q P (wij ) when k is even pairs{i,j}∈X where i>j (k−1)/2 Q P (wij ) ∗ P (wkk ) pairs{i,j}∈X when k is odd where i>j (4.17) 4.1.3 Complexity Analysis: CMM vs. Apriori Since the use of a CMM for ARM is totally novel, we have considered it relevant to provide a theoretical basis for the complexity analysis between the Apriori algorithm and a CMM for ARM. It is important to state that although different Apriori implementations exist, the analysis conducted here is based on a breadth first implementation since it is the gold standard from which many other implementations have been derived. Let m denote the total number of items, from which an n number of transactions have been derived to define a dataset D. Therefore, the time and space complexities can be expressed as follows: 79 Space Complexity Since all the itemsets discovered so far have to be saved by Apriori while the mining is performed, the space required to represent all of them is O(2m ), assuming that each itemset is represented independently in each node of the chosen data structure. In the case of a CMM, this complexity is defined by its weight matrix, in which the input patterns are compressed, and is equal to O(m2 ). Time Complexity In order to determine if a k-itemset (an itemset with k items) is frequent, the counting of its occurrences in D is done by Apriori. Therefore, the complexity of such a calculation is defined by O(nk), whilst an estimation, realised by our methods A and B, produces a complexity of O(k) and O(0.5k) respectively. There is another complexity involved in the case of the CMM, which is produced by the process of learning D and is defined by O(nm2 ). 4.2 Experiments It has been stated above how the resources, needed for the calculation for itemset support, are being captured (learnt) by a CMM during training. Additionally, it has also been proposed how the auto-associative memory based on CMM should be interpreted for retrieving itemset support from its weight matrix. In order to determine the accuracy of the proposed mechanisms for itemsetsupport recalls from a trained CMM, some experiments with real-life datasets, defined in Table 4.1, will be performed in this section. 80 Dataset Chess Connect Mushroom Number of Transactions (n) 3196 67557 8124 Number of Items (m) 75 129 (avg. 43 per transaction) 119 (avg. 23 per transaction) Table 4.1: List of real-life binary training datasets used in the testing of an autoassociative memory for ARM. They are part of the datasets normally used for testing FIM algorithms or FIM benchmarks (Jr. et al., 2004; Goethals and Zaki, 2003). The comparison of the CMM recalls, defining itemset-support estimations, is realised against the zero-error support values given by the Apriori algorithm. In particular, we use the implementation of Borgelt (Borgelt, 2003) of the Apriori algorithm which scans the dataset D in turn and builds a data structure to do the counting of the support of itemsets. It is very important to note that the results of the experiments presented here do not involve calculations for the groups of 1- and 2-itemsets. The latter was concluded because the corresponding support for such itemset groups is a straightforward procedure, as explained above, and which for the benefit of this thesis produces zero-error-support values as Apriori algorithm does. These errorless values have been possible because of the CMM’s natural ability to learn and store this itemset property (support) of the input patterns while it learns the corresponding associations. Therefore, the experimentation carried out here focuses on testing the proposed procedures for the estimation of the support for k-itemsets when k >2. In these experiments, while the method A, defined by Equation 4.13, involves exploiting the independence property of the items for the calculation of the itemset support, the method B, defined by Equation 4.17, uses the information of the support of the 2-itemsets involved in the k-itemset to make a recall. In particular, our experiments consist of querying the associative memory to recall the support for 81 some groups of k-itemsets for testing its accuracy. To determine the accuracy of the generalization given by this memory, an error for each of the different itemset groups, involved in the queries, will be measured. The calculated error will be represented by the well-known RMS (Root-Mean-Square) error defined as follows: E RM S v u n u1 X =t {yi (xi ; W ) − ti }2 n i=1 (4.18) In which, xi represents a k-itemset of a group of n k-itemsets, whose support values yi and ti (fluctuating in the range of 0 -it never occurs- and 100 -it always occurs-) have been calculated respectively by a recall made from a weight matrix W of a trained CMM and a traditional FIM process performed by the Apriori algorithm for comparison. Since it is our aim to evaluate the accuracy of the results given by an autoassociative memory, it is necessary to define which itemsets will be employed for testing. Although it would be ideal to check the response of the trained CMM for the total itemset search space, the fact is that it is unfeasible because of the search-space size. Therefore, some groups representing different support tendencies and sections of the itemset search space have been chosen to be created as follows: Group A The rare itemsets (low-frequent). This group is formed by elements which can have a support fluctuating between 0.001 and 1 percentage of the total transactions of the dataset. This group represents itemsets which rarely appear in the dataset, therefore they can be considered to be outlier 82 Group Type A B C Support constraint applied Itemsets with support in the range of 0.001% – 1% Itemsets with support in the range of 45% – 55% Itemsets with support in the range of 90% – 100% Table 4.2: Support constraint conditions used to form the group of itemsets on which the memories will be tested. associations in the environment. Group B The regular itemsets (semi-frequent). The support values of this group lay between 45 and 55 percent. These are itemsets which appear in a moderate way in the dataset. The associations described by this group can sometimes be interpreted as the obvious knowledge. Nevertheless, their changes of support can be useful to detect abrupt tendencies in the dataset throughout time. Group C The constant itemsets (high-frequent). This is the group of certainty. It contains itemsets which appear frequently in the dataset. An itemset in this group can have a support in the range of 90 to 100 percent. This group describes the common interests among the transactions defined by an environment. After defining the itemset groups on which our proposals will be tested, the elements of each group have been determined by running the Apriori algorithm over the original datasets with the corresponding support constraint conditions defined in Table 4.2. Once the groups have been formed, each member is used as stimulus to query the corresponding memories in order to compare the support values. Before the results of the experiments are shown and discussed, we have plotted each of the final trained W s, which have learnt the associations of the datasets 83 0.9 0.8 0.7 0.6 1 0.5 0.5 0 0 0 0.4 10 10 20 20 0.3 30 30 40 40 0.2 50 50 0.1 60 60 70 70 80 Figure 4.3: 75-by-75 frequency matrix formed by a CMM, from which itemset-support recalls about the Chess dataset will be made. of Table 4.1, in Figures 4.3, 4.4, and 4.5. Each of these matrices represents the available source of knowledge from which each memory will produce with a recall (estimation) of itemset support for the queried itemsets (patterns) through our proposals. The numbers describing the errors calculated for the chosen datasets corresponding to the recalls for 3- and 4-itemset groups are shown in Tables 4.3 and 4.4 respectively. It is important to comment that, due to the range of values available to represent itemset support (in this case from 0 to 100%), the worst error scenario in a recall will be determined by a value of 100. In order to discover the accuracy of recalls made by the CMM with our proposals in situations, which not only involve itemsets with the same size, we have 84 Figure 4.4: 129-by-129 frequency matrix formed by a CMM, from which itemsetsupport recalls about the Connect dataset will be made. Figure 4.5: 119-by-119 frequency matrix formed by a CMM, from which itemsetsupport recalls about the Mushroom dataset will be made. 85 Dataset Chess Connect Mushroom Itemset Group A B C A B C A B C Itemsets in Group (n) 12,340 1,871 167 113,013 1,540 826 21,961 74 - RMS Error Method A Method B 0.790250 0.766740 3.534600 2.769100 0.617510 0.475190 0.21098 0.20019 2.9726 2.3955 0.67045 0.38234 1.0606 1.0402 4.9631 4.1033 - Table 4.3: Error results obtained in the experiments for the support recall for the groups of 3-itemsets, constrained as defined in Table 4.2, made by a CMM through our proposals. Dataset Chess Connect Mushroom Itemset Group A B C A B C A B C Itemsets in Group (n) 269,719 12,359 203 2,776,831 14,743 2,451 243,608 72 - RMS Error Method A Method B 0.724600 0.534530 4.172900 3.185000 0.775860 0.505150 0.19219 0.13916 3.5693 2.8307 0.77525 0.38354 0.76988 0.7246 5.2328 3.5672 - Table 4.4: Error results obtained in the experiments for the support recall for the groups of 4-itemsets, constrained as defined in Table 4.2, made by a CMM through our proposals. set up some experiments with the three selected datasets and summarised the final results in Tables 4.5 and 4.6. In general, these results show that differences exist between the zero-error counted by Apriori and the approximations recalled from the CMM. The best approximations were shown by the proposal which uses the support of the 2itemsets for estimating the support of a k-itemset presented to the memory. 86 Group of Itemsets (k) 1 2 3 4 5 6 7 Number of Itemset (n) 167 203 128 39 4 RMS Error Method A Method B 0.617510 0.47519 0.775860 0.50515 0.912940 0.86483 1.089600 0.9105 1.394400 1.4274 Table 4.5: Error results obtained in the experiments for the support recall for groups of different k-itemsets of the Chess dataset made by a CMM through our proposals. The elements (itemsets) of the groups used to make the CMM recall were defined by the Apriori algorithm through constraining the itemsets with a minimum support between 90 and 100%. Group of Itemsets (k) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Number of Itemset (n) 4985 25,500 88,170 217,705 397,947 550,220 581,647 471,908 293,209 138,294 48,473 12,023 1,896 152 5 RMS Error Method A Method B 2.9141 2.285 3.593 2.6836 4.2165 3.6522 4.8524 3.995 5.5205 5.0014 6.2197 5.2905 6.9495 6.494 7.7125 6.6983 8.5192 8.1773 9.3857 8.3329 10.32 10.098 11.317 10.258 12.416 12.252 13.865 12.791 14.903 14.751 Table 4.6: Error results obtained in the experiments for the support recall for groups of different k-itemsets of the Chess dataset made by a CMM through our proposals. The elements (itemsets) of the groups used to make the CMM recall were defined by the Apriori algorithm by constraining the itemsets with a minimum support between 45 and 100%. 87 Group of Itemsets (k) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Number of Itemset (n) 2,757 13,218 44,721 111,159 208,126 297,836 327,797 277,133 178,389 85,839 29,903 7,135 1,052 76 RMS Error Method A Method B 1.6031 0.98297 1.921 1.0486 2.1543 1.4329 2.3382 1.3499 2.4889 1.6243 2.6207 1.4417 2.7472 1.6749 2.8768 1.4244 3.012 1.6514 3.1502 1.3619 3.2857 1.5753 3.4064 1.2622 3.4853 1.4725 3.4506 1.152 Table 4.7: Error results obtained in the experiments for the support recall for groups of different k-itemsets of the Connect dataset made by a CMM through our proposals. The elements (itemsets) of the groups used to make the CMM recall were defined by the Apriori algorithm by constraining the itemsets with a minimum support between 75 and 100%. Group of Itemsets (k) 1 2 3 4 5 6 Number of Itemset (n) 110 91 37 6 RMS Error Method A Method B 4.2501 3.5735 4.8628 3.2724 5.3181 5.153 5.7803 4.8079 Table 4.8: Error results obtained in the experiments for the support recall for groups of different k-itemsets of the Mushroom dataset made by a CMM through our proposals. The elements (itemsets) of the groups used to make the CMM recall were defined by the Apriori algorithm by constraining the itemsets with a minimum support between 45 and 100%. 88 4.3 Conclusions This chapter has been the first step in the quest for a neural network whose embedded knowledge can be used accurately for the generation of association rules, as if such rules were generated from the original (training) data D. In order to generate such rules, the property of itemset support is necessary, since it is the parameter (metric) to evaluate which itemsets (combinations) from a total space of 2n itemsets (n is the number of items of elements forming the itemsets, patterns, or transactions) are interesting for the process. Therefore, in this chapter, we have investigated if itemset support can be generated from the weight matrix formed by an associative memory. This neural network has been chosen from the large taxonomy of ANNs because its operativity is governed mainly by the use of the association concept among the input patterns. In particular, it has been analysed if this memory, after training, has the knowledge needed to assign a value defining itemset support to the stimulus (queried itemsets) presented to the memory. As a result of our analysis, it has been found that the weighted type of this memory has the natural ability to learn information about the frequency of itemsets defined by the input associations. After discovering that itemset frequencies are embedded in the weight matrix, we defined how it is possible to calculate the support for the group of the 1- and 2-itemsets directly from the weight matrix. These recalled support values from the memory are as errorless as the values given by the well-known Apriori algorithm, which scans and counts the dataset directly to calculate itemset support. In the case of the support for the groups of k-itemsets when 2 > k ≤ m, 89 two methods have been proposed to tackle the problem using the only information available in the matrix, the support information of the groups of 1- and 2-itemsets, to give a recall. Both methods, A and B, assume that items (their events of existence) are probabilistic independent among one another. While the method A uses the individual item probabilities (values defining support in the main diagonal of the matrix) as resources for the estimation, the method B forms pairwise items, whose support values are defined within the matrix, from the queried itemsets in order to give an answer. One of the advantages of using a CMM is that the space complexity generated by its weight matrix, from which itemset support will be recalled and defined by O(2m ), is much smaller than the one formed by its counterpart, the Apriori algorithm, which will be equal to O(m2 ) in the worst scenario. Errors in the recalls have resulted from employing both methods. Nevertheless, the method B has shown slightly better results. In summary, it can be stated that this memory is suitable for the perfect (errorless) calculation of support for the 1- and 2-itemset groups. Nevertheless, improvements for the case of the support for itemsets larger than two items remain an open problem. It is relevant to note that the usage of a CMM for ARM has been conceivable due to its training, which, as shown in figure 4.1, is the result of a superposition of the CMMs defining the input pairs of patterns. 90 Chapter 5 Itemset Support Generation From a Self-Organising Map After concluding in the previous chapter that although a supervised ANN, like an auto-associative memory, has the ability to remember itemset support for the groups of 1- and 2-itemsets perfectly, it struggles in recalling support for larger itemesets, due to overlaps produced by the distribution of the learnt associations in its weight matrix. In contrast to the previous chapter, here we will be looking at an unsupervised ANN rather than a supervised one for the task of building our desired memory. First of all, we will focus on studying a Self-Organising Map, because it has been successfully used for data mining tasks such as data visualisation, clustering and modelling. Additionally, its usage for ARM has already been proposed, but with some limitations. To begin with, a study on the suitability of a SOM for ARM is presented. With the conclusions drawn from the study and considering the biological SOM 91 properties, we will propose how to interpret a trained SOM with patterns, representing associations in an environment, in order to extract itemset support from its nodes. In other words, we propose how to reproduce the counting of patterns from the knowledge embedded in the map by our proposal named PISM (Probabilistic Itemset-support eStimation Mechanism). The accuracy of the itemset-support estimations, made by a SOM for either real-life or artificial datasets, is tested versus itemset-support results calculated by the Apriori algorithm. In order to improve the accuracy of the estimations, the concept of emergent feature maps is also studied. Conclusions on the suitability and considerations for the use of this neural network for ARM are given at the end. 5.1 Considering a SOM for ARM: Principles Motivated by the fact that pattern-occurrence counting values can be reproduced from the knowledge formed within the weight matrix of an auto-associative memory, based on a correlation matrix memory, our attention is now turned to studying whether a property of a network with an unsupervised training, such as the SOM (Self-Organising Map), can be utilised to count itemsets, and thus can become our itemset-support memory for ARM. The SOM is an outstanding neural network, due to its peculiar characteristics for multi-dimensional data clustering and visualisation. As stated in Chapter 3, its advantage is that it captures the knowledge from a dataset without supervision. To use a SOM for a data-mining task (Kaski et al., 1998b), it is necessary to set up first some initial parameters (for example, radius, neighborhood function, 92 and others.). In the SOM training, an NNS (Nearest-Neighbor Search) (Yianilos, 1993) is at the core of the process. This NNS provides a mechanism for the selection of the BMUs (Best-Matching Units) or winners in order to determine the corresponding group of nodes, which triggers the update of the map. Through this update process, the map m re-organises itself iteratively in such a manner that the state M can be reached, so that m → M when t → ∞. In this final state M , the map can be declared to be a model, which has already gained certain information from the input or training dataset in its weight matrix (reference models, codebook), and from which some important clustering and data-visualisation properties can be derived satisfactorily. However, due to the nature of ARM, the model formed will be utilised for a task which concerns neither clustering nor visualisation. Although the training of a SOM is typically performed in a sequential mode, in which the map is updated every time that a single input vector randomly chosen is presented, we have considered the usage of the batch mode, in which the map is updated once when the whole data is presented, because it has been used to propose, for instance, the creation of large maps (Song and Lee, 1998; Kohonen et al., 2000), the formation of maps for non vectorial data (Kohonen and Somervuo, 2002) or string data (Kohonen and Somervuo, 1998), and the speedup of the SOM training through the parallelisation of it (Lawrence et al., 1999a; Kohonen et al., 2000). Moreover, as will be shown further, batch training allows for an easy identification of the distributed knowledge in the neurons during a counting or training epoch. Therefore, a SOM can be updated in batch (Kohonen, 1996; Kohonen et al., 2000) by calculating the new values of its reference vectors associated to its neurons as follows: 93 P j hji (t)Sj (t) mi (t + 1) = P j nV j (t)hji (t) (5.1) This new state of the map (mi (t + 1)) can be understood as the spread of influences from each node mj generated by the corresponding data inputs represented by Sj (t) weighted by a neighborhood kernel function hji (t). The term Sj , defined below in Equation 5.2, describes a sum function of the nvi inputs contained at the Voronoi region Vj = {xi | kxi − mj k < kxi − mk k ∀k 6= j}, corresponding to node mj . Si (t) = nvi X xj (5.2) j=1 In the case of the kernel function, this is normally defined by a Gaussian function as follows: Ã kri − rj k2 hij (t) = exp − 2σ 2 (t) ! (5.3) In which ri and rj represent the positions of nodes mi and mj on the SOM grid, and σ defines the neighborhood radius. One characteristic of this kernel is that it presents its highest value at the origin of the influence (at the winners) and decreases monotonically along the remaining nodes of the map. Considering Equation 5.1, the update of the map can be interpreted as a process which places Gaussian functions close to the values Sj /nV j . Each of these values represents a data point which can be considered to be the mean µj of the nV j numbers of allocated patterns xi at node mj , or the centroid nj of the Voronoi set Vj defined by node mj as follows: 94 nj = 1 X xi nvj x ∈V i (5.4) j After such functions have been placed at the corresponding nodes (the BMUs), their influences need to be propagated to the rest of nodes in the map, in order to produce a new state of the SOM. The strength of the influences is different and is determined by the number of data points allocated at the node and its position in the map. It is worth noting that this interpretation of the update resembles a process of density estimation (Silverman, 1986; Devroye, 1987), in which gaussian functions are often placed at the input values in order to be summed up to determine an estimation fb of the real density distribution f of the input data. Dataset Bin4 Bin6 Bin8 Bin10 Bin12 Bin16 Transactions(itemsets) n 15 63 255 1023 4095 65535 Items m 4 6 8 10 12 16 Description All itemsets derived from 4 items All itemsets derived from 6 items All itemsets derived from 8 items All itemsets derived from 10 items All itemsets derived from 12 items All itemsets derived from 16 items Table 5.1: List of binary-artificial training datasets used in the experiments for analysing SOM properties for ARM. Each dataset contains all the possible n itemsets generated by m items. To identify which SOM characteristics can be used for ARM, experiments were conducted with the datasets described in Table 5.1. It must be noted these data are all artificial, because our interest implies not only to find out how the SOM behaves when it is fed with a data distribution formed with discrete (binary) patterns describing associations, but also to observe the SOM’s response when it is trained with datasets containing the whole itemset data space, which is defined by 2n -1 itemsets derived from n items. 95 For our experiments, the following points apply: i) the use of a batch training method, ii) the Gaussian function is used as the neighborhood function because of its probabilistic properties, and iii) the size of the radius is constant and set to one because in that manner the BMUs will always spread their maximum influence to the entire map. The maps in Figure 5.1 represent the results for this analytical exercise which can be interpreted as follows: Formation of Clusters. To make the clusters visible, the D-matrix1 of each map is plotted and coloured using principal components for colour assignment (Vesanto et al., 2000). The size of the clusters, as well as the strength of the influences, are also directly related to the number of patterns allocated to the BMUs; therefore, in the resulting maps, particularly those with more transactions, it is important to note that the clusters organized around the edges of the square grid are relatively homogenous in size. That is, the hypercube, whose vertices define the different binary patterns of a multi-space, is projected (compressed) into the two-dimensional space bounded by the neurons of the map, forming a circular formation of clusters along with a communal cluster in the middle. A good example, in which this cluster phenomena is visible, is the resulting map for 12 items in Figure 5.1. This cluster size repartition is due to the fact that the input data distribution is controlled by limiting the appearance of any item to 2n /2 times in the dataset. Although, this is something rare in real-life problems, it helps to understand the theoretical scenario when all itemsets exist in the training 1 It is a distance matrix among the most commonly used means for the visualization of the SOM. It investigates the differences of adjacent nodes. In a D-Matrix, the median of the computed distances among the nodes and their neighbor nodes is determined. 96 Figure 5.1: Maps resulting from training a SOM with artificial datasets describing associations. The red hexagons on the gray maps define the hits received from the input patterns during training. Cluster formations are presented with the coloured maps. 97 dataset. Another characteristic that also affects the number of clusters on the maps is their size. The size of the map is often determined heuristically, which normally results in having a codebook which can always be expected to be considerably smaller than the size of the original dataset. In other words, the original dataset with its hidden associations would be compressed and coded in a distributed manner within the nodes of the map in such a way that its interpretation remains unknown. Identification of Winners. One important characteristic of this ANN is that the identification of BMUs is simple. For example, in all figures, the size of the red hexagons represents the number of hits received during an epoch training. The importance of the set of BMUs (nodes with non-zero hits) is high, because they trigger the update of the map at each epoch, and also because it is to these nodes that the input Patterns are allocated. Thus, it can be assumed that any node mj which has been hit mj .] times has a high relevance for the calculation of itemset support iff mj .] = |Vj | ≥ σ (minimum support threshold). The latter initially leads to the conclusion that if a cluster Ci contains strong (that is very dense, very populated or very frequent) BMUs then it can also be considered strong. Using the language of ARM, it is to be expected that if a cluster Ci is found to be frequent on the map, then some of its members, called winners, may also be frequent. It is important to note that the latter is only partially correct because even though a winner is found to be frequent, that fact does not ensure that all the members (items) of the patterns in it would also be considered frequent. As we have initially classified a SOM to be a representative of the projection model defined in (Gardner-Medwin and Barlow, 2001), the number of hits to a node can be interpreted as a property which defines the usage of 98 that node in the process of counting input patterns during an epoch. Therefore, estimations regarding the occurrence frequency of the patterns can be calculated by using such a node property. Due to the way in which a SOM clusters data, it can expected that some BMUs might share similarities among the elements of their patterns even though they will never share the same patterns (ni .S ∩ nj .S = Ø , for all i 6= j). Therefore, it can be assumed that the true support value of an itemset will involve the collection of the corresponding value from the nodes which in turn share the itemset. This can be viewed as a soft clustering procedure in which a data point may have multiple memberships, but the weighted memberships may sum to one. Dependency Amongst Clusters. As a consequence of training, a set S of patterns would have been formed in each BMU. To be using SOMs for FIM, a SOM has to be able to provide the support for any itemset possibly formed by the input patterns. From this perspective, the dependency amongst the binary patterns is relevant. This dependency of the patterns at each BMU can be seen iff they are re-arranged into a hierarchical data structure in order to help the FIM algorithm with counting support from each pattern. This pattern dependency is important since it can be part of a method to calculate the true support of a k-itemset, determining for instance if the kitemset ⊆ m-itemset. To determine whether a pattern is either a parent of or a child of another pattern, a bitwise operator can be adopted as follows: Let A and B be two binary patterns and ∨ be a bitwise OR operator such that the operation A ∨ B can give the following dependencies: 99 • A and B are related directly only if (A ∨ B) gives either A or B as a result, so that A is considered the parent of B, (A ∨ B) = A, meaning that B ⊆ A, otherwise A will be considered the child of B, (A ∨ B) = B, meaning that A ⊆ B. • A partial or zero dependency is defined when (A ∨ B) gives neither A nor B as a result, meaning that a possible dependency exists among the clusters on the map. Following the idea described above, local data structures can be built at each BMU in order to visualise the way in which a SOM splits the datainput-space lattice into the nodes. With the definition of the pattern dependencies, we might build a more complex structure involving all the patterns found and organised by a SOM. Therefore, this idea would lead to the building of a data structure, for instance a tree or trie, whose nodes would contain the pattern definition and its corresponding frequency, which will have to be computed on the-fly. The latter resembles the operation performed by a traditional FIM algorithm so that there would not be a good reason to use a SOM for ARM. At this stage, it can be concluded that even if the separation of the patterns could reduce the computational complexity of the counting, and the identification of important sectors of the input-data space could be detected on the map this neural network is still dependant on the original dataset. In order to overcome the current disadvantage, we could look at two alternatives: The first would involve adopting either the construction of a hierarchical data structure with the input patterns, or their direct use for FIM as proposed in (Shangming Yang, 2004). Nevertheless, both of these proposals would be limiting the use of a SOM to just the formation of clusters, which as commented previously would not be a sufficient justification to claim that association rules can be generated directly from 100 knowledge, derived from this neural network. The second approach is more challenging and interesting, and involves taking this research a step further and investigating a method of decoding the knowledge of the map in order to form an interpretation to eliminate the dependency between the SOM and the training dataset for FIM. Such an interpretation would bring a positive benefit for the use of a SOM for ARM, because the SOM would fulfill the role of a large artificial memory which learns associations, defined in an environment, so that recalls about the support value of any itemset can be made satisfactorily any time that this memory is queried. Therefore, in the next section, we will develop a knowledge extraction mechanism called PISM (Probabilistic Itemset-support eStimation Mechanism) to decode or interpret a trained SOM in order to reproduce the process of the counting of patterns or itemsets from its neurons. 5.2 A Probabilistic Itemset-support Estimation Mechanism To achieve our aim, which corresponds with the development of a SOM-based memory for ARM, we first need to find out how and where the associations are being represented within the SOM. It has already been stated that the representations of the input associations are all distributed among the nodes in the map, but it is necessary to determine which group of nodes in the map will contribute to the itemset-support estimations. In other words, the selection of the nodes, which will serve as the source of knowledge for itemset-support estimations, is indispensable to be realised. 101 Therefore, after observing in our experiments that the set of BMUs is the group responsible for the changes occurring in the map during training, and by following the recommendation made in (Alhoniemi et al., 1999), which states that the information allocated in the BMUs is often an attractive source of information for the development of many applications, we will concentrate on defining an extraction method for the BMUs formed in a training epoch. In order to achieve the selection of such nodes from the trained map, two initial definitions are stated as follows: Definition 2 (Set of Winners) A set W, defining the final winners, is formed by a number of nodes mb from the final map M such that, W = {Mi | Mi .#S > 0 or Mi .S 6= φ} mb = |W | ≤ |M | Where S is a set whose elements are the patterns that hit the node i. Definition 3 (Winning Vector) A vector Z = {z1 , . . . , zn } will be called a final winning reference vector if its respective node forms part of the set W. In addition, zji will be understood as the ith component of Z that has acquired some information about the ith component of some input patterns (xi ) which due to their similarities have been allocated by the training process in the node associated to Mj . Once W = {w1 , w2 , · · · , wmb } has been identified in a converged map M by using Definition 2, the next stage requires the application of some concepts from the Probability Theory to the reference vector of each node in the set W in order to evaluate whether it is feasible to obtain our desired knowledge, the support of itemsets, from these vectors. It is relevant to note that it is not exclusively necessary to extract the set W from the converged map, but it can be assumed 102 that more accurate itemset-support estimations can be made if the latest state of a map is utilized. To define a value Pr that holds the probability of the fact that a node, for instance mj , has become a BMU mc for some input patterns after completing a training epoch, it can be first assumed that becoming a winner in the process is equally likely to happen among all of the nodes in the map; therefore, the probability of each element contained in W can be specified as follows: Definition 4 (Prior Probability: Becoming a Winner) In SOM training, where N defines the number of input patterns contained in D, the probability that a node mi has become mc during an epoch training is defined by the ratio between the number of times that mc has been hit by D and the N number of input occurrences in the training. µ Pr(mi → mc ) = mi .#S N ¶ (5.5) Where #S is understood as the number of data points which belong to the Voronoi set Vi delimited by the winner mi . The previous definition is the prior probability that a node has become a BMU. This Pr(i) appeared in (Alhoniemi et al., 1999) when Alhoniemi et al was giving a probability interpretation of the response of the nodes to a new data sample using Bayes theorem. Initially, the prior probability values of the nodes are all zero, and some will change after the conclusion of each training epoch. In this work, the values given by the converged map will be used. Before continuing to define our extraction mechanism, we consider it impor103 tant to explain how the reference vector Z of a BMU i holds the corresponding probabilities (frequencies) of each pattern component xi of the total number of patterns allocated to that node. Therefore, the explanation is as follows: As mentioned above, the input patterns, representing the associations to be learnt, are formatted binary. Nevertheless, once they are presented to the SOM, they are handled as m-dimensional real vectors due to: i) the use of an Euclidean distance metric for the BMU searching rather than the use of a Hamming distance which could be more appropriated for binary vectors, and ii) the vectors resulting from applying Equation 5.1 are formed by real, but not binary numbers. Therefore, the input patterns can be defined to be real vectors whose components are bistate since they hold zeros and ones and the tendency of having less zeros than ones or vice versa will depend upon the hidden associations defined by them. In other words, each variable or component zi , representing an item behaviour, can be defined to be a bimodal distribution whose peaks will be defined at the values of one and zero. This bistate property of each zi is important for the purpose of this work, as a normal distribution can be created from these values. The different concentrations or densities of these two values will form a density distribution whose mean µ, which is the highest point of the distribution, defines a value containing the percentage of success events (zi =1) for this variable or item. In ARM, this percentage of success of zi of a total number of events N is what the support of zi represents. In other words, the µ of each component measures the number of occurrences of that component. Some examples describing this idea are depicted in Figure 5.2, in which the curve of the density distribution of each bottom graph represents an estimation of the real distribution formed by 104 the n transactions used in each example. For this case, the plotted distributions have been formed by using the concept of a gaussian-kernel estimator (Silverman, 1986; Devroye, 1987). It has been mentioned above that the update of the SOM, particularly Equation 5.1, can be interpreted as an aggregate of influences coming from different BMUs on the map. Hence, If a BMU is about to be updated, it can be derived from 5.1 that the final value for its codeword is a value very close to the mean of all the patterns accumulated in it. The difference between the real mean and the value given by the SOM is due to receiving influences from others weighted by the distances among the nodes and the radius used for the update of the map during training. Then, It can be concluded that the values of the components defined in each BMU are estimations of the real means, and could therefore be used for the calculation of the support instead of the original set of patterns. Reviewing Equation 5.1, which is used for the conventional batch training, and analysing the training procedure, it can be concluded that the updating of all the components zi in the map (codebook) is realized independently. The final value possessed in the component zi is never affected by the updating process of the component zj at any point during training and vice versa. Meanwhile, it has been stated in (Hugh, 1997) that whenever there is a kind of physical independence between events or processes, it shall be assumed that they have mathematical independence. Therefore, assuming that each component zj of a final winning reference vector Z is independent from the others, and that each of these components {z1 , z2 , . . . , zn } defines a probability of appearance of their corresponding elements {x1 , x2 , . . . , xn } in the input patterns, we can proceed to define that: 105 Figure 5.2: This figure illustrates the importance of the mean in the calculation of the support of an item from a trained SOM. Different number of transactions (n) composing of zeros and ones have been used to form the bottom graphs. These graphs show that different concentrations (densities) of these bistate values captured in an item induce the tendency of the distribution of the curve to approach the densest value (e.g., in the left graph the number of failures (zi =0) is greater than the number of successes, therefore the highest point of the distribution tends to be placed at 0). 106 Definition 5 (Independence Amongst Components of a Winner) Let a SOMtraining process be defined as an experiment where the possible outcomes S (S is a sample space in probability terms) are defined by a countable infinite set of data points in Rn (n is the dimensionality of the input data) which are allocated in D. Let two random components zi and zk at the winning vector mj be represented by two discrete random variables, A and B respectively, which describe the probability of their corresponding events of occurrence. Assume that these variables A and B have no influence on each other. Thus, the probability that they both occur together can be described by the joint probability, the product of the probabilities of these two individual variables, i.e. Pr(A = zi , B = zk ) = Pr(A = zi ) ∗ Pr(B = zk ) If the case is to study the joint distribution of n random variables X1 ,. . .,Xn , which conveniently can be represented by a random vector X = (X1 , . . . , Xn ), describing values from z1 ,. . .,zn then the corresponding joint probability can be obtained by, Pr(X1 = z1 , . . . , Xn = zn ) = n Y Pr(Xi = zi ) (5.6) i=1 Definition 5 is possible, due to the fact that all components have been declared independent. It has also been said in (DeGroot, 1975) that n random variables X1 ,. . .,Xn have a discrete joint distribution if the random vector (X1 ,. . .,Xn ) can have only a finite number or an infinite sequence of different possible values (x1 ,. . .,xn ) in Rn . Then, the joint probability function of X1 ,. . .,Xn is defined to be a function f so that for any point (x1 ,. . .,xn ) ∈ Rn , 107 f (x1 , . . . , xn ) = Pr(X1 = z1 , . . . , Xn = zn ) So far, it has been specified that every node in the map has a real value attached defining its prior probability. This prior probability must be zero for all nodes which are not elements in the set W . Consequently, in Equation 5.6, a joint probability is defined describing the fact that some random variables happen together at each winner. Thus, the next step is to define the manner of calculating the total or final probability Pr(E|M ) that some event E, which represents k variables happening together, occurs in the structure defined by a resultingtrained map M . To define the value Pr(E|M ), the concept of probability partition found in (DeGroot, 1975) can be utilized as follows: Definition 6 (Partitioned Data Space) Let S denote a sample space (input data or pattern space) of some experiment (training) and consider k events A1 ,. . . ,Ak in such a way that they are disjoint (they do not share elements). Thus, it is said that these k events form a partition of S. If the k events A1 ,. . . ,Ak form a partition of S, and if B is any other event in S, then the events A1 B, A2 B,. . . ,Ak B will consequently form a partition of B as illustrated in Figure 5.2. Hence, it is possible to write B = (A1 B) ∪ (A2 B) ∪ · · · ∪ (Ak B) Moreover, since the k events (right side of the equation) are disjoint, Pr(B) = k X Pr(Aj B) j=1 Finally, it is known that if Pr(Aj ) >0 for j = 1, 2, . . . , k then Pr(Aj B) = Pr(Aj ) Pr( B| Aj ). Thus, it follows that 108 Pr(B) = k X Pr(Aj ) Pr( B| Aj ) j=1 Where P r( B| Aj ) defines the conditional probability of the event B occurring at the partition defined by the event Aj . Figure 5.3: The figure on the left depicts the intersections of an event B with events A1 ,. . . ,A5 of a partition over S. The figure on the right depicts the concept of Voronoi regions which can be formed on the SOM (the dots represent the codewords while the stars represent the data points assigned to each Voronoi region). For the purpose of this work, it can then be stated that the k events A1 ,. . . ,Ak are the result of the ”best-matching” process during training, and that they represent the group of nodes over which the input data space is split, forming k different events (partitions) which are disjoint (no pattern is shared among them). Similarly, the event B can be seen as any event that is likely to occur defined by some of the elements (items) of the vectors allocated at each node. In summary, the final probability of an event E can be defined as follows: Definition 7 (Total Probability - The Frequency Occurrence of a Pattern -) Having a final map M after training a SOM with D containing discrete patterns (binary patterns), it is possible to obtain the corresponding probability of the event E, describing the fact that k components or items, x1 ,. . .,xk of the input vectors appear together in the training environment D, through calculating the 109 sum of the partial probabilities regarding the event E (z1 ,. . .,zk ) in those neurons, defined by the set W , in which the knowledge of D has been distributed. This calculation is represented by: Pr(E|D) ≈ Pr(E |M ) = mb X Pr(Wi ) Pr(E |Wi ) (5.7) i=1 in which, P r(Wi ) represents the priori probability of each BMU neuron and P r(E|Wi ) defines the probability of the event E in that neuron. This final probability P r(E|M ), whose calculation is depicted in Figure 5.4, can also seen as the estimation the frequency occurrence of the event E in D. It is important to clarify that while Pr(E |M ) is the estimation of the real value Pr(E |D) which normally is calculated by doing the counting of the event or pattern E in the whole dataset D. An Equation summarising this definition is as follows: Pr(E |M ) = mb X Pr(Wi ) i=1 k Y Pr(zj ) (5.8) j=1(z∈Wi ) To conclude the description of this method, it is essential to use the correct terminology in order to describe this proposed method using the frequentitemset-mining language. Therefore, it must be first defined that a k-itemset is a k-multi event E occurring within the components of map M. Each possible kitemset has an associated support value (probability of appearance) in D that can be calculated from M by, Definition 8 (Estimation of Itemset Support from a SOM) Let M be a map resulting from training a SOM with N binary patterns grouped in an environment D representing transactions involving m items. An estimation of the support (supp) ˆ of an itemset X, defining a pattern or event E formed with k items, which is equal to an estimation of the frequency occurrence of such a pattern in the training 110 Figure 5.4: Representation of the Probabilistic Itemset-support Estimation Mechanism (PISM) proposed in this chapter. environment, can be calculated from the embedded and distributed knowledge in M by summing up the probabilities registered by E in the BMUs, defined by a set W , for such D, as follows: supp(X) ˆ = Pr(X |M ) = mb X Pr(Wi ) i=1 k Y Pr(zj ) (5.9) j=1(z∈Wi ) 5.3 Experiments and Results In order to verify the accuracy of the knowledge extraction method described above, which at this stage is only concerned with the estimation of the support metric for any possible itemset derived from the patterns occurring in a training 111 dataset or environment, some datasets described in Tables 5.1 and 5.2 have been used to test it. In all the experiments presented here, the SOM-training method is batch, the radius is set to one, and the neighborhood function is Gaussian unless otherwise stated. Dataset Chess Connect Mushroom Transactions(m) Items(n) 3196 67557 8124 75 129 (Avg. 43) 119 (Avg. 23) Table 5.2: List of real-life binary training datasets used in the testing of PISM for SOM. They have been used in FIM-algorithm benchmarks (Jr. et al., 2004; Goethals and Zaki, 2003). The experiments have been carried out in two stages, involving artificial and real-life datasets. Here, the focus lies on testing how accurate the results given by a SOM via our proposal are when this neural network is being queried to provide the support value of some group of itemsets. A list with all the queries involved in our experiments is presented in Table 5.3, which also defines the way in which each trained map will be queried. To be congruent with the previous chapter, the groups of itemsets for testing our extraction method on the trained SOMs have also been formed by mining the artificial and real-life datasets with some constraints, concerning the support σ and/or the size k of the tested itemsets, through the use of the Apriori implementation developed by Borgelt (Borgelt, 2003). As in the previous chapter, in order to corroborate how far or close the estimations made by our proposal are from the real values, the itemset support given by the Apriori algorithm, implemented by Borgelt, will likewise be used for comparison. 112 Query All 1Itemsets 2Itemsets 3Itemsets 4Itemsets 1to3Itemsets 45to100Itemsets 75to100Itemsets 90to100Itemsets Description Applied to Maps Trained With All itemsets All 1-itemsets All 2-itemsets All 3-itemsets All 4-itemsets All k-itemsets where 1 ≤ k ≥ 3 All k-itemsets where 45 ≤ σ ≥ 100 All k-itemsets where 75 ≤ σ ≥ 100 All k-itemsets where 90 ≤ σ ≥ 100 All artificial datasets All real-life datasets All real-life datasets All real-life datasets Chess dataset All real-life datasets Chess, mushroom datasets Connect dataset Chess dataset Table 5.3: List of queries used to form the groups of itemsets used for the testing of the itemset-support estimations from SOMs via PISM. In this case, k means the size of the itemsets and σ refers to the support used to form such itemset groups. Each experiment begins with the training of a SOM M with a dataset or environment D. Consequently, a group of itemsets L along with their real support property, representing the itemsets satisfying some constraints C, is generated by mining D with the Apriori algorithm. The itemsets existing in L are then used as stimuli to make M recall their support or frequency occurrence values via our method from the knowledge embedded in its group of BMUs. Once the real and estimated itemset-support values are collected, a generalization error is calculated. To evaluate the effectiveness of a SOM in recalling itemset support, the RMS error, defined by Equation 5.10, has been applied. E RM S v u N u1 X =t {y(xn ; m∗ ) − tn }2 N i=1 (5.10) In this particular case, y stands for the method proposed here for calculating itemset-support from a trained map m∗ which is queried to recall the support of an itemset x. The comparison of y is then done versus t which holds the support 113 of x generated by the Apriori algorithm over the original dataset. N stands for the number of patterns contained in the itemset group requested. For example, N will be equal to 75 if a SOM, trained with the Chess dataset, is queried to recall support values for the group of 1-itemsets defined for such a dataset. In some of the graphs shown below, along the x-axis, the itemsets, for which the SOM is queried, are arranged. Each itemset is plotted based on their order of appearance (from left to right, from bottom to top) in the lattice structure representing the complete data space formed by the k items. An example of a lattice showing this arrangement can be seen in Figure 2.1 in Chapter 2. In the case of the artificial datasets, the maps are queried for all the support values for all of the possible itemsets (combinations) that can be formed by the n items or attributes of the dataset. This is to determine if our events E, representing patterns or associations in the environment, whose final probability or frequency occurrence is defined by the probabilities of the individual random variables representing the items, are independent or pairwise independent. This concern is important because if it were found that components of the events are only pairwise independent, then the estimations made by this method for events, involving the joint probability of more than 3 variables or items, could present large differences to the real frequency counters. Therefore, it could not be claimed that a SOM works as an itemset-support memory for ARM. In Figure 5.5, the results for the artificial datasets obtained from applying PISM, in particular from applying Equation 5.9, to make SOM recall itemset support are shown. 114 60 SOM (PISM) Apriori Implementation 50 Support (%) 40 30 20 10 0 15 k−itemsets (1<=k<=4) 60 SOM (PISM) Apriori Implementation 50 Support (%) 40 30 20 10 0 255 k−itemsets (1<=k<=8) 60 SOM (PISM) Apriori Implementation 50 Support (%) 40 30 20 10 0 65535 k−itemsets (1<=k<=16) Figure 5.5: Results for the support value of 15 itemsets (top graph), 255 itemsets (centre graph) and 65535 (bottom graph) obtained after using PISM in order to satisfy the query All- to the map trained with the dataset Bin4, Bin8 and Bin16 respectively. For reference, the values corresponding to the same queries using an Apriori implementation are also plotted. 115 To provide an indication of the behavior of a SOM and how accurate the results can be during training, some temporal (intermediate) stages of a SOM before it converges are plotted in Figure 5.6. It should be noted in these two figures that even though the mappings, derived from a SOM training, represent the same input environment, it does not imply that the same results will be generated by them. The variation in the results is the consequence of the map initialisation, that for this work, is done randomly. To overcome this unstable situation, the map could then have been initialised linearly. It is relevant to point out from Figure 5.6 that since the very first training iteration (epoch) both SOMs are able to provide a good estimation for the real itemset support. 70 SOM (PISM) Epoch 1 SOM (PISM) Epoch 3 SOM (PISM) Epoch 5 Apriori Implemetation Support (%) 60 50 40 30 20 10 0 15 k−itemsets (1<=k<=4) 80 SOM (PISM) Epoch 1 SOM (PISM) Epoch 3 SOM (PISM) Epoch 5 Apriori Implemetation Support (%) 60 40 20 0 15 k−itemsets (1<=k<=4) Figure 5.6: Intermediate results (The support values of 15 itemsets) generated from using PISM for the query -All- to the map being trained with dataset Bin4x100. In both cases, the SOM needs five epochs to converge but after the first epoch, good estimations can be formed for the support of itemsets. The small difference in the performance between these two exercises is due to the type of initialisation chosen. To give a taste of the quality of the estimations given by a SOM via the pro116 posed method for some of the query cases for real-life datasets, the graphs in Figures 5.7, 5.8, and 5.9 are depicted. These resulting mappings of the trained SOMs compared to the Apriori results have, at first glance, a better accuracy than the artificial ones. An explanation for this improvement in the itemset-support mapping is that in the real-life datasets the distribution of the patterns is unbalanced, that is, the fact of finding only a single repetition for each possible different pattern (itemset) of the distribution is only remotely likely to happen. The latter is a phenomenon happening in the artificial datasets used in this experimentation. SOM (PISM) Apriori Implementation S u p p o r t (%) 100 50 0 75 100 50 0 1−i t e m s e t s S u p p o r t (%) 100 50 0 2775 100 50 0 2−i t e m s e t s S u p p o r t (%) 50 67525 100 50 0 3−i t e m s e t s 67525 3−i t e m s e t s S u p p o r t (%) 100 50 0 2775 2−i t e m s e t s 100 0 75 1−i t e m s e t s 100 50 0 1215450 4−i t e m s e t s 1215450 4−i t e m s e t s Figure 5.7: Results (plots on the right) obtained after using PISM in order to satisfy the queries -1Itemset : 4Itemset- to the map trained with the dataset Chess. For reference, the values corresponding to the same queries (plots on the left) against the dataset Chess using an Apriori implementation are also plotted. In order to assess the capability of a SOM to generalise the recall of the support of any itemset, a series of three experiments has initially been conducted with the Chess, Mushroom and Connect datasets for the groups of the itemsets 117 SOM (PISM) Apriori Implementation 100 80 80 S u p p o r t (%) 100 60 40 20 0 60 40 20 0 119 1−i t e m s e t s 100 80 80 S u p p o r t (%) 100 60 40 20 0 60 40 20 0 7021 2−i t e m s e t s 7021 2−i t e m s e t s 100 80 80 S u p p o r t (%) 100 60 40 20 0 119 1−i t e m s e t s 60 40 20 0 273819 3−i t e m s e t s 273819 3−i t e m s e t s Figure 5.8: Results (plots on the right) obtained after using PISM in order to satisfy the queries -1Itemset : 3Itemset- to the map trained with the dataset Mushroom. For reference, the values corresponding to the same queries (plots on the left) against the dataset Mushroom using an Apriori implementation are also plotted. SOM (PISM) Apriori Implementation 100 80 80 S u p p o r t (%) 100 60 40 20 0 60 40 20 0 129 1−i t e m s e t s 100 80 80 S u p p o r t (%) 100 60 40 20 0 60 40 20 0 8256 2−i t e m s e t s 8256 2−i t e m s e t s 100 80 80 S u p p o r t (%) 100 60 40 20 0 129 1−i t e m s e t s 60 40 20 0 349504 3−i t e m s e t s 349504 3−i t e m s e t s Figure 5.9: Results (plots on the right) obtained after using PISM in order to satisfy the queries -1Itemset : 3Itemset- to the map trained with the dataset Connect. For reference, the values corresponding to the same queries (plots on the left) against the dataset Connect using an Apriori implementation are also plotted. 118 plotted in Figures 5.7, 5.8 and 5.9. In these experiments, we are interested in evaluating the effectiveness of our method applied to a SOM with random initialisation in recalling itemset support for itemset groups whose components are formed by the same number of items. The corresponding generalisation errors for these experiments are summarised in Tables 5.4, 5.5, and 5.6. These numbers have been obtained by applying Equation 5.10 to the results given by a SOM and our method, and original dataset and the Apriori algorithm. Experiment 1 2 3 Itemset Group 1-itemsets 2-itemsets 3-itemsets 1-itemsets 2-itemsets 3-itemsets 1-itemsets 2-itemsets 3-itemsets 1 0.13781 1.3735 1.4338 0.27685 1.426 1.4803 0.21732 1.3974 1.4593 Rectangular Radius 0.5 0.14819 0.96662 1.0097 0.15956 0.95871 1.0012 0.14176 0.99191 1.037 0.001 0.26277 0.8871 0.92083 0.16423 0.75058 0.78645 0.14409 0.79192 0.83168 1 0.15412 1.3771 1.4415 0.15149 1.3789 1.4431 0.15006 1.3699 1.4314 Hexagonal Radius 0.5 0.18899 1.0398 1.0841 0.12718 0.99795 1.0454 0.16442 1.0199 1.0659 0.001 0.15363 0.80162 0.84045 0.1863 0.82116 0.85914 0.15158 0.78599 0.82551 Table 5.4: Generalisation errors produced by a trained SOM through PISM for the queries 1Itemsets, 2Itemsets and 3Itemsets derived from the Chess dataset. Experiment 1 2 3 Itemset Group 1-itemsets 2-itemsets 3-itemsets 1-itemsets 2-itemsets 3-itemsets 1-itemsets 2-itemsets 3-itemsets 1 0.14403 0.42695 0.2484 0.19336 0.44692 0.25762 0.21152 0.4413 0.25409 Rectangular Radius 0.5 0.14272 0.31001 0.18097 0.095607 0.28158 0.16553 0.14741 0.29438 0.17267 0.001 0.16962 0.32908 0.193 0.15078 0.30919 0.18251 0.18457 0.34385 0.20083 1 0.14957 0.46244 0.27029 0.1531 0.45812 0.26885 0.16683 0.4298 0.25081 Hexagonal Radius 0.5 0.11507 0.28882 0.17016 0.10954 0.29399 0.1724 0.11674 0.30569 0.18004 Table 5.5: Generalisation errors produced by a trained SOM through PISM for the queries 1Itemsets, 2Itemsets and 3Itemsets derived from the Mushroom dataset. 119 0.001 0.13477 0.3464 0.2034 0.15362 0.32803 0.1929 0.12932 0.31145 0.18481 Experiment 1 2 3 Itemset Group 1-itemsets 2-itemsets 3-itemsets 1-itemsets 2-itemsets 3-itemsets 1-itemsets 2-itemsets 3-itemsets 1 0.35658 0.76868 0.6471 0.44143 0.80008 0.66575 0.39652 0.76256 0.63775 Rectangular Radius 0.5 0.49537 0.68955 0.55741 0.38671 0.65154 0.53933 0.33373 0.63865 0.53422 0.001 0.78376 0.77088 0.57361 0.83169 0.80109 0.59207 0.8609 0.84214 0.6246 1 0.45956 0.80169 0.66536 0.38639 0.7705 0.64628 0.33847 0.76385 0.64553 Hexagonal Radius 0.5 0.36677 0.65929 0.5489 0.28075 0.63333 0.53553 0.78373 0.83437 0.6366 Table 5.6: Generalisation errors produced by a trained SOM through PISM for the queries 1Itemsets, 2Itemsets and 3Itemsets derived from the Connect dataset. It is important to note that errors for three different values of radius for different map layouts have been calculated. Radius has been varied to determine the best parametric state of a SOM for FIM. The tuning of the radius is done since this parameter can work as the regulator of the BMU influences exchanged during training, and its definition can affect the accuracy of the final mapping. The way in which the nodes are distributed in the SOM, hexagonal or rectangular, has also been tested since we have assumed that different organisations of neurons can also influence the final estimated itemset-support values. Because the six maps of each experiment were initialised randomly, it can be stated that there is no relationship amongst their results. Nevertheless, what can be stated to have a relationship is the three values, representing errors for the three groups of itemsets, because these values have been calculated from the same trained map at each experiment. By using such relationships, a tendency in the resulting errors for the tested groups for each dataset can be stated as follows: • In case of the Chess dataset, Table 5.4, the error tends to increase while k, which defines the size of the itemsets, increases without any exception 120 0.001 0.74751 0.75174 0.56305 0.82694 0.79409 0.58539 0.6665 0.71637 0.54759 of the type of radius employed in the experiments. No clear tendency is presented regarding on which map layout provides the best results, so the layout can be stated to be irrelevant for this dataset in the estimation of itemset support. • For the case of the Mushroom dataset, Table 5.5, the error tends to form a distribution with a mode at the 2-itemsets without any exception of the type of radius. As the above case, a tendency to give more accurate results, amongst the two map layouts, is not clear either. • In the last case, regarding the results for the Connect dataset, Table 5.6, the error also tends to form a distribution with a mode at the 2-itemsets, as similar to the previous case, but no defined order between the extreme error values is presented. Since it was not possible to establish which radius is the best for the estimation of itemset support with the results given above, the next experiments will involve initialising all the maps, utilising different radius’, in the same manner. To produce such initialisation in the maps, a linear initialisation will be applied, because in this way the weight vectors are initialised along the linear subspace spanned by the two principal eigenvectors of the input dataset (Kohonen, 1996). Only one experiment is shown for each group of itemsets in Table 5.7, because the same results will always be obtained for all cases. This is, a map trained in batch with a linear initilisation will converge to a final state in an identical manner because its initial state is always the same. Unlike our previous experiments, in this case, the results maintain a relationship for the different radius and itemset sizes. Therefore, in general terms, it can be stated that while the radius decreases for training, the generalisation error 121 Dataset Itemset Group Chess Mushroom Connect 1-itemsets 2-itemsets 3-itemsets 1-itemsets 2-itemsets 3-itemsets 1-itemsets 2-itemsets 3-itemsets 1 0.2074 1.3238 1.3837 0.24362 0.41104 0.23652 0.8835 0.95011 0.72913 Rectangular Radius 0.5 0.13494 0.95034 0.99443 0.28344 0.35578 0.20278 0.67506 0.75157 0.58338 0.001 0.15624 0.5906 0.61271 0.15767 0.34597 0.20182 0.74965 0.66644 0.47813 1 0.26896 1.3455 1.4024 0.24715 0.42121 0.24215 0.87661 0.95628 0.73671 Hexagonal Radius 0.5 0.15379 1.0044 1.0498 0.29623 0.36697 0.20841 0.7407 0.79004 0.60596 0.001 0.15624 0.5906 0.61271 0.15767 0.34597 0.20182 0.74965 0.66644 0.47813 Table 5.7: Generalisation errors for the trained SOMs with linear initialisation. also decreases. It can also be noted that the error values for the different itemset groups, given by, either the rectangular, or hexagonal maps with a radius set to 0.001, are all the same respectively. In the cases, in which the radius is greater than 0.001, it is observed that the maps with a rectangular layout give better estimations than their hexagonal counterparts. In summary, based on the results in Table 5.7, some temporal conclusions can be made as follows: • It can temporally be determined that the value of the error tends to grow while k, which is the number of components (items) in the itemsets, does it as well. • It can also be concluded that the generalisation error has a direct relationship with the amount of influence defined by the radius used for training. • The best error values come from the maps whose radius parameter has been reduced to 0.001. Therefore, it can be assumed that performing noexchange-of-BMU-influence, i.e., setting the radius to zero, which reduces the SOM to a Vector Quantization algorithm, could generate even better estimations for itemset support. In order to find out if the above conclusions can satisfy more realistic mining 122 activities, in which support for different groups of k-itemsets with different sizes is needed, some trained SOMs have also been tested on the queries, constraining by the support property of the itemsets in the dataset, defined in Table 5.3. To visualise the SOM behavior in this type of mining exercises, we have plotted, in Figure 5.10, an example of the SOM estimations against the real support values to model their discrepancies. The complete results for these new experiments, involving different ranges of varied groups of k-itemsets, are summarized in Table 5.8. For the 3−itemset group For the 4−itemset group 100 Real Itemset Support Real Itemset Support 100 98 96 94 92 90 88 90 92 94 96 98 98 96 94 92 90 88 100 90 Estimated Itemset Support For the 5−itemset group 96 98 96 94 92 90 92 94 96 98 Estimated Itemset Support 93 92 91 90 89 90 91 92 93 Estimated Itemset Support For the 7−itemset group Real Itemset Support 91.5 91 Hexagonal SOM Rectangular SOM 90.5 90 89 100 94 Real Itemset Support Real Itemset Support 94 For the 6−itemset group 98 90 88 92 Estimated Itemset Support Apriori 89.5 90 90.5 91 91.5 Estimated Itemset Support Figure 5.10: Distribution of the itemset-support estimations made by SOM via our method for the query -90to100Itemsets- for the Chess dataset. The corresponding errors are summarized in Table 5.8. 123 94 Dataset Query Chess 90to100Itemsets Chess 45to100Itemsets Mushroom 45to100Itemsets Connect 75to100Itemsets k 3 4 5 6 7 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 3 4 5 6 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Map Layout Hexagonal Rectangular 0.42918 0.3668 0.52008 0.4352 0.61963 0.50871 0.76948 0.62251 1.0274 0.81303 0.92081 0.89465 1.0748 1.0476 1.2055 1.1724 1.327 1.2829 1.4407 1.3794 1.5395 1.4548 1.6126 1.4983 1.6496 1.4996 1.648 1.4559 1.6202 1.3801 1.5947 1.3037 1.6045 1.2653 1.6755 1.2926 1.8418 1.4084 2.0514 1.5868 0.47635 0.44559 0.54856 0.49902 0.62168 0.49366 0.71689 0.51841 1.6103 1.6109 1.6982 1.6883 1.7487 1.7322 1.7817 1.7626 1.8063 1.7886 1.8296 1.8168 1.858 1.8532 1.8969 1.9027 1.9503 1.9686 2.0217 2.0538 2.1142 2.1612 2.23 2.293 2.3675 2.4485 2.5059 2.6096 2.24 2.4146 Itemsets per group 167 203 128 39 4 4985 25500 88170 217705 397947 550220 581647 471908 293209 138294 48473 12023 1896 152 5 100 91 37 6 2757 13218 44721 111159 208126 297836 327797 277133 178389 85839 29903 7135 1052 76 1 Table 5.8: Generalised errors for trained SOMs with linear initialisation on different ranges of groups of k-itemsets. 124 Roughly speaking, it has been noted again that the rectangular maps produce better estimations than the hexagonal ones. Nevertheless, the differences established between them are not considerable. The tendency of the error to growth, while k increases, is also presented. However, in some cases, it tends to stay steady or even to decrease as shown in Figure 5.11. For Chess dataset with 90to100Itemsets Query 1.2 For Chess dataset with 45to100Itemsets Query 2.2 Hexagonal SOM Rectangular SOM 1 2 Hexagonal SOM Rectangular SOM 1.8 0.8 1.6 0.6 1.4 1.2 0.4 1 0.2 3 4 5 6 7 For Chess Mushroom with 45to100Itemsets Query 0.8 Hexagonal SOM 0.75 Rectangular SOM 0.8 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 For Chess Connect with 75to100Itemsets Query 3 Hexagonal SOM Rectangular SOM 0.7 2.5 0.65 0.6 0.55 2 0.5 0.45 0.4 3 4 5 6 1.5 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Figure 5.11: Generalising errors for the results given by Table 5.8. While the x-axis represents the different itemset groups, the y-axis determines the calculated error. Reviewing the numbers defined by the error tables, it can be determined that even though the generalisation errors look relatively small for the experiments, some considerable differences between the estimated and real support values can be found for some itemsets. Although these differences tend to get diminished during training, their final values could cause misleading support values during the detection of frequent itemsets in an ARM exercise so that it is important to 125 minimize such misleading effects. As an explanation for these discrepancies in the support results, two options may be considered: a) the type of metric used to organise the patterns on the map, or b) the property of assuming that the items (events) of the patterns are independent at each BMU. Moreover, both reasons could be directly influenced by the number of neurons contained in the map since this characteristic of the map is responsible for limiting the node-space available for any new incoming pattern. In other words, the larger the number of nodes available in the map, the better the distribution of the patterns along the map, and therefore the better the resolution of the outcome. The latter is a concept known as emergent feature maps, in which their structure is defined by large number of neurons, defined by Ultsch (Ultsch, 1999) which has shown to have the potential for data mining tasks (e.g., classification and clustering). Emergence occurs in natural- as well as artificial-systems, and refers to the ability of a system to produce a phenomenon on a new, higher level, as a result of the cooperation of many elementary processes (Ultsch, 1999). That is, new data features can emerge from structures formed by the cooperation of a large number of neurons. The latter is a characteristic that it is not presented in the traditional SOMs in which the number of neurons is controlled and limited (this number is approximately equal to the number of clusters) to show emergence. Therefore, a series of experiments with the Chess dataset have been set up to evaluate if the concept above defined could improve the response of a SOM for FIM. In these experiments, different sizes of a SOM have been used and the results are shown in Figures 5.12 and 5.13. The sizes of the map involved in these experiments range from two nodes to three times the map size H, which defines the number of neurons in the map 126 Error calculated For 1−itemsets 0.6 0.4 0.2 0 2 0.5H H 1.5H Size of the map 2H 2.5H 3H 2H 2.5H 3H 2H 2.5H 3H Error Calculated For 2−itemsets 3 2 1 0 2 0.5H H 1.5H Size of Map Error Calculated For 3−itemsets 3 2 1 2 0.5H H 1.5H Size of Map Figure 5.12: Distribution of the itemset-support generalisation error made by SOM via our method for the queries: 1Itemsets, 2Itemsets and 3Itemsets for the Chess dataset, when the size of the map, representing an itemset-support memory for ARM, increases. needed for mapping the n number of input patterns contained in the training dataset. H is normally calculated heuristically (Vesanto et al., 1999) by Equation 5.11. In this case, it has resulted in being 289 for the 3196 patterns existing in the Chess dataset. √ H=5 n (5.11) At first suspected and then corroborated by the results shown in the Figures 5.12, and 5.13, the error for the recall of the support of k-itemsets (where k is greater than 1) tends to decrease while the number of nodes increases in the map. The improvement is due to the fact that the input patterns develop a better organization on the map, which also breaks down the dependency property among the items of the input patterns. That is, patterns which are truly related remain closer on the map but they do not share the same node. 127 15 3−itemset Group 4−itemset Group 5−itemset Group 6−itemset Group 7−itemset Group 8−itemset Group 9−itemset Group 10−itemset Group 11−itemset Group 12−itemset Group 13−itemset Group 14−itemset Group 15−itemset Group 16−itemset Group 17−itemset Group 1.6 1.4 Error Error 10 1.8 1.2 1 5 0.8 0.6 0.4 0 2 0.5H H 1.5H 2H 2.5H 3H Size of Map 3H Size of Map Figure 5.13: Distribution of the itemset-support generalisation error made by SOM via our method for the query 45to100Itemsets for the Chess dataset, when the size of the map, representing an itemset-support memory for ARM, increases. This improvement in the error for the group of the 1-itemsets does not represent the behaviour explained above. In this particular case, it starts with a value very close to zero since there are just two nodes which accumulate the total input patterns so that the values, representing the calculated support, are placed closer to their corresponding means because there are not too many BMU influences to combine. The contrary effect occurs when the map size is increased; this has the result that the error tends to increase, due to the fact that some BMU influences are presented in the updating of the map. Nevertheless, the likely error stays steady and low along the experiments. In a more real scenario, in which the value for different itemset groups is 128 extracted from a SOM, for instance, the query 45to100Itemsets over the Chess dataset, it is evident how the error in the estimations tend to decrease while H increases in the experiments. This is due to the better distribution of the patterns along the nodes; therefore, it can be concluded the use of the concept of emergent feature maps is advantageous for the quality of the itemset-support estimations made by this SOM-based memory for ARM. 5.4 Conclusions In the discovery of association rules from datasets, the task of FIM has to be performed first, since it provides the raw material to form the possible rules. To measure which itemsets from the database are interesting for the data mining task, the support of itemsets has to be calculated. Motivated for the results obtained from the knowledge formed with an auto-associative memory in the previous chapter, we focus on exploring an unsupervised neural network. In particular, we investigate the use of a SOM for ARM. In this work, the exploration of a novel application for SOM, which refers to using its codebook for the estimation of itemset support, has been undertaken. Unlike other proposals (Changchien and Lu, 2001; Shangming Yang, 2004), which form the literature of this topic, in this work, the suitability of estimating ”itemset support” from the weight matrix of this neural network has been investigated. In particular, the calculation of the support has been proposed by an extraction mechanism called PISM (Probabilistic Itemset-support eStimation Mechanism), which uses only the winners formed in the final map for forming itemset-support estimations. Thus, the input dataset can be discarded after training, since the winners have gathered enough capability (information) to map the 129 associations occurring in the multi-dimensional input patterns. To validate the suitability of a SOM for this type of data mining task, results of some experiments have also been given versus an implementation of the Apriori algorithm. Numeric comparisons have also been done by using an error metric between the estimated value extracted from a SOM against a traditional FIM algorithm. The results of the experiments have shown that the suitability of a SOM for ARM is realistic, in particular if the concept of emergent SOM is utilized. In summary, we have satisfactorily tackled the problem of how to reproduce the counting of patterns, occurring in a high dimensional space of an environment, with the learnt information or knowledge embedded in the two-dimensional space formed by a trained SOM. Therefore, it can be concluded that a SOM can be seen as a good candidate for our desired memory, due to its ability to learn information about the frequency-occurrence of patterns hidden in training associations. 130 Chapter 6 Incremental Training for Incremental ARM: A SOM Model Because itemset support is a very important metric for the generation of association rules, an extraction mechanism for SOMs, which decodes the formed codebook, has been proposed to estimate itemset support in the previous chapter. As a result of our proposal, the generation of association rules directly from the knowledge of a trained SOM has become feasible; therefore, unlike other related proposals (Shangming Yang, 2004; Changchien and Lu, 2001), we have stated that the training data is no longer needed for ARM. Nevertheless, a problem, affecting the validity of the itemset-support knowledge embedded in the trained SOM, emerges as soon as the database1 used for its training starts acquiring new transactions. This is, the itemset information in the SOM is not still valid to describe the current state of the database. To tackle this new problem, resulted from the dynamics of the data, incremental-training 1 Throughout this chapter, the term database will be used instead of the term dataset to refer to the data source because we part from the fact that data have been recorded and stored in a model beforehand. Moreover, we believe the use of database is more appropriated for the overtaking problem since dataset is often used in literature to describe a data file with a finite number of samples (transactions, patterns). 131 mechanisms can be used; nevertheless, their definitions have been based on nonbatch procedures. Because the concept of batch training is important for itemset-support estimation and recall from a trained SOM, we undertake the task of developing an incremental batch training mechanism for SOMs called Bincremental-SOM in this chapter. In particular, we will propose how the itemset knowledge embedded in the map should be updated to maintain it valid while the original environment changes periodically. 6.1 Introduction Previously in Chapter 5, we have stated that itemset support can be estimated from a trained SOM, especially from its BMUs. These estimations have been possible because the codebook has been decoded probabilistically. As the codebook stores information about itemset support, we have considered the SOM to be an artificial memory from which the support of an itemset X can be recalled anytime that it is needed. Therefore, we have proposed that the SOM can take the role of an itemset-support provider, instead of the original database, in the ARM framework. Additionally, this resulting trained SOM can be also seen as an abstract descriptive model Msom formed with the associations happening in the training database D. In the real world, the possibility that data are static is almost impossible. Especially, if we refer to data which are used for analysis which typically tend to change their state as a result of events occurring in their environment. This type of environments are called non-stationary, since new states appear as a result of 132 changes throughout time. For instance, new shopping patterns may emerge in the transactions of a database as a result of the introduction or promotion of new products in the market. Because of all the changes or dynamics presented in a database, any knowledge derived from it, producing either predictive or descriptive models, tends to loose accuracy in representing the current state of the underlying database. The ignorance of this inconvenient-but-realistic data characteristic in the development of data-mining approaches becomes a serious problem for data analysis. In particular, misleading and wrong decisions can be made by end users for not having the latest state of the data represented in their models. For example, a mortgage can be initially given to a bad-credit client just because, in current usage, the model is not considering the most up-to-date tendencies in the market. Unprofitable selling policies can be applied to some items in a supermarket if the rules used do not catch the actual tendency of the customer-shopping behavior. Therefore, it has become important to consider developing algorithms which can cope with this undesirable data property in order to keep the models alive, updated, or valid as reasonably as possible. As part of the DM toolbox, any SOM-based approach, such as our itemsetsupport memory, suffers from the problem stated above because it is well-known that any trained SOM only learns the state presented in the input database at the moment of its training. One way of making a SOM learn a new state of D is by employing the simplest solution to tackle the problem, which involves retraining the SOM with the latest information contained in D (history and changes together). Nevertheless, this solution is neither practical nor optimal because it 133 requires keeping the entire D throughout time to perform the training and consequently no advantage can be taken from the past knowledge learnt by its nodes. In the ANN world, this problem has been tackled by performing incremental training of neural networks, which allows them to update their knowledge with the current information of their environment without loosing or forgetting the information already embedded in their weight matrix. When a neural network learns incrementally, it can also be established that it addresses the elasticityplasticity dilemma, since it adapts its internal structure in order to capture the basis of the new incoming inputs without forgetting the past experiences. During the past years, proposals related to SOM technology have appeared to tackle the lack of training under non-stationary environments. For instance, GSOM (Alahakoon et al., 2000a) is focused on developing a self-organisingand-growing structure for continuous learning. This variant of a SOM is trained under a sequential mechanism which grows while it learns through the usage of some heuristics which determine if the deletion or insertion of nodes into the structure is suitable. Since our aim is to update the model Msom , representing an artificial memory full of information about itemset support, and having in mind that the itemsetsupport extraction mechanism proposed assumes that Msom has been formed by a batch training, we can re-state that the underlying problem has turned out to involve not only updating the map under non-stationary conditions but also doing it by a batch-incremental training. According to Hung et al (Hung S., 2004), the limitation of learning models in a non-stationary environment has been addressed by introducing the concept of non-batch learning. This concept involves techniques such as online learning, lifelong learning, incremental learning and 134 knowledge transfer, which have all pointed out the limitation of employing batch learning for tasks concerning non-stationary environments. The main criticism of the batch methods refers to the need of keeping the entire database for training which results impractical for these environments. Therefore, we focus here on defining a batch training mechanism for SOM suitable for non-stationary environments. In particular, the latter is necessary because our interest lies in gaining some insight into the topic regarding the problem of incremental association rule mining in which the maintenance of the frequent itemsets and their support, according to Cheung et al (Cheung et al., 1996b), is relevant to maintain the association rules up-to-date. In other words, the work proposed below should be considered as a proposal needed to perform incremental ARM from a neural-network standpoint. The key insight in the maintenance of itemset support will result from the exploitation of incremental learning properties of neural networks. The idea will then involve studying whether the dynamics, derived from the presence of data chunks (defining inserts) from D, can be incorporated incrementally to the model Msom (t), formed at time t, in order to produce Msom (t + k), which defines the latest state of D. Our approach must be interpreted as an incremental learning task for SOMs which makes the map retain the past knowledge while the latest state of the support of itemsets in the environment is learning. 6.2 Batch SOM for Non-stationary Environments Practically speaking, we can state that a database itself represents a non-stationary environment formed by different phases throughout time. These phases, defin- 135 ing the size-varied groups of transactions or data chunks, can normally occur at different speed rates which adds a new complexity to the incremental problem. Nevertheless, we will assume in this case that among the different phases forming the environment (database), there is always a time space when the data can be buffered in order to be posteriorly presented to the SOM for learning. 6.2.1 The Problem Definition Let D be a database and t a metric to measure time. Thus, a data distribution that represents D at the time t can be expressed by D(t). Let D+ be a group of new data points, representing itemsets, that makes D pass from the state t to t+x; S such that D(t+x) = D(t) D+ . Let Msom be a SOM-based model resulting from training a SOM with input data. Thereby, Msom (D(t)) defines a SOM describing some concepts, for instance, the topology, the distribution or the associativity of the attributes or items of D(t). The problem is then to produce a model Msom (D(t + x)) by not using traditional procedures, because they require the presence of the entire dataset D(t+x) for a solution. Instead, it is necessary to do it in batch to produce a satisfactory 0 approximation model Msom (D(t + x)) in comparison to the possible traditional 0 final model Msom (D(t + x)), so that Msom (D(t + x)) ≈ Msom (D(t + x)). The main characteristic of the wanted mechanism is that the old data chunks D(i), for all i < j in which j defines the index of the latest data chunk in the environment, will not be needed for generating such an approximation. However, the use of some knowledge K from Msom (D(j − 1)) can be considered. In terms of the above definition, the algorithm we aim for can be defined as 136 follows: Msom (D(t))0 = γ( K[Msom (D(t − 1))] , D+ ) ≈ Msom (D(t)) (6.1) Where γ() is the target algorithm with inputs parameters: K[Msom (D(t−1))] and D+ that define some knowledge of the SOM generated at phase (t-1) and the new group of transactions of D respectively. 6.2.2 Interpretation by Node Influences of the Batch Training SOM training can typically be realised under two modes: sequential or batch, whose usage depends exclusively on the characteristics of the task to be tackled. The differences between them are basically the manner in which they perform the update of the map m. Overall, both modes always look for the best group of reference vectors (weight matrix) which can map and quantize the distribution and information of the training data accurately. The steps needed for doing the SOM training in batch are summarized in detail in Figure 6.1. Figure 6.1: Algorithm SOM training in batch. In the search task for BMUs, a distance metric, for instance the Euclidean distance, is needed to perform the comparison between the vectors of the map and the inputs (step 4). A new state in the map at the epoch i will result from 137 using Equation 6.2 (step 6). A final map, representing the information of D, will be created after the total number of epochs has been reached. P j hji (t)Sj (t) mi (t + 1) = P j nV j (t)hji (t) (6.2) From Equation 6.2, it can be deduced that each node contributes to the modification of the map until it converges. In particular, it can be stated that each node mi contributes with its own Si to the update of the map at each epoch. The term S, which is defined below, is used to define the concentration of input patterns at each node. Si (t) = nvi X xj (6.3) j=1 Here xj refers to the nvi different patterns that have chosen the node mi as their BMU. S can also be understood as a collection of the different data points allocated at the Voronoi region Vi = {xi | kxi − mi k < kxi − mk k ∀k 6= i}. An expanded version of Equation 6.2, which governs training, can be derived as follows: mi (t + 1) = h1i (t)S1 (t) + · · · + hji (t)Sj (t) + · · · + hmi (t)Sm (t) nV1 (t)h1i (t) + · · · + nVj (t)hji (t) + · · · + nVm (t)hmi (t) (6.4) In this representation of the batch-training equation, the way in which each of the units contribute to the update of the map is even more evident. Although it seems that all the nodes take part in the update operation, the reality is that not all of them have the resources to provide an influence to such a change. The lack of contribution from some nodes is due to the fact that they have not become BMUs; therefore, they do not have any data point in their regions which can be shared with the rest of the map. For instance, if inputs have chosen only nodes 138 mj and mk (out of a total of m nodes) as BMUs during a training epoch, then a particular situation, describing the update of node mi from nodes m1 , mj and mk , can be expressed decomposing Equation 6.4 as follows: ∆i (m1 (t)) = ∆i (V1 (t)) = 0 ∆i (mj (t)) = hji (t)Sj (t) nV1 (t)h1i (t)+···+nVj (t)hji (t)+···+nVk (t)hki (t)+···+nVm (t)hmi (t) ∆i (Vj (t)) = ∆i (bmuj (t)) 6= 0 ∆i (mk (t)) = h1i (t)S1 (t) nV1 (t)h1i (t)+···+nVj (t)hji (t)+···+nVk (t)hki (t)+···+nVm (t)hmi (t) hki (t)Sk (t) nV1 (t)h1i (t)+···+nVj (t)hji (t)+···+nVk (t)hki (t)+···+nVm (t)hmi (t) ∆i (Vk (t)) = ∆i (bmuk (t)) 6= 0 (6.5) Where ∆k (mj (t)) defines the contribution or influence given to the node mk from the node mj or the Voronoi region Vj at time t. In the particular case of the term ∆i (m1 (t)) (top term), its influence becomes zero because this node was not able to allocate any pattern input in its region (|V1 | = 0). After observing that only some nodes contribute to the map formation, a new formulation of the batch training equation can be expressed by, mi (t + 1) = X ∆i (mj (t)) (6.6) mj ∈W Which represents the update of the map in terms of the influential factors generated by a set W containing nodes satisfying the condition of: W = {mi | mi .|S| > 0 or mi .S 6= φ} Where S is a set whose elements are patterns that have hit the node mi . 139 (6.7) In order to use Equation 6.6 for the training of the SOM, it is necessary to state mathematically how each of the node influence can be calculated. Therefore, if we employ one of the non-zero-result-influential components stated in 6.5, for instance, the term ∆i (mj (t)), and considering that two nodes become BMUs, then an expression, defining the influence from node mj to mi , can be stated as follows: ∆i (mj (t)) = hji (t)Sj (t) nVj (t)hji (t) + nVk (t)hki (t) In which, it is important to note that the total influence generated from mj to mi depends not only on the state produced by mj but also on external influences formed at others BMUs. This external influence is represented in the denominator of the expression, and represents the number of data points allocated into those other BMUs weighted by the corresponding neighborhood function originating from them. Hence, a total influence generated from mj to mi at time t can be described by, ∆i (mj (t)) = Sj (t) i h nVk (t)hki (t) nVj (t) + hji (t) It is important to point out that the final response from node mj results in a value almost equal to the centroid of its Voronoi region Vj defined by P nj = n1v xi . The difference between these two terms can be stated to be j xi ∈Vj controlled by an influential coefficient β, which defines the ratio of the influences of other BMUs apart from mj to the distance between mi and mj . Therefore, it can be stated that the influence generated at a node mj towards a node mi in a SOM training in batch can be defined by: 140 ∆i (mj (t)) = Sj (t) nVj (t) + βji (6.8) In which βji is defined by, P βji = (k∈W )∧(k6=j) nVk (t)hki (t) hji (t) (6.9) The tendency of the possible differences between the resulting value ∆i (mj (t)) and the corresponding value of the centroid mj (t) is represented by the graph in Figure 6.2. Final Node Influence Sj(t)/n vj(t) Di (mj(t)) 0 Influence from other nodes Figure 6.2: Tendency of the final influence given by the node mj depending on the strength of the influences received from other nodes. 6.2.3 The Algorithm In order to satisfy the requirements stated in Equation 6.1, we need to determine which information from an old trained map is relevant to keep for its maintenance throughout time. This is, we need to identify K since it defines a knowl- 141 edge about the past of an environment. Therefore, we have proposed K to be defined by the last group of BMUs since they have triggered the last change of the available map. In other words, a historical knowledge K about a training environment D up to time t can be formed from a model Msom (D(t)) by extracting the following properties of each BMU: • The reference vector, since it summarises all the patterns grouped at the node. • The hit histogram, which also defines a prior probability of the node. This is, the number of times that the corresponding node has been hit by the inputs. Once this knowledge has been obtained, the next step is to define a procedure to re-use it in a training process in which a SOM is about to learn the new state D(t + x) drawn from the original environment. The latter is necessary otherwise the past knowledge will be forgotten as soon as the new training with the latest state in D begins. To perform the re-usage of the past knowledge, we propose adding a new term to Equation 6.6 based on the fact that a batch SOM training is the result of summing up neural influences. Therefore, we propose that the SOM training in batch should be defined by, mi (t + 1) = X ∆i (mj (t)) + X ∆i (K[MSOM (D(t))]) (6.10) mj ∈W In which, t of ∆i (K[MSOM (D(t))]) refers to the previous state defined in the non-stationary environment rather than describing the t epoch in the current training. The insertion of this new component to the traditional training of the SOM is proposed as a medium to keep track of the old data because it defines 142 not only the past organization of data (centroids) but also how strong or weak the pattern populations allocated to the BMUs were. Based on the above explained, our desired γ() algorithm is defined as in Figure 6.3. The first function is responsible for triggering the learning of the data at each of the stages existing in the non-stationary environment. The second function, which is called as a result of the appearance of a new data chunk in the environment, is in charge of not only learning the new data chunk in the environment, but also retaining the old knowledge captured by the previous trained map. In other words, both sources, new patterns and reference vectors in the BMUs of the previous SOM, serve as vector inputs for the training of the new map (Step 4). Moreover, a type of linear initialisation occurs with our approach at the moment of the learning of a new data chunk, because the old map is also re-used to initialise the new one (Step 2). Function Batch-Incremental () 1) t=0 // t defines the current stage in the training environment 2) Mt=0 // it represents the dynamic SOM 3) while (t<endofEnvironment) 4) Mt+1 = UpdateSOMIncludingPast(Mt, D(t+1)) 5) t++ 6) end // this function returns a trained SOM with information defining the current and past data chunks. Function UpdateSOMIncludingPast (Mt, D(t+1)) 1) K = extract-bmus-knowledge(Mt) // knowledge about the past is captured 2) mi= initialisation (K) // knowledge is also used for initialisation of a SOM 3) for (i=1;i=numepochs; i++) do begin 4) forall patterns p Î D(t+x) È K do begin 5) call LookingforBMU(p); 6) end 7) mi = update-map() // the batch equation is employed 8) end 9) return (mi) // resulting map describing the data chunks of the environment up to the time t+1 Figure 6.3: Incremental algorithm proposed for SOM in batch. While the first (top) function triggers the learning at each stage of a non-environment, the second (bottom) function performs the training of the SOM with the current data chunk and the old information coming from the set of best matching units of the latest trained map. 143 6.2.4 Experiments To test our approach defined above, we have re-created the experiment conditions presented in (Furao and Hasegawa, 2004), in which a neural-network training is tested for non-stationary environments by using diverse data chunks to represent different data distributions. In our case, the training data space, which will be learnt incrementally by our approach, is shown in Figure 6.4. To simulate a nonstationary environment, a SOM will be fed with these data regions (A1, A2, A3, B, C and D - defining different topologies -) in the order defined in Table 6.1. Figure 6.4: Representation of a training data space describing a non-stationary environment. Phase environment I II III IV V VI Inputs chunk knowledge A1 no A2 yes(A1) A3 yes(A2) B yes(A3) C yes(B) D yes(C) Table 6.1: Data order followed in the incremental batch training. The aim of this experiment is to observe if the internal structure of the SOM, trained incrementally by using the algorithm defined above, is able to map all of 144 the different topologies represented in the training data. In other words, it is desirable that the presentation of a new data topology (data chunk) does not make the nodes forget previous knowledge or mappings about other topologies. The mappings, formed at each of the six-phase training of a non-stationary environment by a SOM trained with our proposal, are shown in Figure 6.5. In all the phases apart from the first one, the SOM is trained with the corresponding data chunk (changes in the data) along with the historical knowledge (the group of the BMUs) describing the previous phases. It can be noted that in each of the different phases the SOM not only covers the topology of the corresponding data chuck for the phases, but also uses some of its nodes to retain the knowledge learnt previously. 6.3 Itemset Support Maintenance by Incremental SOM Training After defining a method which allows the batch-incremental training of a SOM, we will focus in this section on investigating whether such a proposal can be used to maintain the knowledge itemset support embedded in a trained map throughout time. This is, in the next experiments we will evaluate if a trained SOM, acting as an artificial itemset-support memory, is able to update its knowledge with the changes occurring in a non-stationary environment that will be simulated by partitioning a FIM real-life dataset. 6.3.1 Experiments To evaluate the suitability of our batch-incremental-training algorithm for SOMs as an approach for the maintenance of the knowledge (itemset support) learnt by 145 Environment III 100 Environment IV A1 100 A1 B 50 50 A2 A2 0 0 −50 −50 A3 A3 −100 −100 0 70 0 70 Environment V Environment VI 100 100 A1 A1 B 50 B 50 A2 A2 0 D 0 C C −50 −50 A3 A3 −100 −100 0 70 0 Figure 6.5: Topologies formed by a SOM trained with our incremental batch approach through the six different phases defining the non-stationary environment represented in Figure 6.4. The black dots define the structure of the trained map. The data points used for the training of a SOM at each phase of the environment, according to the order in Table 6.1, are defined by the green and blue dots, which represent respectively the old knowledge (data extracted from the BMUs) and the current data chunk. 146 70 a SOM from dynamic data, a series of experiments will be conducted by simulating three different non-stationary environments derived from the Chess dataset (a real-life dataset with 3196 transactions defined by 75 items). In each environment, the dataset is separated into different data chunks with different sizes to model different phases of changes in which the database can get involved as a result of the dynamic of the environment. It is important to state that the radius parameter of the batch training equation, which is responsible for shrinking the neighborhoods and tuning the trained map for detecting fine data structures (Kohonen, 1996), will be maintaining a constant value of 1 in our experiments, since we are interested in capturing in the final map the global tendencies of itemset support in the environment. The general conditions describing each of the environments are summarized in Table 6.2. Environment Data-chunk Size Number of Phases I 800 transactions per chunk (Fixed) 4 II 400 transactions per chuck (Fixed) 8 III From 200 to 600 transactions per chunk (Unfixed) 9 Table 6.2: Definition of the non-stationary environments using the Chess dataset. In the first two environments, each data chunk has the same (fixed) number of transactions, while in the case of the third environment, the number of transactions was chosen randomly (unfixed). The idea behind setting the environments as phases is because the resulting SOM will be queried to estimate support for some groups of itemsets after its training has been performed with the data chunk representing the corresponding phase. In all experiments, the map size has been set up to a fixed number of nodes (about 900 nodes) that represent about one quarter of the total number of transactions in the Chess dataset. 147 To measure the efficiency of our method to update a SOM for FIM purposes throughout time, two other approaches, representing traditional mechanisms to perform the update, are going to be tested for comparison. The first approach (Chunk-SOM) corresponds to the training of a SOM with only the corresponding data chunk which defines the changes occurred in the database at the phase i of the environment. This approach can be understood as the worst possible scenario to address the maintenance of the map because the accuracy of the results (itemset support) will be depending on how well the current data chunk (current sampling) represents the overall tendency of the associations in the entire D throughout time. Relying on this approach for itemset-support recall, as further proven by the results, is risky, because although all of the data chunks are drawn from the same data distribution D from the environment, the fact is that they do not necessarily imply having the same behavior in their transactions. The second approach to be evaluated will be our batch incremental training for SOM (Bincremental-SOM). This represents the use of a batch-incrementaltraining for SOMs for the maintenance of itemset support. This method is characterized not only by learning the current data chunk at each phase k, but also by utilizing some knowledge from the latest state of the SOM in the environment, because it has retained information about, for instance, the topology and distribution of the past phases in the training environment. The premise followed by our approach is that there is no reason to retain the old data chunks in the environment any longer once they have been learnt by a SOM. Representing the third approach (Allchunks-SOM), a naive retraining of the 148 SOM will be used to update the map. This approach involves the training of the SOM with all the data chucks produced up to the current phase k in the environment. The reason for using this approach is that its results represent the best theoretical itemset-support approximations that can be generated from the environment. This approach has the big disadvantage that all data chunks, representing the changes, have to get involved in the training of the map. To make the comparison between our approach and the others, some error readings will be calculated from the maps formed at each of the phases of the environments. The metrics used to evaluate the accuracy and quality of the maps are described as follows: • The RMS (Root-Mean-Squared) error: This error metric, defined below in Equation 6.11, is used to measure the accuracy of the generalization on the support recall for the groups of the 1- and 2-itemsets. To calculate this error, the target t will be represented by the results obtained from applying the Apriori algorithm to the data chunk in turn. It is important to note that N, which defines the number of itemsets in the recall, may vary during an environment since data changes can also involve the apparition of new items throughout time. ERM S v u N u1 X t = {y(xn ; m∗ ) − tn }2 N i=1 (6.11) • The average quantization Error : This is a well-known error metric used to measure the quality of a map based on the resolution of the mapping from a Vector Quantization point of view, in which the aim is to look for a group of vectors (codebook) that can represent the distribution of the input data source in the most suitable form. The error is defined by Equation 6.12 in 149 which xi , mc and N define respectively an input vector from D, the BMU for xi and the total number of patterns in D. In our case, as the SOM will be changing and accumulating new knowledge, this error is going to be calculated to have an idea on how well the approaches summarise the past and current data chunks at each phase. As we assume that recently added transactions to a dynamic database can be more interesting than those inserted long ago in the sense that they reflect the current tendencies more closely, it will be very interesting to determine which data, between the history and the latest state of the database (added transactions), are better represented by the approaches tested. Eq = N 1 X kxi − mc k N i=1 (6.12) In addition to these two metrics, we have also recorded other values to observe the manner in which the nodes in the maps are being used along the phases of the environments. Figures 6.6 and 6.7 are shown to illustrate the response of a SOM regarding the support estimation for the groups of 1- and 2-itemsets during the four phases defining the environment I. One of the characteristics that can be observed in these results is the magnitude of the differences between the first approach (Chunk-SOM) and Apriori, in comparison to the other two (Bincremental-SOM and Allchunks-SOM). The first approach, as predicted above, is by far the worst approximation under non-stationary circumstances; therefore, no reliable results to describe the support of the itemsets can be derived from it. Moreover, the differences tend to become larger along the phases of the environment. 150 Chunk−SOM Bincremental−SOM Allchunks−SOM 0.4 0.4 0.4 0.2 0.2 0.2 0 0 0 −0.2 −0.2 −0.2 −0.4 66 −0.4 66 −0.4 50 50 50 0 0 0 −50 71 40 −50 71 −50 50 50 0 0 66 71 20 0 −20 −40 72 −50 72 −50 100 100 100 50 50 50 0 0 0 −50 −50 −50 −100 75 −100 75 −100 Figure 6.6: Differences between estimations (approximations) and calculations (real values) made respectively by a trained SOM and the Apriori algorithm for the group of 1-itemsets throughout the four phases of the environment I. The first column from the left describes the type of estimations that can be produced with a SOM trained with only the data chunk in turn in the environment (Chunk-SOM). The second column represents the estimations that can be made from a SOM trained with our incremental approach (Bincremental-SOM). The last column shows the estimation made by a SOM trained with always all the data chunks available en the environment (Allchunks-SOM). 151 72 75 Chunk−SOM Bincremental−SOM Allchunks−SOM 20 20 20 10 10 10 0 0 0 −10 2145 −10 2145 −10 50 50 50 0 0 0 −50 2485 −50 2485 −50 40 40 40 0 0 0 −40 2556 −40 2556 −40 100 100 100 50 50 50 0 0 0 −50 −50 −50 −100 2775 −100 2775 −100 Figure 6.7: Differences between estimations (approximations) and calculations (real values) made respectively by a trained SOM and the Apriori algorithm for the group of 2-itemsets throughout the four phases of the environment I. The first column from the left describes the type of estimations that can be produced with a SOM trained with only the data chunk in turn in the environment (Chunk-SOM). The second column represents the estimations that can be made from a SOM trained with our incremental approach (Bincremental-SOM). The last column shows the estimation made by a SOM trained with always all the data chunks available en the environment (Allchunks-SOM). 152 2145 2485 2556 2775 The behaviour of the RMS error, measuring the generalisation of the support recall for the 1- and 2-itemsets for the first environment, is shown in Figures 6.8 and 6.9. In these figures, it can be observed that even if, the approaches have started on the same condition (same initialisation, same inputs) and have provided the same result after concluding the first phase, they all follow different tendencies, in which the first approach stands out particularly for giving the poorest results. In the case of our approach, even if it presents some differences with the best case (a SOM trained always with all data chunks) its performance can be stated to be satisfactory, since it remains steady during the incorporation of new transactions, in particular, if we consider that its training has involved just the new data chunk in turn, along with the knowledge extracted from the latest trained SOM. Allchunks−SOM Bincremental−SOM Chunk−SOM 0.5 0.4 0.3 0.2 0.1 0 1 2 3 4 Phase Figure 6.8: The RMS error during the phases of the environment I for the group of the 1-itemsets. In Figures 6.10 and 6.11, we show the values recorded on the quantization 153 3 Chunk−SOM Bincremental−SOM Allchunks−SOM 2.5 2 1.5 1 0.5 0 1 2 3 4 Phase Figure 6.9: The RMS error during the phases of the environment I for the group of the 2-itemsets. 1.9 Chunk−SOM Bincremental−SOM Allchunks−SOM 1.8 1.7 1.6 1.5 1.4 1.3 1.2 1.1 1 0.9 1 2 3 4 Phase Figure 6.10: Error describing the quality of the quantization of the approaches tested for the data chunk describing the changes at each phase of the environment I. 154 3 Allchunks−SOM Bincremental−SOM Chunk−SOM 2.5 2 1.5 1 0.5 0 1 2 3 4 Phase Figure 6.11: Error describing the quality of the quantization of the approaches tested for the data chunks describing the previous phases (history) at each phase of the environment I. error at each phase k for the k th data chunk and all the past ones. These figures help analysing the abilities of quantization of each of the approaches tested. A perfect quantization of some data is defined by a quantization error equal to zero, and refers to the fact that the vectors defined, in this case by the SOM nodes, describe the distribution of the input data perfectly. In particular, the quatization of the maps produced for the data chunk provided at each phase is shown in Figure 6.10. Initially, it was expected that the first approach (Chunk-SOM) was going to generate the best results for this task, since the input data for its training at each phase is the same data used for the measurement of the error. In other words, the best results were expected from the first approach because its nodes (vectors) do not get distracted by any other inputs during training unlike the maps of the other two approaches. Therefore, the nodes can focus particularly on forming the mapping of the current data 155 chunk. Nevertheless, the results show that the converged map based on our approach (Bincremental-SOM) provides the best mapping resolution for the group of transactions of the corresponding data chunk at each phase in the environment. A reason for the decrease in the quantization error in our approach can be claimed to be due to the fact that our bath-incremental SOM trained takes advantages of the past knowledge. The key of the improvement is that, for instance, a map formed, based on our proposal at phase k, has already an advantages in its initial state since its vectors already map some regions of the data space defined by the database; therefore, it can be claimed that our proposals produces the re-use of past knowledge. Hence, the maps towards the end of the environment are created faster and produce a better approximation of the perfect quantization model. Although the improvement observed in our approach could be claimed to be replicated by using linear initialisation in the formation of the map, the fact is that in the best of the approaches (Allchunks-SOM), which was initialised linearly, this positive characteristic is not presented. In fact, in this task the worst quantization for the latest chunk at each phase was given by the SOM trained with all data chunks, which makes us conclude that the vectors of the map got distracted in the mapping of the most recent data chunks, due to the existence of others (past data chunks to be mapped too) in the training. A reduction of the quantization error under non-stationary conditions is a very positive characteristic for our approach because we can state that the organisation of the map via our algorithm forms a stronger mapping to the newest data chuck rather than the old ones. Hence, we can be sure that, in the recall of support, the latest tendencies of the environment are going to be considered. 156 In the case of the quatization for the old data chunks at each phase, the results are shown in Figure 6.11. In this case, all approaches start with a zero error since in the initial phase there is no history to remember. In the consequent phases, the best organization of vectors is given by the third approach since its training always involves the presence of past and current chunks. For this task, the results of our approach seem positive if we keep in mind that there were no past data for its training. In other words, our approach still remembers the past phases of the database satisfactorily. The response of a SOM with our approach under the conditions set up for the environment II, which has been formed by splitting the data chunks of the environment I in half, regarding the RMS error for the itemset-support generalization for the group of 1- and 2-itemsets are shown in Figures 6.12 and 6.13 respectively. 0.8 Allchunks−SOM Bincremental−SOM Chunk−SOM 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 1 2 3 4 5 6 7 8 Phase Figure 6.12: RMS Error during the phases of environment II for the group of the 1itemsets. 157 5 Allchunks−SOM Bincrement−SOM Chunk−SOM 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 1 2 3 4 5 6 7 8 Phase Figure 6.13: RMS Error during the phases of environment II for the group of the 2itemsets. In overall, the accuracy of the recalls of the approaches was noticed to be similar to the ones presented for the first environment. The good characteristic of our approach to be pointed out in this case is that even if there are more phases in the environment, the error does not tend to increase substantially. Satisfactorily, the property of mapping the latest data chunk better than the history is still observable in the results for the quantization errors shown in Figures 6.14 and 6.15. In the environment III, the size of the data chunks has been varied and selected randomly, giving as a result a nine-phase environment. The corresponding results for the RMS and quantization errors are plotted in Figures 6.16, 6.17, 6.18, and 6.19. 158 2 Allchunks−SOM Bincremental−SOM Chunk−SOM 1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 1 2 3 4 5 6 7 8 Phase Figure 6.14: Error describing the quality of the quantization of the approaches tested for the data chunk describing the changes at each phase of the environment II. 3 2.5 2 1.5 1 0.5 Allchunks−SOM Bincremental−SOM Chunk−SOM 0 1 2 3 4 5 6 7 8 Phase Figure 6.15: Error describing the quality of the quantization of the approaches tested for the data chunks describing the previous phases (history) at each phase of the environment II. 159 1 Chunk−SOM Bincremental−SOM Allchunks−SOM 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 1 2 3 4 5 Phase 6 7 8 9 Figure 6.16: RMS Error during the phases of environment III for the group of the 1-itemsets. 2 Chunk−SOM Bincremental−SOM Allchunks−SOM 1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0 1 2 3 4 5 Phase 6 7 8 9 Figure 6.17: RMS Error during the phases of environment III for the group of the 2-itemsets. 160 3 2.5 2 1.5 1 0.5 Chunk−SOM BIncremental−SOM Allchunks−SOM 0 1 2 3 4 5 Phase 6 7 8 9 Figure 6.18: Error describing the quality of the quantization of the approaches tested for the data chunk describing the changes at each phase of the environment III. 2 Chunk−SOM BIncremental−SOM Allchunk−SOM 1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 1 2 3 4 5 Phase 6 7 8 9 Figure 6.19: Error describing the quality of the quantization of the approaches tested for the data chunks describing the previous phases (history) at each phase of the environment III. 161 In general, we can state that the three approaches maintain the same behaviour as the environments with fixed-size data chunks. The results for all of the experiments for the three environments are summarized in Tables 6.3 to 6.5. Approach First Second Third Phase 1 2 3 4 1 2 3 4 1 2 3 4 RMS Error 1-itemsets 2-itemsets 0.11048 0.70959 9.7047 8.8469 10.817 9.6593 12.546 10.687 0.11048 0.70959 0.11357 0.91807 0.12896 1.0835 0.17268 1.1946 0.11048 0.70959 0.10551 0.81733 0.070201 0.90658 0.062413 0.9667 Present 484 435 397 393 484 526 546 545 484 593 615 649 BMU History 0 274 398 521 0 303 502 577 0 326 507 569 Shared 0 157 174 243 0 245 402 436 0 326 507 569 Quantization Error History Current chunk 0 1.1426 2.3581 1.1681 2.1766 1.3377 2.5064 1.3793 0 1.1426 1.4863 1.4542 1.6577 1.1557 1.8495 0.92327 0 1.1426 1.3588 1.4247 1.4937 1.7444 1.6426 1.8763 Table 6.3: Results obtained of the approaches tested for Environment I. Some records, regarding the behavior of the use of the BMUs on the maps during the training phases, have also been added to our result tables in order to analyse the usage and organization of the mappings formed by the different approaches. The column BMU-Present defines the total number of nodes which ended up becoming BMUs for the corresponding inputs. The column BMUHistory specifies the number of BMUs which have been used to map the old data chunks. The column BMU-Shared represents the number of BMUs which have been shared in the SOM to map both the most current and old data chunks. Taking the results of Table 6.5 for discussion, since the settings of the environment from which they have been derived are closer to a real-life environment, it can be noted that our approach (second approach) keeps using less numbers of nodes than the third approach (theoretically the best approach since the map 162 Approach First Second Third Phase 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 RMS Error 1-itemsets 2-itemsets 0.10288 0.63257 7.8216 7.5049 10.973 10.156 15.424 14.1 14.226 12.628 16.862 14.779 17.325 14.748 14.449 12.382 0.10288 0.63257 0.13455 0.80306 0.14465 0.93041 0.14651 1.0392 0.15682 1.1311 0.16921 1.2214 0.17049 1.2807 0.1813 1.342 0.10288 0.63257 0.11048 0.70959 0.062258 0.7758 0.10551 0.81733 0.076753 0.8352 0.070201 0.90658 0.10098 0.93237 0.062413 0.9667 Present 325 319 296 317 291 281 286 247 325 448 439 445 489 478 493 497 325 484 529 593 613 615 651 649 BMU History 0 189 310 372 365 395 468 482 0 261 339 402 526 555 586 613 0 264 377 465 556 559 604 610 Shared 0 106 165 196 107 151 145 161 0 221 258 288 398 379 405 416 0 264 377 465 556 559 604 610 Quantization Error History Current chunk 0 0.89182 2.0268 0.88066 2.3137 0.92952 2.5348 0.89395 2.3648 1.0259 2.5568 1.0009 2.6447 1.0125 2.7223 1.0748 0 0.89182 1.2076 1.2426 1.4575 0.9156 1.6167 0.72646 1.6942 0.66307 1.8229 0.57516 1.8978 0.54156 1.98787 0.49316 0 0.89182 1.1315 1.1536 1.2825 1.3432 1.392 1.3908 1.4305 1.6856 1.5432 1.7477 1.6021 1.8078 1.6702 1.9172 Table 6.4: Results obtained of the approaches tested for Environment II. 163 Approach First Second Third Phase 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 RMS Error 1-itemsets 2-itemsets 0.091058 0.42505 5.1857 5.0553 11.573 10.909 16.689 15.39 13.922 12.775 13.733 12.242 16.585 14.597 15.307 12.985 15.566 13.375 0.091058 0.42505 0.16707 0.76713 0.23229 0.8775 0.26526 1.0197 0.25449 1.0779 0.21993 1.1506 0.22947 1.2419 0.24148 1.2712 0.27288 1.3688 0.091058 0.42505 0.10355 0.71514 0.1079 0.73557 0.061232 0.79409 0.077156 0.79983 0.078521 0.84692 0.070201 0.90658 0.10605 0.92097 0.062413 0.9667 Present 189 364 243 223 312 284 258 322 238 189 430 435 416 440 472 462 471 469 189 435 500 513 606 619 615 628 649 BMU History 0 107 209 337 358 390 404 458 466 0 161 343 406 430 495 527 529 582 0 126 367 427 483 576 573 578 624 Shared 0 67 109 165 182 128 141 172 148 0 151 275 285 294 359 357 356 381 0 126 367 427 483 576 573 578 624 Quantization Error History Current chunk 0 0.53331 1.8826 0.99159 2.3294 0.79141 2.4964 0.67263 2.4863 0.90183 2.3275 1.0423 2.5412 0.92153 2.6158 1.1486 2.79 0.87699 0 0.53331 1.0576 1.1491 1.397 0.8237 1.535 0.62643 1.6422 0.71565 1.7038 0.66279 1.8217 0.52506 1.8959 0.61216 1.9895 0.41183 0 0.53331 1.1129 1.1107 1.2155 1.2916 1.3085 1.3741 1.3954 1.423 1.4519 1.7477 1.5472 1.7535 1.6186 1.8095 1.6792 1.9126 Table 6.5: Results obtained of the approaches tested for Environment III. 164 is always built with the whole database) for building the mapping for the data chunks during the environment. The latter can be understood as a good feature for our approach since we part from the fact that the mapping formed by these two approaches (Bincremental-SOM and Allchunks-SOM) give similar results for itemset-support recalls by involving a different number of nodes in the answer. Furthermore, as depicted in Figure 6.20, which shows the runtime for the evaluated methods for the experiment regarding the non-stationary environment with unfixed data chunks, our batch incremental converges by consuming the same or less time than the best approach (Allchunks-SOM) during the learning of the non-stationary environment. 80 Allchucks−SOM Chuck−SOM BIncremental−SOM 70 Time (sec) 60 50 40 30 20 10 1 2 3 4 5 Phase 6 7 8 9 Figure 6.20: Runtime for the three approaches evaluated on the environment III. A contradictory event, regarding the mapping of the old data chunks, seems to occur with our approach since results show that it tends to gradually forget the reference of the past due to employing an incremental procedure. For instance, maps formed after phase five need more BMUs than the ones obtained in its training for producing the best mapping for the past chunks. Even if the 165 latter could be seen as a negative feature, the actual fact is that it is a positive one, because the map gives more attention to the mapping of the latest chunk rather than the historical ones; therefore, the recall will be influenced by the latest tendencies of the changes rather than the very old ones. As a consequence of the latter, we can also observe that the number of shared BMUs between the past and present data at each phase does not tend to agree, as in the third case in which the sharing is complete. Therefore, we can assume that some BMUs need to be combined with others in our approach for some reduction on the recall error (RMS). Nevertheless, promoting such a node fusion could result in updating some nodes (vectors), which already satisfactorily map the most current state of the database. In conclusion, it can be stated that a tradeoff has to be made between a better generalization and a better quantization of the data chunks. 6.4 Conclusions In the previous chapters we have proposed how two different neural networks can be used for ARM by building itemset-support memories based on them. Since these artificial memories may be learning from non-stationary environments, we have stated that the update of the knowledge in them is necessary and relevant for the development of the neural-based framework for ARM. Hence, in this chapter we have looked at tackling the maintenance of the knowledge in a memory based on a self-organising map. To achieve our goal, we have investigated how to perform such an update in the memory throughout time by the usage of incremental training. Since the concept of batch training has been important for the definition of a method for the recall of itemset-support from a SOM, and knowing that this type of training has 166 been stated as unsuitable for non-stationary environments, we have proposed an incremental training method for SOM in batch called Bincremental-SOM which allows the new associations to be learnt without forgetting the previous knowledge. To evaluate the accuracy of the proposed method, we set up experiments which simulate some non-stationary situations with the information contained in a dataset used for FIM studies. To compare the results given by our approach, we also trained two other maps with the latest state and all of the partial states occurring in the environment. It can be concluded from our results, shown above, that a SOM-based memory is able to keep its knowledge updated by performing its training with our method. It was also observed that the quantization generated by the SOM trained with our approach, for the newest group of inserts in the environment, is even better than the one produced with the map trained only with the corresponding data chunk. Therefore, it can be stated that the hypotheses generated, which in this case represent itemset-support estimations, will be produced with the latest itemset tendencies in the environment, but without forgetting the past ones. 167 Chapter 7 Conclusions and Future Work In this thesis, we have looked at the suitability of ANNs (Artificial Neural Networks) for the descriptive data-mining task of ARM (Association Rule Mining). In particular, we have followed the premise that a neural-based framework can be built for tasks like association rule mining because concepts, which are all implicated in the generation of these rules, such as association, frequency, counting, and information storage, are also involved in the learning process performed by humans which artificial neural networks target to imitate. In order to begin with the development of such a neural-based framework and considering the importance of itemset support for the generation of this type of rules, we have focused on the development of the memory stage. This stage is in charge of learning, storing, maintaining, and recalling of the property of itemset support defined by the learnt associations, in order to supply with a frequencystatistical (support) information about the itemsets to other stages in the framework; for instance, processes controlling the generation of rules (in control of the FIM logic). In the quest for the most suitable neural network to become an itemset168 support (pattern-frequency) memory, we have studied two neural networks: a self-organising map and an auto-associative memory. Our studies have focused on determining if these two ANNs have the ability to learn frequency-statistical information from the taught associations in order to make estimations about the support of the itemsets of an environment. In other words, we have proposed how these two neural networks can reproduce the values which normally result from the counting of discrete patterns describing associations (itemsets). Additionally, since data often describe events occurring in a non-stationary environment, we have also proposed how the itemset-support knowledge embedded in the weight matrix, generated by a self-organising map, can be updated while the environment changes throughout time. Therefore, in order to complete this thesis, the final conclusions , of the work proposed, are presented in this chapter. Answers to our research questions stated in Section 1.4 are also given. Moreover, we will establish some links to other pieces of research and guidelines for future work which can contribute to the continuation with (1) the quest for an understanding of the counting of patterns with artificial neural networks, and (2) the development of an ANN-based framework for ARM or similar tasks. 7.1 Final Results Neural network technology has been successfully used to tackle problems aiming prediction and clustering. Nevertheless, their use for tasks like ARM is still unclear and uncertain because the field lacks of research on this topic. For this reason, we have studied the usage of ANNs for ARM. 169 After we established that ARM is a mechanical process whose realisation involves some of the concepts presented in the learning process of biological systems (Gardner-Medwin and Barlow, 2001), and motivated for imitating the human behavior in the generation of association rules with ANNs, we have conducted research on the suitability of ANNs for ARM in this thesis. In particular, we have followed the hypothesis that the embedded knowledge formed by an ANN, as a result of its training with some environment defining associations, can be used for the generation of association rules mainly because: • We have created the baseline for the generation of such symbolic representations describing associations from the knowledge of two neural networks. This is, association rules, for instance, like those in Figure 7.1, can be generated in future from the knowledge embedded in our studied ANNs because we have proposed in this thesis mechanisms, through which the support of itemsets, being the raw material needed for the rules, can be estimated from the weight matrix of a CMM and a SOM. Nevertheless, as stated later, the definition of the logic of the rule generation mechanism remains open. • We have pointed out that both technologies, ANNs and ARM, handle the concept of association in one form or another to form knowledge from data. • We have stated all the different alternatives in which a neural network can participate within the current ARM framework. • We have categorized ARM as a mechanical inference task which can be performed satisfactorily by humans through the counting and the making of associations among the existing elements of data. Therefore, it was 170 assumed that ANNs can be used for ARM only if they are able to reproduce the pattern counting done by their biological counterparts. Figure 7.1: Example of association rules which can be further generated from the knowledge learnt by a neural network about the Mushroom dataset defined in (D.J. Newman and Merz, 1998). All these rules, describing the associativity among the attributes of a dataset, have the format of: if (list of items or attributes) then (list of items or attributes) with [support=% and confidence=%]. A detailed explanation about the links between ARM and ANNs has been given in Section 1.2 as a part of the motivations for this work. In order to gain some insight into the use of ANNs for ARM, we have also 171 proposed the idea that the current ARM framework could be transformed into a framework whose core can be represented by a neural network architecture (defined in Chapter 3). As a first stage of the neural framework for ARM, we have proposed to build an ANN-based artificial memory, which can learn, store, maintain and recall knowledge about the occurrence of patterns in the training dataset to support other stages of the framework with this knowledge. In other words, we have looked at building an ANN-based memory which can provide a good estimation about itemset support when it is queried. Our research has been carried out principally to establish extraction mechanisms through which an auto-associatory memory (correlation matrix memory) and a self-organising map can infer itemset support from their knowledge formed after their training. Our main aim has involved reproducing the results given by a traditional pattern counting process such as, for instance, the itemset-support values given by the Apriori algorithm in an association rule mining process, by using only the knowledge embedded in the mapping or weight matrix of our two ANN candidates. We have used the Apriori algorithm for comparison, because, as commented in previous sections of the thesis, Apriori0 s itemset-support calculations are errorless since it counts directly the itemset occurrences within the data. Moreover, it is the gold standard and historically most important algorithm for ARM, from which many other algorithms have been derived in one way or another. In other words, approximation methods, like our estimations here produced, should be compared to the real values, like the ones generated by Apriori. Discovering and emulating counting abilities with our ANN candidates has been established as relevant for the success of this thesis for the following rea- 172 sons: 1. We identified the relevance of itemset support for the support-confidence framework in ARM and therefore for the generation of rules. Hence, we stated that in order to produce rules from ANNs, it is first necessary to reproduce itemset support from them since it is the metric to determine the frequent itemsets from which rules can be derived. 2. The achievement and viability to build itemset-support memories is directly related to the counting of patterns with ANNs. Moreover, the reproduction of counting with ANNs is challenging as well as important since it takes part in the learning process performed by biological systems. 3. In order to have a mainly neural-based framework for ARM or similar tasks, we defined that the first stage would incorporate an ANN, acting as a memory, because it conveys autonomy to the framework in the sense that its hypotheses or rules will depend exclusively on the knowledge accumulated by the artificial neurons. The latter is an activity occurring within the brain, which stores knowledge (experiences) throughout time to be employed for different purposes. Moreover, we have also defined that the existence of a mapping (feature knowledge space) formed with the associations would make the framework be independent from retaining the learnt environment to provide answers after learning it. The latter is a drawback presented in most of the related work summarised in Chapter 3. Since the counting of patterns as well as the learning of an environment can occur under totally dynamic conditions (non-stationary), we also looked at proposing how to prepare the knowledge in the memory for these environments. Therefore, we have proposed a method to update the knowledge embedded in a 173 self-organising map which learns from an environment with dynamic conditions. Because our work involves reproducing counting properties on two wellknown neural models rather than theoretical ones, our work can also be understood as a continuation of theoretical work in (Gardner-Medwin and Barlow, 2001). Moreover, our approaches have taken into account the combinatorics of patterns described by the associations among their elements. It is important to state that our work does not claim that our interpretations, of how the counting of patterns is performed by the two neural networks studied here, happen in biological systems. Nevertheless, they serve to form a better understanding of what may be happening in the brain. Specific conclusions of each piece of research can be summarized as follows: Auto-Associative Memory Model (Chapter 4) In this chapter, it has been studied whether a weighted auto-associative memory, based on a CMM (Correlation Matrix Memory), could represent our desired itemset-support memory. This memory was investigated because it exploits the concept of association among the input patterns to learn about its environment. As a result of our analysis, it was found that a weighted m-by-m correlation matrix memory, in which m defines the number of items or attributes forming the patterns, has the natural ability to learn statistical-frequency about the discrete patterns used for its training. After discovering such ability, we also pointed out that, because of the symmetry of its weight matrix, a CMM only needs around half of the elements to represent the different learnt occurrence frequencies in the training data. Since it has been advised in (Gardner-Medwin and Barlow, 2001) that it is convenient 174 for the counting of patterns to have direct representations, one-to-one relationships, between patterns (itemsets) and neurons rather than distributed ones, we focused on looking for those direct representations on the m2 nodes defined by this memory. It was satisfactorily possible to identify one-to-one relationships between nodes and some thought associations or itemsets. We concluded that frequency values and therefore itemset support can be calculated straight for m(m + 1)/2 patterns out of 2m patterns that idealistically we wanted this memory to represent (if each node maps a different pattern). In particular, we have stated that the support values for the groups of 1- and 2-itemsets given by this memory are errorless as the values given by the well-known Apriori algorithm which scans and counts the patterns directly in the original data to calculate their itemset support. In the case of the remainder k-itemsets, for 3 ≥ k ≤ m, we looked at defining a distributed method to infer their frequencies needed for support calculations. Hence, two methods, A and B, have been proposed to estimate support (supp) ˆ by using the distributed knowledge embedded within m(m + 1)/2 nodes of the memory which defines directly single- and pairwise-event frequencies. In the definition of both methods, it was assumed that the elements, defining a pattern or itemset, are independent; therefore, an estimation about its support can be generated through the calculation of the joint probability among the elements xi Q of an itemset X, such that supp(X) ˆ = supp(xi ). xi ∈X In our method A, we proposed that the support of a k-itemset can be estimated by using the k individual item probabilities (values defining support in the main diagonal of the matrix) as the resources for the estimation. In the method B, we estimated itemset support by first forming approximately (k/2) pairwise 175 events (2-itemsets), whose support values are directly defined within the matrix, with the elements of the queried k-itemset. Therefore, in this case, we have employed elements lying off the main diagonal for itemset-support estimation. After testing our proposed methods with real-life datasets used in frequent itemset mining benchmarks, it was noticed that the itemset-support estimations made with the knowledge embedded in the weight matrix of this memory and our two methods, present discrepancies with the real value support calculated by the Apriori algorithm. Overlaps of itemset support, occurring among the nodes forming this memory, have been blamed to be the cause of the support differences. Thus, we strongly believe that better estimations can be obtained iff the neurons of this memory are arranged in a different way. Self-Organising Map Model (Chapter 5) In contrast with Chapter 4, in which a supervised trained ANN was studied, we focused here on using an unsupervised trained neural network for our needs. In particular, the suitability of a SOM (Self-Organising Map) to become our itemset-support memory was investigated. This work involved analysing SOM training procedure, in order to identify how its embedded knowledge, about the high-dimensional input data, can be interpreted for producing itemset-support estimations. The idea of estimating itemset support directly from the trained map was developed because other approaches (Changchien and Lu, 2001; Shangming Yang, 2004), which have also explored the use of this ANN for ARM, suffer from the drawback of limiting SOM usage to only cluster the data in order to form links between the original data and the SOM clusters for ARM. Moreover, we wanted to evaluate the abil- 176 ity of a SOM to reproduce the counting of patterns of a high-dimensional space with the coded information about it distributed in the two-dimensional space described by its neurons. To define our mechanism for itemset support estimation, the following concepts were important: • The identification of the best matching units because they control the update of the map during training. • The prior probability associated to each node because it defines the number of times that a node has been hit by input patterns. • The components of the codewords or reference vectors associated with the BMUs because each of them represents a random variable whose value is an approximation of the expected value, or mean, of the represented item. • The concept of batch training. • The concept of the partition of events which has been stated to take part in the training of a SOM. • The concept of independence among the items. Thus, the corresponding support of an itemset has been calculated by finding the joint probability among the corresponding variables. The estimations derived from a trained SOM via our proposed mechanism were also tested against their counterparts provided by the Apriori algorithm. As we have also stated the SOM to be a representative of the theoretical projection model defined in (Gardner-Medwin and Barlow, 2001), it was investigated to produce representations among input patterns and neurons which cannot 177 be derived directly, as in the case of the auto-associative memory, but distributed. In other words, our mechanism uses the information of the reference vectors associated to the BMUs to estimate itemset support. In our experiments, we also analysed other factors which can affect the accuracy of the SOM estimations. For instance, we performed some experiments, involving different values of the radius and map layouts, to determine the best parameters for SOM for ARM. According to our results, we have concluded that rectangular SOMs provide more accurate estimations than the hexagonal maps. The best estimations, in terms of the radius utilized, were given by the map formed with the minimum value; therefore, we have concluded that the best scenario will be reached if the SOM algorithm is reduced as a pure Vector Quantization algorithm in which influences among the nodes are not carried out. It is important to state that the best estimations also occur when the initialisation of the maps were done linearly, which is not a characteristic in the biological systems; hence, to be more congruent with what happens in real life, it can be stated that a value, such as, for instance, 0.5 for the radius, which controls the dispersion of the influence during training, is enough to have satisfactory values of itemset support. Since we wanted to improve the results obtained by a SOM, we investigated likewise if the concept of emergent feature maps, proposed by Ultsch (Ultsch, 1999), could be used for this aim. Our results with this concept, which involves creating large maps for mining tasks, showed an improvement in the itemsetsupport estimations. The reason for this improvement was that since there were more nodes in the memory, the input patterns had a better way to get distributed along the map. Thus, it has been concluded that among a better distribution of 178 the input associations in the map, a better estimation will be produced certainly. Comparing The Proposed Memories By summarizing in Table 7.1 some of the characteristics of the studied neural networks, conclusions regarding their use for ARM, involving a training data with n transactions derived from an itemset space defined by m items, can be drawn as follows: Characteristic Total nodes in the network Type of training Epochs for training Nodes updated for learning Nodes utilized for support recalling Pattern representation for support recalling Main usage AAM-based Memory m2 supervised 1 k2 1, k or k/2 direct(for k < 3), distributed(for k > 2 ) pattern association pattern matching SOM-based Memory √ 5 n (Vesanto et al., 2000) unsupervised several 1 √ mb ≤ 5 n always distributed clustering and data visualisation Table 7.1: Characteristics of the two itemset-support memories based on the studied neural networks. m defines the total number of items from which the n transactions or itemsets of a training data D are derived. mb corresponds to the number of BMUs formed in an epoch training. In the case of an AAM-based memory, the maximum number of nodes needed to define this neural network depends directly on the m number of items forming the training data, rather than, as a SOM-based memory, on the n number of itemsets in the training dataset D. The training of an AAM-based memory is supervised, while the training of a SOM-based memory is unsupervised. When they are both learning from an environment, an input k-itemset provokes k 2 nodes in an AAM-based memory to be updated, while only one node is directly hit by the pattern in the case of 179 a SOM-based memory; nevertheless, in practise, this node has to propagate its influence to the rest of the map. Therefore, it can be stated that the presence of an input itemset updates all nodes forming the SOM-based memory. The training of the AAM-based memory requires only one pass over the environment to collect knowledge, while in the case of the SOM-based memory, even if it can produce estimations after the first training epoch, it needs more passes to converge and therefore provides more accurate results. Once these memories have been trained, recalling or estimation of support for a queried k-itemset from a AAM-based memory requires: one node for the 1and 2-itemsets since direct pattern representations were found and defined, and k or k/2 nodes when k > 2 through our proposed methods which use individualor paired-itemset probabilities respectively. In the case of a SOM-based memory, since the knowledge has been distributed along the BMU in the map, a mb number of nodes is used from which k components of their reference vectors are taken into account for an estimation. Based on the results shown in Tables 4.6 and 4.5 in Chapter 4, and 5.8 in Chapter 5, which have been summarised here in Tables 7.2 and 7.3, it can be concluded that even if our AAM-based memory gives errorless itemset support for the group of the 1- and 2-itemsets, its performance is beaten generally by its SOM-based counterpart whose estimations are made depending on the good distribution of the association within its nodes. Nevertheless, in terms of the steadiness of the itemset-support estimations, the AAM-based memory has an advantages over a SOM-based memory since the latter depends on initial factors for the distribution of its knowledge and therefore for the itemset-support recalls. 180 Itemset Size k 3 4 5 6 7 SOM Hexagonal Rectangular 0.42918 0.3668 0.52008 0.4352 0.61963 0.50871 0.76948 0.62251 1.0274 0.81303 CMM Method A Method B 0.617510 0.47519 0.775860 0.50515 0.912940 0.86483 1.089600 0.9105 1.394400 1.4274 Itemsets Tested 167 203 128 39 4 Table 7.2: Comparison of the generalised errors between our approaches for SOM and CMM for the different number of tested itemsets, which resulted a query from the Chess dataset with a minimal support between 90 and 100 %. Itemset Size k 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 SOM Hexagonal Rectangular 0.92081 0.89465 1.0748 1.0476 1.2055 1.1724 1.327 1.2829 1.4407 1.3794 1.5395 1.4548 1.6126 1.4983 1.6496 1.4996 1.648 1.4559 1.6202 1.3801 1.5947 1.3037 1.6045 1.2653 1.6755 1.2926 1.8418 1.4084 2.0514 1.5868 CMM Method A Method B 2.9141 2.285 3.593 2.6836 4.2165 3.6522 4.8524 3.995 5.5205 5.0014 6.2197 5.2905 6.9495 6.494 7.7125 6.6983 8.5192 8.1773 9.3857 8.3329 10.32 10.098 11.317 10.258 12.416 12.252 13.865 12.791 14.903 14.751 Itemsets per group 4985 25500 88170 217705 397947 550220 581647 471908 293209 138294 48473 12023 1896 152 5 Table 7.3: Comparison of the generalised errors between our approaches for SOM and CMM for the different number of tested itemsets, which resulted a query from the Chess dataset with a minimal support between 45 and 100 %. Maintaining Pattern Frequency Throughout Time with a SOM (Chapter 6) Since the knowledge of a SOM-based memory can become invalid as a result of changes occurring in the training environment, we have looked at proposing a mechanism to maintain it. In particular, we tackled the problem of training a SOM incrementally in batch because: 181 • In order to generate association rules from the weight matrix of a trained SOM, the procedure to estimate itemset-support assumes that its has been in batch. • Batch mechanisms have not been considered suitable for non-stationary environments. Therefore, we have proposed an incremental batch mechanism for SOM which performs by using the new data occurring in the environment and knowledge generated from the last state of the SOM-based memory because it represents the old data states in the environment. To test our approach, we set a non-stationary environment artificially with the associations of the Chess dataset by forming data chunks to represent the different phases of it. We compared the itemset-support estimations made by a SOM-based memory trained with our approach to results given by memories trained with two traditional approaches which involve the retraining of SOM with all data available and making estimations with the latest occurred data chuck in the environment. Our experiments have shown that, with our method, the update of the map throughout time is possible. Even though we were not able to improve the results given by the memory, which was always trained with all data in the environment, interesting properties were identified in our results as follows: • It was noticed that the error describing the quality of the itemset-support recalls generally grows linearly. • The quantization of the latest data chunk in the environment produced by a memory trained with our approach is better than the one formed with a 182 memory which just learns that data chunk. This positive property of our method is because we have exploited the re-use of knowledge. • Since a SOM trained with our approach is better at quantizing the present data chunks than the old ones, it has been stated that the rules derived from this trained SOM will positively reflect the latest itemset tendencies rather than the old ones. Therefore, we have concluded that batch training mechanisms should be considered as a real alternative for the update of a SOM for non-stationary environments. Moreover, we have moved another step forward in the development of the neural-based framework for ARM since we have given the property of learning incrementally to the artificial memory. The latter is important and necessary because it represents the basis to perform incremental ARM (Cheung et al., 1996b) whose aim is to maintain the rules while the environment changes. Answering Our Research Questions With the conclusions drawn about this thesis above, our research questions, as stated in the introduction, can be answered as follows: • Can they be used for descriptive-data-mining techniques in which the aim is to represent the data in form of association rules? Yes, they can. One manner to describe data is by producing association rules from data. To form such rules it is first necessary to determine the basic but important itemset property of support. With the results of the experiments conducted in this thesis on two different neural networks, it can then be concluded that this itemset property can be estimated from the knowledge embedded in their weight matrix. 183 Therefore, in order to produce association rules from data, the next step would be to define a FIM logic ( a version of the Apriori algorithm to mine the feature space defined by the weight matrix formed by our neural networks) which would aim to traverse the structure defined in the ANN in the best way. • Would the knowledge learnt by an ANN be useful for describing associations among the elements of a database? Yes, it would be useful to describe associations among the elements of events from an environment. With this work, we have proven that the support value associated to those associations or itemsets can be either calculated directly from the weight matrix for some cases (the case of an AAMbased memory), or estimated indirectly through the use of the methods proposed here, which decode the distributed knowledge in order to form itemset-support estimations. • Could it be possible that our chosen neural networks can have some knowledge describing the frequency of patterns distributed along their nodes? I.e., Could the results of counting be generated from our ANN candidates? Yes, our chosen ANN candidates have shown that they count patterns implicity while they learn them. This has been concluded since we discovered that they keep the knowledge needed to reproduce values, defining the occurrence of patterns in a group of events, which are normally obtained by scanning the events of the original environment for the count of patterns. Nevertheless, it has also been found out that the organization of the learnt patterns within the networks can produce discrepancies between the estimation and the real frequency of the patterns. Therefore, new internal organizations should be investigated in future. 184 • Could it be possible to have an itemset-support memory based on an ANN, as a substitute for the original database, to take part in such framework? The reason for retaining the original data in the ARM framework is because it is the only source from which the itemset support can be calculated throughout time. Nevertheless, our results show that a trained neural network is able to reproduce those values sufficiently well. Therefore, it can be stated that these ANNs, acting as memories which can learn, retain and recall knowledge about itemset support, can take the role of the original data source in the ARM framework. • Could it continue accumulating knowledge of the pattern frequency throughout time while the original data environment changes? Yes, it could. The counting of patterns under a non-stationary environment has been investigated with a self-organising map, and the results have shown that knowledge embedded in map can be updated while the original environment adds new events to its definition. Additionally, results of our experiments have also highlighted a positive property on the recalls made by a SOM since they tend to be highly influenced by the latest state in the environment rather that the old ones. 7.1.1 Contributions • We have investigated the usage of ANNs for the transduction task of association rule mining by initially stating that the generation of rules can be done by humans and we have aimed to reproduce this human behavior with ANNs. In specific, we have investigated if association rules can be formed from the knowledge formed by two ANNs: an auto-associative 185 memory and a self-organising map. • We have worked on making the use of ANN for ARM possible by developing itemset support memories based on an auto-associative memory and a self-organising map which are able to reproduce an estimation of the itemset support after they have learnt associations describing an environment. • We have reproduced the counting of patterns with the knowledge embedded in the ANNs rather than using the original dataset. In other words, we have studied and proposed methods, decoding the created weight matrix, through which the two neural networks studied can reproduce the counting of patterns, which is traditionally done by scanning the original data. • We have contributed to the work on the counting of patterns with neural networks, since we have studied two well-known neural networks rather than forming theoretical ones. Moreover, our studies have included the combinatoric aspect of patterns which in this case is represented by the term of itemsets. • Since data is dynamic, we have also proposed how a memory based on a self-organising map can keep its knowledge updated while the original environment changes as a result of the incorporation of new patterns. 7.2 Future Work Based on the results, findings, and conclusions drawn by this thesis, we define the future work in this section , organised into different research paths, that can be pursued for the development of the topic. 186 7.2.1 For The Auto-associativity-based Memory It has been found that overlaps create differences in the estimation of support since all patterns are piled up together in the same number of nodes. It happens that while this ANN is learning, input patterns, such as, for instance, X and Y defined respectively by 0011 and 1011 are stored in the same group of m2 nodes; therefore, when the memory is queried to recall support for the pattern of 1010, the estimation produces a difference with the real value because of the pattern overlaps. To tackle this drawback, we can consider that patterns with similar prefix can be grouped together in different branches of a neural network tree structure. Nevertheless, this way of solving the problem will cross with some of the work in the ARM field which has already proposed their split or organization into tree- or trie-based structures for their counting. Therefore, we propose to look for other alternatives to decode the knowledge of this ANN, or combining it with other neural networks, to provide more accurate results for itemsets whose size is large than 2 items. We have also discovered that the mapping, formed by the auto-associative memory based on a correlation matrix memory, results in a data structure which is similar to the matrix structure proposed in (Agarwal et al., 2001) to perform frequent itemset mining by using lexicographic tree structures. These triangular matrices have also been used by other approaches, for instance in (Bodon, 2003), for the specific discovery of the 2-itemsets because of its compact memory representation. 7.2.2 For The Self-organising-map-based Memory Our experiments and results have shown that the itemset-support estimations made by this ANN are directly influenced by the way in which the knowledge, about the input associations, is spread in the map. Therefore, a better organisation of the knowledge has to be found and applied in order to improve the results 187 given by this thesis. One way of doing it is by exploring the use of a more convenient distance metric for binary data in the training of this ANN because this metric is responsible for allocating the input patterns to the nodes of the map during training. To begin with this research path, a revision of the work proposed in (Leisch et al., 1998; Fernando Lourenco and Bacao, 2004) is necessary since other metrics, different from the traditional Euclidean distance, have been investigated for the formation of the map. Moreover, the study of the work of Lebbah et al. (Lebbah et al., 2000), in which a batch version of the Kohonen SOM algorithm dedicated to binary data is proposed, is important. Another alternative is to propose a new distance metric from scratch which we believe should consider the following: Such a metric should produce formations of triangular binary matrices at each BMU with the input itemsets. Therefore, it will be a metric which compares the training itemsets based not only on their item differences but also in the hierarchial property of them. These matrices aim to organise the knowledge of the SOM in a manner that a more accurate itemset-support estimations can be produced. Some examples of these matrices are given in Figure 7.2. Once these matrices have been formed and the corresponding codebook of the map calculated during training, the estimation of itemset support should be obtained as follows: supp(X) ˆ = mb X P (mj ) ∗ minsupp(xi ∈ X) (7.1) j=1 In which the probability of the event, representing an association or an itemset X, is defined by the minimal support among the items xi belonging to X rather 188 Figure 7.2: Triangular binary formations formed with the itemset search space formed with 3 and 4 items. than by the product defined by the joint probability of them which was defined in Chapter 5 and re-state here in Equation 7.2. It is important to note that the prior probability P(mj ) of the mb number of BMUs will still be necessary for itemset-support estimation. supp(X) ˆ = mb X (P (mj ) j=1 Y xi ) (7.2) xi ∈X Equation 7.1 can be also seen as a simple optimization of the current proposed estimation method of itemset support, because it requires less number of operations to make an estimation. Moreover, the quality of its estimations generated can be expected to be better than our current results since input patterns are better organised in the map. For instance, let Mjt be a binary triangular matrix allocated at the BMU mj representing input itemsets involving four items (a,b,c,d). Because of the hierarchy existing among the input itemsets used to build Mjt , which will be summarised by the reference vector formed in mj , an 189 ordering, based on the support property, appears in the items of the matrix such that it is defined by either a ≥ b ≥ c ≥ d or a ≤ b ≤ c ≤ d. Therefore, if the map is queried to recall the support of an itemset, for instance X = abc, the answer, representing itemset-support estimation of X, would involve determining the support of either a or c depending on the ordering formed in the BMUs of the map. Since we have noticed that the quality of the estimations made by this neural network is highly dependent on its initial state and the radius utilised for its training, it can be thought of evaluating approaches which cope with this instability property of the SOM which produces under- or over-representation of the training data. Therefore, we propose to investigate alternative initilisation schemes for SOM, for instance those proposed in (Su et al., 1999; Salem et al., 2003; Attik et al., 2005), in order to improve the results given by a random initilised SOM and to avoid the use of linear initialisation which has the inconvenience that requires a lot of computations for the initial weights when the input data is large. In this thesis, we have initially explored the generation of association rules known as categorical association rules because they only describe facts refereing to the associations defined among items without considering any quantitative aspect of the variables they represent. To overcome this drawback of the categorical rules, another type of rules, known as quantitative association rules (Srikant and Agrawal, 1996; Wijsen and Meersman, 1998; Aumann and Lindell, 2003), have been already proposed for the description of data. Since the formation of these rules involves the usage of statistics to define them, it can be assumed that SOM also has the natural ability to generate them since the information about the 190 distribution of the variables learnt is coded in the nodes. Therefore, the development of mechanisms for this type of rules is needed. One way of tackling this aim is by considering the work in (Giraudel and Lek, 2001) in which the statistical properties of SOM have been investigated, or by taking into consideration the use of a SOM as a probability density estimator for classification problems (Yin and Allinson, 2001). Hence, this idea would involve to approximate the distributions of the variables modeled by the input data in order to determine how strongly or weakly they participate in the original environment and therefore in the future rules. When it has been demonstrated that quantitative association rules can be generated from the knowledge embedded in a trained SOM, it can be considered to explore the work of Lebbah et al. (Lebbah et al., 2005) in which the creation of a mapping for the analysis of mixed (numerical and binary) data is proposed. This proposal may be useful for the generation of mixed association rules which will be composed by both quantitative and qualitative components. Taking into account that the formation of rules from a SOM is produced by decoding the information of its codebook or mapping, which represents a summary of the input data formed by the ability of SOM for vector quantization, it can be concluded that the final vectors in the map can be understood as a group of representatives of the itemset patterns existing in the training environment. Hence, it is possible to create a link between the generation of association rules from a neural network and the work proposed by Yan et al. (Yan et al., 2005a), which examines how to summarise a collection of itemsets using a finite number of K representatives. In our case, this wanted itemset summary will be given by the vectors created at each BMUs in the trained map. 191 Since the calculation of itemset support and the generation of rules throughout time implies developing mechanism for non-stationary environments, the idea of using self-growing approaches derived from the traditional SOM, which control the size of the map during training, can be considered for studying if these can be trained in batch mode. To begin with this idea, the work of (Alahakoon et al., 2000b) should be analysed. 7.2.3 For the Quality of the Itemset-support Estimation One important aspect, regarding the itemset-support estimations made by any neural network, is to define a manner in which the quality of its answers (estimations) can be measured. This is, although in this thesis we have used the RMS error for measuring the quality of the estimations, it has been necessary to know the real support values to perform such an error calculation; therefore, we propose that it should be investigated a manner on how to measure such approximations or estimations with considering that the real value is totally unknown. 7.2.4 ANNs-based Candidate Generation Procedures Another topic which is worth investigating is whether itemset support can be inferred or predicted with neural networks. Itemset-support inference is a topic which has not been studied extensively in the ARM field. For instance, an algorithm called PASCAL has been proposed in (Bastide et al., 2000) as an optimization of the Apriori algorithm based on counting inference. This is, some candidates during the mining process derive their support from the frequent itemsets already discovered rather than performing the typical counting over the original dataset. Although this approach reduces the number of candidates by inferring the support for some of them, it can still happen that the rest of them, whose sup192 port need to be checked by the algorithms, may become unfrequent. Therefore, it is important to count with candidate procedures which return exactly those itemsets that will certainly turn out frequent in order to not waste time checking fake candidates. Moreover, candidate generation is an important topic in ARM since it is necessary to determine how it influences the complexity of the discovery of frequent itemsets. Based on the above described and knowing a priori that the support of itemsets forms a function f (x) as in Figure 7.3, in which x represents itemsets from a search space in a lexical order, we propose to investigate if an approximation of that function could be generated with a neural network by interpolating a series of n points, representing some itemsets from the space with their corresponding support. To approach this goal, we can start from the fact that a negative correlation exists between the size of the itemsets and their corresponding support. In other words, the support tends to decrease while the number of items in a pattern increases. Hence, we propose to train an RBF (Radial Basis Functions) neural networks (Bishop, 1995; Buhmann, 2003) in order to investigate if a candidate procedure can be defined with it. This goal can also be understood as a task which aims for the prediction of itemset-support having as input knowledge some information about the total itemset search space. 7.2.5 Distributed Association Rule Mining The collection of data is often realized in distributed manner in real life. This is, databases, representing and collecting huge amounts of local data, are set along different sites in order to manage them in a strategic and satisfactory manner. 193 100 80 80 S u p p o r t (%) S u p p o r t (%) 100 60 40 20 60 40 20 0 0 75 2775 2−i t e m s e t s 100 100 80 80 S u p p o r t (%) S u p p o r t (%) 1−i t e m s e t s 60 40 20 60 40 20 0 67525 0 3−i t e m s e t s 1215450 4−i t e m s e t s Figure 7.3: Function generated with the support of some groups of itemsets derived from the Chess dataset. Nevertheless, as the need of analysing them as a grand total arises, the development of distributed mining algorithms become crucial. In ARM, this problem has been introduced in (Cheung et al., 1996a). In order to tackle distributed ARM, a distributed database has been treated as a large database formed by partitions representing each of the local sites in the system. The common problem that approaches on this category have had to deal with refers to the amount of data that should be transmitted amongst the sites to achieve the generation of rules. For instance, as a first attempt, a solution could involve moving either the original data partition or the corresponding set of the frequent itemsets at each site; however, both solutions result in producing an overutilisation of the channel because they can be large and contain data which will turn out to be irrelevant. 194 Figure 7.4: Incremental SOM-based approach for distributed ARM. The rules will be generated from the latest trained SOM. Based on the above defined, it can be thought of extending our work as follows: As a first approach, depicted in Figure 7.4, it can be investigated the usage of a SOM moving from site to site and learning incrementally as defined in Chapter 6. This is, a SOM would be learning the local data distribution at each site until it reaches a site in which the generation of rules will be performed by using the mechanism proposed in Chapter 5. Since new tendencies can occur in the different sites, it will be important if the approach integrates a self-growing node structure rather than a traditional SOM which makes use of a fixed structure. A second approach, given n Figure 7.5, would involve having local SOMs at each site to learn the local data distributions, which will then be queried remotely from, or moved to, a central location in which ARM will take place. 195 Figure 7.5: Local SOM-based approach for distributed ARM. While the local maps are queried remotely in the model on the left, the trained maps are transmitted to be the source from rules will be generated in the model on the right. It is also important to notice that the development of distributed algorithms has a strong relationship with parallel-based algorithms (Zaki, 1999); therefore, a SOM can be stated to be a strong candidate for this type of mining tasks since its training has already been performed in parallel based on its batch mode (Lawrence et al., 1999b; Porrmann et al., 2003). 7.2.6 The Itemset Concept in Dynamic Data ARM over Data Streams In this thesis, we have started gaining some inside into the topic of generating association rules from neural networks. Nevertheless, not all data is static in real life as it is defined by the traditional ARM. This is, data, like data streams, can 196 arrive indefinitely and continuously; therefore, its mining becomes a complex task since its availability often occurs in short periods of time. Because research involving data streams is still at an early stage in the field of ARM (Jiang and Gruenwald, 2006), we suggest that it should be investigated if a neural network is able to perform such a mining task. The idea, which is depicted in Figure 7.6, consists in developing a two-stage proposal. In other words, the data stream will first be learnt in a way similar way to that used with the static data in this thesis, and then rules will be derived from the ANN. Figure 7.6: A neural-based approach for ARM for data streams. The analysis of this type of data has also been considered with approaches based on ANNs. For instance, the usage of a self-organising map and an ART has been considered in (Laerhoven et al., 2001; Laerhoven, 2001) and (Rajaraman and Tan, 2001). As this type of data comes constantly, we could consider using the incremental proposal made for SOM in this thesis for its learning. Nevertheless, these data can change so fast that there may not be a time to buffer it. Hence, the 197 sequential training of SOM should be considered necessary. The problem with that sequential training of SOM is that the identification of the BMUs is not as direct and easy as in the batch training since the map is updated every time that a new pattern is presented. One way of identifying these nodes is by projecting the training patterns into the map once the training has been finished. However, as the data is not kept, this option is not feasible. Hence, one first problem would be to find a method to keep track of the BMUs while the training is done in order to use our itemset-support estimation method for the generation of association rules. Sequential Pattern Mining The task of sequential pattern mining, introduced in (Agrawal and Srikant, 1995), is similar to the traditional problem of ARM; nevertheless, its definition considers the concept of time in the generation of knowledge. This is, the transactions, representing the input data, are defined by an Id costumer, an itemset, and a time stamp. Alike ARM, the aim is to identify interesting patterns; however, in this particular case, the target is to discover sequences, which are ordered lists of itemsets, satisfying a minimum support constraint from the given data. Since in this thesis, we have defined how to estimate itemset support from two trained ANNs, our approaches can be used as a basis for a proposal for this type of descriptive data-mining task. However, the main issue to be tackled is the way in which the input data will be presented to the ANN. In other words, a transformation method for the data is needed in order to form the corresponding vectorial patterns required to feed the neural network. 198 Appendix A The Apriori Algorithm Since we have used the well-known Apriori Algorithm to evaluate and compare the itemset support estimations derived from our ANN candidates through our proposals, such an algorithm will be defined in this section. The Apriori Algorithm was independently proposed in (Agrawal and Srikant, 1994; Mannila et al., 1994). This algorithm performs the generation of association rules by following the stages established in the confidence-support framework (Agrawal et al., 1993). This is, first, the frequent itemsets are generated, and then the corresponding association rules are formed from such frequent itemsets. Apriori tackles the complexity of the discovery of frequent itemsets by performing: • The identification of only those itemsets whose support satisfy a minimum threshold. • The formation of new itemsets, known as candidates, by joining the already discovered frequent itemsets. This is, candidates representing the set of itemsets with length k are derived from a combination of the set of the frequent k-1-itemsets. 199 • The calculation of the candidate support by counting the number of occurrences of the itemset in turn directly from the high dimensional space generated by the input data. • The traversal of the itemset space in a BFS (Breadth-First Search) way. • The pruning of fake candidates by considering the anti-monotonic property of itemset support: All subsets of a frequent itemset must be also frequent (Agrawal and Srikant, 1994). A complete specification of the algorithm is given in Figure A.1. Figure A.1: The Apriori algorithm. This figure was extracted from the original paper of Agrawal (Agrawal and Srikant, 1994). The top pseudocode describes the main steps of Apriori. The bottom SQL query defines the way in which candidates are formed during a mining process. It is important to state that this algorithm is historically important for ARM, because it has influenced the development of new ARM algorithms in one way or another since its introduction in 1994. 200 Appendix B The Neural Network Candidate Algorithms Since two well-known neural network are part of our studies, we define below their corresponding training algorithms employed in this thesis. Auto-Associative Memory: Correlation Matrix Memory A CMM (Correlation Matrix Memory), as shown in Figure B.1, is a single-layer memory whose size depends on the problem to be tackled. For instance, while a grid of n-by-m nodes is used to define a hetero-associative memory, a grid of m-by-m nodes defines an auto-associative memory. Moreover, depending on the format of its inputs, a CMM can have binary (Austin and Stonham, 1987) or real synapses (Kohonen, 1978; Haykin, 1999; Ham and Kostanic, 2001). This supervised memory is trained as follows: 1. A pair of input patterns X,Y is presented to the memory. 2. Its weight matrix is updated by a supervised Hebbian rule with the information defined by the input pair. 3. The steps 1 and 2 are performed until no more pairs are available. 201 Figure B.1: An Associative memory based on a CMM. In the case of a binary memory, its training is commanded by Equation B.1, while the training of its numeric or weighted counterpart is defined by Equation B.2. wij = m _ yik xjk (B.1) yik xjk (B.2) k=1 wij = m X k=1 The training in Equation B.1 and B.2 is defined respectively by a superposition and a sum function of the m matrices derived from by the m training pairs. The idea behind its training is to accumulate information of the input associations in such a way that when a stimulus is presented, this memory is able to remember the corresponding associated patterns. Self-Organising Map The self-organising map (Kohonen, 1996) is one of the most utilized neural networks for data-mining problems because of its unsupervised property to learn knowledge from data. The typical structure of a SOM is a two-dimensional ar- 202 range of neurons which re-organises itself during training. Each neuron or node has associated a vector which models and summarises input patterns coming from the training data. Its training is based on a search and an update mechanisms. The former is responsible for allocating the input patterns to the neurons which map them the best. Each of these neurons mc are called winners or BMUs (Best Matching Units). One mc satisfies for some inputs x the following condition: ∀i, kx(t) − mc (t)k < kx(t) − mi (t)k. The update mechanism establishes when and how the map will be updated in order to learn the input data. Two modes, sequential and batch, are often used to accomplish such a task (Kohonen, 1996; Kohonen, 1998). Both modes make this ANN converge through an iterative data presentation. Nevertheless, they differ from each other in when the update of the network will take place. This is, while in a sequential or incremental mode the map is updated every time that a new stimulus is presented, in a batch mode the map is modified after all training vectors are propagated. In general, the steps involving a SOM training can be summarised as follows: • Initialisation of the network. This means assigning either random or linear values to the nodes of the map. • The presentation of a new input and the determination of its BMU. This is, a search process, which compares the current input to the nodes in order to discover the best match for it, is needed. • The update of the map. In a sequential mode, this is done by Equation B.3, while Equation B.4 governs the corresponding batch mode. 203 mi (t + 1) = mi (t) + hc(x),i (x(t) − mi (t)) (B.3) P j hji (t)Sj (t) mi (t + 1) = P j nV j (t)hji (t) (B.4) Both equations use a neighborhood or kernel function h(), defined in Equation B.5, to spread the influence of the current input(s) along the map. α is known as the learning-rate factor which holds a value between 0 and 1 and decreases monotonically in the sequential mode. On the other hand, this factor is constant and equal to one in the batch mode. ri and rj (rc ) ∈ <2 define the grid positions of a node receiving and producing stimulation respectively. Ã hc(x),i kri − rj k2 = hji (t) = α(t) exp − 2σ 2 (t) ! (B.5) In the case of Equation B.4, the term Sj , described by Equation B.6, represents the the concentration of nvi input patterns allocated to each node. In other words, it is a sum of all data points contained at some Voronoi region region Vj = {xi | kxi − mj k < kxi − mk k ∀k 6= j} defined by a node mj . Si (t) = nvi X xj (B.6) j=1 This neural network has been employed for a large variety of problem domains, which will not be summarised here, but it can be consulted in (Oja et al., 2003) for instance. Nevertheless, for the case of data mining, a SOM has been an effective neural architecture for tasks like the visualization of high-dimensional data (Vesanto, 1999; Flexer, 1999), classifiers (Yin and Allinson, 2001), the de- 204 scription of data through clustering (Vesanto and Alhoniemi, 2000), the generation of rules through the interpretation of its knowledge (Malone et al., 2006), among others. Furthermore, a SOM has also served as the basis to build more complex neural architectures. For instance, networks whose structure is not only organised, but also grows autonomously during training (Alahakoon et al., 2000a) for the modeling of dynamic data. 205 Bibliography Aaron Ceglar, J. F. R. (2006). Association mining. ACM Computing Surveys, 38(2):1–42. Agarwal, R. C., Aggarwal, C. C., and Prasad, V. V. V. (2001). A tree projection algorithm for generation of frequent item sets. Journal of Parallel and Distributed Computing, 61(3):350–371. Aggarwal, C. C. and Yu, P. S. (1998). A new framework for itemset generation. In PODS, pages 18–24. ACM Press. Agrawal, R., Imielinski, T., and Swami, A. N. (1993). Mining association rules between sets of items in large databases. In Buneman, P. and Jajodia, S., editors, SIGMOD Conference, pages 207–216. ACM Press. Agrawal, R. and Srikant, R. (1994). Fast algorithms for mining association rules in large databases. In Bocca, J. B., Jarke, M., and Zaniolo, C., editors, VLDB, pages 487–499. Morgan Kaufmann. Agrawal, R. and Srikant, R. (1995). Mining sequential patterns. In Yu, P. S. and Chen, A. S. P., editors, Eleventh International Conference on Data Engineering, pages 3–14, Taipei, Taiwan. IEEE Computer Society Press. Alahakoon, D., Halgamuge, S. K., and Srinivasan, B. (2000a). Dynamic self- 206 organizing maps with controlled growth for knowledge discovery. IEEE Transactions on Neural Networks, 11(3):601–614. Alahakoon, D., Halgamuge, S. K., and Srinivasan, B. (2000b). Dynamic selforganizing maps with controlled growth for knowledge discovery. IEEENN, 11(3):601. Alatas, B. and Akin, E. (2006). An efficient genetic algorithm for automated mining of both positive and negative quantitative association rules. Soft Comput, 10(3):230–237. Alhoniemi, E., Himberg, J., and Vesanto, J. (1999). Probabilistic measures for responses of self-organizing map units. In Proc. of International ICSC Congress on Computational Intelligence Methods and Applications (CIMA’99), pages 286–290, Rochester, N.Y., USA. ICSC Academic Press. Andrews, R., Diederich, J., and Tickle, A. B. (1995). Survey and critique of techniques for extracting rules from trained artificical neural networks. Knowledge Based Systems (UK), 8(6):378–389. Aras, N., Altinel, I. K., and Oommen, J. (2003). A kohonen-like decom- position method for the euclidean traveling salesman problem-KNIES/spl I.bar/DECOMPOSE. IEEE-NN, 14:869–890. Attik, M., Bougrain, L., and Alexandre, F. (2005). Self-organizing map initialization. In Duch, W., Kacprzyk, J., Oja, E., and Zadrozny, S., editors, ICANN (1), volume 3696 of Lecture Notes in Computer Science, pages 357– 362. Springer. Aumann, Y. and Lindell, Y. (2003). A statistical theory for quantitative association rules. J. Intell. Inf. Syst, 20(3):255–283. 207 Austin, J. (1995). Distributed associative memories for high speed symbolic reasoning. International Journal on Fuzzy Sets and Systems, 82:223–233. Austin, J. (1996). Associative memories and the application of neural networks to vision. In Handbook of Neural Computation. Institute of Physics and Oxford University Press. Austin, J., Kennedy, J., and Lees, K. (1995). The advanced uncertain reasoning architecture. Weightless Neural Network Workshop. Austin, J. and Stonham, T. (1987). An associative memory for use in image recognition and occlusion analysis. Image and Vision Computing, 5(4):251– 261. Bastide, Y., Taouil, R., Pasquier, N., Stumme, G., and Lakhal, L. (2000). Mining frequent patterns with counting inference. SIGKDD Explorations, 2(2):66– 75. Benitez, J. M., Castro, J. L., and Requena, I. (1997). Are artificial neural networks black boxes? IEEE Transactions on Neural Networks, 8(5):1156– 1164. Bishop, C. M. (1995). Neural networks for Pattern Recognition. Oxford University Press. Bodon, F. (2003). A fast apriori implementation. In Goethals, B. and Zaki, M. J., editors, Proceedings of the IEEE ICDM Workshop on Frequent Itemset Mining Implementations (FIMI’03), volume 90 of CEUR Workshop Proceedings, Melbourne, Florida, USA. Bodon, F. (2006). A survey on frequent itemset mining. Technical report, Budapest University of Technology and Economics. 208 Borgelt, C. (2003). Efficient implementations of apriori and eclat. Brin, S., Motwani, R., and Silverstein, C. (1997). Beyond market baskets: generalizing association rules to correlations. In SIGMOD ’97: Proceedings of the 1997 ACM SIGMOD international conference on Management of data, pages 265–276, New York, NY, USA. ACM Press. Browne A., Hudson B., W. D. P. P. (2003). Knowledge extraction from neural networks. In Proceedings of the 29th Annual Conference of the IEEE Industrial Electronics Society, Roanoke, Virginia, USA, pages 1909–1913. Buhmann, M. D. (2003). Radial Basis Functions Theory and Implementations. Cambridge Monographs on Applied and Computational Mathematics (No. 12). Cambridge. Burdick, D., Calimlim, M., and Gehrke, J. (2001). Mafia: A maximal frequent itemset algorithm for transactional databases. In Proceedings of the 17th International Conference on Data Engineering, pages 443–452, Washington, DC, USA. IEEE Computer Society. Carpenter, G. A. and Grossberg, S. (1989). Search mechanisms for adaptive resonance theory (ART) architectures. In IEEE International Joint Conference on Neural Networks (3rd IJCNN’89), volume I, pages I–201–I–205, Washington DC. IEEE. Boston U. Changchien, S. W. and Lu, T.-C. (2001). Mining association rules procedure to support on-line recommendation by customers and products fragmentation. Expert Systems with Applications, 20(4):325–335. Cheung, Han, Ng, Fu, and Fu (1996a). A fast distributed algorithm for mining association rules. In PDIS: International Conference on Parallel and 209 Distributed Information Systems. IEEE Computer Society Technical Committee on Data Engineering, and ACM SIGMOD. Cheung, D. W.-L., Han, J., Ng, V., and Wong, C. Y. (1996b). Maintenance of discovered association rules in large databases: An incremental updating technique. In ICDE, pages 106–114. Cios, K. (2000). Data Mining Methods for Knowledge Discovery. Kluwer Academic. Coenen, F., Goulbourne, G., and Leng, P. (2004a). Tree structures for mining association rules. Data Mining and Knowledge Discovery, 8(1):25–51. Coenen, F., Leng, P., and Ahmed, S. (2004b). Data structure for association rule mining: T-trees and p-trees. Knowledge and Data Engineering, IEEE Transactions on, 16(6):774–778. Craven, M. and Shavlik, J. (1999). Rule extraction: Where do we go from here? Craven, M. and Shavlik, J. W. (1993). Learning symbolic rules using artificial neural networks. In ICML, pages 73–80. Craven, M. W. and Shavlik, J. W. (1997). Using neural networks for data mining. Future Generation Computer Systems, 13(2–3):211–229. DeGroot, M. (1975). Probability and Statistics. Addison-Wesley. Devroye, L. (1987). A Course in Density Estimation. Birkhauser, Boston. D.J. Newman, S. Hettich, C. B. and Merz, C. (1998). UCI repository of machine learning databases. Duch, W., Adamczak, R., and Grabczewski, K. (1996). Extraction of logical rules from training data using backpropagation networks. In First Polish Conference on Theory and Applications of Artificial Intelligence. 210 Duda, R. O., Hart, P. E., and Stork, D. G. (2000). Pattern Classification (2nd Edition). Wiley-Interscience. Eggermont, J. (1998). Rule-extraction and learning in the BP-SOM architecture. El-Haji, M. and Zaiane, O. (2003). Inverted matrix: Efficient discovery of frequent items in large datasets in the context of interactive mining. Eom, J.-H. (2006). Neural feature association rule mining for protein interaction prediction. In Wang, J., Yi, Z., Zurada, J. M., Lu, B.-L., and Yin, H., editors, ISNN (2), volume 3973 of Lecture Notes in Computer Science, pages 690– 695. Springer. Eom, J.-H., Chang, J. H., and Zhang, B.-T. (2004). Prediction of implicit proteinprotein interaction by optimal associative feature mining. In Yang, Z. R., Everson, R. M., and Yin, H., editors, IDEAL, volume 3177 of Lecture Notes in Computer Science, pages 85–91. Springer. Eom, J.-H. and Zhang, B.-T. (2004). Adaptive neural network-based clustering of yeast protein-protein interactions. In Das, G. and Gulati, V. P., editors, CIT, volume 3356 of Lecture Notes in Computer Science, pages 49–57. Springer. Eom, J.-H. and Zhang, B.-T. (2005). Prediction of east protein-protein interactions by neural feature association rule. In Duch, W., Kacprzyk, J., Oja, E., and Zadrozny, S., editors, ICANN (2), volume 3697 of Lecture Notes in Computer Science, pages 491–496. Springer. Fernando Lourenco, V. L. and Bacao, F. (2004). Binary-based similarity measures for categorical data and their application in self-organizing maps. In JOCLAD 2004 - XI Jornadas de Classificacao e Analise de Dados. 211 Flexer, A. (1999). On the use of self-organizing maps for clustering and visualization. In Principles of Data Mining and Knowledge Discovery, pages 80–88. Furao, S. and Hasegawa, O. (2004). An incremental neural network for nonstationary unsupervised learning. In ICONIP, pages 641–646. Gaber, J., Bahi, J. M., and El-Ghazawi, T. A. (2000a). Parallel mining of association rules with a hopfield type neural network. In ICTAI, page 90. IEEE Computer Society. Gaber, K., Bahi, J., and El-Ghazawi, T. (2000b). Parallel mining of association rules with a hopfield type neural network. In 12th IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2000), pages 90–93, Vancouver, Canada. Gardner-Medwin, A. R. and Barlow, H. B. (2001). The limits of counting accuracy in distributed neural representations. Neural Comput., 13(3):477–504. Giraudel, J. L. and Lek, S. (2001). A comparison of self-organizing map algorithm and some conventional statistical methods for ecological community ordination. ECOLOGICAL MODELLING, 146(1–3):329–339. Goethals, B. (2002). Efficient Frequent Pattern Mining. PhD thesis, University of Limburg, Belgium. Goethals, B. (2003). Frequent itemset mining implementations repository. Goethals, B. (2004). Memory issues in frequent itemset mining. In Haddad, H., Omicini, A., Wainwright, R. L., and Liebrock, L. M., editors, SAC, pages 530–534. ACM. 212 Goethals, B. and Zaki, M. J., editors (2003). FIMI ’03, Frequent Itemset Mining Implementations, Proceedings of the ICDM 2003 Workshop on Frequent Itemset Mining Implementations, 19 December 2003, Melbourne, Florida, USA, volume 90 of CEUR Workshop Proceedings. CEUR-WS.org. Gouda, K. and Zaki, M. J. (2001). Efficiently mining maximal frequent itemsets. In ICDM, pages 163–170. Grahne, G. and Zhu, J. (2003). ing frequent itemsets. Efficiently using prefix-trees in min- In Proceeding of the First IEEE ICDM Workshop on Frequent Itemset Mining Implementations (FIMI’03). http://www.cs.concordia.ca/db/dbdm/dm.html. Grahne, G. and Zhu, J. (2005). Fast algorithms for frequent itemset mining using fp-trees. IEEE Transactions on Knowledge and Data Engineering, 17(10):1347–1362. Gunopulos, D., Mannila, H., and Saluja, S. (1997). Discovering all most specific sentences by randomized algorithms. In Afrati, F. N. and Kolaitis, P. G., editors, Database Theory - ICDT ’97, 6th International Conference, Delphi, Greece, January 8-10, 1997, Proceedings, volume 1186 of Lecture Notes in Computer Science, pages 215–229. Springer. Gupta, G., Strehl, A., and Ghosh, J. (1999). Distance based clustering of association rules. Ham, F. M. and Kostanic, I. (2001). Principles of neurocomputing for science and engineering. McGraw-Hill. Hammer, B., Rechtien, A., Strickert, M., and Villmann, T. (2002). Rule extraction from self-organizing networks. In Dorronsoro, J. R., editor, 213 ICANN, volume 2415 of Lecture Notes in Computer Science, pages 877– 883. Springer. Han, E.-H., Karypis, G., and Kumar, V. (1997). Scalable parallel data mining for association rules. pages 277–288. Han, J. and Kamber, M. (2000). Data Mining: Concepts and Techniques. Morgan Kaufmann. Han, J., Pei, J., and Yin, Y. (2000a). Mining frequent patterns without candidate generation. In Chen, W., Naughton, J. F., and Bernstein, P. A., editors, SIGMOD Conference, pages 1–12. ACM. Han, J., Pei, J., and Yin, Y. (2000b). Mining frequent patterns without candidate generation. In Chen, W., Naughton, J., and Bernstein, P. A., editors, 2000 ACM SIGMOD Intl. Conference on Management of Data, pages 1–12. ACM Press. Han, J., Pei, J., and Yin, Y. (2000c). Mining frequent patterns without candidate generation. In SIGMOD ’00: Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 1–12, New York, NY, USA. ACM Press. Hand, D., Mannila, H., and Smyth, P. (2001). Principles of Data Mining. The MIT Press, Cambridge, Massachusetts. Haykin, S. (1999). Neural networks: A comprehensive foundation. PrenticeHall, New York. Heskes, T. (2001). Self-organizing maps, vector quantization, and mixture modeling. IEEE-EC, 12:1299–1305. 214 Hilderman, R. and Hamilton, H. (1999). Knowledge discovery and interestingness measures: A survey. Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings National Academy Science USA, 79:2554–2558. Hugh, G. (1997). Discrete Probability. Springer, New York. Hung S., W. S. (2004). A time-based self-organising model for document clustering. In The International Joint Conference on Neural Networks. Jacobsson, H. (2005). Rule extraction from recurrent neural networks: A taxonomy and review. Neural Comput., 17(6):1223–1263. Jiang, N. and Gruenwald, L. (2006). Research issues in data stream association rule mining. SIGMOD Rec., 35(1):14–19. Jin, R. and Agrawal, G. (2002). Shared memory parallelization of data mining algorithms: Techniques. Jolliffe, I. T. (1986). Principal Component Analysis. Series in Statistics. Springer-Verlag. Joshi, M. V., Han, E.-H., Karypis, G., and Kumar, V. (1999). Efficient parallel algorithms for mining associations. In Large-Scale Parallel Data Mining, pages 83–126. Jr., R. J. B., Goethals, B., and Zaki, M. J., editors (2004). FIMI ’04, Proceedings of the IEEE ICDM Workshop on Frequent Itemset Mining Implementations, Brighton, UK, November 1, 2004, volume 126 of CEUR Workshop Proceedings. CEUR-WS.org. 215 Kantardzic, M. (2002). Data Mining: Concepts, Models, Methods, and Algorithms. IEEE Press and John Wiley. Kaski, S., Nikkilä, J., and Kohonen, T. (1998a). Methods for interpreting a selforganized map in data analysis. In ESANN, pages 185–190. Kaski, S., Nikkilä, J., and Kohonen, T. (1998b). Methods for interpreting a self-organized map in data analysis. In Verleysen, M., editor, Proceedings of ESANN’98, 6th European Symposium on Artificial Neural Networks, Bruges, April 22–24, pages 185–190. D-Facto, Brussels, Belgium. Kecman, V. V. (2001). Learning and soft computing: support vector machines, neural networks, and fuzzy logic models. Complex adaptive systems. pubMIT, pub-MIT:adr. Kiang, M. Y. (2001). Extending the kohonen self-organizing map networks for clustering analysis. COMPUTATIONAL STATISTICS & DATA ANALYSIS, 38(2):161–180. Kimball, R. (1996). The Data Warehouse Toolkit: Practical Techniques for Building Dimensional Data Warehouses. John Wiley. Kohonen, T. (1978). Associative memory. Springer-Verlag, Berlin. Kohonen, T. (1996). Self-Organizing Maps. Springer-Verlag. It explains SOM technology. Kohonen, T. (1998). The self-organizing map. Neurocomputing, 21(1-3):1–6. Kohonen, T., Kaski, S., Lagus, K., Salojärvi, J., nad V. Paatero, J. H., and Saarela, A. (2000). Self-organization of a massive document collection. IEEE Transactions on Neural Networks, 11:574–85. 216 Kohonen, T. and Somervuo, P. (1998). Hself-organizing maps of symbol strings. Neurocomputing, 21(1–3):19–30. Kohonen, T. and Somervuo, P. (2002). How to make large self-organizing maps for nonvectorial data. Neural Networks, 15(8–9):945–952. Krishnan, R., Sivakumar, G., and Bhattacharya, P. (1999). A search technique for rule extraction from trained neural networks. Pattern Recognition Letters, 20(3):273–280. Laerhoven, K. V. (2001). Combining the self-organizing map and K-means clustering for on-line classification of sensor data. In Dorffner, G., Bischof, H., and Hornik, K., editors, ICANN, volume 2130 of Lecture Notes in Computer Science, pages 464–469. Springer. Laerhoven, K. V., Aidoo, K. A., and Lowette, S. (2001). Real-time analysis of data from many sensors with neural networks. In ISWC, pages 115–122. IEEE Computer Society. Lawrence, R. D., Almasi, G. S., and Rushmeier, H. E. (1999a). A scalable parallel algorithm for self-organizing maps with applications to sparse data mining problems. Data Min. Knowl. Discov., 3(2):171–195. Lawrence, R. D., Almasi, G. S., and Rushmeier, H. E. (1999b). A scalable parallel algorithm for self-organizing maps with applications to sparse data mining problems. Data Min. Knowl. Discov, 3(2):171–195. Lebbah, M., Badran, F., and Thiria, S. (2000). Topological map for binary data. In ESANN, pages 267–272. Lebbah, M., Chazottes, A., Badran, F., and Thiria, S. (2005). Mixed topological map. In ESANN, pages 357–362. 217 Leisch, F., Weingessel, A., and Dimitriadou, E. (1998). Competitive learning for binary valued data. In Niklasson, L., Bodén, M., and Ziemke, T., editors, Proceedings of the 8th International Conference on Artificial Neural Networks (ICANN 98), volume 2, pages 779–784, Skövde, Sweden. Springer. Liu, J., Pan, Y., Wang, K., and Han, J. (2002). Mining frequent item sets by opportunistic projection. Lu, H. J., Setiono, R., and Liu, H. (1996). Effective data mining using neural networks. Ieee Trans. On Knowledge And Data Engineering, 8:957–961. Malone, J., McGarry, K., Wermter, S., and Bowerman, C. (2006). Data mining using rule extraction from kohonen self-organising maps. Neural Computing and Applications, 15(1):9–17. Mannila, H. and Toivonen, H. (1997). Levelwise search and borders of theories in knowledge discovery. Data Mining and Knowledge Discovery, 1(3):241– 258. Mannila, H., Toivonen, H., and Verkamo, A. I. (1994). Efficient algorithms for discovering association rules. In Fayyad, U. M. and Uthurusamy, R., editors, AAAI Workshop on Knowledge Discovery in Databases (KDD-94), pages 181–192, Seattle, Washington. AAAI Press. McGarry, K. J., Wermter, S., and MacIntyre, J. (1999). Knowledge extraction from radial basis function networks and multi-layer perceptrons. In IEEE International Conference on Neural Networks (IJCNN’99), volume IV, pages 2494–2497, Washington DC. IEEE. Meo, R. (2003). Replacing support in association rule mining. Technical Report RT70-2003, Universita degli Studi di Torino. 218 Mitra, S., Pal, S. K., and Mitra, P. (2002). Data mining in soft computing framework: a survey. IEEE-EC, 13:3–14. Oja, M., Kaski, S., and Kohonen, T. (2003). Bibliography of self-organizing map (som) papers: 1998-2001 addendum. Neural Computing Surveys, 3:1–156. O’Keefe, S. (1995). Neural Networks for FAX Image Analysis. PhD thesis, University of York. Omiecinski, E. (2003). Alternative interest measures for mining associations in databases. IEEE Trans. Knowl. Data Eng., 15(1):57–69. Pasquier, N., Bastide, Y., Taouil, R., and Lakhal, L. (1998). Pruning closed itemset lattices for association rules. Proceedings of the BDA French Conference on Advanced Databases, October 1998. To appear. Pasquier, N., Bastide, Y., Taouil, R., and Lakhal, L. (1999). Discovering frequent closed itemsets for association rules. Lecture Notes in Computer Science, 1540:398–416. Pei, J., Han, J., and Mao, R. (2000). CLOSET: An efficient algorithm for mining frequent closed itemsets. In ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery, pages 21–30. Piatetsky-Shapiro, G. (2000). Knowledge discovery in databases: 10 years after. hardcopy. Piatetsky-Shapiro, G. and Frawley, W. J., editors (1991). Knowledge Discovery in Databases. AAAI/MIT Press. Porrmann, M., Witkowski, U., and Ruckert, U. (2003). A massively parallel architecture for self-organizing feature maps. IEEE-NN, 14:1110–1121. 219 Rácz, B., Bodon, F., and Schmidt-Thieme, L. (2005). Benchmarking frequent itemset mining algorithms: from measurement to analysis. In Goethals, B., Nijssen, S., and Zaki, M. J., editors, Proceedings of ACM SIGKDD International Workshop on Open Source Data Mining (OSDM’05), pages 36–45, Chicago, IL, USA. Rajaraman, K. and Tan, A.-H. (2001). Topic detection, tracking, and trend analysis using self-organizing neural networks. In Cheung, D. W.-L., Williams, G. J., and Li, Q., editors, PAKDD, volume 2035 of Lecture Notes in Computer Science, pages 102–107. Springer. Roberto J. Bayardo, J. (1998). Efficiently mining long patterns from databases. SIGMOD Rec., 27(2):85–93. Salem, A.-B. M., Syiam, M. M., and Ayad, A. F. (2003). Improving selforganizing feature map (sofm) training algorithm using k-means initialization. In ICEIS (2), pages 399–405. Sallans, B. (1997). Data mining for association rules with unsupervised neural networks: Csc final project. Setiono, R. (2000). Extracting ¡em¿M¡/em¿-of-¡em¿N¡/em¿ rules from trained neural networks. IEEE-NN, 11(2):512. Shangming Yang, Y. Z. (2004). Self-organizing feature map based data mining. volume 3173, pages 193–198. Shenoy, P., Haritsa, J. R., Sundarshan, S., Bhalotia, G., Bawa, M., and Shah, D. (2000). Turbo-charging vertical mining of large databases. pages 22–33. Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis. Chapman and Hall, New York. 220 Song, H.-H. and Lee, S.-W. (1998). A self-organizing neural tree for large-set pattern classification. Neural Networks, IEEE Transactions on, 9(3):369– 380. SPSS (1968). Clementine. Srikant, R. and Agrawal, R. (1996). Mining quantitative association rules in large relational tables. In Jagadish, H. V. and Mumick, I. S., editors, Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data, pages 1–12, Montreal, Quebec, Canada. Su, M.-C. and Chang, H.-T. (2001). A new model of self-organizing neural networks and its application in data projection. IEEE-NN, 12:153–158. Su, M.-C., Liu, T.-K., and Chang, H.-T. (1999). An efficient initialization scheme for the self-organizing feature map algorithm. In IEEE International Conference on Neural Networks (IJCNN’99), volume III, pages 1906–1910, Washington DC. IEEE. Taha, I. A. and Ghosh, J. (1999). Symbolic interpretation of artificial neural networks. Knowledge and Data Engineering, 11(3):448–463. Tan, P.-N., Kumar, V., and Srivastava, J. (2002). Selecting the right interestingness measure for association patterns. In KDD ’02: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 32–41, New York, NY, USA. ACM Press. Tickle, A. B., Andrews, R., Golea, M., and Diederich, J. (1998). The truth will come to light: Directions and challenges in extracting the knowledge embedded within trained artificial neural networks. IEEE-NN, 9(6):1057. Toivonen, H. (1996). Sampling large databases for association rules. In Vijayaraman, T. M., Buchmann, A. P., Mohan, C., and Sarda, N. L., editors, 221 In Proc. 1996 Int. Conf. Very Large Data Bases, pages 134–145. Morgan Kaufman. Towell, G. G. and Shavlik, J. W. (1993). Extracting refined rules from knowledge-based neural networks. Machine Learning, 13:71–101. Tsukimoto, H. (2000). Extracting rules from trained neural networks. IEEE-NN, 11(2):377. Ultsch, A. (1999). Data mining and knowledge discovery with emergent selforganizing feature maps for multivariate time series. Ultsch, A. and Siemon, H. P. (1990). Kohonen’s self organizing feature maps for exploratory data analysis. In INNC Paris 90, pages 305–308. Universit”at Dortmund. Vapnik, V. (1998). Statistical Learning Theory. Wiley. Vázquez, J. M., Macı́as, J. L. Á., and Santos, J. C. R. (2002). Discovering numeric association rules via evolutionary algorithm. In Cheng, M.-S., Yu, P. S., and Liu, B., editors, PAKDD, volume 2336 of Lecture Notes in Computer Science, pages 40–51. Springer. Veloso, A. (2003). New parallel algorithms for frequent itemset mining in large databases. Vesanto, J. (1999). SOM-based data visualization methods. Intelligent-DataAnalysis, 3:111–26. Vesanto, J. and Ahola, J. (1999). Hunting for correlations in data using the selforganizing map. In Proc. of International ICSC Congress on Computational Intelligence Methods and Applications (CIMA’99), Rochester, New York, USA, June 22–25, pages 279–285. ICSC Academic Press. 222 Vesanto, J. and Alhoniemi, E. (2000). Clustering of the self-organizing map. IEEE Transactions on Neural Networks, 11(3):586–600. Vesanto, J., Himberg, J., Alhoniemi, E., and Parhankangas, J. (1999). Selforganizing map in matlab: the SOM toolbox. In Proc. of Matlab DSP Conference 1999, Espoo, Finland, November 16–17, pages 35–40. Vesanto, J., Himberg, J., Alhoniemi, E., and Parhankangas, J. (2000). SOM toolbox for matlab 5. Technical report. Wijsen, J. and Meersman, R. (1998). On the complexity of mining quantitative association rules. Data Min. Knowl. Discov, 2(3):263–281. Woon, Y.-K., Ng, W.-K., and Lim, E.-P. (2004). A support-ordered trie for fast frequent itemset discovery. IEEE Transactions on Knowledge and Data Engineering, 16(7):875–879. Yan, X., Cheng, H., Han, J., and Xin, D. (2005a). Summarizing itemset patterns: a profile-based approach. In Grossman, R., Bayardo, R., and Bennett, K. P., editors, KDD, pages 314–323. ACM. Yan, X., Zhang, C., and Zhang, S. (2005b). Armga: Identifying interesting association rules with genetic algorithms. Applied Artificial Intelligence, 19(7):677–689. Yianilos (1993). Data structures and algorithms for nearest neighbor search in general metric spaces. In SODA: ACM-SIAM Symposium on Discrete Algorithms (A Conference on Theoretical and Experimental Analysis of Discrete Algorithms). Yin, H. and Allinson, N. M. (2001). Self-organizing mixture networks for probability density estimation. IEEE-NN, 12:405–411. 223 Zaki, M. J. (1999). Parallel and distributed association mining: A survey. IEEE Concurrency, 7(4):14–25. Zaki, M. J. (2000). Scalable algorithms for association mining. IEEE Trans. Knowl. Data Eng, 12(2):372–390. Zaki, M. J. and Hsiao, C.-J. (2002). Charm: An efficient algorithm for closed itemset mining. In Grossman, R. L., Han, J., Kumar, V., Mannila, H., and Motwani, R., editors, SDM. SIAM. Zaki, M. J., Parthasarathy, S., Li, W., and Ogihara, M. (1996). Evaluation of sampling for data mining of association rules. Technical Report TR617. Zaki, M. J., Parthasarathy, S., Ogihara, M., and Li, W. (1997a). New algorithms for fast discovery of association rules. In Heckerman, D., Mannila, H., Pregibon, D., Uthurusamy, R., and Park, M., editors, In 3rd Intl. Conf. on Knowledge Discovery and Data Mining, pages 283–296. AAAI Press. Zaki, M. J., Parthasarathy, S., Ogihara, M., and Li, W. (1997b). Parallel algorithms for discovery of association rules. Data Mining and Knowledge Discovery, 1(4):343–373. Zhang (2000). Neural networks for classification: A survey. IEEETSMC: IEEE Transactions on Systems, Man, and Cybernetics, 30:451–462. 224