Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Proceedings of The Workshop on Mining Complex Patterns Editors Annalisa Appice (University of Bari, Italy) Michelangelo Ceci (University of Bari, Italy) Corrado Loglisci (University of Bari, Italy) Giuseppe Manco (ICAR-CNR, Italy) Preface The International Workshop on Mining Complex Patterns (MCP 2011) was held in Mondello (Palermo), Italy, on September 17th 2011 in conjunction with AI*IA 2011: the 12th International Conference of Italian Association for Artificial Intelligence (AI*IA 2011). During the last two decades, studies in Machine Learning have paved the way to the definition of efficient and stable data mining and knowledge discovery algorithms. Data mining and knowledge discovery can today be considered as stable fields with numerous efficient algorithms and studies that have been proposed to extract knowledge in different forms from data. Although, most existing data mining approaches look for patterns in tabular data (which are typically obtained from relational databases), algorithmic extensions are recently investigated to massive datasets representing complex interactions between several entities from a variety of sources. These interactions may be spanned at multiple levels of granularity as well as at spatial and temporal dimensions. Our purpose in this workshop was to bring together researchers and practitioners of data mining interested in methods and applications where complex patterns in expressive languages are extracted from text/hypertext data, networks and graphs, event or log data, biological sequences, spatio-temporal data, sensor data and streams, and so on. Twelve contributions were originally submitted, of which seven were accepted for the oral presentation. Each submission was evaluated by three independent referees. Besides paper presentations, the scientific programme also featured an invited talk by Sašo Džeroski (Department of Knowledge Technologies, Jozef Stefan Institute, Ljubljana, Slovenia). We would like to thank the invited speaker, all the authors who submitted papers and all the workshop participants. We are also grateful to members of the program committee for their thorough work in reviewing submitted contributions with expertise and patience, and the members of AI*IA. Mondello, September 2011 Annalisa Appice Michelangelo Ceci Corrado Loglisci Giuseppe Manco Organization Workshop Chairs Annalisa Appice Università degli Studi di Bari Aldo Moro webpage: http://www.di.uniba.it/~appice/ mailto: [email protected] Michelangelo Ceci Università degli Studi di Bari Aldo Moro webpage: http://www.di.uniba.it/~ceci/ mailto: [email protected] Corrado Loglisci Università degli Studi di Bari Aldo Moro webpage: http://www.di.uniba.it/~loglisci/ mailto: [email protected] Giuseppe Manco Institute for High Performance Computing and Networks Italian National Research Council, Rende (CS) webpage: http://www.icar.cnr.it/manco mailto: [email protected] Program Committee Fabrizio Angiulli (Università della Calabria) Tania Cerquitelli (Politecnico di Torino) Saso Dzeroski (Jozef Stefan Institute) Nicola Fanizzi (Università degli Studi di Bari "Aldo Moro") Stefano Ferilli (Università degli Studi di Bari "Aldo Moro") Joao Gama (University of Porto) Elio Masciari (Institute for High Performance Computing and Networks Italian National Research Council, Rende (CS)) Rosa Meo (Università degli Studi di Torino) Andrea Passerini (Università di Trento) Zbigniew W. Ras (University of North Carolina and Warsaw University of Technology) Chiara Renso (KDD Lab, Pisa) Fabrizio Riguzzi (Università di Ferrara) Alessandro Sperduti (Università degli studi di Padova) Franco Turini (Università di Pisa) Alfonso Urso (Institute for High Performance Computing and Networks Italian National Research Council, Palermo) Table of Contents An Ontology of Data Mining ..................................................................................... Sašo Džeroski 1 Cooperating Techniques for Extracting Conceptual Taxonomies from Text............. Stefano Ferilli, Fabio Leuzzi and Fulvio Rotella 2 PatTexSum: A Pattern-based Text Summarizer......................................................... Elena Baralis, Luca Cagliero, Alessandro Fiori and Saima Jabeen 14 An Expectation Maximization Algorithm for Probabilistic Logic Programs............. Elena Bellodi and Fabrizio Riguzzi 26 Clustering XML Documents by Structure: a Hierarchical Approach......................... Gianni Costa, Giuseppe Manco, Riccardo Ortale, Ettore Ritacco 38 Outlier Detection For XML Documents..................................................................... Giuseppe Manco and Elio Masciari 46 P2P support for OWL-S discovery............................................................................. Domenico Redavid and Stefano Ferilli and Floriana Esposito 54 Marine Traffic Engineering through Relational Data Mining.................................... Antonio Bruno and Annalisa Appice 66 An Ontology of Data Mining Sašo Džeroski Jožef Stefan Institute, Department of Knowledge Technologies, Ljubljana, Slovenia [email protected] Abstract. We have developed OntoDM[2][1][3], an ontology of the scientific domain of data mining aimed at describing data mining investigations. It represents entities such as data, data mining tasks and algorithms, and generalizations (output by the latter). In contrast to other ontologies of data mining, OntoDM is a deep ontology, general purpose rather than tailor made, and compliant to best practices in ontology engineering. OntoDM allows us to cover a large part of the diversity in data mining research, including recently developed approaches to mining structured data and constraint-based data mining. The talk will describe the OntoDM ontology and how standard and more recent data mining approaches are represented within it. Two use cases will be described, one of which concerns QSAR modeling in drug design investigations. References 1. P. Panov, S. Džeroski, and L. Soldatova. Ontodm: An ontology of data mining. In Proceedings of the 2008 IEEE International Conference on Data Mining Workshops, pages 752–760, Washington, DC, USA, 2008. IEEE Computer Society. 2. P. Panov, L. Soldatova, and S. Džeroski. Representing entities in the ontodm data mining ontology. In S. Džeroski, B. Goethals, and P. Panov, editors, Inductive Databases and Constraint-Based Data Mining, pages 27–55. Springer, 2010. 3. P. Panov, L. N. Soldatova, and S. Džeroski. Towards an ontology of data mining investigations. In J. Gama, V. S. Costa, A. M. Jorge, and P. Brazdil, editors, Discovery Science, volume 5808 of Lecture Notes in Computer Science, pages 257– 271. Springer, 2009. 1 Cooperating Techniques for Extracting Conceptual Taxonomies from Text S. Ferilli1,2 , F. Leuzzi1 , and F. Rotella1 1 2 Dipartimento di Informatica – Università di Bari [email protected] {fabio.leuzzi, rotella.fulvio}@gmail.com Centro Interdipartimentale per la Logica e sue Applicazioni – Università di Bari Abstract. The current abundance of electronic documents requires automatic techniques that support the users in understanding their content and extracting useful information. To this aim, it is important to have conceptual taxonomies that express common sense and implicit relationships among concepts. This work proposes a mix of several techniques that are brought to cooperation for learning them automatically. Although the work is at a preliminary stage, interesting initial results suggest to go on extending and improving the approach. 1 Introduction The spread of electronic documents and document repositories has generated the need for automatic techniques to understand and handle the documents content, in order to help the user in satisfying his information needs without being overwhelmed by the huge amount of available data. Since most of these data are in textual form, and since text explicitly refers to concepts, most work has focussed on Natural Language Processing (NLP). Obtaining automatically Full Text Understanding is not trivial, due to the intrinsic ambiguity of natural language and to the huge amount of required common sense and linguistic/conceptual background knowledge. Nevertheless, even small portions of such a knowledge may significantly improve understanding performance, at least in limited domains. Although standard tools, techniques and representation formalisms are still missing, lexical and/or conceptual taxonomies can provide a useful support to many NLP tasks. Unfortunately, manually building this kind of resources is very costly and error-prone, which is a strong motivation towards automatic construction of conceptual networks by mining large amounts of documents in natural language. This work aims at partially simulating some human abilities in this field, such as extracting the concepts expressed in given texts and assessing their relevance; obtaining a practical description of the concepts underlying the terms, which in turn would allow to generalize concepts having similar descriptions; and applying some kind of reasoning ‘by association’, that looks for possible indirect connections between two identified concepts. The system will take in input texts in natural language, and process them to build a conceptual network that supports the above objectives. The resulting network can be visualized by means of a suitable interface and translated into a First-Order Logic 2 (FOL) formalism, to allow the subsequent exploitation of logic inference engines in applications that use that knowledge. Our proposal consists in a mix of existing tools and techniques, that are brought to cooperation in order to reach the above objectives, extended and supported by novel techniques when needed. The next section briefly recalls related work. Then, Section 3 describes the mixed approach, and discusses the novel parts in more detail. A preliminary evaluation of the proposal is reported in Section 4, while Section 5 concludes the paper and outlines future work issues. 2 Related Work Many works exist aimed at building taxonomies and ontologies from text. A few examples: [10, 9] build ontologies from natural language text by labelling the taxonomical relations only, while we label also non-taxonomical ones with actions (verbs); [14] builds a taxonomy considering only concepts that are present in a domain but do not appear in others, while we are interested in all recognized concepts independently of their being generic or domain-specific; [13] defines a language to build formal ontologies, while we are interested in the lexical level. As regards our proposal, a first functionality that we needed is syntactic analysis of the input text. We exploited the Stanford Parser and Stanford Dependencies [7, 1], two very effective tools that can identify the most likely syntactic structure of sentences (including active and passive forms), and label their components as ‘subject’ or ‘(direct/indirect) object’. Moreover, they normalize the words in the input text using lemmatization instead of stemming, which allows to distinguish their grammatical role and is more comfortable to read by humans. We also exploited the Weka project [5], that provides a set of tools to carry out several learning and Data Mining (DM) tasks, including clustering, classification, regression, discovery of association rules and visualization. Another technique that inspired our work is the one described in [8] to semiautomatically extract a domain-specific ontology from free text, without using external resources but focussing the analysis on Hub Words (i.e., words having high frequency). After building the ontology, the adaptation of a Hub Word t is ranked according to its ‘Hub Weight’: W (t) = αw0 + βn + γ n X w(ti ) i=1 where w0 is a given initial weight, n is the number of relationships in which t is involved, w(ti ) is the tf ∗idf weight of the i-th word related to t, and α+β+γ = 1. A task aimed at identifying most important words in a text, to be used as main concepts for inclusion in the taxonomy, is Keyword Extraction (KE). Among the several proposals available in the literature, we selected two techniques that can work on single documents (rather than requiring a whole corpus) and are based on different and complementary approaches, so that they can together provide an added value. The quantitative approach in [12] is based on 3 the assumption that the relevance of a term in a document is proportional to how frequently it co-occurs with a subset of most frequent terms in that document. The χ2 statistic is exploited to check whether the co-occurrences establish a significant deviation from chance. To improve orthogonality, the reference frequent terms are preliminarily grouped exploiting similarity-based clustering (using similar distribution of co-occurrence with other terms) and pairwise clustering (based on frequent co-occurrences). The qualitative approach in [3], based on WordNet [2] and its extension WordNet Domains [11], focusses on the meaning of terms instead of their frequency and determines keywords as terms associated to the concepts referring to the main subject domain discussed in the text. It exploits a density measure that determines how much a term is related to different concepts (in case of polysemy), how much a concept is associated to a given domain, and how relevant a domain is for a text. Lastly, we need in some steps of our technique to assess the similarity among concepts in a given conceptual taxonomy. A classical, general measure, is the Hamming distance [6], that works on pairs of equal-lenght vectorial descriptions and counts the minimum number of changes required to turn one into the other. Other measures, specific for conceptual taxonomies, are sf F a [4] (that adopts a global approach based on the whole set of hypernyms) and sf W P [16] (that focuses on a particular path between the nodes to be compared). 3 Proposed Approach In the following, we will assume that each term in the text corresponds to an underlying concept (phrases can be preliminarily extracted using suitable techniques, and handled as single terms). A concept is described by a set of characterizing attributes and/or by the concepts that interact with it in the world described by the corpus. The outcome is a graph, where nodes are the concepts recognized in the text, and edges represent the relationships among these nodes, expressed by verbs in the text (whose direction denotes their role in the relationship). This can be interpreted as a semantic network. 3.1 Identification of Relevant Concepts The input text is preliminarily processed by the Stanford Parser in order to extract the syntactic structure of the sentences that make it up. In particular, we are interested only in (active or passive) sentences of the form subject-verb(direct/indirect)complement, from which we extract the corresponding triples hsubject, verb, complement i that will provide the concepts (the subject s and complement s) and attributes (verbs) for the taxonomy. Indirect complements are treated as direct ones, by embedding the corresponding preposition into the verb: e.g., to put, to put on and to put across are considered as three different verbs, and sentence John puts on a hat returns the triple hJohn,put on,hati, in which John and hat are concepts associated to attribute put on, indicating that John can put on something, while a hat can be put on). Triples/sentences involving 4 verb ‘to be’ or nouns with adjectives provide immediate hints to build the subclass structure in the taxonomy: for instance, “The dog is a domestic animal...” yields the relationships is a(dog, animal) and is a(domestic animal,animal). The whole set of triples is represented in a Concepts×Attributes matrix V that recalls the classical Terms×Documents Vector Space Model (VSM) [15]. The matrix is filled according to the following scheme (resembling tf · idf ): fi,j |A| Vi,j = P · log |{j : ci ∈ aj }| k fk,j where: – fP i,j is the frequency of the i-th concept co-occurring with the j-th attribute; – k fk,j is the sum of the co-occurrences of all concepts with the j-th attribute; – A is the entire set of attributes; – |{j : ci ∈ aj }| is the number of attributes with which the concept ci cooccurrs (i.e., for which fi,j 6= 0). Its values represent the term frequency tf, as an indicator of the relevance of the term in the text at hand (no idf is considered, to allow the incremental addition of new texts without the need of recomputing this statistic). A clustering step (typical in Text Mining) can be performed on V to identify groups of elements having similar features (i.e., involved in the same verbal relationships). The underlying idea is that concepts belonging to the same cluster should share some semantics. For instance, if concepts dog, John, bear, meal, cow all share attributes eat, sleep, drink, run, they might be sufficiently close to each other to fall in the same cluster, indicating a possible underlying semantic (indeed, they are all animals). Since the number of clusters to be found is not known in advance, we exploit the EM clustering approach provided by Weka based on the Euclidean distance applied row vectors representing concepts in V. Then, the application on the input texts of various Keyword Extraction techniques, based on different (and complementary) aspects, perspectives and theoretical principles, allows to identify relevant concepts. We use the quantitative approach based on co-occurrences kc [12], the qualitative one based on WordNet kw [3] and a psychological one based on word positions kp . The psychological approach is novel, and is based on the consideration that humans tend to place relevant terms/concepts toward the start and end of sentences and discourses, where the attention of the reader/listener is higher. In our approach, the chance of a term being a keyword is assigned simply according to its position in the sentence/discourse, according to a mixture model determined by mixing two Gaussian curves whose peaks are placed around the extremes of the portion of text to be examined. The information about concepts and attributes is exploited to compute a Relevance Weight W (·) for each node in the network. Then, nodes are ranked by decreasing Relevance Weight, and a suitable cutpoint in the ranking is determined to distinguish relevant concepts from irrelevant ones. We cut the list at 5 the first item ck in the ranking such that: W (ck ) − W (ck+1 ) ≥ p · max (W (ci ) − W (ci+1 )) i=0,...,n−1 i.e., the difference in relevance weight from the next item is greater or equal than the maximum difference between all pairs of adjacent items, smoothed by a user-defined parameter p ∈ [0, 1]. Computation of Relevance Weight Identifying key concepts in a text is more complex than just identifying keywords. Inspired to the Hub Words approach, we compute for each extracted concept a Relevance Weight expressing its importance in the extracted network, by combining different values associated to different perspectives: given a node/concept c, P w(c) k(c) e(c) dM − d(c) (c,c) w(c) W (c) = α +ǫ +β +γ +δ maxc w(c) maxc e(c) e(c) dM maxc k(c) where α, β, γ, δ, ǫ are weights summing up to 1, and: – – – – – – w(c) is an initial weight assigned to node c; e(c) is the number of edges of any kind involving node c; (c, c) denotes an edge involving node c; dM is the largest distance between any two nodes in the whole vector space; d(c) is the distance of node c from the center of the corresponding cluster; k(c) is the keyword weight associated to node c. The first term represents the initial weight provided by V, normalized by the maximum initial weight among all nodes. The second term considers the number of connections (edges) of any category (verbal or taxonomic relationships) in which c is involved, normalized by the maximum number of connections of any node in the network. The third term (Neighborhood Weight Summary) considers the average initial weight of all neighbors of c (just summing up the weights the final value would be proportional to the number of neighbors, that is already considered in the previous term). The fourth term represents the Closeness to Center of the cluster, i.e. the distance of c from the center of its cluster, normalized by the maximum distance between any two instances in the whole vector space. The last term takes into account the outcome of the three KE techniques on the given text, suitably weighted: k(c) = ζkc (c) + ηkw (c) + θkp (c) where ζ, η and θ are weights ranging in [0, 1] and summing up to 1. These terms were designed to be independent of each other. A partial interaction is present only between the second and the third ones, but is significantly smoothed due to the applied normalizations. 6 3.2 Generalization of Similar Concepts To generalize two or more concepts (G generalizes A if anything that can be labeled as A can be labeled as G as well, but not vice-versa), we propose to exploit WordNet and use the set of connections of each concept with its direct neighbors as a description of the underlying concept. Three steps are involved in this procedure: 1. Grouping similar concepts, in which all concepts are grossly partitioned to obtain subsets of similar concepts; 2. Word Sense Disambiguation, that associates a single synset to each term by solving possible ambiguities using the domain of discourse (Algorithm 1); 3. Computation of taxonomic similarity, in which WordNet is exploited to confirm the validity of the groups found in step 1 (Algorithm 2). As to step 1, we build a Concepts×Concepts matrix C where Ci,j = 1 if there is at least a relationship between concepts i and j, or Ci,j = 0 otherwise. Each row in C can be interpreted as a description of the associated concept in terms of its relationships to other concepts, and exploited for applying a pairwise clustering procedure based on Hamming distance. In detail, for each possible pair of different row and column items whose corresponding row and column are not null and whose similarity passes a given threshold: if neither is in a cluster yet, a new cluster containing those objects is created; otherwise, if either item is already in a cluster, the other is added to the same cluster; otherwise (both already belong to different clusters) their clusters are merged. Items whose similarity with all other items does not pass the threshold result in singleton clusters. This clustering procedure alone might not be reliable, because terms that occur seldom in the corpus have few connections (which would affect their cluster assignment due to underspecification) and because the expressive power of this formalism is too low to represent complex contexts (which would affect even more important concepts). For this reason, the support of an external resource might be desirable. We consider WordNet as a sensible candidate for this, and try to map each concept in the network to the corresponding synset (a non trivial problem due to the typical polysemy of many words) using the one domain per discourse assumption as a simple criterion for Word Sense Disambiguation: the meanings of close words in a text tend to refer to the same domain, and such a domain is probably the dominant one among the words in that portion of text. Thus, WordNet allows to check and confirm/reject the similarity of concepts belonging to the same cluster, by considering all possible pairs of words whose similarity is above a given threshold. The pair (say {A, B}) with largest similarity value is generalized with their most specific common subsumer (hypernym) G in WordNet; then the other pairs in the same cluster that share at least one of the currently generalized terms, and whose least common hypernym is again G, are progressively added to the generalization. Similarity is determined using a mix of the measures proposed in [4] and in [16], to consider both the global similarity 7 Algorithm 1 Find “best synset” for a word Input: word t, list of domains with Output: best synset for word t. weights. best synset ← empty best domain ← empty for all synset(st ) do max weight ← −∞ optimal domain ← empty for all domains(ds ) do if weight(ds ) > max weight then max weight ← weight(ds ) optimal domain ← ds end if end for if max weight > weight(best domain) then best synset ← st best domain ← optimal domain end if end for and the actual viability of the specific candidate generalization: sf (A, B) = sfF a (A, B) · sfW P (A, B) 3.3 Reasoning ‘by association’ Reasoning ‘by association’ means finding a path of pairwise related concepts that establishes an indirect interaction between two concepts c′ and c′′ in the semantic network. We propose to look for such a path using a Breadth-First Search (BFS) technique, applied to both concepts under consideration.The expansion steps of the two processes are interleaved, checking at each step whether the new set of concepts just introduced has a non-empty intersection with the set of concepts of the other process. When this happens, all the concepts in such an intersection identify one or more shortest paths connecting c′ and c′′ , that can be retrieved by tracing back the parent nodes at each level in both directions up to the roots c′ and c′′ . Since this path is made up of concepts only, to obtain a more sensible ‘reasoning’ it must be filled with the specific kind of interaction represented by the labels of edges (verbs) that connect adjacent concepts in the chain. 4 Evaluation The proposed approach was evaluated using ad-hoc tests that may indicate its strengths and weaknesses. Due to lack of space, only a few selected outcomes will be reported here. Although preliminary, these results seem enough to suggest that the approach is promising. The following default weights for the Relevance Weight components were empirically adopted: – α = 0.1 to increase the impact of most frequent concept (according to tf ); 8 Algorithm 2 Effective generalization research. Input: the set of C clusters returned by pair-wise Output: set of candidate generalizations. clustering; T similarity threshold. generalizations ← empty set for all c ∈ C do good pairs ← empty set for all pair(Oi , Oj ) | i, j ∈ c do if similarity score(pair(Oi , Oj )) > T then good pairs.add(pair(Oi, Oj ), wordnet hypernym(pair(Oi, Oj ))) end if if good pairs 6= empty set then new set ← {good pairs.getBestP air, good pairs.getSimilarP airs} generalizations.add(new set) end if end for end for good pairs → all pairs that passed T , with the most specific common hypernym discovered in WordNet good pairs.getBestPair → the pair that has the best similarity score. good pairs.getSimilarPairs → the pairs that involve one of two objects of the best pair, that have satisfied the similarity score and have the same hypernym as the best pair wordnet hypernym → the most specific common hypernym discovered in WordNet for the two passed object. – β = 0.1 to keep low the impact of co-occurrences between nodes; – γ = 0.3 to increase the impact of less frequent nodes if they are linked to relevant nodes; – δ = 0.25 to increase the impact of the clustering outcome; – ǫ = 0.25 as for δ, to increase the impact of keywords. while those for the KE techniques were taken as ζ = 0.45, η = 0.45 and θ = 0.1 (to reduce the impact of the psychological perspective, that is more naive compared to the others). 4.1 Recognition of relevant concepts We exploited a dataset made up of documents concerning social networks of socio-political and economic subject. Table 1 shows on the top the settings used for three different runs, concerning the Relevance Weight components: W =A+B+C +D+E and the cutpoint value for selecting relevant concepts. The corresponding outcomes (at the bottom) show that the default set of parameter values yields 3 relevant concepts, having very close weights. Component D determines the inclusion of the very unfrequent concepts (see column A) access and subset (0.001 and 6.32 E-4, respectively) as relevant ones. They benefit from the large initial weight of network, to which they are connected. Using the second set of parameter values, the predominance of component A in the overall computation, 9 Table 1. Three parameter choices and corresponding outcome of relevant concepts. Test # α β γ δ ǫ p 1 0.10 0.10 0.30 0.25 0.25 1.0 2 0.20 0.15 0.15 0.25 0.25 0.7 3 0.15 0.25 0.30 0.15 0.15 1.0 Test # Concept # A B C D E network 0.100 0.100 0.021 0.178 0.250 1 access 0.001 0.001 0.154 0.239 0.250 subset 6.32E-4 0.001 0.150 0.239 0.250 2 network 0.200 0.150 0.0105 0.178 0.250 network 0.150 0.25 0.021 0.146 0.150 user 0.127 0.195 0.022 0.146 0.150 3 number 0.113 0.187 0.022 0.146 0.150 individual 0.103 0.174 0.020 0.146 0.150 W 0.649 0.646 0.641 0.789 0.717 0.641 0.619 0.594 Table 2. Pairwise clustering statistics. Dataset MNC (0.001) MNC (0.0001) Vector size B 3 2 1838 P 3 1 1599 B+P 5 1 3070 and the cutpoint threshold lowered to 70%, cause the frequency-based approach associated to the initial weight to give neat predominance to the first concept in the ranking. Using the third set of parameter values, the threshold is again 100% and the other weights are such that the frequency-based approach expressed by component A is balanced by the number of links affecting the node and by the weight of its neighbors. Thus, both nodes with highest frequency and nodes that are central in the network are considered relevant. Overall, concept network is always present, while the other concepts significantly vary depending on the parameter values. 4.2 Concept Generalization Two toy experiments are reported for concept generalization. The maximum threshold for the Hamming distance was set to 0.001 and 0.0001, respectively, while the minimum threshold of taxonomic similarity was fixed at 0.4 in both. Two datasets on Social networks were exploited: a book (B) and a collection of scientific papers (P ) concerning socio-political and economic discussions. Observing the outcome, three aspects can be emphasized: the level of detail of the concept descriptions that in pairwise clustering satisfy the criterion, the intuitivity of the generalizations supported by WordNet Domains, and the values of the single conceptual similarity measures applied to synsets in WordNet. In Table 2, MNC is the Max Number of Connections detected among all concept descriptions that have been agglomerated at least once in the pairwise clustering. Note that all descriptions which have never been agglomerated, are 10 Table 3. Generalizations for different pairwise clustering thresholds (Thr.) and minimum similarity threshold 0.4 (top), and corresponding conceptual similarity scores (bottom). Thr. Dataset Subsumer parent B [110399491] human action P 0.001 [100030358] Subs. Domain Concepts adopter [109772448] person dad [109988063] discussion [107138085] factotum judgement [100874067] psychiatrist [110488016] B + P dr. [110020890] medicine abortionist [109757175] specialist [110632576] physiological state dependence [114062725] B physiology [114034177] affliction [114213199] mental attitude marxism [106215618] 0.0001 P psychology [106193203] standpoint [106210363] feeling psychological dislike [107501545] B+P [100026192] features satisfaction [107531255] # Pairs F a score W P score 1 adopter, dad 0.733 0.857 2 discussion, judgement 0.731 0.769 psychiatrist, abortionist 0.739 0.889 3 psychiatrist, specialist 0.728 0.889 4 dependence, affliction 0.687 0.750 5 marxism, standpoint 0.661 0.625 6 dislike, satisfaction 0.678 0.714 Conc. Domain factotum person factotum law medicine medicine medicine physiology medicine politics factotum psychological features psychological features Score 0.628 0.562 0.657 0.647 0.516 0.413 0.485 considered as single instance in a separated cluster. Hence, the concepts recognized as similar have very few neighbors, suggesting that concepts become ungeneralizable as their number of connections grows. Although in general this is a limitation, such a cautious behavior is to be preferred until an effective generalization technique is provided, that ensures the quality of its outcomes (wrong generalizations might spoil subsequent results in cascade). It is worth emphasizing that not only sensible generalizations are returned, but their domain is also consistent with those of the generalized concepts. This happens with both thresholds (0.001 and 0.0001), that return respectively 23 and 30 candidate generalizations (due to space limitations, Table 3 reports only a representative sample, including a generalization for each dataset used). Analyzing the two conceptual similarity measures used for generalization reveals that, for almost all pairs, both yield very high values, leading to final scores that neatly pass the 0.4 threshold, and sf W P is always greater than sf F a . Since the former is more related to a specific path, and hence to the goodness of the chosen subsumer, this confirms the previous outcomes (suggesting that the chosen subsumer is close to the generalized concepts). In the sample reported in Table 3, only case 5 disagrees with these considerations. 4.3 Reasoning by association Table 4 shows a sample of outcomes of reasoning by association. E.g., case 5 explains the relationship between freedom and internet as follows: the adult 11 Table 4. Exampes of reasoning by associations (start and target nodes in emphasis). # 1 2 3 4 5 Subject Verb flexibility convert people settle, desire, do at, extinguish people use, revolution myspace develop people member people erode, combine computer acknowledge internet extend majority erode majority erode, do adult use facebook acknowledge adult write adult use technology acknowledge internet acknowledge Complement life life myspace headline threat technology technology neighbor internet facebook platform platform freedom platform platform technology write about freedom, and use platform, that is recognized as a technology, as well as internet. 5 Conclusions This work proposed an approach to automatic conceptual taxonomy extraction from natural language texts. It works by mixing different techniques in order to identify relevant terms/concepts in the text, group them by similarity and generalize them to identify portions of a hierarchy. Preliminary experiments show that the approach can be viable, although extensions and refinements are needed to improve its effectiveness. In particular, a study on how to set standard suitable weights for concept relevance assessment is needed. A reliable outcome might help users in understanding the text content and machines to automatically performing some kind of reasoning on the resulting taxonomy. References [1] Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. Generating typed dependency parses from phrase structure trees. In LREC, 2006. [2] Christiane Fellbaum, editor. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA, 1998. [3] S. Ferilli, M. Biba, T.M. Basile, and F. Esposito. Combining qualitative and quantitative keyword extraction methods with document layout analysis. In Postproceedings of the 5th Italian Research Conference on Digital Library Management Systems (IRCDL-2009), pages 22–33, 2009. 12 [4] S. Ferilli, M. Biba, N. Di Mauro, T.M. Basile, and F. Esposito. Plugging taxonomic similarity in first-order logic horn clauses comparison. In Emergent Perspectives in Artificial Intelligence, Lecture Notes in Artificial Intelligence, pages 131–140. Springer, 2009. [5] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I.H. Witten. The weka data mining software: an update. SIGKDD Explorations, 11(1):10–18, 2009. [6] R.W. Hamming. Error detecting and error correcting codes. Bell System Technical Journal, 29(2):147–160, 1950. [7] Dan Klein and Christopher D. Manning. Fast exact inference with a factored model for natural language parsing. In Advances in Neural Information Processing Systems, volume 15. MIT Press, 2003. [8] Sang Ok Koo, Soo Yeon Lim, and Sang-Jo Lee. Constructing an ontology based on hub words. In ISMIS’03, pages 93–97, 2003. [9] A. Maedche and S. Staab. Mining ontologies from text. In EKAW, pages 189–202, 2000. [10] A. Maedche and S. Staab. The text-to-onto ontology learning environment. In ICCS-2000 - Eight International Conference on Conceptual Structures, Software Demonstration, 2000. [11] Bernardo Magnini and Gabriela Cavagli. Integrating subject field codes into wordnet. pages 1413–1418, 2000. [12] Yutaka Matsuo and Mitsuru Ishizuka. Keyword extraction from a single document using word co-occurrence statistical information. International Journal on Artificial Intelligence Tools, 13:2004, 2003. [13] N. Ogata. A formal ontology discovery from web documents. In Web Intelligence: Research and Development, First Asia-Pacific Conference (WI 2001), number 2198 in Lecture Notes on Artificial Intelligence, pages 514–519. Springer-Verlag, 2001. [14] Alessandro Cucchiarelli Paola Velardi, Roberto Navigli and Francesca Neri. Evaluation of OntoLearn, a methodology for automatic population of domain ontologies. In Paul Buitelaar, Philipp Cimiano, and Bernardo Magnini, editors, Ontology Learning from Text: Methods, Applications and Evaluation. IOS Press, 2006. [15] G. Salton, A. Wong, and C. S. Yang. A vector space model for automatic indexing. Commun. ACM, 18:613–620, November 1975. [16] Zhibiao Wu and Martha Palmer. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Computational Linguistics, pages 133–138, Morristown, NJ, USA, 1994. Association for Computational Linguistics. 13 PatTexSum : A pattern-based text summarizer Elena Baralis, Luca Cagliero, Alessandro Fiori, and Saima Jabeen elena.baralis,luca.cagliero,alessandro.fiori,[email protected] Affiliation: Politecnico di Torino. Corso Duca degli Abruzzi, 24 10129 Torino, Italy. Tel: +390110907194 Fax:+390110907099 Abstract. In the last decade the growth of the Internet has made a huge amount of textual documents available in the electronic form. Text summarization is commonly based on clustering or graph-based methods and usually considers the bag-of-word sentence representation. Frequent itemset mining is a widely exploratory technique to discover relevant correlations among data. The well-established application of frequent itemsets to large transactional datasets prompts their usage in the context of document summarization as well. This paper proposes a novel multi-document summarizer, namely PatTexSum (Pattern-based Text Summarizer), that is mainly based on a pattern-based model, i.e., a model composed of frequent itemsets. Unlike previously proposed approaches, PatTexSum selects most representative and not redundant sentences to include in the summary by considering both (i) the most informative and non-redundant itemsets extracted from document collections tailored to the transactional data format, and (ii) a sentence score, based on the tf-idf statistics. Experiments conducted on a collection of real news articles show the effectiveness of the proposed approach. 1 Introduction From the birth of the Internet on, analysts may progressively access and analyze larger data collections. Since the large majority of the information is available in textual form, a challenging task is to convey the most relevant information provided by textual documents into short and concise summaries. Many document summarization approaches have been proposed in literature. Most of them select the most representative sentences to include in the summaries by means of the following approaches: (i) clustering (e.g., [13, 20]), (ii) graph-based methods (e.g., [12]), and (iii) linear programming (e.g., [15]). Clustering-based approaches exploit clustering algorithms to group sentences and select representatives among each group. For instance, MEAD [13] evaluates the similarity between the document sentences and the centroids and selects, similarly to [6], the most relevant sentences among each document cluster based on the tf-idf statistical measure [16]. Differently, in [20] an incremental hierarchical clustering algorithm is exploited to update summaries over time. The graph-based approaches try to represent correlations among sentences by 14 means of a graph-based model. According to this model, sentences are represented by graph nodes, while the edges weigh the strength of the correlation between couples of sentences. The most representative sentences are selected according to graph-based indexing strategies. For instance, [12] proposes to rank sentences based on the eigenvector centrality computed by means of the wellknown PageRank algorithm [5]. Finally, the linear programming methods identify the most representative sentences by maximizing ad-hoc object functions. For instance, in [15] the authors formalized the extractive summarization task as a maximum coverage problem with the Knapsack constraints based on the the bag-of word sentence representation and enforce additional constraints based on sentence relevance within each document. Most the aforementioned approaches rely on the bag-of-word sentence representation and make use of well-founded statistical measures (e.g., the tf-idf measure [16]). Frequent itemset mining is a widely exploratory technique, first introduced in [1] in the context of market basket analysis, to discover correlations that frequently occur in the analyzed data. A number of approaches focus on discovering frequent itemsets from transactional data and then selecting their most informative yet non-redundant subset by means of postpruning. To address this issue, static approaches (e.g., [4, 8]) compare the observed frequency (i.e., the support) of each itemset in the source transactional data against some null hypotheses (i.e., their expected frequency). Differently, dynamic approaches (e.g., [9, 18]) make often use of the maximum entropy model to take previously selected patterns into account and, thus, reduce model redundancy. Although the discovery and selection of valuable frequent itemsets from transactional data is well-established, to the best of our knowledge their usage in document summarization has never been investigated yet. PatTexSum (Pattern-based Text Summarizer) is a novel multi-document summarization approach that exploits a pattern-based model to select the most representative and not redundant sentences belonging to the document collection. It focuses on combining the effectiveness of pattern-based models, composed of highly informative and non-redundant itemsets, to represent correlations among data with the discriminating power of a sentence evaluation measure, based the tf-idf statistics. Pattern-based model generation focuses on extracting and selecting valuable frequent itemsets from a transactional representation of the document collection. To this aim, an efficient and effective approach, recently proposed in [11] in the context of transactional data, is adopted. [11] succinctly summarizes transactional data by adopting an heuristics to solve the maximum entropy model that allows on-the-fly evaluating itemsets during their extraction. This feature makes this approach particularly appealing for its application in text summarization. To effectively discriminate among sentences, an evaluation score, computed from their bag-of-word representation and based on the well-founded tf-idf statistic [16], is also considered. PatTexSum combines the information discovered from both transactional and bag-of-word data representations and adopts an effective greedy approach, first proposed in [2], to solve the problem of selecting sentences that cover at best the pattern-based model. 15 To evaluate the PatTexSum performance a suite of experiments on a collection of news articles has been performed. Results, reported in Section 3, show that PatTexSum significantly outperforms mostly used previous summarizers in terms of precision, recall, and F-measure. This paper is organized as follows. Section 2 presents the proposed method and thoroughly describes its main steps. Section 3 assesses the effectiveness of the PatTexSum framework in summarizing textual documents, while Section 4 draws conclusions and presents future developments of this work. 2 The PatTexSum method PatTexSum focuses on summarizing collections of textual documents by exploiting a two-way data representation. Pattern-based model generation relies on a transactional representation of the document sentences, while the relevance score evaluation, based on the tf-idf statistic, is based on the bag-of-word sentence representation. A greedy approach is used to effectively combine knowledge discovered from both data representations and select most representative sentences to include in the summary. Figure 1 shows the main steps behind the proposed approach, which will be thoroughly described in the following. Fig. 1. The PatTexSum method 2.1 Document representation PatTexSum exploits two different document/sentence representations: (i) the traditional bag-of-word (BOW) representation and (ii) the transactional data 16 format. The raw document content is first preprocessed to make it suitable for the data mining and knowledge discovery process. Stopwords, numbers, and website URLs are removed to avoid noisy information, while the Wordnet stemming algorithm [3] is applied to reduce document words to their base or root form (i.e., the stem). Let D={d1 , . . . , dn } be a document collection, where each document dk is composed of a set sentences Sk ={s1k , . . . , szk }. Documents are composed of a sequence of sentences, each one composed of a set of words. The BOW representation of the j-th sentence sjk belonging to the k-th document dk of the collection D is the set of all word stems (i.e., terms) occurring in sjk . Consider now the set trjk ={w1 , . . . , wl } where trjk ⊆ sjk and wq 6= wr ∀ q 6= r. It includes the subset of distinct terms occurring in the sentence sjk . To tailor document sentences to the transactional data format, we consider each document sentence as a transaction whose items are distinct terms taken from its BOW representation, i.e., trjk is the transaction that corresponds to the document sentence sjk . A transactional representation T of the document collection D is the union of all transactions trjk corresponding to each sentence sjk belonging to any document dk ∈ D. The document collection is associated with the statistical measure of the term frequency-inverse document frequency (tf-idf) that evaluates the relevance of a word in the whole collection. A more detailed description of the tf-idf statistic follows. The whole document content could be represented in a matrix form T C, in which each row represents a distinct term of the document collection while each column corresponds to a document. Each element tcik of the matrix T C is the tf-idf value associated with a term wi in the document dk belonging to the whole collection D. It is computed as follows: tcik = P nik r∈{q : wq ∈dk } nrk · log |D| |{dk ∈ D : wi ∈ dk }| (1) where nik is the number of occurrences P of i-th term wi in the k-th document dk , D is the collection of documents, r∈{q : wq ∈dk } nrk is the sum of the number of occurrences of all terms in the k-th document dk , and log |{dk ∈D|D| : wi ∈dk }| represents the inverse document frequency of term wi . 2.2 The pattern-based model generation Frequent itemset mining is a well-established data mining approach that focuses on discovering recurrences, i.e., itemsets, that frequently occur in the source data. An itemset I of length k, i.e., a k-itemset, is a set of k distinct items. Let T be the document collection in the transactional data format. We denote as D(I) the set of transactions supported by I, i.e., D(I) = {trjk ∈ T | I ⊆ trjk }. The support of an itemset I is the observed frequency of occurrence of I in D, i.e., sup(I)= D(I) |T | . Since the problem of discovering all itemsets in a transactional dataset is computationally intractable [1], itemset mining is commonly driven by a minimum support threshold min sup. 17 Given a minimum support threshold min sup and a model size p, PatTexSum generates a pattern-based model that includes the most informative yet non-redundant set of p frequent itemsets discovered from the document collection T tailored to the transactional data format (Cf. Section 2.1). Among the large set of previously proposed approaches focused on succinctly representing transactional data by means of itemsets [8, 17, 18], we adopt a method recently proposed in [11]. Unlike previous approaches, it exploits an entropy-based heuristic to drive the mining process and select most informative yer not redundant itemsets without the need of postpruning. Its efficiency and effectiveness in discovering succinct transactional data summaries makes it particularly suitable for the application to text summarization. 2.3 Sentence evaluation and selection The PatTexSum method exploits the pattern-based model to evaluate and select most relevant sentences to include in the summary. Sentence evaluation and selection steps consider (i) a sentence relevance score that combines the tf-idf statistic [16] associated with each sentence term, and (ii) the sentence coverage with respect to the generated pattern-based model (Cf. Section 2.2). In the following we formalize both sentence coverage and relevance. Sentence relevance score The relevance score of a sentence is evaluated by using the bag-of-word document representation. It is computed as the sum of the tf-idf values (Cf. Formula 1) of each term belonging to the sentence in the document collection. In Formula 2 the score expression for a generic sentence sjk belonging to the document collection D is reported P i | wi ∈sjk tcik SR(sjk ) = (2) |tjk | P where |tjk | is the number of distinct terms occurring in sjk , and i | wi ∈sjk tcik is the sum of the tf-idf values associated with terms (i.e., word stems) in sjk (Cf. Formula 1). Sentence model coverage The sentence coverage measures the pertinence of each sentence to the generated pattern-based model. To this aim, it considers document sentences tailored to the transactional data format. Let D be the collection of documents, i.e., a set of sentences. We first associate with each sentence sjk ∈ D a binary vector, denoted in the following as sentence coverage vector (SC), SCjk ={sc1 , . . . , scp } where p is the number of itemsets belonging to the model and sci = 1trjk (Ii ) indicates whether itemset Ii is included or not in trjk . More formally, 1trjk is an indicator function defined as follows: ( 1 if Ii ⊆ trjk , 1trjk (Ii ) = (3) 0 otherwise 18 Algorithm 1 Sentence selection - Greedy approach Input: set of sentence relevance scores SR, set of sentence coverage vectors SC, tf-idf matrix T C Output: summary S 1: {Initializations} 2: S = ∅ 3: ESC = ∅ {set of eligible sentence coverage vectors} 4: SC ∗ = all zeros() {summary coverage vector with only 0s} 5: {Cycle until either SC ∗ contains only 1s or all the SC vectors contain only zeros} 6: while not (summary coverage vector all ones() or sentence coverage vectors only zeros()) do 7: {Determine the sentences with the highest number of ones} 8: ESC = max ones sentences() 9: if ESC != ∅ then 10: {Select the sentence with maximum relevance score} 11: SCbest = ESC[1] 12: for all t ∈ ESC[2 :] do 13: if SRt > SRbest then 14: SCbest =SCt 15: end if 16: end for 17: {Update sets and summary coverage vector} 18: S = S ∪ SCbest 19: SC ∗ = SC ∗ OR SCbest 20: ESC = ESC \ SCbest 21: {Update the sentence coverage vectors belonging to V} 22: for all SCi in SC do 23: SCi = SCi AND SC ∗ 24: end for 25: else 26: break 27: end if 28: end while 29: return S The coverage of a sentence sjk with respect to the pattern-based model is defined as the number of 1’s that occur in the corresponding coverage vector SCjk . We formalize the problem of selecting the most informative and not redundant sentences according to the pattern-based model as a set covering problem. The set covering problem A set covering algorithm focuses on selecting the minimum set of sentences, of arbitrary size l, whose logic OR of coverage vectors, i.e., SC ∗ =SC1 ∨ . . . ∨ SCl , generates a binary vector composed of all 1’s. This implies that each itemset belonging to the model covers at least one sentence. The SC ∗ vector will be denoted as the summary coverage vector throughout the paper. The set covering problem is known to be NP-hard. To solve the problem, we adopt a greedy strategy that we already proved to be effective in summarization of biological microarray data [2]. In order to build an accurate yet concise summary, the sentence coverage with respect to the pattern-based model is considered as the most discriminative feature, i.e., sentences that cover the maximum number of itemsets belonging to the model are selected firstly. At equal terms, the sentence with maximal coverage that is characterized by the highest relevance score SR is preferred. The adopted algorithm identifies, at each step, the sentence sjk with the best complementary vector SCjk with respect to the current summary coverage 19 vector SC ∗ . The pseudo-code of the greedy approach is reported in Algorithm 1. It takes in input the set of sentence relevance scores SR, the set of sentence coverage vectors SC, and the tf-idf matrix T C. It produces the summary S, i.e., the minimal subset of most representative sentences. The first step is the variable initialization and the sentence coverage vector computation (lines 1-4). Next, the sentence with maximum coverage, i.e., the one whose coverage vector contains the highest number of ones, is iteratively selected (line 7). At equal terms, the sentence with maximum relevance score (Cf. Formula 2) is preferred (lines 12-16). Finally the selected sentence is included in the summary S while the summary and sentence coverage vectors are updated (lines 18-24). The procedure iterates until either the summary coverage vector contains only ones, i.e., the model is fully covered by the summary, or the remaining sentences are not covered by any itemset, i.e., the remaining sentences are not pertinent to the model (line 6). Experimental results, reported in Section 3, show that the proposed summarization method performs better than exclusively considering either sentence coverage or sentence relevance. 3 Experimental results We conducted a set of experiments to address the following issues: (i) the effectiveness of the proposed summarization approach against two widely used summarizers, i.e., the Open Text Summarizer (OTS) [14] and TexLexAn [19] (Section 3.1), and (ii) the impact of the pattern-based model size and the support threshold on the performance of PatTexSum (Section 3.2). We evaluated all the summarization approaches on a collection of real-life news articles. To this aim, the 10 top-ranked news documents, provided by the Google web search engine (http://www.google.com), that concern the following recent news topics have been selected: Natural Disaster: Earthquake in Spain 2011 Royal Wedding: Prince William and Kate Middleton wedding Technology: Microsoft purchased Skype Education: Wealthy parents could buy their children places at elite universities – Sport:Australia defeat Pakistan in Azlan shah Hockey – – – – The datasets relative to the above news categories are made available for research purposes, upon request to the authors. To compare the results by PatTexSum with OTS [14] and TexLexAn [19], we used the ROUGE [10] toolkit (version 1.5.5), which is widely applied by Document Understanding Conference (DUC) for document summarization performance evaluation1 . It measures the quality of a summary by counting the unit overlaps between the candidate summary and a set of reference summaries. 1 The provided command is: ROUGE-1.5.5.pl -e data -x -m -2 4 -u -c 95 -r 1000 -n 4 -f A -p 0.5 -t 0 -d -a 20 Intuitively, the summarizer that achieves the highest ROUGE scores could be considered as the most effective one. Several automatic evaluation scores are implemented in ROUGE. For the sake of brevity, we reported only ROUGE-2 and ROUGE-4 as representative scores. Analogous results have been obtained for the other scores. Since a ”golden summary” (i.e., the optimal document collection summary) is not available for web news document, we performed a leave-one-out cross validation. More specifically, for each category we summarized nine out of ten news documents and we compared the resulting summary with the remaining (not yet considered) document, which has been selected as golden summary at this stage. Next, we tested all other possible combinations by varying the golden summary and we computed the average performance results, in terms of precision, recall, and F-measure, achieved by each summarizer for both ROUGE2 and ROUGE-4. 3.1 Performance comparison and validation We evaluated the performance, in terms of ROUGE-2 and ROUGE-4 precision (Pr), recall (R), and F-measure (F), of PatTexSum against OTS and TexLexAn. For both OTS and TexLexAn we adopted the configuration suggested by the respective authors. For PatTexSum we enforced a minimum support threshold min sup=1.5% and we tuned the value of the pattern-based model size p to its best value for each considered dataset. A more detailed discussion on the impact of both min sup and p on the performance of PatTexSum is reported in Section 3.2. PatTexSum performs better than the other considered summarizers on all tested datasets. To validate the statistical significance of PatTexSum performance improvement against OTS and TexLexAn, we used the paired t-test [7] at significance level p− value = 0.05 for all evaluated datasets and measures. For ROUGE-2, PatTexSum provides significantly better results than OTS, whose summarization approach is mainly based on tf-idf measure, and TexLexAn in terms of precision and/or recall on 3 out of 5 datasets (i.e., Natural disaster, Technology and Sports). Moreover, PatTexSum significantly outperforms TexLexAx and OTS in terms of F-measure (i.e., the harmonic average of precision and recall [16]) on, respectively, 2 and 3 of them (i.e., Natural disaster and Technology for both, and Sports for TexLexAn). Similar results were obtained for ROUGE-4. 3.2 PatTexSum parameter analysis We analyzed the impact of the minimum support threshold and the patternbased model size, i.e., the number of generated itemsets, on the performance 21 PatTexSum OTS TexLexAn p R Pr F R Pr F R Pr F Natural Disaster 16 0.116 0.288 0.141 0.040 0.120 0.053 0.038 0.114 0.045 Royal Wedding 12 0.036 0.215 0.058 0.034 0.174 0.054 0.030 0.150 0.047 Technology 5 0.141 0.465 0.210 0.042 0.208 0.067 0.042 0.172 0.065 Sports 10 0.145 0.297 0.189 0.055 0.133 0.075 0.071 0.149 0.093 Education 8 0.039 0.241 0.064 0.036 0.170 0.054 0.034 0.150 0.051 Table 1. Performance comparison in terms of ROUGE-2 score. dataset PatTexSum OTS TexLexAn p R Pr F R Pr F R Pr F Natural-Disaster 16 0.060 0.125 0.068 0.005 0.012 0.006 0.005 0.011 0.006 Royal-wedding 12 0.009 0.082 0.015 0.003 0.018 0.005 0.003 0.018 0.005 Technology 5 0.113 0.356 0.167 0.009 0.065 0.016 0.003 0.011 0.005 Sports 10 0.059 0.112 0.077 0.004 0.010 0.006 0.022 0.036 0.027 Education 8 0.017 0.141 0.030 0.003 0.012 0.005 0.003 0.009 0.004 Table 2. Performance comparison in terms of ROUGE-4 score. dataset of the PatTexSum summarizer. To also test the impact of the tf-idf statistic on the performance of the pattern-based summarizer, we entail (i) neglecting the relevance score evaluation (i.e., by simply selecting the top-ranked maximal coverage sentence provided by the itemset miner [11]), and (ii) considering other statistical measures in place of the tf-idf score. Among all the evaluated scores, the tf-idf statistic turns out to be most effective measure in discriminating among sentences. In Figures 2(a) and 2(b) we reported the F-measure achieved by PatTexSum, by either considering or not the relevance score in the sentence evaluation, and by varying, respectively, the support threshold on Technology and the model size on the Natural Disaster document collection. For the sake of brevity, we reported only the results obtained with the ROUGE-4 score. Analogous results have been obtained for the other ROUGE scores, for precision and recall measures, and for all other configurations. The usage of the relevance score based on the tf-idf statistic always improves the performance of PatTexSum in the range of those values of p and min sup yielding the highest F-measure. This improvement is due to its ability to well discriminate sentence term occurrence among documents. When higher support thresholds (e.g., 5%) are enforced, many informative patterns are discarded, thus the model becomes too general to yield high summarization performance. Oppositely, when very low support thresholds (e.g., 0.1%) are enforced, data overfitting occurs, i.e., the model is too much specialized to effectively and concisely summarize the whole document collection content. At medium support thresholds (e.g., 1.5%) the best balancing between model specialization and generalization is achieved, thus, PatTexSum produces very concise yet informative summaries. 22 0.08 0.2 With SR Without SR 0.18 0.07 0.16 0.06 F-measure F-measure 0.14 0.12 0.1 0.08 0.06 0.05 0.04 0.03 0.04 0.02 0.02 With SR Without SR 0.01 0 0 0.5 1 1.5 2 2.5 3 Minsup (%) 3.5 4 4.5 6 5 (a) Technology. p=5. Impact of the support threshold. 8 10 12 14 16 p 18 20 22 24 (b) Natural Disaster. min sup=1.5%. Impact of the pattern-based model size. Fig. 2. PatTexSum performance analysis by either considering or not of the relevance score (SR). Rouge-4 score. F-measure. The model size may also significantly affect the summarization performance. When a limited number of itemsets (e.g., p = 6) is selected, the relevant knowledge hidden in the news category Natural Disaster is not yet fully covered by the extracted patterns (see Figure 2(b)), thus the generated summaries are not highly informative. When p = 16 the pattern-based model provides the most informative and non-redundant knowledge. Consequently, the multi-document pattern-based summarization becomes very effective. When a higher number of itemsets is included in the model, the quality of the generated summaries worsens as the model is still informative but redundant. The best values of model size and support threshold achieved by each news category depend on the analyzed document term distribution. 4 Conclusions and future works This paper presents a multi-document summarizer that combines the knowledge provided by a pattern-based model, composed of frequent itemsets, with a statistical evaluation, based on the well-founded tf-idf measure, to select the most representative and not redundant sentences. Albeit the application of frequent itemsets to represent most valuable correlations among transactional data is well-established, their usage in text summarization has never been investigated so far. The proposed summarizer exploits a greedy approach to combine knowledge discovered from two different data representations, i.e., the transactional and bag-of-word representations, and select the minimal set of most relevant sentences. Experiments conducted on real-life news articles show both the effectiveness and the efficiency of the proposed text summarization method. 23 Future works will address: (i) the extension of the proposed approach to address the problem of incremental summary updating, and (ii) the exploitation of new techniques to address the set covering problem. References 1. R. Agrawal, T. Imieliński, and A. Swami. Mining association rules between sets of items in large databases. In ACM SIGMOD Record, volume 22, pages 207–216. 2. E. Baralis, G. Bruno, and A. Fiori. Minimum number of genes for microarray feature selection. 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC-08), pages 5692–5695, 2008. 3. S. Bird, E. Klein, and E. Loper. Natural language processing with Python. O’Reilly Media, 2009. 4. S. Brin, R. Motwani, and C. Silverstein. Beyond market baskets: Generalizing association rules to correlations. In SIGMOD Conference, pages 265–276, 1997. 5. S. Brin and L. Page. The anatomy of a large-scale hypertextual web search engine. In Proceedings of the seventh international conference on World Wide Web 7, pages 107–117, 1998. 6. J. M. Conroy, J. Goldstein, J. D. Schlesinger, and D. P. Oleary. Left-brain/rightbrain multi-document summarization. In In Proceedings of the Document Understanding Conference, 2004. 7. T. G. Dietterich. Approximate statistical test for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1998. 8. S. Jaroszewicz and D. A. Simovici. Interestingness of frequent itemsets using bayesian networks as background knowledge. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 178–186, 2004. 9. K.-N. Kontonasios and T. D. Bie. An information-theoretic approach to finding informative noisy tiles in binary databases. In SIAM International Conference on Data Mining, pages 153–164, 2010. 10. C.-Y. Lin and E. Hovy. Automatic evaluation of summaries using n-gram cooccurrence statistics. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, pages 71–78, 2003. 11. M. Mampaey, N. Tatti, and J. Vreeken. Tell me what I need to know: Succinctly summarizing data with itemsets. In Proceedings of the 17th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2011. 12. D. R. Radev. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22:2004, 2004. 13. D. R. Radev, H. Jing, M. Stys, and D. Tam. Centroid-based summarization of multiple documents. Information Processing and Management, 40(6):919 – 938, 2004. 14. N. Rotem. Open text summarizer (ots). Retrieved July, 3(2006):2006, 2003. 15. H. Takamura and M. Okumura. Text summarization model based on the budgeted median problem. In Proceeding of the 18th ACM conference on Information and knowledge management, pages 1589–1592, 2009. 16. P. Tan, M. Steinbach, V. Kumar, et al. Introduction to data mining. Pearson Addison Wesley Boston, 2006. 24 17. N. Tatti. Probably the best itemsets. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 293–302, 2010. 18. N. Tatti and H. Heikinheimo. Decomposable families of itemsets. In Proceedings of the European conference on Machine Learning and Knowledge Discovery in Databases - Part II, pages 472–487, 2008. 19. TexLexAn. Texlexan: An open-source text summarizer, 2011. 20. D. Wang and T. Li. Document update summarization using incremental hierarchical clustering. In Proceedings of the 19th ACM international conference on Information and knowledge management, pages 279–288, 2010. 25 An Expectation Maximization Algorithm for Probabilistic Logic Programs Elena Bellodi and Fabrizio Riguzzi ENDIF – Università di Ferrara – Via Saragat, 1 – 44122 Ferrara, Italy. {elena.bellodi,fabrizio.riguzzi}@unife.it Abstract. Recently much work in Machine Learning has concentrated on representation languages able to combine logic and probability, leading to the birth of a whole field called Statistical Relational Learning. In this paper we present a technique for parameter learning targeted to a family of formalisms where uncertainty is represented using Logic Programming tools - the so-called Probabilistic Logic Programs such as ICL, PRISM, ProbLog and LPAD. Since their equivalent Bayesian networks contain hidden variables, an EM algorithm is adopted. To speed the computation expectations are computed directly on the Binary Decision Diagrams that are built for inference. The resulting system, called EMBLEM for “EM over BDDs for probabilistic Logic programs Efficient Mining”, has been applied to various datasets and showed good performances both in terms of speed and memory. 1 Introduction In the field of Statistical Relational Learning (SRL) logical-statistical languages are used to effectively learn in complex domains involving relations and uncertainty. They have been successfully applied in social networks analysis, entity recognition, information extraction, etc. Similarly, a large number of works in Logic Programming has attempted to combine logic and probability, among which the distribution semantics [11] is a prominent approach. It underlies for example PRISM [11], the Independent Choice Logic, Logic Programs with Annotated Disjunctions (LPADs) [15], ProbLog [3] and CP-logic. The approach is appealing because efficient inference algorithms appeared [3,9], which adopt Binary Decision Diagrams (BDD). In this paper we present the EMBLEM system for “EM over BDDs for probabilistic Logic programs Efficient Mining” that learns parameters of probabilistic logic programs under the distribution semantics by using an Expectation Maximization (EM) algorithm: it is an iterative method to estimate some unknown parameters Θ of a model, given a dataset where some of the data is missing,to find maximum likelihood estimates of Θ. The translation of these programs into graphical models requires the use of hidden variables and therefore of EM: the main characteristic of our system is the computation of expectations using BDDs. Since there are transformations with linear complexity that can convert a program in a language into the others[2], we will use LPADs for their general syntax. 26 EMBLEM has been tested on the IMDB, Cora and UW-CSE datasets and compared with RIB [10], LeProbLog [3], Alchemy [8] and CEM, an implementation of EM based on the cplint interpreter [9]. The paper is organized as follows. Section 2 presents LPADs and Section 3 describes EMBLEM. Section 4 presents experimental results. Section 5 discusses related works and Section 6 concludes the paper. 2 Logic Programs with Annotated Disjunctions Formally a Logic Program with Annotated Disjunctions [15] consists of a finite set of annotated disjunctive clauses. An annotated disjunctive clause Ci is of the form hi1 : Πi1 ; . . . ; hini : Πini : −bi1 , . . . , bimi . In such a clause hi1 , . . . hini are logical atoms and bi1 , . . . , bimi are P logical literals, {Πi1 , . . . , Πini } are real i numbers in the interval [0, 1] such that P nk=1 Πik ≤ 1. bi1 , . . . , bimi is called the ni body and is indicated with body(Ci ). If k=1 Πik < 1 the head of the annotated disjunctive clause implicitly contains an extra atom null Pnithat does not appear in the body of any clause and whose annotation is 1 − k=1 Πik . We denote by ground(T ) the grounding of an LPAD T . An atomic choice is a triple (Ci , θj , k) where Ci ∈ T , θj is a substitution that grounds Ci and k ∈ {1, . . . , ni }. (Ci , θj , k) means that, for the ground clause Ci θj , the head hik was chosen. In practice Ci θj corresponds to a random variable Xij and an atomic choice (Ci , θj , k) to an assignment Xij = k. A set of atomic choices κ is consistent if (C, θ, i) ∈ κ, (C, θ, j) ∈ κ ⇒ i = j, i.e., only one head is selected for the same ground clause. A composite choice κ is a consistent set of atomic choices. The probability P (κ) of a composite choiceQ κ is the product of the probabilities of the individual atomic choices, i.e. P (κ) = (Ci ,θj ,k)∈κ Πik . A selection σ is a composite choice that, for each clause Ci θj in ground(T ), contains an atomic choice (Ci , θj , k). We denote the set of all selections σ of a program T by ST . A selection σ identifies a normal logic program wσ defined as wσ = {(hik ← body(Ci ))θj |(Ci , θj , k) ∈ σ}. wσ is called a world of T . Since selections are composite choices we can assign a probability to possible worlds: Q P (wσ ) = P (σ) = (Ci ,θj ,k)∈σ Πik . We consider only sound LPADs in which every possible world has a total well-founded model. Subsequently we will write wσ |= Q to mean that the query Q is true in the well-founded model of the program wσ . P The probability of a query Q according to an LPAD T is given by P (Q) = σ∈E(Q) P (σ) where E(Q) is {σ ∈ ST , wσ |= Q}, i.e., the set of selections corresponding to worlds where the query is true. To reduce the computational cost of answering queries in our experiments, random variables can be directly associated to clauses rather than to their ground instantiations: atomic choices then take the form (Ci , k), meaning that head hik is selected from program clause Ci , i.e., that Xi = k. Example 1. The following LPAD T encodes a very simple model of the development of an epidemic or pandemic: 27 C1 C2 C3 C4 = epidemic : 0.6; pandemic : 0.3 : −f lu(X), cold. = cold : 0.7. = f lu(david). = f lu(robert). Clause C1 has two groundings, C1 θ1 with θ1 = {X/david} and C1 θ2 with θ2 = {X/robert}, so there are two random variables X11 and X12 . The possible worlds in which a query is true can be represented using a Multivalued Decision Diagram (MDD). An MDD represents a function f (X) taking Boolean values on a set of multivalued variables X by means of a rooted graph that has one level for each variable. Each node is associated to the variable of its level and has one child for each possible value of the variable. The leaves store either 0 or 1. Given values for all the variables X, we can compute the value of f (X) by traversing the graph starting from the root and returning the value associated to the leaf that is reached. A MDD can be used to represent the set E(Q) by considering the multivalued variable Xij associated to Ci θj of ground(T ). Xij has values {1, . . . , ni } and the atomic choice (Ci , θj , k) corresponds to the propositional equation Xij = k. If we represent with an MDD W V the function f (X) = σ∈E(Q) (Ci ,θj ,k)∈σ Xij = k then the MDD will have a path to a 1-leaf for each possible world where Q is true. While building MDDs simplification operations can be applied that delete or merge nodes. In this way a reduced MDD is obtained with respect to a Multivalued Decision Tree (MDT), i.e., a MDD in which every node has a single parent, all the children belong to the level immediately below and all the variables have at least one node. For example, the reduced MDD corresponding to the query epidemic from Example 1 is shown in Figure 1(a). The labels on the edges represent the values of the variable associated to the node: nodes at first and second level have three outgoing edges, corresponding to the values of X11 and X12 , since C1 has three head atoms (epidemic, pandemic, null); similarly X21 has two values since C2 has two head atoms (cold, null), hence the associated node has two outgoing edges. X11 k 1 k k kkkk 3 kkkk 1 k k TTTT 2 TTT2T 1 TTTT T X12 1 n1 R X111 3 2 1 X21 2 X121 X211 3 0 ; n2 n3 Y V T R O L 1 (b) BDD. (a) MDD. Fig. 1. Decision diagrams for Example 1. 28 H 0 It is often unfeasible to find all the worlds where the query is true so inference algorithms find instead explanations for it, i.e. composite choices such that the query is true in all the worlds whose selections are a superset of them. Explanations however, differently from possible worlds, are not necessarily mutually exclusive with respect to each other, but exploiting the fact that MDDs split paths on the basis of the values of a variable and the branches are mutually disjoint, the probability of the query can be computed. Most packages for the manipulation of a decision diagram are however restricted to work on Binary Decision Diagrams (BDD), i.e., decision diagrams where all the variables are Boolean. A node n in a BDD has two children: the 1-child, indicated with child1 (n), and the 0-child, indicated with child0 (n). The 0-branch, the one going to the 0-child, is drawn with a dashed line. To work on MDDs with a BDD package we must represent multivalued variables by means of binary variables. For a multivalued variable Xij , corresponding to ground clause Ci θj , having ni values we use ni − 1 Boolean variables Xij1 , . . . , Xijni −1 and we represent the equation Xij = k for k = 1, . . . ni − 1 by means of the conjunction Xij1 ∧ Xij2 ∧ . . . ∧ Xijk−1 ∧ Xijk , and the equation Xij = ni by means of the conjunction Xij1 ∧Xij2 ∧. . . ∧Xijni −1 . BDDs obtained in this way can be used as well for computing the probability of queries by associating to each Boolean variable Xijk a parameter πik that represents P (Xijk = 1). If we define g(i) = {j|θj is a substitution grounding Ci } then P (Xijk = 1) = πik for all j ∈ g(i). The parameters are obtained from those of multivalued variables up to k = ni − 1. Figure 1(b) shows in this way: πi1 = Πi1 , . . . πik = Qk−1Πik j=1 (1−πij ) the reduced BDD corresponding to the MDD on the left, with binary variables for each level. 3 EMBLEM EMBLEM applies the algorithm for performing EM over BDDs, proposed in [14,6], to the problem of learning the parameters of an LPAD. EMBLEM takes as input a number of goals that represent the examples and for each one generates the BDD encoding its explanations. The examples are organized in a set of interpretations (sets of ground facts) each describing a portion of the domain of interest. The queries correspond to ground atoms whose predicate has been indicated as “target” by the user. The predicates can be treated as closed-world or open-world. In the first case the body of clauses with a target predicate in the head is resolved only with facts in the interpretation, in the second case it is resolved both with facts in the interpretation and with clauses in the theory. If the last option is set and the theory is cyclic, we use a depth bound on SLD-derivations to avoid going into infinite loops. Given a program containing the clauses C1 and C2 from Example 1 and the interpretation {epidemic, f lu(david), f lu(robert)}, we obtain the BDD in Figure 1(b) that represents the query epidemic. Then EMBLEM enters the EM cycle, in which the steps of expectation and maximization are repeated until the log-likelihood of the examples reaches a local maximum. For a single example Q: 29 – Expectation: computes E[cik0 |Q] and E[cik1 |Q] for all rules Ci and k = 1, . . . , ni − 1, where cikx is the number of times a variable Xijk takes value x P for x ∈ {0, 1}, with j in g(i). E[cikx |Q] is given by j∈g(i) P (Xijk = x|Q). – Maximization: computes πik for all rules Ci and k = 1, . . . , ni − 1: πik = E[cik1 |Q] E[cik0 |Q]+E[cik1 |Q] If we have more than one example the contributions of each example simply sum up when computing E[cijx ]. P (Xijk =x,Q) with P (Xijk = x|Q) is given by P (Xijk = x|Q) = P (Q) P (Xijk = x, Q) = X P (Q, Xijk = x, σ) σ∈E(Q) = X P (Q|σ)P (Xijk = x|σ)P (σ) σ∈E(Q) = X P (Xijk = x|σ)P (σ) σ∈E(Q) where P (Xijk = 1|σ) = 1 if (Ci , θj , k) ∈ σ for k = 1, . . . , ni − 1 and 0 otherwise. Since there is a one to one correspondence between the worlds where Q is true and the paths to a 1 leaf in a Binary Decision Tree (a MDT with binary variables), Y X P (Xijk = x|ρ) π(d) P (Xijk = x, Q) = ρ∈R(Q) d∈ρ where ρ is a path and if σ corresponds to ρ then P (Xijk = x|σ)=P (Xijk = x|ρ). R(Q) is the set of paths in the BDD for query Q that lead to a 1 leaf, d is an edge of ρ and π(d) is the probability associated to the edge: if d is the 1-branch from a node associated to a variable Xijk , then π(d) = πik , if d is the 0-branch from a node associated to a variable Xijk , then π(d) = 1 − πik . Now consider a BDT in which only the merge rule is applied, fusing together identical sub-diagrams. The resulting diagram, that we call Complete Binary Decision Diagram (CBDD), is such that every path contains a node for every level. For a CBDD P (Xijk = x, Q) can be further expanded as X Y P (Xijk = x, Q) = π(d) ρ∈R(Q)∧(Xijk =x)∈ρ d∈ρ where (Xijk = x) ∈ ρ means that ρ contains an x-edge from a node associated to Xijk . We can then write X Y Y P (Xijk = x, Q) = π(d) π(d) n∈N (Q)∧v(n)=Xijk ∧ρn ∈Rn (Q)∧ρn ∈Rn (Q,x) d∈ρn d∈ρn where N (Q) is the set of nodes of the BDD, v(n) is the variable associated to node n, Rn (Q) is the set containing the paths from the root to n and Rn (Q, x) 30 is the set of paths from n to the 1 leaf through its x-child. X X Y Y X π(d) π(d) P (Xijk = x, Q) = n∈N (Q)∧v(n)=Xijk ρn ∈Rn (Q) ρn ∈Rn (Q,x) d∈ρn = X X n∈N (Q)∧v(n)=Xijk = X Y X π(d) ρn ∈Rn (Q) d∈ρn ρn ∈Rn (Q,x) d∈ρn Y π(d) d∈ρn F (n)B(childx (n))πikx n∈N (Q)∧v(n)=Xijk where πikx is πik if x=1 and (1 − πik ) if x=0. F (n) is the forward probability [6], the probability mass of the paths from the root to n, while B(n) is the backward probability [6], the probability mass of paths from n to the 1 leaf. If root is the root of a tree for a query Q then B(root) = P (Q). The expression F (n)B(childx (n))πikx represents the sum of the probabilities of all the paths passing through the x-edge of node n and is indicated with ex (n). Thus X P (Xijk = x, Q) = ex (n) (1) n∈N (Q),v(n)=Xijk For the case of a BDD, i.e., a diagram obtained by applying also the deletion rule, Formula 1 is no longer valid since also paths where there is no node associated to Xijk can contribute to P (Xijk = x, Q). These paths might have been obtained from a BDD having a node m associated to variable Xijk that is a descendant of n along the 0-branch and whose outgoing edges both point to child0 (n). The correction of formula (1) to take into account of this aspect is applied in the Expectation step. We now describe EMBLEM in detail. EMBLEM’s main procedure consists of a cycle in which the procedures Expectation and Maximization are repeatedly called. The first one returns the log likelihood LL of the data that is used in the stopping criterion: EMBLEM stops when the difference between LL of the current iteration and the one of the previous iteration drops below a threshold ǫ or when this difference is below a fraction δ of the current LL. Procedure Expectation takes as input a list of BDDs, one for each example, and computes the expectation for each one, i.e. P (Q, Xijk = x) for x all P variables Xijk in the BDD. In the procedure we use η (i, k) to indicate j∈g(i) P (Q, Xijk = x). Expectation first calls GetForward and GetBackward that compute the forward, the backward probability of nodes and η x (i, k) for non-deleted paths only. Then it updates η x (i, k) to take into account deleted paths. The expectations are updated in this way: for all rules i and k = 1 to ni − 1 E[cikx ] = E[cikx ] + η x (i, k)/P (Q),where P (Q) is the backward probability of the root. Procedure Maximization computes the parameters values for the next EM iteration. Procedure GetForward traverses the diagram one level at a time starting from the root level, where F(root)=1, and for each node n computes its contribution to the forward probabilities of its children. Function GetBackward 31 computes the backward probability of nodes by traversing recursively the tree from the root to the leaves. More details can be found in [1]. 4 Experiments EMBLEM has been tested over three real world datasets: IMDB1 , UW-CSE2 and Cora3. We implemented EMBLEM in Yap Prolog4 and we compared it with RIB [10]; CEM, an implementation of EM based on the cplint inference library [9]; LeProblog [4], and Alchemy [8]. All experiments were performed on Linux machines with an Intel Core 2 Duo E6550 (2333 MHz) and 4 GB of RAM. To compare our results with LeProbLog and Alchemy we exploited the translations of LPADs into ProbLog [2] and MLN [10] respectively. For the probabilistic logic programming systems (EMBLEM, RIB, CEM and LeProbLog) we consider various options: associating a distinct random variable to each grounding of a probabilistic clause or a single random variable to a nonground clause, to express whether the clause is used or not (the latter case makes the problem easier); putting a limit on the depth of derivations, thus eliminating explanations associated to derivations exceeding the limit (necessary for problems that contain cyclic clauses, such as transitive closure clauses); setting the number of restarts for EM based algorithms. All experiments for probabilistic logic programming systems have been performed using open-world predicates. IMDB regards movies, actors, directors and movie genres and is divided into five mega-examples. We performed training on four mega-examples and testing on the remaining one. Then we drew a Precision-Recall curve and computed the Area Under the Curve (AUCPR and AUCROC). We defined 4 different LPADs, two for predicting the target predicate sameperson/2, and two for predicting samemovie/2. We had one positive example for each fact that is true in the data, while we sampled from the complete set of false facts three times the number of true instances in order to generate negative examples. For predicting sameperson/2 we used the same LPAD of [10]. We ran EMBLEM on it with the following settings: no depth bound (theory is acyclic), random variables associated to instantiations of the clauses (learning time is very low) and a number of restarts chosen to match the execution time of EMBLEM with that of the fastest other algorithm. The queries that LeProbLog take as input are obtained by annotating with 1.0 each positive example for sameperson/2 and with 0.0 each negative example for sameperson/2 obtained by random sampling. We ran LeProbLog for a maximum of 100 iterations or until the difference in Mean Squared Error (MSE) between two iterations got smaller than 10−5 ; this was done also in all the subsequent experiments. For Alchemy we used the preconditioned rescaled conjugate 1 2 3 4 http://alchemy.cs.washington.edu/data/imdb http://alchemy.cs.washington.edu/data/uw-cse http://alchemy.cs.washington.edu/data/cora http://www.dcc.fc.up.pt/~ vsc/Yap 32 gradient discriminative algorithm for every dataset and in this case we specified sameperson/2 as the only non-evidence predicate. A second LPAD, also taken from [10], has been created to evaluate the performance of the algorithms when some atoms are unseen. The settings are the same as the ones for the previous LPAD. In this experiment Alchemy was run with the −withEM option that turns on EM learning. Table 1 shows the AUCPR and AUCROC averaged over the five folds for EMBLEM, RIB, LeProbLog, CEM and Alchemy. Results for the two LPADs are shown respectively in the IMDB-SP and IMDBu-SP rows. Table 2 shows the learning times in hours. For predicting samemovie/2 we used the LPAD: samemovie(X,Y):p:- movie(X,M),movie(Y,M),actor(M). samemovie(X,Y):p:- movie(X,M),movie(Y,M),director(M). samemovie(X,Y):p:- movie(X,A),movie(Y,B),actor(A),director(B), workedunder(A,B). samemovie(X,Y):p:- movie(X,A),movie(Y,B),director(A),director(B), genre(A,G),genre(B,G). To test the behaviour when unseen predicates are present, we transformed the program for samemovie/2 as we did for sameperson/2 [10]. We ran EMBLEM on them with no depth bound, one variable for each instantiation of a rule and one random restart. With regard to LeProbLog and Alchemy, we ran them with the same settings as IMDB-SP and IMDBu-SP, by replacing sameperson with samemovie. Table 1 shows, in the IMDB-SM and IMDBu-SM rows, the average AUCPR and AUCROC for EMBLEM, LeProblog and Alchemy. For RIB and CEM we obtained a lack of memory error (indicated with “me”). The Cora database contains citations to computer science research papers. For each citation we know the title, authors, venue and the words that appear in them. The task is to determine which citations are referring to the same paper, by predicting the predicate samebib(cit1,cit2). From the MLN proposed in [13]5 we obtained two LPADs. The first contains 559 rules and differs from the direct translation of the MLN because rules involving words are instantiated with the different constants, only positive literals for the hasword predicates are used and transitive rules are not included. The Cora dataset comprises five mega-examples each containing facts for the four predicates samebib/2, samevenue/2, sametitle/2 and sameauthor/2, which have been set as target predicates. We ran EMBLEM on this LPAD with no depth bound (theory is acyclic), a single variable for each instantiation of a rule (learning time is reasonable) and a number of restarts chosen to match the execution time of EMBLEM with that of the fastest other algorithm. The second LPAD adds to the previous one the transitive rules for the predicates samebib/2, samevenue/2, sametitle/2, for a total of 563 rules. In this case we had to run EMBLEM with a depth bound equal to two (theory becomes cyclic and with higher values of depth learning time was overlong) and a single 5 Available at http://alchemy.cs.washington.edu/mlns/er. 33 variable for each non-ground rule (LPAD too complex to be treated with a variable for each instantiation); the number of restarts was one. As for LeProbLog, we separately learned the four predicates because learning the whole theory at once would give a lack of memory error. We annotated with 1.0 each positive example for samebib/2, sameauthor/2, sametitle/2, samevenue/2 and with 0.0 the negative examples for the same predicates, which were contained in the dataset provided with the MLN. As for Alchemy, we learned weights with the four predicates as the non-evidence predicates. Table 1 shows in the Cora and CoraT (Cora transitive) rows the average AUCPR and AUCROC obtained by training on four mega-examples and testing on the remaining one. CEM and Alchemy on CoraT gave a memory error while RIB was not applicable because it was not possible to split the input examples into smaller independent interpretations as required by RIB. The UW-CSE dataset contains information about the Computer Science department of the University of Washington through 22 different predicates, such as yearsInProgram/2, advisedBy/2, taughtBy/3 and is split into five mega-examples. The goal here is to predict the advisedBy/2 predicate, namely the fact that a person is advised by another person: this was our target predicate. The negative examples have been generated by applying the closed world assumption to advisedBy/2. The theory used was obtained from the MLN of [12]6 and contains 86 rules. We ran EMBLEM on it with a single variable for each instantiation of a rule, a depth bound of two (cyclic theory) and one random restart (to limit time, in comparison with the other faster algorithms). The annotated queries that LeProbLog takes as input have been created by annotating with 1.0 each positive example and with 0.0 each negative example for advisedBy/2. As for Alchemy, we learned weights with advisedBy/2 as the only non-evidence predicate. Table 1 shows the AUCPR and AUCROC averaged over the five mega-examples for all the algorithms. Table 3 shows the p-value of a paired two-tailed t-test at the 5% significance level of the difference in AUCPR and AUCROC between EMBLEM and RIB/LeProbLog/CEM/Alchemy (significant differences in bold). From the results we can observe that over IMDB EMBLEM has comparable performances with CEM for IMDB-SP, with similar execution time. On IMDBu-SP it has better performances than all other systems(see AUCPR), with a learning time equal to the fastest other algorithm. On IMDB-SM it reaches the highest area value in less time (only one restart is needed). On IMDBu-SM it still reaches the highest area with one restart but with a longer execution time. Over Cora it has comparable performances with the best other system CEM but in a significantly lower time and over CoraT is one of the few systems to be able to complete learning, with better performances in terms of area (especially AUCPR) and time. Over UW-CSE it has significant better performances with respect to all the algorithms. Longer learning times are needed for EMBLEM on IMDBu-SM and UW-CSE datasets, but in both cases AUCPR achieves significantly higher values. LeProblog reveals itself to be the closest 6 Available at http://alchemy.cs.washington.edu/mlns/uw-cse. 34 Table 1. Results of the experiments on all datasets. IMDBu refers to the IMDB dataset with the theory containing unseen predicates. CoraT refers to the theory containing transitive rules. Numbers in parenthesis followed by r mean the number of random restarts (when different from one) to reach the area specified. “me” means memory error during learning, “no” means that the algorithm was not applicable. AUCPR is the area under the Precision-Recall curve, AUCROC is the area under the ROC curve, both averaged over the five folds. E is EMBLEM, R is RIB, L is LeProbLog, C is CEM, A is Alchemy. Dataset E IMDB-SP 0.202(500r) IMDBu-SP 0.175(40r) IMDB-SM 1.000 IMDBu-SM 1.000 Cora 0.995(120r) CoraT 0.991 UW-CSE 0.883 AUCPR R L 0.199 0.096 0.166 0.134 me 0.933 me 0.933 0.939 0.905 no 0.970 me 0.270 C 0.202 0.120 0.537 0.515 0.995 me 0.644 AUCROC A E R L C A 0.107 0.931(500r) 0.929 0.870 0.930 0.907 0.020 0.900(40r) 0.897 0.921 0.885 0.494 0.820 1.000 me 0.983 0.709 0.925 0.338 1.000 me 0.983 0.442 0.544 0.469 1.000(120r) 0.992 0.994 0.999 0.704 me 0.999 no 0.998 me me 0.294 0.993 me 0.932 0.873 0.961 Table 2. Execution time in hours of the experiments on all datasets. R is RIB, L is LeProbLog, C is CEM and A is Alchemy. Time(h) EMBLEM R L C A IMDB-SP 0.01 0.016 0.35 0.01 1.54 IMDBu-SP 0.01 0.0098 0.23 0.012 1.54 IMDB-SM 0.00036 me 0.005 0.0051 0.0026 IMDBu-SM 3.22 me 0.0121 0.0467 0.0108 Cora 2.48 2.49 13.25 11.95 1.30 CoraT 0.38 no 4.61 me me UW-CSE 2.81 me 1.49 0.53 1.95 Dataset Table 3. Results of t-test on all datasets, relative to AUCPR and AUCROC. p is the p-value of a paired two-tailed t-test (significant differences at the 5% level in bold) between EMBLEM and all the others. R is RIB, L is LeProbLog, C is CEM, A is Alchemy. p - AUCPR p - AUCROC E-R E-L E-C E-A E-R E-L E-C IMDB-SP 0.2167 0.0126 0.3739 0.0134 0.3436 0.0012 0.3507 IMDBu-SP 0.1276 0.1995 0.001 4.5234e-5 0.2176 0.1402 0.0019 IMDB-SM me 0.3739 0.0241 0.1790 me 0.3739 0.018 IMDBu-SM me 0.3739 0.2780 2.2270e-4 me 0.3739 0.055 Cora 0.011 0.0729 1 0.0068 0.0493 0.0686 0.4569 CoraT no 0.0464 me me no 0.053 me UW-CSE me 1.5017e-4 0.0088 4.9921e-4 me 0.0048 0.2911 Dataset 35 E-A 0.015 1.01e-5 0.2556 6.54e-4 0.0327 me 0.0048 system to EMBLEM from the point of view of performances, able in addition to always complete learning, but with longer times (except for IMDBu-SM and UW-CSE). Looking at the overall results, AUCPR and AUCROC are higher or equal for EMBLEM than the other systems except on IMDBu-SP, where LeProbLog achieves a non-statistically significant higher AUCROC. Differences between EMBLEM and the other systems are statistically significant in 22 out of 43 cases. 5 Related Work Our work has close connection with various other works. [6] proposed an EM algorithm for learning the parameters of Boolean random variables given observations of the values of a Boolean function over them, represented by a BDD. EMBLEM is an application of that algorithm to probabilistic logic programs. Independently [14] also proposed an EM algorithm over BDD to learn parameters for the CPT-L language. [5] presented the CoPrEM algorithm that performs EM for the ProbLog language. We differ from this work in the construction of BDDs: they build a BDD for an interpretation while we build it for single ground atoms for the specified target predicate(s), the one(s) for which we are interested in good predictions. Moreover CoPrEM treats missing nodes as if they were there and updates the counts accordingly. Other approaches for learning probabilistic logic programs employ constraint techniques, or use EM, or adopt gradient descent. Among the approaches that use EM, [7] first proposed to use it to induce parameters and the Structural EM algorithm to induce ground LPADs structures. Their EM algorithm however works on the underlying Bayesian network. RIB [10] performs parameter learning using the information bottleneck approach, which is an extension of EM targeted especially towards hidden variables. Among the works that use a gradient descent technique we remind LeProbLog [4], which tries to find the parameters of a ProbLog program that minimize the MSE of the query probability and uses BDD to compute the gradient. Alchemy [8] is a state of the art SRL system that offers various tools for inference, weight learning and structure learning of Markov Logic Networks (MLNs). MLNs significantly differ from the languages under the distribution semantics since they extend first-order logic by attaching weights to logical formulas, but do not allow to exploit logic programming techniques. 6 Conclusions We have proposed a technique which applies an EM algorithm to BDDs for learning the parameters of Logic Programs with Annotated Disjunctions. It can be applied to all languages that are based on the distribution semantics and exploits the BDDs that are built during inference to efficiently compute the expectations for hidden variables. We executed the algorithm over the real datasets IMDB, UW-CSE and Cora, and evaluated its performances - together with four 36 other systems - through the AUCPR. These results show that EMBLEM uses less memory than RIB, CEM and Alchemy, allowing it to solve larger problems. Moreover its speed allows to perform a high number of restarts making it escape local maxima. In the future we plan to extend EMBLEM for learning LPADs structure. References 1. Bellodi, E., Riguzzi, F.: EM over binary decision diagrams for probabilistic logic programs. Tech. Rep. CS-2011-01, ENDIF, Università di Ferrara (2011) 2. De Raedt, L., Demoen, B., Fierens, D., Gutmann, B., Janssens, G., Kimmig, A., Landwehr, N., Mantadelis, T., Meert, W., Rocha, R., Santos Costa, V., Thon, I., Vennekens, J.: Towards digesting the alphabet-soup of statistical relational learning. In: NIPS Workshop on Probabilistic Programming (2008) 3. De Raedt, L., Kimmig, A., Toivonen, H.: ProbLog: A probabilistic prolog and its application in link discovery. In: International Joint Conference on Artificial Intelligence. pp. 2462–2467 (2007) 4. Gutmann, B., Kimmig, A., Kersting, K., Raedt, L.D.: Parameter learning in probabilistic databases: A least squares approach. In: European Conference on Machine Learning. LNCS, vol. 5211, pp. 473–488. Springer (2008) 5. Gutmann, B., Thon, I., De Raedt, L.: Learning the parameters of probabilistic logic programs from interpretations. Tech. Rep. CW 584, Department of Computer Science, Katholieke Universiteit Leuven, Belgium (June 2010) 6. Ishihata, M., Kameya, Y., Sato, T., Minato, S.: Propositionalizing the em algorithm by bdds. Tech. Rep. TR08-0004, CS Dept., Tokyo Institute of Technology (2008) 7. Meert, W., Struyf, J., Blockeel, H.: Learning ground CP-Logic theories by leveraging Bayesian network learning techniques. Fund. Inf. 89(1), 131–160 (2008) 8. Richardson, M., Domingos, P.: Markov logic networks. Mach. Learn. 62(1-2), 107– 136 (2006) 9. Riguzzi, F.: Extended semantics and inference for the Independent Choice Logic. Log. J. IGPL 17(6), 589–629 (2009) 10. Riguzzi, F., Mauro, N.D.: Applying the information bottleneck to statistical relational learning. Mach. Learn. (2011), to appear 11. Sato, T.: A statistical learning method for logic programs with distribution semantics. In: International Conference on Logic Programming. pp. 715–729. MIT Press (1995) 12. Singla, P., Domingos, P.: Discriminative training of Markov logic networks. In: National Conference on Artificial Intelligence. pp. 868–873. AAAI Press/The MIT Press (2005) 13. Singla, P., Domingos, P.: Entity resolution with Markov logic. In: International Conference on Data Mining. pp. 572–582. IEEE Computer Society (2006) 14. Thon, I., Landwehr, N., Raedt, L.D.: A simple model for sequences of relational state descriptions. In: European conference on Machine Learning. LNCS, vol. 5212, pp. 506–521. Springer (2008) 15. Vennekens, J., Verbaeten, S., Bruynooghe, M.: Logic programs with annotated disjunctions. In: International Conference on Logic Programming. LNCS, vol. 3131, pp. 195–209. Springer (2004) 37 Clustering XML Documents by Structure: a Hierarchical Approach (Extended Abstract) G. Costa, G. Manco, R. Ortale, and E. Ritacco ICAR-CNR Via Bucci 41c 87036 Rende (CS) - Italy Abstract. A new parameter-free approach to clustering XML documents by structure is proposed. The idea is to consider various forms of structural patterns occurring in the XML documents to form a hierarchy of nested clusters. At any level in the hierarchy, clusters explain how the XML documents can be grouped on the basis of common structural patterns of the form considered at that level. The resulting explanation is progressively refined at the subsequent level, where another type of structural patterns is used to divide the individual clusters from the above level into subgroups, revealing meaningful and previously uncaught structural differences. Each cluster in the hierarchy is summarized through a novel technique into a corresponding representative, that provides a clear and differentiated understanding of the structural information within the cluster. 1 Introduction The problem of clustering XML documents by structure has been extensively investigated, with the consequent development of several approaches, such as [5, 7–10]. XML trees can share various forms of common structural components, ranging from simple node/edge and pairwise tags [11], to more complex substructures such as groups of siblings, paths (either root-to-node [11] or root-toleaf [7]), as well as subtrees or even summaries [9, 10]. Therefore, if the addressed form of structural patterns does not accord with the underlying properties of XML data, valuable relationships of structural resemblance between the XML documents can be missed, with a consequent degrade of clustering effectiveness. Moreover, judging differences only in terms of one type of structural components may not suffice to effectively separate the available XML documents. This paper proposes a new hierarchical approach to clustering that considers various forms of structural patterns in the XML documents to progressively derive a hierarchy of nested clusters. In addition, the characterization of each cluster is accomplished by means of new summarization method, aimed at subsuming the structural properties within each cluster in terms of strongly representative substructures. 38 2 Partitioning XML Trees We introduce the notation used throughout the paper as well as some basic concepts. The structure of XML documents without references can be modeled in terms of rooted ordered labeled trees, that represent the hierarchical relationships among the document elements (i.e., nodes). Definition 1. XML Tree. An XML tree is a rooted, labeled, ordered tree, represented as a tuple t = (rt , Vt , Et , λt ), whose individual components have the following meaning. Vt a set of nodes and rt ∈ Vt is the root node of t, i.e. the only node with no entering edges. Et ⊆ Vt × Vt is a set of edges, catching the parent-child relationships between nodes of t. Finally, λt : Vt 7→ Σ is a node labeling function and Σ is an alphabet of node tags (i.e., labels). t u Parent-child relationship in t is denoted by ni ≺ nj , where ni , nj ∈ Vt such that ∃(ni , nj ) ∈ Et , ni is the parent while nj is the child. Ancestor-descendant relationship is indicated as ni ≺p nj , where p is the distance (in nodes) between the ancestor and the descendant (ni ≺1 nj is equivalent to ni ≺ nj ). Tree-like structures are also used to represent generic structural patterns occurring across a collection of XML trees. Definition 2. Substructure. Let t and s be two XML trees. s is said to be a substructure of t, if there exists a total function ϕ : Vs → Vt , that satisfies the following conditions for each n, ni , nj ∈ Vs . Firstly, (ni , nj ) ∈ Es iff ϕ(ni ) ≺p ϕ(nj ) in t with p ≥ 1. Secondly, λs (n) = λt [ϕ(n)]. t u The mapping ϕ preserves node labels and hierarchical relationships. In this latter regard, depending on the value of p, two definitions of substructures can be distinguished. In the simplest case p = 1 and a substructure s is simply an induced tree pattern that matches a contiguous portion of t, this is indicated as s v t. When p ≥ 1 [6, 12], s matches not necessarily contiguous portions of t, this is denoted as s ⊆ t and s is also said to be an embedded tree pattern of t. Our clustering is based on structural similiraty. Two documents are similar if they share some elements, which can be nodes, edges (parent-child relationship), paths (ancestor-descendant relationship), etc... For this reason we choose to cluster the documents in a multi-stage way, considering one by one a set of elements, belonging to a specific feature space (nodes, edge, paths, etc). At a given stage i, finding clusters in the high-dimensional feature space S (i) is a challenging issue for various reasons [3]. The XML trees are partitioned by the AT-DC algorithm [3], which is an effective hierarchical and parameter-free method for transactional clustering. The main clustering procedure is reported in fig. 1. It consists of m stages of clustering (line 1). The end user incorporates (at line 1) valuable domain knowledge and application semantics into the clustering process, by establishing the most appropriate set of structural features S (i) for each stage as well as the overall number m of stages. 39 Generate-Hierarchy(D) Input: a set D = {t1 , . . . , tN } of XML trees; Output: a set ∪i P (i) of multiple cluster partitions; 1: let S (i) be the set of features at stage i, with i = 1, . . . , m; 2: let i ← 1; 3: let P ← {D }; 4: while i ≤ m do 5: while P = 6 ∅ do 6: let C be a cluster in P; 7: P ← P − C; 8: R ← Generate-Clusters(C, S (i) ); 9: for each C 0 ∈ R do 10: let C 0 ← R − {C 0 } be the set of siblings of C 0 ; 11: end for 12: P (i) ← P (i) ∪ R; 13: end while 14: for each C ∈ P (i) do 15: Rep(C) ← MineRep(C, C, α); 16: end for 17: P ← P (i) ; 18: i ← i + 1; 19: end while 20: RETURN ∪i P (i) ; Fig. 1. The hierarchical clustering process The generic stage i (lines 4-19) consists of two phases: cluster separation and summarization. Cluster separation exploits AT-DC to divide the individual clusters belonging to the current partition P with respect to the feature space S (i) (lines 5 - 13). At the beginning, i.e. when i = 1, the current partition P includes a single cluster, which coincides with the whole dataset D of XML trees (line 3). The partition P (i) resulting at the end of stage i (line 13) is itself a collection of partitions. More precisely, at the current stage i, each parent cluster C from P (i−1) is divided into an appropriate number of child clusters (line 8), which together form the partition R of the foresaid C. At this point, each child cluster C 0 in R is associated (lines 9-11) with its siblings C 0 = R − C (for the cluster summarization purpose) and R is then added to the ongoing P (i) . Cluster summarization (lines 14-16) is applied to each cluster C from the obtained P (i) . It consists of a procedure, discussed in section 3, which associates C with a set Rep(C) of representative substructures, that subsume the structural information within C. P (i) becomes (at line 17) the current partition P for the subsequent stage i + 1. At this stage, AT-DC is re-applied to further divide every cluster C ∈ P (i) with respect to another set of structural features, i.e., S (i+1) . The choice of a distinct feature space at each stage guarantees a progressively increasing degree of structural homogeneity. Moreover, at each distinct stage, representatives provide a summarization of the tree structures within the corresponding clusters in terms of (a combination of) the structural features considered a that particular stage. Hence, the representative of a subcluster highlights local patterns of structural homogeneity, that are not caught by the representative of the parent cluster. 40 3 Cluster Summarization The representative of a cluster of XML trees is modeled as a set of highly representative tree patterns, which provide an intelligible summarization of the most relevant structural properties in the cluster. Notice that, as mentioned before, a cluster is already characterized by a set of relevant features. However, features can be combined further, and they do not necessarily allow to distinguish among different clusters. A set of tree patterns is actually viewed as the representative of a cluster of XML trees, if its frequency in the cluster is much higher then elsewhere, and it exhibits a strong degree of correlation with the documents of the cluster. A representative can be computed by merging patterns. To avoid combinatory explosion, we consider only two types of tree pattern composition, namely parent-child and sibling tree patterns composition. Definition 3. Parent-child tree pattern. A parent-child tree pattern is an arrangement of two basic tree patterns, in which one of the two tree patterns is rooted at some leaf node of the other tree pattern. Let si and sj be two generic tree patterns. Also, assume that l is some leaf node of si . The operator si /l sj defines a new parent-child tree pattern s, such that |Vs | = |Vsi | + |Vsj | and |Es | = |Esi | + |Esj | + 1, wherein the root rsj of sj is a child of l. Given any two tree patterns si and sj , the set of all possible parent-child tree patterns in which the root of sj is a child of the individual leaves of si is denoted as [ {si /l sj } si / sj = l∈Lsi where Lsi represents the set of leaves of si . t u A parent-child tree pattern is a vertical arrangement of two component tree patterns. Instead, a sibling tree pattern follows from an horizontal arrangement of its components. Definition 4. Sibling tree pattern. Given two tree patterns with a same label at their roots, a sibling tree pattern is a composite structure, whose root-to-leaf paths are the union of the root-to-leaf paths in the two component patterns. Let si and sj be two tree patterns such that λsi (rsi ) = λsj (rsj ). The Representative Discovery Procedure is an Apriori-based technique whose candidate generation phase is performed through these compositions. 4 Evaluation The behavior of the devised clustering approach is now investigated through an empirical evaluation with three objectives: the assessment of clustering quality, the evaluation of cluster-summarization and a performance comparison. 41 All experiments were conducted on a Windows machine, with an Intel Itanium processor, 2Gb of memory and 2Ghz of clock speed. Standard benchmark data sets were employed for a direct comparison against the competitors. Realworld data, named Real, encompasses the following collections: Astronomy (217 documents), Forum (264 messages), News (64 documents), Sigmod (51 documents), Wrapper (53 documents). The distribution of tags within the above documents is quite heterogeneous, due to the complexity of the DTDs associated with the classes, and to the documents’ semantics. Three further synthetic data sets were generated from as many collections of DTSs reported in [5]. The first synthesized data set, referred to as Synth1, comprises 1000 XML documents produced from a collection of 10 heterogeneous DTDs (illustrated in fig. 6 of [5]), that were individually used to generate 100 XML documents. These DTDs exhibit strong structural differences and, hence, most clustering algorithms can produce high-quality results. A finer evaluation can be obtained by investigating the behavior of the compared algorithms on a collection of XML documents, that are very similar to one another from a structural point of view. To perform such a test, a second synthesized data set, referred to as Synth2 and consisting of 3000 XML documents, was assembled from 3 homogeneous DTDs (illustrated in fig. 7 of [5]), individually used to generate 1000 XML documents. Experiments over Synth2 clearly highlight the ability of the competitors at operating in extremely-challenging applicative-settings, wherein the XML documents share multiple forms of structural patterns. Additionally, Synth3 is a collection consisting of the synthesized documents in [7], which exhibit a 30% degree of overlap. Synth3 allows us to compare the effectiveness of the devised approach to the approach proposed in [7]. Clustering effectiveness is evaluated over each partition Pi = {C1 , . . . , Ck } and it is measured in terms of average precision and recall [1]. Table 1 shows the results of clustering on such collections. As we can see, precision and recall are optimal, even for the collection Synth2 of homogeneous documents. Collection N. of Docs Classes Clusters Avg Precision Avg Recall Avg Γ Time Real Synth1 Synth2 Synth3 Synth4 649 1000 3000 1400 800 5 10 3 7 8 5 10 3 7 10 1 1 1 1 1 1 1 1 1 0.8 0.9558 0.9455 0.3833 0.7875 0.7127 20.48s 13.32s 7.5s 2.68s 3.68s Table 1. Evaluation of separability and homogeneity To deeply investigate the Generate-Hierarchy procedure effectiveness, we produce a new data set Synth4 which requires the multi-layer clustering over all the features we consider. It is composed by 800 documents whose schema is shown in fig. 2. The DTDs capture substantial similarities and differences. In particular, all dataset exhibit different paths (but they can share some edges). The documents in DTD4 can be further split, since they can exhibit trees with paths ending 42 Fig. 2. DTDs for the Synth3 dataset in the node A6. Also, node frequencies in DTD4 can substantially differ, thus differentiating this DTD from the others even at a node level. This situation is fully captured by the clustering algorithm, as shown in fig. 3(a). DTD4 is separated from DTD3 at the node level and further split in two subclusters at the edge level, according to whether edges contain or not edge (A9, A6). Also, the trees containing such an edge can be further split according on whether or not they contain the path from A10 to A6. Notice that, by the contrary, DTD8 does not behave similarly, since there is no such a node like A6 that differentiates the trees in the class. (a) Cluster hierarchy for Synth3 (b) Cluster hierarchy for Sigmod Fig. 3. The evaluation of the multi-stage clustering is further confirmed by experimenting on Sigmod. As already mentioned, this dataset consists of documents complying with three different DTDs. In particular, the distribution of the documents is unbalanced, since one of the DTDs, named IndexTermsPage, contains much more documents the the other ones. Figure 3(b) shows that GenerateHierarchy separates all documents complying to different DTDs and further splits the documents in the class related to IndexTermsPage, according to whether or not these documents contain the optional elements described in the DTD (mainly, categoryAndSubjectDescriptorsTuple, category, content and term). In particular, the separation of such a class leads to two subclasses C1 43 and C2 , that can be described by two DTDs, both subsumed by IndexTermsPage. The difference between C1 and C2 is the absence (in C1 ) and the presence in C2 , of the elements of IndexTermsPage. The evaluation of the accuracy of cluster summarization is inspired to an idea originally proposed in [5] for a different purpose, i.e., measuring the structural homogeneity of a set of intermediate clusters obtained while partitioning a collection of XML documents. Let t be an XML tree and R a set of substructures. The representativeness γ(R, t) of R with respect to t is the fraction of nodes in t matched by the embedded substructures of R: γ(R, t) = | ∪s∈R,s⊆t {n ∈ V|Vs 7→ V ⊆ Vt }| |Vt | where, Vt and Vs are the sets of nodes of, respectively, the XML tree t and the generic substructure s. V is instead the subset of the nodes in t matched by the nodes of s (which is the meaning of notation Vs 7→ V). Representativeness can be easily generalized to clusters. The representativeness Γ [Rep(C)] of the representative Rep(C) with respect to a cluster C can be defined as the average representativeness of the documents in the cluster. A connection between cluster representativeness and structural homogeneity explains unexpectedly low Γ values. Representativeness is high when the representative frequently occurs in a cluster but not in the other ones. In the case of homogeneous documents the erasure of candidate structures is very frequent: the only structures that survives are very specific, so they are infrequent in the cluster (for example in Synth2 ). Table 1 shows the average Γ value exhibited in each experiment. In order to evaluate the scalability of the algorithm, we used the DTDs for Synth1 and produced respectively 100, 1000, 10.000 and 100.000 documents with 2, 4, and 8 clusters. The results are shown in fig. 4. As we can see, the algorithm is linear both in the number of documents and in the number of clusters. Fig. 4. Performance in milliseconds for data sets of different size At the end of this intensive empirical evaluation, the devised approach can be compared against a state-of-the-art competitor, namely the XProj approach [5]. By looking at the performance of XProj reported in [5], it can be observed that 44 our clustering approach attains the same quality. However, two strong advantages of the proposed approach are: the development of a hierarchy of nested clusters, explaining multiple forms of structural relationships in the data; the summarization of a cluster of XML documents, which provides an intelligible subsumption of its structural properties. Also, notice that the scalability of our approach is orders of magnitude higher than the one of XProj. Finally, the devised approach is fully-automatic (i.e., parameter-free), whereas the optimal performance of XProj, on each data set, is the consequence of a complex setting process. 5 Conclusions A new approach to clustering of XML documents was proposed, that produces a hierarchy of nested clusters. Along the paths from the root to the leafs of the hierarchy, the approach progressively separates the XML data by looking at the occurrence of different types of structural patterns in their structures. Also, each cluster in the hierarchy is subsumed, through a novel summarization method, by a set of representative substructures, that provide an understanding of the structural properties considered in the cluster. A comparative evaluation proved that the devised approach is on a par and even better than established competitors in terms of effectiveness, scalability and cluster summarization. References 1. R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. AddisonWesley, 1999. 2. R. Baumgartener, S. Flesca, and G. Gottlob. Visual web information extraction with lixto. In Procs. VLDB’01 Conf., pages 119 – 128, 2001. 3. E. Cesario, G. Manco, and R. Ortale. Top-down parameter-free clustering of highdimensional categorical data. IEEE TKDE, 19(12):1607 – 1624, 2007. 4. V. Crescenzi, G. Mecca, and P. Merialdo. Roadrunner: Towards automatic data extraction from large web sites. In Procs. VLDB’01 Conf., pages 109–118, 2001. 5. C. C. Aggarwal et al. Xproj: A framework for projected structural clustering of xml documents. In Procs. SIGKDD’07 Conf., pages 46 – 55, 2007. 6. C. Wang et al. Efficient pattern-growth methods for frequent tree pattern mining. In Procs. PAKDD’04 Conf., pages 441 – 451, 2004. 7. G. Costa et al. A tree-based approach to clustering xml documents by structure. In Procs. PKDD’04 Conf., pages 137 – 148, 2004. 8. M. L. Lee et al. Xclust: Clustering xml schemas for effective integration. In Procs. CIKM’02 Conf., pages 292 – 299, 2002. 9. T. Dalamagas et al. A methodology for clustering xml documents by structure. Information Systems, 31(3):187 – 228, 2006. 10. W. Lian et al. An efficient and scalable algorithm for clustering xml documents by structure. IEEE TKDE, 16(1):82 – 96, 2004. 11. S. Helmer. Measuring the structural similarity of semistructured documents using entropy. In Procs. VLDB’07 Conf., pages 1022 – 1032, 2007. 12. M. J. Zaki. Efficiently mining frequent trees in a forest: Algorithms and applications. IEEE TKDE, 17(8):1021 – 1035, 2005. 45 Outlier Detection For XML Documents (Extended Abstract) Giuseppe Manco and Elio Masciari ICAR-CNR {manco,masciari}@icar.cnr.it Abstract. XML (eXtensible Markup Language) became in recent years the new standard for data representation and exchange on the WWW. This has resulted in a great need for data cleaning techniques in order to identify outlying data. In this paper, we present a technique for outlier detection that single out anomalies with respect to a relevant group of objects. We exploit a suitable encoding of XML documents that are encoded as signals of fixed frequency that can be transformed using Fourier Transforms. Outlier are identified by simply looking at the signal spectra. The results show the effectiveness of our approach. 1 Introduction An outlier is an observation that differs so much from other observations as to arouse suspicion that it was generated by a different mechanism [8]. There exist several approaches to the identification of outliers, namely, statistical-based [5], deviation-based [4], distance-based [3], density-based [6], projection-based [1], MDEF-based [12], and others. Abstracting from the specific method being exploited the general outlier detection task is the problem of identifying deviations from the general patterns characterizing a data set. Detecting outliers is important in many application scenarios, as an example it can be used for improving data cleaning approaches, where outliers are often data noise or errors diminishing the accuracy of data mining. Outlier detection is also the core of applications such as fraud detection, stock market analysis, intrusion detection, marketing, network sensors, and email spam detection, where irregular patterns entail special attention. Due to the increasing usage of semi-structured data models like XML (eXtensible Markup Language), that is the new standard for data representation and exchange on the Web there is a great need for outlier detection strategies entailed for such data. Although outlier detection methods are well established for relational data, adapting them directly to XML data is unfeasible because XML and relational data models differ in several aspects. First, XML data contain multiple levels of nested elements (or attributes) organized in a tree-based structure, whereas relational data models have a flat tabular structure. Indeed, the hierarchical structure of XML data induce an ordering lacking in relational data. Also, the modeling objectives for XML and relational data are different, and therefore different relationships are represented. In relational data models, the primary-foreign key relationships between entities form the basis for data normalization and referential integrity. On the contrary, relationships between the XML elements are encoded in hierarchies, often with direct semantic correspondence to the real-world relations such as containment and composition. Despite its importance, XML outlier detection has not been paid the attention it deserves. There exists few works addressing structural and attribute outlier detection for XML. The main distinction is between class outlier and attribute outlier, i.e. respectively outlier based on the overall structure of the document and outlier based on univariate points that exhibit deviating correlation 46 behavior with respect to other attributes[10]. In [10] an approach is presented for correlation based attribute outlier detection, while main approaches for class outlier detection try to adapt the techniques defined for the relational setting to the semistructured one. They have been mainly proposed for data cleaning purposes like in [15, 14, 16]. Our approach. We will tackle in this paper the class outlier detection problem. The basic intuition exploited in this paper is that an XML document has a “natural” interpretation as a time series (namely, a discrete-time signal), in which numeric values summarize some relevant features of the elements enclosed within the document. We can get an example evidence of this observation by simply indenting all the tags in a given document according to their nesting level. Indeed, the sequence of indentation marks (as they appear within the document rotated by 90 degrees), can be looked at as a time series, whose shape roughly describes the document’s structure. Hence, a key tool in the analysis of time-series data is the use of the Discrete Fourier Transform (DFT): some useful properties of the Fourier transform, such as energy concentration or invariance under shifts, enable to analyze and manipulate signals in a very powerful way. The choice of comparing the frequency spectra follows both effectiveness and efficiency issues. Indeed, the exploitation of DFT leads to abstract from structural details which, in most application contexts, should not affect the similarity estimation (such as, e.g., different numbers of occurrences of a shared element or small shifts in the actual positions where it appears). This eventually makes the comparison less sensitive to minor mismatches. Moreover, a frequency based approach allows to estimate the similarity through simple measures (e.g., vector distances) which are computationally less expensive than techniques based on the direct comparison of the original document structures. To summarize, we propose to represent the structure of an XML document as a time series, in which each occurrence of a tag in a given context corresponds to an impulse. By analyzing the frequency spectra of the signals, we can hence state the degree of (structural) similarity between documents. It is worth noticing that the overall cost of the proposed approach is only O(N log N ), where N is the maximum number of tags in the documents to be compared. Once defined an effective distance measure we will exploit a distance-based outlier detection algorithm in order to single out the outlying documents. 2 Problem Statement and Overview of the Proposal We begin by presenting the basic notation for XML documents that will be used hereafter. An XML document is characterized by tags, i.e., terms enclosed between angled brackets. Tags define the structure of an XML document and provide the semantics of the information enclosed. A tag is associated with a tag name (representing the term enclosed between angled brackets), and can appear in two forms: either as a start-tag (e.g., <author>), or as an end-tag (characterized by the / symbol, like, e.g., in </author>). Finally, a tag instance denotes the occurrence of a tag within a certain document. It is required that, in a well-formed XML document, tags are properly nested, i.e. each start-tag has a corresponding end-tag at the same level. Therefore, an XML document can be considered as an ordered tree, where each node (an element) represents a portion of the document, delimited by a pair of start-tag and end-tag instances, and denoted by the tag name associated with the instances. The structure of an XML document corresponds to the shape of the associated tree. In a tree, several types of structural information can be detected, which correspond to different refinement levels: for example, attribute/element labels, edges, paths, subtrees, etc. Defining the similarity among two documents essentially means 47 choosing an appropriate refinement level and comparing the documents according to the features they exhibit at the chosen level. Different choices may result in rather dissimilar behaviors: in particular, comparing simple structural components (such as, e.g., labels or edges) allows for an efficient computation of the similarity, but typically produces loose-grain similarity values. On the other hand, complex structural components would make the computation of similarity inefficient, and hence unpractical. Consider, for example, the documents represented in Fig. 1. If a comparison of nodes or edges is exploited, documents book1 and book2 appear to be similar, even though the subtrees rooted at the book element appear with different frequencies. Accounting for frequencies does not always help: for example, if the order of appearance of the subtrees of the xml element in book3 were changed, the resulting tree still should have the same number of nodes, edges and even paths. In principle, approaches based on tree-edit distance [11] can better quantify the difference between XML trees; however, they turn out to be too expensive in many applications contexts, as they are generally quadratic w.r.t. document sizes. Finally, notice that solutions based on detecting local substructures [9] to be used as features may be even hard to handle, for they showing two main disadvantages: first, they may imply ineffective representations of the trees in high dimensional spaces, and second, costly feature extraction algorithms are required. xml xml book book xml book book title book title author title (a) book1 author title author book title author (b) book2 author title author book title author name email (c) book3 Fig. 1. book1 and book2 have the same elements, but with different cardinality. By contrast, book3 induces a different structure for the author element. In our opinion, an effective notion of structural similarity should take into account a number of issues. First of all, it is important to notice that each document may induce a definition of the elements involved. Thus, an appropriate comparison between two documents should rely on the comparison of such definitions: the more different they are, the more dissimilar are the documents and this information has to be exploited for signaling candidate outliers. Our main objective is the development of an efficient method which is able to approximate the above features at best. Thus, we can state the problem of finding the structural similarity in a set of XML documents as follows. Given a set D = {d1 , . . . , dn } of XML documents, we aim at building a similarity matrix S, i.e., a matrix representing, for each pair (di , dj ) of documents in D, an optimal measure of similarity sij . Here, optimality refers to the capability of reflecting the above described differences. Observe that we do not address here the problem of finding which parts of two documents are similar or not, as, e.g., tree-edit based techniques do. We propose a technique which is essentially based on the idea of associating each document with a time series representing, in a suitable way, both its basic elements and their relationships within the document. More precisely, we can assume a preorder visit of the tree-structure of an XML docu- 48 ment. As soon as we visit a node of the tree, we emit an impulse encoding the information corresponding to the tag. The resulting signal shall represent the original document as a time series, from which relevant features characterizing a document can be efficiently extracted. As a consequence, the comparison of two documents can be accomplished by looking at their corresponding signals. The main features of the approach can be summarized as follows: 1) Each element is encoded as a real value. Hence, the differences in the values of the sequence provide for an evaluation of the differences in the elements contained by the documents; 2) The substructures in the documents are encoded using different signal shapes. As a consequence, the analysis of the shapes in the sequences realizes the comparison of the definitions of the elements. 3) Context information can be used to encode both basic elements and substructures, so that the analysis can be tuned to handle in a different way mismatches which occur at different hierarchical levels. In a sense, the analysis of the way the signal shapes differ can be interpreted as the detection of different definitions for the elements involved in the documents. Moreover, the analysis of the frequencies of common signal shapes can be seen as the detection of the differences between the occurrences associated with a repetition marker. In this context, the proposed approach can be seen as an efficient technique, which can satisfactorily evaluate how much two documents are similar w.r.t. the structural features previously discussed. Notably, the use of time-series for representing complex XML structures, combined with an efficient frequency-based distance function, is the key for quickly evaluating structural similarities: if N is the maximum number of tags in two documents, they can be compared in only O(N log N ) time. In particular, the use of DFT supports the above described notion of similarity: if two documents share many elements having a similar definition, they will be recognized as similar, even when there are repeated and/or optional sub-elements. Indeed, working on frequency spectra makes the comparison less sensitive to the differences in the number of occurrences of a given element and to small shifts in the actual positions where it occurs in the documents. The details on representing an XML document as a signal are omitted here due to space limitations, a complete explanation can be found in [7]. 3 Comparing Documents using DFT Once defined a proper document encoding, we can now detail the similarity measures for XML documents, sketched in section 1. As already mentioned, we can assume that we are visiting the tree-structure of an XML document d (using a preorder visit) starting from an initial time t0 . We also assume that each tag instance occurs after a fixed time interval ∆. The total time spent to visit the document is N ∆, where N is the size of tags(d). During the visit, as we find a tag, we produce an impulse which depends on a particular tag encoding function e and on the overall structure of the document (i.e., the document encoding function enc). As a result of the above physical simulation, the visit of the document produces a signal hd (t), which usually changes its intensity in the time interval [t0 , t0 + N ∆). The intensity variations are directly related to the opening/closure of tags: ½ [enc(d)](k) if t0 + k∆ ≤ t < t0 + (k + 1)∆ hd (t) = 0 if t < t0 or t ≥ t0 + N ∆ Comparing such signals, however, might be as difficult as comparing the original documents. Indeed, comparing documents having different lengths, requires 49 the combination of both resizing and alignment operations. Moreover, the intensity of a signal strongly depends on the encoding scheme adopted, which can in turn depend from the context (as in the case, e.g., of the multilevel encoding scheme). In order to compare two documents di and dj , hence, we can exploit the properties of the corresponding transforms. In particular [2, 13], a possibility is to exploit that, by Parseval’s theorem, energy (total power) is an invariant in the transformation (and hence the information provided by the encoding remains unchanged in the transform). However, a more effective discrimination can exploit the difference in the magnitude of frequency components: in a sense, we are interested (i ) in abstracting from the length of the document, and (ii ) in knowing whether a given subsequence (representing a subtree in the XML document) exhibits a certain regularity, no matter where the subsequence is located within the signal. In particular, we aim at considering as (almost) similar documents exhibiting the same subtrees, even if they appear at different positions. Now, as the encoding guarantees that each relevant subsequence is associated with a group of frequency components, the comparison of their magnitudes allows the detection of similarities and differences between documents. Observe that measuring the energy of the difference signal would result in a low similarity value. On the other side, if the phases of the documents’ transforms are disregarded, documents are more likely to be considered as similar. A viable approximation can be the interpolation of the missing coefficients starting from the available ones. It is worth noticing that the approximation error due to interpolation is inversely proportional to min(Ndi , Ndj ): the more elements are available in a document d, the better the DFT approximates the (continuous) Fourier Transform of the signal hd (t), and consequently the higher is the degree of reliability of interpolation. As a practical consequence, the approach is expected to exhibit good results with large documents, providing poorer performances with small documents. Definition 1. Let d1 , d2 be two XML documents, and enc a document encoding function, such that h1 = enc(d1 ) and h2 = enc(d2 ). Let DFT be the Discrete Fourier Transform of the (normalized) signals. We define the Discrete Fourier Transform distance of the documents as the approximation of the difference of the magnitudes of the DFT of the two encoded documents: 12 X ¡¯ ¯ ¯ ¯¢2 ¯[DFT(h ¯ ¯ ˜ ¯ ˜ dist(d1 , d2 ) = 1 )](k) − [DFT(h 2 )](k) M/2 k=1 ˜ is an interpolation of DFT to the frequencies appearing in both d1 where DFT and d2 , and M is the total number of points appearing in the interpolation, i.e., M = Ndi + Ndj − 1 points. u t 3.1 Outlier Identification Once defined a technique to state the similarity between two XML documents we need to define a strategy that, exploiting such a technique, identifies anomalies in the data set. Definition 2 (Fourier Based XML Outlier). Given a set of XML documents S, a positive integer k, and a positive real number R, a documents d ∈ S is a DB(k, R)-outlier, or a distance-based outlier with respect to parameters k and R, if less than k objects in S lie within distance R from o w.r.t. our distance metric. 50 The threshold values R and k has to be chosen depending on the scenario being monitored. Once defined our notion of outlier, we can design an effective method for outlier detection. Algorithm 1 Function Compute Outlier: INPUT: a set of XML documents S = {d1 , · · · , dn }, a pair of threshold values R and k, an XML document dnew ; OUTPUT: Y es if dnew is an outlier no otherwise; begin temp = 0 For each di ∈ S do dist = computeDF T Distance(di , dnew ) if dist > R temp = temp + 1 if temp > k return Y es return N o; end Function computeDF T Distance evaluates the DF T distance between the XML documents being analyzed and the XML documents previously collected. If the computed distance is greater than the threshold distance set by the user we will increase an auxiliary variable temp. If temp is greater than the threshold value k the document is marked as outlying. Proposition 1. Algorithm 1 works in time O(|S|Ṅ log(N )). The running time can be trivially computed by observing that for each document being analyzed we have to compute the Fourier Transformation and this operation is performed in O(N log(N )) time that is the dominant operation for the Algorithm. 4 Experimental Results In this section, we present the experiments we performed to assess the effectiveness of the proposed approach in detecting outliers. To this purpose, a collection of tests is performed, and for each test some relevant groups of homogeneous documents (document classes) are considered. The direct result of each test is a similarity matrix representing the degree of similarity for each pair of documents in the data set and the number of detected outliers. The evaluation of the results relies on some a priori knowledge about the document classes being used that was obtained by domain experts or available from the datasets providers. We performed several experiments on a wide variety of real datasets. More in detail we report here due to space limitations the results on the following data. The documents used belong to two main classes:1) Astronomy, a data set containing 217 documents extracted from an XML-based metadata repository, that describes an archive of publications owned by the Astronomical Data Center at NASA/GSFC (http://adc.gsfc.nasa.gov/); 2) Sigmod, a data set composed of 51 XML documents containing issues of SIGMOD Record. Such documents were obtained from the XML version of the ACM SIGMOD Web site. For each class we added some outlying documents by perturbating the original DTD’s. We compared our approach with the one proposed in [16], we refer to it as N oise. In order to perform a simple quantitative analysis we produce for each test a similarity matrix, aimed at evaluating the resulting neighbors similarities (i.e., the average of the values computed for documents belonging to the same class), and to compare them with the outer similarities (i.e., the similarity computed by considering only documents belonging to different classes). To this 51 purpose, values inside the matrix can be aggregated according to the class of membership of the related elements: given a set of documents belonging to n prior classes, a similarity matrix S about these documents can be summarized by a n × n matrix CS , where the generic element CS (i, j) represents the average similarity between class i and class j. P x,y∈Ci ,x6=y DIST (x,y) iff i = j i |×(|Ci |−1) CS (i, j) = Px∈C|C,y∈C DIST (x,y) i j otherwise |Ci |×|Cj | where DIST (x, y) is the chosen distance metric (N oise metric or our F ourier metric). The above definition is significant since we normalize the metric value so eventually we can use different approaches, this will allow us to compare performance in the ideal setting for any approach. The higher are the values on the diagonal of the corresponding CS matrix w.r.t. those outside the diagonal, the higher is the ability of the similarity measure to separate different classes. In the following we report a similarity matrix for each dataset being considered, as it will be clear the reported results show that our technique is quite effective for outlier detection. In particular, the similarity matrix will give an intuition about the ability of the approach to catch the neighboring documents for each class while the number of outliers detected for each dataset is reported in a separate table. We used for the experiments the following parameter values: as k value the maximum number of documents supposed to belong to each class and as R value the average distance between documents belonging to the same class. Measuring Effectiveness for Astronomy. For this dataset our prior knowledge is the partition of the documents into two classes. As it is easy to see in Figure 2(a) and (b) F ourier outperforms N oise by allowing a perfect assignment of the proper neighboring class to each document. N oise Class 1 Class 2 F ourier2D Class 1 Class 2 Class 1 Class 2 0.6250 1 1 0.6250 Class 1 0.9790 0.8528 Class 2 0.8528 0.9915 (a) (b) Fig. 2. N oise and F ourier similarity matrices for Astronomy dataset In Figure 3 the number of detected outlier is reported. The actual number of outliers for each class is 7 so as it is easy to see F ourier exactly detected all the outliers in the dataset, such a result is quite understandable considering that the similarity matrix for F ourier exactly recognized the neighboring documents. M ethod N oise F ourier Class 1 Class 2 5 7 4 7 Fig. 3. N oise and F ourier2D number of detected outliers for Astronomy dataset Measuring Effectiveness for Sigmod. In this case there were 3 main classes as it is shown in Figure 4(a) and (b). Also in this case F ourier outperforms N oise. As we can see, differences among the various classes are marked with higher precision by F ourier. This is mainly due to the fact that our approach is quite discriminative since it takes into account all the document features. For this dataset the number of actual outliers was 8 for each class, as it is easy to see in Figure 5 F ourier still outperforms N oise for this dataset. 52 N oise Class 1 Class 2 Class 3 Class 1 0.9986 0.7759 0.7055 Class 2 0.7759 0.9889 0.7566 Class 3 0.7055 0.7566 0.9920 F ourier Class 1 Class 2 Class 3 Class 1 Class 2 Class 3 0.9885 0.7439 0.7108 0.7439 0.9899 0.7223 0.7108 0.7223 0.9874 (a) (b) Fig. 4. N oise and F ourier similarity matrices for Sigmod dataset M ethod Class 1 Class 2 Class 3 N oise F ourier 6 7 4 8 5 8 Fig. 5. N oise and F ourier number of detected outliers for Sigmod dataset 5 Conclusion In this paper we addressed the problem of detecting outliers in XML data. The technique we have proposed is mainly based on the idea of representing a document as a signal. Thereby, the similarity between two documents can be computed by analyzing their Fourier transforms thus defining a distance measure that can be exploited to define distance based outliers. Experimental results showed the effectiveness of the approach in detecting outlying XML documents. References 1. C.C. Aggarwal and P. Yu. Outlier detection for high dimensional data. In SIGMOD01, 2001. 2. R. Agrawal, C. Faloutsos, and A. Swami. Efficient Similarity Search in Sequence Databases. In FODO’93, pages 69–84, 1993. 3. F. Angiulli and F. Fassetti. Dolphin: An efficient algorithm for mining distancebased outliers in very large datasets. TKDD, 3(1), 2009. 4. A. Arning, C. Aggarwal, and P. Raghavan. A linear method for deviation detection in large databases. In KDD’96, page 164169, 1996. 5. V. Barnett and T. Lewis. Outliers in Statistical Data. John Wiley & Sons, 1994. 6. M.M. Breunig, H. Kriegel, R. Ng, and J. Sander. Lof: Identifying density-based local outliers. In SIGMOD00, 2000. 7. S. Flesca, G. Manco, E. Masciari, L. Pontieri, and A. Pugliese. Fast detection of xml structural similarity. IEEE TKDE, 17(2):160–175, 2005. 8. D. Hawkins. Identification of Outliers. Monographs on Applied Probability and Statistics. Chapman & Hall, 1980. 9. H. Kashima and T. Koyanagi. Kernels for semi-structured data. In Procs. Int. Conf. on Machine Learning (ICML’02), pages 291–298, 2002. 10. Judice L. Y. Koh, Mong Li Lee, Wynne Hsu, and Wee Tiong Ang. Correlationbased attribute outlier detection in xml. In Proceedings of the 2008 IEEE 24th International Conference on Data Engineering, pages 1522–1524, Washington, DC, USA, 2008. IEEE Computer Society. 11. A. Nierman and H.V. Jagadish. Evaluating structural similarity in XML documents. In Procs. 5th Int. Workshop on the Web and Databases (WebDB 2002), 2002. 12. S. Papadimitriou, H. Kitagawa, P. Gibbons, and C. Faloutsos. Loci: Fast outlier detection using the local correlation integral. In ICDE’03, page 315326, 2003. 13. D. Rafiei and A. Mendelzon. Efficient retrieval of similar time series. In FODO’98, 1998. 14. Choh Man Teng. Polishing blemishes: Issues in data correction. IEEE Intelligent Systems, 19:34–39, 2004. 15. Melanie Weis and Felix Naumann. Dogmatix tracks down duplicates in xml. In Proceedings of the 2005 ACM SIGMOD international conference on Management of data, SIGMOD ’05, pages 431–442, New York, NY, USA, 2005. ACM. 16. Xingquan Zhu and Xindong Wu. Class noise vs. attribute noise: a quantitative study of their impacts. Artif. Intell. Rev., 22:177–210, November 2004. 53 P2P support for OWL-S discovery Domenico Redavid† , Stefano Ferilli? , and Floriana Esposito? † ? Artificial Brain S.r.l., Bari, Italy Computer Science Department, University of Bari “Aldo Moro”, Italy [email protected] {ferilli, esposito}@di.uniba.it Abstract. The discovery of Web services is often influenced by the rigid structure of registries containing their XML description. In recent years some methods that replace the traditional UDDI registry with Peer To Peer networks for the creation of catalogs of Web services have been proposed in order to make this structure flexible and usable. This paper proposes a different view by placing the semantic description of services as content of P2P networks and showing that all the needed information for an efficient Web service discovery is already contained in its OWL-S description. 1 Introduction The discovery of Web services (Ws)1 is achieved through Universal Description, Discovery and Integration (UDDI)2 , which provides a standard mechanism to register and search WS descriptions. An UDDI registry is an indexed database that contains instances of Web Services Description Language (WSDL)3 in turn based on eXtensible Markup Language (XML)4 and independent from hardware platforms. A requester needing to use a service queries the UDDI registry to find the one that best meets its needs. The register returns an access point and WSDL description which are then used by the requester to build needed SOAP5 messages to communicate with the actual service. The UDDI registry is supported by a worldwide network of nodes connected in a federation. When a client sends information to a registry, this is propagated to other nodes. In this way it implements data redundancy, providing a certain degree of reliability. However, data replication implies lower consistency and is not a scalable approach. Another limitation of UDDI is the search mechanism: it can focus only on a single search criterion such as name, location, category 1 2 3 4 5 W3C Web of Services - http://www.w3.org/standards/webofservices/ Universal Description, Discovery and Integration v3.0.2 (UDDI), OASIS Specification - http://uddi.org/pubs/uddi_v3.htm Web Services Description Language (WSDL) Version 2.0 Part 0: Primer, W3C Recommendation 26 June 2007 - http://www.w3.org/TR/wsdl20-primer/ W3C XML Technology - http://www.w3.org/standards/xml/ SOAP Version 1.2 Part 0: Primer (Second Edition), W3C Recommendation 27 April 2007 - http://www.w3.org/TR/soap12-part0/ 54 of business, etc.. Within the Service-Oriented Architecture (SOA) [3], the register has a role similar to yellow pages where a list of services can be found. To fully exploit the potential of this type of architecture, the register should be consultable not only by humans but also by software systems that need to find, select and compose services in an automatic way. In recent years research has been focusing on Peer-to-Peer (P2P) technologies [2] that offer Distributed Hash Table (DHT) [11] functionalities. A P2P network provides a typical distributed decentralized approach where multiple computers are interconnected and communicate by exchanging messages. DHT partitions the items of a key set among participating nodes, and can send messages to the owner of a given key in an efficient manner. The P2P network with DHT support is scalable and solves the problem of data redundancy, but supports only exact match for keywords. The inclusion in the P2P network of references to service semantics could be a turning point, because such information could be exploited for the automatic discovery of services with the help of semantic matchmaking techniques. In this paper we discuss how this vision can be realized. Section 2 introduces the basic concepts related to WS registries and Catalogs based on P2P protocols, and the OWL-S language for the representation of the semantics associated with services. Section 3 describes an implementation of the P2P network created by means of the Open Chord API and OWL-S. Finally, Section 4 presents an analysis on the potential of the proposed approach. 2 2.1 Background Web service registries Web services are software systems identified by means of a Web address and designed to support interoperability between computers on a network. They have public interfaces defined and described as XML documents in a format, such as WSDL, that can be processed by a machine in an automatic way. Their definitions can be sought from other software systems, which can directly interact with the Web service operations described in the interface by activating the appropriate messages enclosed in an SOAP envelope. These messages are usually transported via the Hyper Text Transport Protocol (HTTP) and formatted according to the XML standard. For the purposes of this discussion is important to point out what are the current approaches to the organization of Web services. These approaches can be broadly classified as centralized or decentralized. The traditional centralized approach includes UDDI, where a central registry is used to store descriptions of Web services. The current UDDI approach attempts to mitigate the disadvantages of centralization by replicating the entire information on different sites. Replication, however, may temporarily improve performance if the number of UDDI users is limited. But with the rise of the replicated sites, decreases the consistency of the duplicated data. The replication of UDDI data is not a scalable approach. For this reason, different approaches on decentralized registries have been proposed in order to connect individual customers through the P2P network. Since this technology organizes peers into a hypercube, the 55 management becomes inefficient in the presence of a large amount of data. A solution to this problem is given by [14] where a method for the reduction in size of the indexing scheme that maps the multidimensional information space with physical peers is presented. However, this method does not use semantic descriptions. The Web Service discovery based on P2P is also discussed in [18] and [5] where ad hoc model frameworks are proposed for this purpose. As a starting point, in the next section we will analyze a proposed approach that combines ontologies and P2P based on DHT as a sophisticated solution to these issues. 2.2 Web Service catalog system based on DHT Without a central registry, the easiest way to find out the location of a service in a distributed system is to send the query to each participant (service provider). While this approach might work for a small number of service providers, it is certainly not scalable in a large distributed system. When a system includes thousands of nodes, facilities that allow the selection of a subset of nodes that will be fitted with the functionality exposed in the catalogs are needed. The new generation of P2P systems includes complete DHT features [11] for decentralized applications. Some groups have proposed innovative approaches such as CAN [11], Pastry [13] and Chord [17], which eliminate the defects of the first P2P systems like Gnutella6 and Napster7 . Although they are implemented in different ways, all these systems have interfaces to support access to DHT. These interfaces permit to request shared information. In contrast to UDDI, the P2P network content is usually described and indexed locally within each peer, while search queries are propagated through the network. A central index that spans the whole network is not required. Given a key, the corresponding data items can be efficiently located using up to O(log(n)) network messages, where n is the total number of nodes in the system [17]. In addition, distributed systems evolve while remaining scalable to a large number of nodes. Current efforts are directed towards this functionality in order to provide a catalog of services fully distributed and scalable. The approach chosen for the purposes of this paper is Chord because it proposes an original approach to the problem of efficient location and is able to maintain the bandwidth close to the optimal in the management of arrival and departure of competing nodes [8]. Chord uses routed queries to locate a key, minimizing the number of visits on large amount of nodes. What distinguishes Chord from other P2P applications is the ease of use and provable performance and correctness. In essence, Chord supports one operation: given a key, it is mapped on a node. The location data can be implemented by associating each key with a datum. In detail, it routes a key through a sequence of O(log(n)) other nodes toward the destination. This requires that a Chord node has information about O(log(n)) other nodes for efficient routing. When the information is out of date, it is proved that the performance degrades gracefully. This is important in practice because nodes will 6 7 Gnutella Web site - http://rfc-gnutella.sourceforge.net/ Napster Web site- http://free.napster.com/ 56 join and leave arbitrarily, and consistency about O(log(n)) nodes may be hard to maintain. Only one piece of information per node needs to be correct in order to guarantee correct routing of queries. The Chord protocol uses the SHA − 1 hash functions to assign a m-bit identifier to each node and key. Furthermore, it uses Consistent Hash functions that allow nodes to leave and enter the network with minimal disruption [6]. The integer m is chosen to be large enough to make negligible the probability that two nodes (keys) received the same identifier. The hash function calculates the key identifier performing hashing over the IP address of the node. The key identifiers of the nodes are arranged in a circular ring of dimension 2m called Chord ring. The identifiers on the Chord ring are numbered from 0 to 2m − 1. A key is assigned to a node whose identifier is equal to or greater than the key identifier. This node is called the successor of the node k and is the first node k on the circle in a clockwise direction. When a node n wants to find a certain key k, it uses a lookup function that returns the successor of n if k is between n and its successor, otherwise forwards the query in the circle. Furthermore, in order to provide more efficient lookup, parts of the routing information are stored in the nodes. In particular, each node n maintains a routing table with at most m entries (where m is the number of bits in the identifiers), called finger table. Stabilization primitives are used to maintain updated finger tables, as well as the Chord ring itself. This structure can be used to create Web Service Catalogs. For example, in [19] each node in the system is a service provider or requester so that both these two actors are connected together in the Chord ring. When a service provider Ni wants to publish a service, creates the service catalog item, i.e., the tuple C = key, Summary. The Chord protocol routes the catalog information to the corresponding node of the system in accordance with the key in the catalog. Thus, each node in the system contains part of information of catalog and all the nodes together constitute the global catalog system implementing the functionality of a traditional UDDI registry. With WSDL, a Web Service can be expressed as a set of operations, each of which implements a certain amount of functions. An operation is specified by its name and the types of input and output messages. The service name is used as the key catalog information for the DHT hashing algorithm. In line with this, the operations included in the service and messages associated with these operations are used as a summary. The catalog for a Web service W si has the structure: CW si = (Key, Summary, N ), where: – Key is the name of W si , – Summary contains the operations and its messages included in W si , – N is the node that publishes W si . In the same paper [19] is proposed a mapping between the information contained in the nodes (related to the WSDL) with ontology classes that represent them in OWL-S services. In contrast, our approach foresees that the information contained in the catalogs are taken directly from the OWL-S descriptions already available online. 57 2.3 Web Ontology Language for Services (OWL-S) Semantic Web Services[9] provide an ontological framework for describing services, messages, and concepts in a machine-readable format, enabling logical reasoning on service descriptions. The Web Ontology Language for Services (OWL-S) provides a Semantic Web Services framework on which an abstract description of a service can be formalised. It is an upper ontology described with OWL8 whose root class is Service, therefore, every described service maps onto an instance of this concept. The upper level Service class is associated with three other classes: Service Profile. The service profile specifies the functionality of a service. This concept is the top-level starting point for the customizations of the OWL-S model that supports the retrieval of suitable services based on their semantic description. It describes the service by providing several types of information: • Human readable information: such as the service description, service name, contact information, etc.; • Functionalities: i.e. parameter type identifiers, identifiers for the input and output, parameters of service methods, preconditions, results, etc.; • Service parameters: which include parameter identifiers (e.g. name, value) used by the service; • Service categories: these include identifiers for defining the category of service, i.e. category name, taxonomy, value, code; Service Model. The service model exposes to clients how to use the service, by detailing the semantic content of requests, the conditions under which particular outcomes will occur, and, where necessary, the step by step processes leading to those outcomes. In other words, it describes how to ask for (invoke) the service and what happens when the service is carried out. From the point of view of the processes, the service model defines the concept Process that describes the composition of one or more services in terms of their constituent processes. A Process can be atomic, composite or simple: an atomic process is a description of a non-decomposable service that expects one message and returns one message in response. A composite process consists of a set of processes within some control structures defining a workflow. Whereas a simple process provides a service abstraction that allows to view a composite service as an atomic one. Each process can have any number of inputs, a set of preconditions, all of which must hold for the process to be successfully invoked, and any number of results (outputs and/or effects) that come from a successful execution of the service. Service Grounding. A grounding is a mapping from an abstract to a concrete specification of those service description elements that are required for interacting with the service. In general, a grounding indicates a communication protocol, a message format and other service-specific details (e.g., port 8 OWL Web Ontology Language, W3C Recommendation 10 February 2004 - http: //www.w3.org/TR/owl-features/ 58 numbers, the serialization techniques of inputs and outputs, etc.). From the point of view of processes, a service grounding enables the transformation from inputs and outputs of an atomic process into a concrete atomic process grounding constructs. Fig. 1. Schema Mapping WSDL-OWL-S As we can see from Figure 1, OWL-S grounding maps the semantic description of the service with the corresponding WSDL. This means that the information contained in the Summary of Catalog shown in the previous section can be directly obtained from OWL-S. Since each OWL-S instance, as well as all its constituent parts, has its own URI, such information is always available online. 3 OWL-S discovery with P2P 3.1 Open Chord Open Chord9 is an open source implementation of Chord. Its architecture consists of three levels (Figure 2). On the lower level is located the implementation of the communication protocol used (Communication Layer), based on a network protocol (such as Java Socket). Currently, two implementations are provided: Local communication protocol, that has been developed for testing purposes, and Socket-based protocol, that facilitates reliable communication between Open Chord peers based on TCP/IP sockets. 9 Open Chord Web site - http://open-chord.sourceforge.net/ 59 Fig. 2. Open CHORD architecture The abstraction level (Communication abstraction layer ) provides two abstract classes that must be implemented to realize the communication protocol: – Proxy, that represents a reference to remote peers in the Open Chord overlay network. – Endpoint, that provides a connection point for remote peers conform to a specific communication protocol. Concrete implementations for a communication protocol are determined with help of the URL of a peer. The Chord logic level, which implements the functionality of Chord, offers two interfaces for Java applications that abstract from the implementation of the Chord DHT routing. Both interfaces (i.e., Chord and AsynChord ), which can be used by an application built on-top of Open Chord to retrieve, remove, and store data in synchronous and asynchronous way from/to the underlying DHT, provide some common methods that are important to create, join, and leave an Open Chord DHT. The Chord logic level is also responsible for data replication and maintenance of the necessary properties to keep running the DHT, as described in [17]. 3.2 A prototype implementation To simulate an OWL-S P2P network using the Open Chord API a simple graphical application that displays a drop down menu consisting of the items File, Edit, and View has been developed. By selecting ’Create Network’ from ’File’ menu a 60 Fig. 3. Screenshot of the prototype single peer will be created, consisting simply of a new URL. Only the first node has the ability to create a new network, to add other nodes the join method of the interface Chord will be invoked. This method works similarly to the method used to create the network, but in addition to the node that is to be added, an existing URL, that is already part of the network, is required. This is called bootstrap peer. To test the operations of the P2P network we have taken a set of services from the dataset OWLS-TC10 , placed them in a local folder and inserted them in the network using the function ’Create nodes with’ from the ’File’ menu that automatically creates a number of nodes equal to the number of files in the local folder. Subsequently, by selecting ’Insert’ from ’Edit’ menu, a dialog appears that allows to insert new nodes in the network individually, specifying the URL of the bootstrap node and the URL of the new node which must be different from those used for the other nodes in the network. The next step is the population of the network. To work with DHT the choice of the key is a fundamental step. Our key was the output value of the OWL-S profile of the selected services. The output value is extracted parsing the service profile available online. The value we have associated with the key is the URI of the OWL-S service itself. Selecting ’Populate the Network’ from ’File’ menu this procedure is executed in automatic way for all services contained in the local folder. The synchronous retrieval of the value associated with the key is carried out invoking the Open Chord method retrieve(Key). The result is an array of strings containing the URL of zero or more OWL-S services, depending on whether the searched key is the output of one or more services inserted in the network. Figure 10 OWLS-TC service retrieval test collection - http://projects.semwebcentral.org/ projects/owls-tc 61 Fig. 4. An example of results setting BOOK as key 4 shows the results obtained for the key BOOK using the ’Search. . . ’ pop up opening from ’Modify’ menu. By applying methods that use lexical ontologies (e.g., WordNet11 ) you can obtain the synonyms of the key. Invoking the method retrieve(Key) on synonyms, those services that do not have as output the initial key are found, providing a solution to the problem of exact matching between the searched key and the output of the service. 4 Discussion The use of P2P networks for SWS discovery opens the way for the application of intelligent methods to satisfy the requests of Web users. The availability of semantic descriptions of services allows the realization of new scenarios in which the weight of the reasoning for the attainment of a goal moves increasingly towards software systems. In the scenario shown in Figure 5, a user queries a software agent, capable of interpreting natural language, asking him to find a service that returns a certain result (Goal). The agent uses the P2P network for the discovery of services that may be suitable to meet the demand. Since the P2P network returns the semantic descriptions of services, the agent can apply automated reasoning methods to select and compose the most appropriate services with respect to the available user inputs. If the services are described by using different ontologies, it will use semantic alignment tools12 and approaches in the literature [4, 1] during the execution of these operations. This scenario 11 12 WordNet, a lexical database for English - http://wordnet.princeton.edu/ wordnet/ Alignment API and Alignment Server - http://alignapi.gforge.inria.fr/ 62 Fig. 5. Scenario describes the automatic orchestration of SWS and is particularly suitable for use with OWL-S [12]. Looking in particular to the discovery use case, there are various matchmaking techniques that exploit the OWL-S description to determine which services are best suited to fulfil the request. Srinivasan et al. [16] propose an enhancement of CODE OWL-S IDE [15] where the used matching procedure is based on the algorithm described in [10]. It defines a flexible matching mechanism based on subsumption in Description Logics. More sophisticated solutions are provided by OWLS-MX [7], an hybrid Semantic Web Service matchmaker that retrieves services for a given query written in OWL-S itself. In other words for every OWL-S service representing the description of the desired service (query), it returns an ordered set of relevant services ranked according to their degree of (syntactic and/or semantic) matching with the query. This approach complements logic based reasoning with approximate matching relying on Information Retrieval metrics. Figure 6 illustrates the graphical user interface (GUI) of a software component that we have developed as a support for the test of matchmaking on retrieved services. At the top there are two text fields designed to insert user inputs and the searched output of the service, respectively. By pressing the Ok button the request is processed. The results will vary depending on the chosen sort order (syntactic, semantic or hybrid) that reflect those available with OWLS-MX API13 . Finally, clicking on a listed service the following information 13 OWLS-MX Semantic Web Service Matchmaker API - http://www.semwebcentral. org/projects/owls-mx/ 63 Fig. 6. Service discovery GUI will be displayed: name, URI, inputs and outputs. The work presented in this paper represents only a starting point towards a SWS discovery system based solely on semantic descriptions of services. Future work includes extensive use of the annotations included in the OWL-S profile for the selection of services that best meet the user needs. For this purpose, the domain ontologies matchmaking methods will be combined with lexical ontology based approaches in order to analyze the lexical text descriptions of the service during the discovery process. References [1] David, J., Euzenat, J., Scharffe, F., dos Santos, C.T.: The alignment api 4.0. Semantic Web 2(1), 3–10 (2011) [2] Doyle, J.F.: Peer-to-peer: harnessing the power of disruptive technologies. Ubiquity 2001 (May 2001) [3] Erl, T.: Service-Oriented Architecture: Concepts, Technology, and Design. Prentice Hall PTR, Upper Saddle River, NJ, USA (2005) [4] Euzenat, J., Shvaiko, P.: Ontology matching. Springer-Verlag, Heidelberg (DE) (2007) [5] Gharzouli, M., Boufaida, M.: Pm4sws: A p2p model for semantic web services discovery and composition. Journal of Advances in Information Technology 2(1) (2011) [6] Karger, D.R., Lehman, E., Leighton, F.T., Panigrahy, R., Levine, M.S., Lewin, D.: Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the world wide web. In: STOC. pp. 654–663 (1997) [7] Klusch, M., Fries, B., Sycara, K.: Automated semantic web service discovery with OWLS-MX. In: AAMAS ’06: Proceedings of the fifth international joint conference 64 [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] on Autonomous agents and multiagent systems. pp. 915–922. ACM Press, New York, NY, USA (2006) Liben-Nowell, D., Balakrishnan, H., Karger, D.R.: Observations on the dynamic evolution of peer-to-peer networks. In: Druschel, P., Kaashoek, M.F., Rowstron, A.I.T. (eds.) IPTPS. Lecture Notes in Computer Science, vol. 2429, pp. 22–33. Springer (2002) McIlraith, S.A., Son, T.C., Zeng, H.: Semantic Web Services. IEEE Intelligent Systems 16(2), 46–53 (2001) Paolucci, M., Kawamura, T., Payne, T.R., Sycara, K.P.: Semantic Matching of Web Services Capabilities. In: ISWC ’02: Proceedings of the First International Semantic Web Conference on The Semantic Web. pp. 333–347. Springer-Verlag, London, UK (2002) Ratnasamy, S., Francis, P., Handley, M., Karp, R.M., Shenker, S.: A scalable content-addressable network. In: SIGCOMM. pp. 161–172 (2001) Redavid, D., Esposito, F., Iannone, L.: A comparative study on semantic web services frameworks from the dynamic orchestration perspective. In: In Proceedings of International Conference on Knowledge Engineering and Ontology Development (KEOD). pp. 355–359 (October 2010) Rowstron, A.I.T., Druschel, P.: Pastry: Scalable, decentralized object location, and routing for large-scale peer-to-peer systems. In: Guerraoui, R. (ed.) Middleware. Lecture Notes in Computer Science, vol. 2218, pp. 329–350. Springer (2001) Schmidt, C., Parashar, M.: Flexible information discovery in decentralized distributed systems. In: HPDC. pp. 226–235. IEEE Computer Society (2003) Srinivasan, N., Paolucci, M., Sycara, K.: CODE: A Development Environment for OWL-S Web services. Tech. Rep. CMU-RI-TR-05-48, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA (October 2005) Srinivasan, N., Paolucci, M., Sycara, K.: Semantic Web Service Discovery in the OWL-S IDE. In: HICSS ’06: Proceedings of the 39th Annual Hawaii International Conference on System Sciences. p. 109.2. IEEE Computer Society, Washington, DC, USA (2006) Stoica, I., Morris, R., Liben-Nowell, D., Karger, D.R., Kaashoek, M.F., Dabek, F., Balakrishnan, H.: Chord: a scalable peer-to-peer lookup protocol for internet applications. IEEE/ACM Trans. Netw. 11(1), 17–32 (2003) Xu, B., Chen, D.: Semantic web services discovery in p2p environment. In: ICPP Workshops. p. 60. IEEE Computer Society (2007) Yu, S., Zhu, Q., Xia, X., Le, J.: A novel web service catalog system supporting distributed service publication and discovery. In: Ni, J., Dongarra, J. (eds.) IMSCCS (1). pp. 595–602. IEEE Computer Society (2006) 65 Marine Traffic Engineering through Relational Data Mining Antonio Bruno1 and Annalisa Appice1,2 1 Dipartimento di Informatica, Università degli Studi di Bari Aldo Moro via Orabona, 4 - 70126 Bari - Italy 2 CILA (Centro Interdipartimentale per la ricerca in Logica e Applicazioni) [email protected],[email protected] Abstract. The automatic discovery of maritime traffic models can achieve useful information for the identification, tracking and monitoring of vessels. Frequent patterns represent a means to build human understandable representations of the maritime traffic models. This paper describes the application of a multi-relational method of frequent pattern discovery into the marine traffic investigation. Multi-relational data mining is here demanded for the variety of the data and the multiplicity of the vessel positions (latitude-longitude) continuously transmitted by the AIS (Automatic Identification System) installed on shipboard. This variety of information leads to a relational (or complex) representation of the vessels which by the way permits to naturally model the total temporal order over consecutive AIS transmissions of a vessel. The viability of relational frequent patterns as a model of the maritime traffic is assessed on navigation data truly collected in the gulf of Taranto. 1 Introduction Marine traffic engineering is a research field originally defined in 1970s [10] at the aim of investigating the marine traffic data and building a human interpretable model of the maritime traffic. Through the understanding of this model, the Vessel Traffic Service (VTS) would improve the port and fairway facilities as well as the traffic regulation. Intuitively, the complexity of building a significant maritime traffic model resides in the requirement of a model able to reflect the spatial distribution and the temporal characteristics of the traffic flow. Although, the marine traffic engineering was a popular research field between the 1970s and the 1980s, after the 1990s, the relevant literature and research projects in this field appeared less frequently. This little interest to research in marine traffic engineering was caused to the actual difficulty in collecting traffic data. In fact, the required observation time was long and several technological limitations raised in the observation time. Today, the data collection problem is definitely overcome. The widespread use of Automatic Identification System (AIS) has had a significant impact on the maritime technology and any VTS is now fit to obtain a large volume of traffic information which comprises the timestamped latitude and longitude of the monitored vessels. On the other hand, 66 the galloping developments in data mining research have paved the way to face the problem of automatically analyzing this large volume of traffic data, by the now available, in order to extract the knowledge required to feed the service marine traffic management and the VTS decision making systems. Both these factors, traffic data availability and data mining techniques, have boosted the recent renewed scientific interest towards the marine traffic engineering. Clustering [9], classification [5] and association rule discovery [11] techniques have been employed to analyze AIS data and discover characteristics and/or rules for the marine traffic flow forecast and the development and programming of marine traffic engineering. Although, these studies have proved that data mining techniques are able to provide the extra aid for the situational awareness in maritime traffic control, it is a fact that no marine traffic model described in these works is able to capture the truly temporal characteristics of each AIS transmission. In fact, AIS transmissions are timestamped, but a traditional data mining technique looses the time label of the AIS data, and represents a navigation trajectory as a set, rather than a sequence, of consecutive latitude-longitude vessel positions. In this paper, we resort to multi-relational data mining to address the task of learning a human interpretable model of the maritime traffic in the sea ports, where several vessels are entering and leaving the port. The innovative contribution of this work is that, at the best of our knowledge, this is the first study in maritime traffic engineering which correctly spans the traffic data over several data tables (or relations) of a relational database and discover relational patterns (i.e. patterns which may involve several relations at once) to describe the traffic maritime model. In this multi-relational representation, we are able to model vessel data and AIS data as distinct relational data tables (one for each data type). This leads to distinguish between the reference objects of analysis (vessel data) and the task-relevant objects (AIS data), and to represent their interactions. The modeled interactions also include the total temporal order over the AIS transmissions for the same vessel. SPADA [6] is a multi-relational data mining method that discovers relational patterns and association rules. Relational patterns extracted by SPADA have been proved to be effective for the capture of the behavioral model underlying census data [1] and workflow data [12]. In the case of traffic data, we use SPADA to discover interesting associations between a vessel (reference objects) and a navigation trajectory. Each navigation trajectory represents a spatio-temporal pattern obtained by tracing subsequent AIS transmissions (task-relevant objects) of vessels. This kind of spatio-temporal rules automatically identify the well traveled navigation courses. This information can be employed in several ways. To opportunely arrange the navigation traffic incoming a gulf in order to avoid collision or traffic jams. To discover vessels which suspiciously deviates from the planned navigation course. The main limitation of SPADA in this application is the high computational complexity which makes the analysis of large databases practically unfeasible. To overcome this limitation, we run SPADA by considering the distributed version of SPADA described in [2]. 67 In order to prove the viability of the multi-relational approach in marine traffic engineering, we describe a relational representation of the traffic data derived from monitoring vessels entered and left the gulf of Taranto (South of Italy) between September 1, 2010 (00:04:23) and October 9, 2010 (23:58:52) (Section 2) and we briefly illustrate the multi-relational method for relational pattern discovery (Section 3). Finally, we comment significance of navigation traffic model we have extracted and its viability in the marine traffic engineering (Section 4). Finally, some conclusions are drawn. 2 Marine Traffic Data For this study, we consider the navigation traffic data collected for 106 vessels entering and/or leaving the gulf of Taranto between September 1, 2010 (00:04:23) and October 9, 2010 (23:58:52). The traffic data are obtained from [13]. As in [11], the area of the gulf is converted into a geographic grid of 0.005◦ × 0.005◦ squared cells. Each cell of the grid is then enumerated by a progressive number. For each vessel, the following data are collected: – the name of the vessel, – the MMSI, that is, a numeric code that unambiguously identifies the vessel, – the vessel category, that is, wing, pleasure craft, tug, low enforcement, cargo, tanker or other, and – the sequence of AIS messages which were sent by the transceiver installed on shipboard. The AIS transceiver sends dynamic messages every two to thirty seconds depending on the vessel speed, and every three minutes while the vessel is at the anchor. As we are interested in describing the observable change of the vessel position within the geographic grid, we decide to consider only those AIS transmissions which reflect a change of the cell occupied by the vessel. Each AIS message includes the following data: – – – – – the the the the the vessel MMSI; received time (day-month-year hour-minutes-seconds); latitude and longitude of the vessel; course over ground; vessel speed; The latitude and longitude coordinates of each AIS transmission are transformed into the identifier of the cell containing the coordinates. By following the suggestion reported in [11], the course over ground is discretized every 45◦ thus obtaining N, E, W, S, NE, NW, SE, SW, while the speed is discretized in low, medium and high. After this transformation, properties of vessels (name and category), data of the AIS transmission (cell, speed, direction) and interaction between vessel and transmitted AIS data are stored as ground atoms into the extensional part of a deductive database. An example of data stored in the 68 database for the vessel named ALIDA S is reported below. mmssi(247205900). name(247205900, alida s). category(247205900, cargo). ais(247205900, 2010-10-07 19:51:30). ais(247205900, 2010-10-07 20:45:26). ais(247205900, 2010-10-07 21:50:19). ais(247205900, 2010-10-07 21:55:23). cell(247205900, 2010-10-07 19:51:30, 312). cell(247205900, 2010-10-07 20:45:26, 313). cell(247205900, 2010-10-07 21:50:19, 312). cell(247205900, 2010-10-07 21:55:23, 311). direction(247205900, 2010-10-07 19:51:30, northwest). direction(247205900, 2010-10-07 20:45:26, northwest). direction(247205900, 2010-10-07 21:50:19, northwest). direction(247205900, 2010-10-07 21:55:23, northwest). speed(247205900, 2010-10-07 19:51:30, medium). speed(247205900, 2010-10-07 20:45:26, medium). speed(247205900, 2010-10-07 21:50:19, low). speed(247205900, 2010-10-07 21:55:23, low). The key predicate mmsi() identifies the reference object (vessel) of the unit of analysis. The property predicates name(), category(), position(), direction() and speed () define the value (in bold) taken by an attribute of an object (reference object as for name() and category() or task relevant object as for position(), direction() and speed ()). Finally the structural predicate ais() relates reference objects (vessel) with task-relevant objects (AIS transmissions). This way, the extensional part of deductive database for SPADA is fed with 19137 atoms partitioned between 106 units of analysis. 3 Maritime Traffic Model Discovery Studies for association rule discovery in Multi-Relational Data Mining [6] are rooted in the field of Inductive Logic Programming (ILP) [8]. In ILP both relational data and relational patterns are expressed in a first-order logic and the logical notions of generality order and of the downward/upward refinement operator on the space of patterns are used to define both the search space and the search strategy. In the specific case of SPADA, properties of both reference and task relevant objects are represented as the extensional part DE of a deductive database D [4], while the domain knowledge is represented as a normal logic program which defines the intensional part DI of the deductive database D. In the application of SPADA in the marine traffic engineering, the extensional database stores information on the traffic data (e.g., vessel and AIS data) as reported in Section 2, while the intensional database includes the definition 69 of relations which are implicit in data, but useful for capturing the model underlying the data. In this study, the intensional part of database surely includes some definition of a relation next (which makes explicit the temporal order over the AIS transmissions that is implicit in the timestamp of each transmission). A possible definition of the relation next is the following: next(V, A1, A2) ← ais(V, T1), ais(V, T2), cell(V, T1, A1), cell(V, T2, A2), A1= A2, T1<T2, not(ais(V,T), T1<T, T<T2) which defines the direct sequence relation between two consecutive AIS transmissions of the same vessel. In SPADA, the set of ground atoms in DE is partitioned into a number of non-intersecting subsets D[e] (unit of analysis) each of which includes facts concerning the AIS transmissions involved in a specific vessel trip e. The partitioning of DE is coherent with the individual-centered representation of training data [3], which has both theoretical (PAC-learnability) and computational advantages (smaller hypothesis space and more efficient search). The discovery process is performed by resorting to the classical levelwise method described in [7], with the variant that the syntactic ordering between patterns is based on θ-subsumption. By SPADA, fragments of the traffic models underlying the navigations of the various traced vessels can be expressed in the form of relational navigation rules in the form: mmsi(V ) ⇒ µ(V ) [s, c], where mmsi(V ) is the atom that identifies a vessel, while µ(V ) is a conjunction of atoms which provides a description of a fragment of the navigation trajectory traced for V . Each atom in µ(V ) describes either the next relation between AIS transmissions or a property of the vessel (type or length) or a datum included in the AIS transmission (id of the crossed geographical cell, navigation direction, velocity). An example of discovered association rule is the following: vessel(V)⇒ cell(V,T,123), next(V,123,124), next(V,124,125) [s=63%, c=100%] The support s estimates the probability p(vessel(V ) µ(V )) on D. This means that s% of the units of analysis D[e] are covered by vessel(V ) µ(V ), that is a substitution θ = {V ← e}·θ1 exists such that vessel(V ) µ(V )θ ⊆ D[e]. The confidence c estimates the probability p(µ(V )|vessel(V )). Our proposal is to employ SPADA in order to process large traffic data volume and to collect the navigation rules discovered by SPADA in order to obtain an interpretable description of the model underlying the maritime traffic. As the 70 navigation rules describe fragments of the trajectories frequently crossed by the monitored vessels, they are then visualized in a GIS environment for the human interpretation. At the aim of this study, we have further extended SPADA by integrating a rule post-processing module which filters out uninteresting rules and ranks the output of the filtering phase on the basis of the rule significance. Then, the top-k rules compose the maritime traffic model. Interesting rules correspond to non-redundant rules. Formally, let R be the navigation rule set output by SPADA. A rule r ∈ R is labeled as redundant in R iff there exists a rule r ∈ R and the substitution θ such that rθ ⊂ r . For example, let us consider the set of navigation rules which comprises: r1: vessel(V)⇒ cell(V,T,123). r2: vessel(V)⇒ cell(V,T,123), next(V,123,124). r3: vessel(V)⇒ cell(V,T,123), next(V,123,124), next(V,124,125). Both r1 and r2 are redundant in R due to the presence of r3. Redundant rules are implicit in non-redundant rules (although, they may have different support, they are always frequent rules), hence we can filtered out the redundant navigation rules without loosing any knowledge in the maritime traffic model which is built finally. Filtered rules are ranked on the basis of significance expressed by pattern length (number of atoms in the rule and support value). By decreasing k, we prune less significant knowledge in the model. 4 Maritime Traffic Models A relational model of the maritime traffic in the gulf of Taranto (South of Italy) was extracted by considering two experimental settings, denoted as S1 and S2. In the former setting (S1), the intensional part is populated with the definition of the ternary “next” predicate formulated as in Section 2. In the latter setting (S2), the intensional part is populated with an intensional definition of both a new “cell” predicate and a “next” predicate which incorporate the information on the speed and direction of navigation as follows: cell(V, T, C, S, D) ← cell(V, T, C), speed(V, T, S), direction(V, T, D). next(V, A1, A2, S, D) ← ais(V, T1), ais(V, T2), cell(V, T1, A1), cell(V, T2, A2), speed(V, T2, S), direction(V, T2, S) A1= A2, T1<T2, not(ais(V, T, A), T1<T, T<T2). In both settings, SPADA is run to discover relational rules with 0.1 as minimum support and 3 as minimum pattern length. In the first setting, SPADA 71 outputs the geometrical description fragments of navigation trajectories incoming and leaving the gulf of Taranto. The number of discovered rules is 126. After filtering out redundant rules, 41 rules are ranked according to the significance criterion. The top ranked navigation rule is reported below: vessel(V)⇒ category(V,cargo), cell(V, T, 903), next(V, 903, 904), next(V, 904, 944), next(V, 944, 945), next(V, 945, 946), next(V, 946, 986), next(V, 986, 987). [s=10.3%, c=100%] This rule states that 10.3% vessels monitored in the gulf of Taranto in the period under study are cargo vessels which follow a navigation trajectory crossing across the cells identified by 903, 904, 944, 945, 946, 986 and 987 in this order. The maritime traffic model obtained by selecting the top-5 navigation rules is plotted in Figure 1. By visualizing this model we are able to see the geometrical representation of maritime trajectories which may be busy in the gulf of Taranto. This information may be employed from the service maritime management in order to opportunely program the maritime traffic in the gulf of Taranto and avoid gridlock or vessel accident. In the second setting, SPADA discovers a more detailed description of the navigation trajectories frequently crossed in the gulf. In fact, the description mined for each navigation trajectory now comprises both direction and velocity of the vessel at each crossed cell in the trajectory. With this setting, SPADA discovers 11 navigation rules. After filtering out redundant rules, 8 rules are ranked according to the significance criterion. The top ranked navigation rule is reported below: vessel(V)⇒ category(V,cargo), cell(V, T, 945, low, north east), next(V, 945, 946, low, north east), next(V, 946, 986, low, north east). [s=11.3%, c=100%] This rule states that 11.3% of vessels in this study move across the cells 945, 946 and 986 maintaining a low velocity and north-east navigatition dierection. Although this navigation rule describes a shorter trajectory than the top ranked rule of the first setting, it provides a deeper insight in the navigation behaviour (velocity and direction) of vessels crossing these cells, which where ignored before. 5 Conclusions In this paper, we presented a preliminary study of the application of relational data mining to the marine traffic engineering. Relational data mining is here demanded to represent multiplicity and variety of data continuously transmitting 72 (a) Visualization. (b) Ranking. Fig. 1. The top-5 relational models of the incoming and outcoming navigation trajectories frequently crossed in the gulf of Taranto. from a vessel during the navigation time. In particular, we prove the viability of a multi-relational approach to obtain human interpretable maritime models of the maritime traffic by considering the AIS data transmitted from vessels in the gulf of Taranto. The results are encouraging and open appealing and novel directions of research in the field of the marine traffic engineering. As future work, we plan to explore the task of discovering relational rules which include a disjunction of atoms in the rule body in order to describe those trajectories which include one or more ramification in the path. Additionally, we intend to use the discovered navigation trajectories to obtain a prediction model that permits to predict the position of a vessel at any future time. This task requires the consideration of either geographical constraints such as the presence of the mainland (or in general physical obstacles) or navigation constraints such as velocity, direction, timetable and so on. Acknowledgment This work is partial fulfillment of the research objective of ATENEO-2010 project entitled “Modelli e Metodi Computazionali per la Scoperta di Conoscenza in Dati Spazio-Temporali”. References 1. A. Appice, M. Ceci, A. Lanza, F. A. Lisi, and D. Malerba. Discovery of spatial association rules in geo-referenced census data: A relational mining approach. Intelligent Data Analysis, 7(6):541–566, 2003. 2. A. Appice, M. Ceci, A. Turi, and D. Malerba. A parallel, distributed algorithm for relational frequent pattern discovery from very large data sets. Intell. Data Anal., 15(1):69–88, 2011. 73 3. H. Blockeel and M. Sebag. Scalability and efficiency in multi-relational data mining. SIGKDD Explorations, 5(1):17–30, 2003. 4. S. Ceri, G. Gottlob, and L. Tanca. Logic Programming and Databases. SpringerVerlag New York, Inc., New York, NY, USA, 1990. 5. R. Lagerweij. Learning a Model of Ship Movements. Thesis for Bachelor of Science - Artificial Intelligence, University of Amsterdam, 2009. 6. F. A. Lisi and D. Malerba. Inducing multi-level association rules from multiple relations. Machine Learning, 55(2):175–210, 2004. 7. H. Mannila and H. Toivonen. Levelwise search and borders of theories in knowledge discovery. Data Mining and Knowledge Discovery, 1(3):241–258, 1997. 8. S. Muggleton. Inductive Logic Programming. Academic Press, London, 1992. 9. C. Tang and Z. Shao. Modelling urban land use change using geographically weighted regression and the implications for sustainable environmental planning. In Q. Peng, K. C. P. Wang, Y. Qiu, Y. Pu, X. Luo, and B. Shuai, editors, Proceedings of the 2nd International Conference on Transportation Engineering, pages 4465– 4470. ASCE, American Society of Civil Engineering, 2009. 10. S. Toyoda and Y. Fujii. Marine traffic engineering. The Journal of Navigation, 24:24–34, 1971. 11. M.-C. Tsou. Discovering knowledge from ais database for application in vts. The Journal of Navigation, 63:449–469, 2010. 12. A. Turi, A. Appice, M. Ceci, and D. Malerba. A grid-based multi-relational approach to process mining. In S. S. Bhowmick, J. Küng, and R. Wagner, editors, Proceedings of the 19th International Conference on Database and Expert Systems Applications, DEXA 2008, volume 5181 of Lecture Notes in Computer Science, pages 701–709. Springer, 2008. 13. web url:. http://www.marinetraffic.com/ais. 74 Author Index Annalisa Appice, 66 Saima Jabeen, 14 Elena Baralis, 14 Elena Bellodi, 26 Antonio Bruno, 66 Luca Cagliero, 14 Gianni Costa, 38 Fabio Leuzzi, 2 Giuseppe Manco, 38, 46 Elio Masciari, 46 Riccardo Ortale, 38 Sašo Džeroski, 1 Floriana Esposito, 54 Stefano Ferilli, 2, 54 Alessandro Fiori,14 Domenico Redavid, 54 Fabrizio Riguzzi, 26 Ettore Ritacco, 38 Fulvio Rotella, 2