Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Open Database Connectivity wikipedia , lookup
Microsoft SQL Server wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Entity–attribute–value model wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Versant Object Database wikipedia , lookup
Clusterpoint wikipedia , lookup
Keyword-based Search in a Relational Database Daniël Suelmann Advisor: dr. George M. Welling Bachelor‟s Thesis Department of Information Science Faculty of Arts University of Groningen August 2009 Abstract A relational database is often operated by means of a structured query language (SQL). When composing SQLqueries one must have an understanding of the SQL syntax to be able to produce a query the database can execute. Additionally, one must be familiar with the attributes and relations in the database to be able to retrieve the data desired. When one wants to search the available data stored in the relational database, these requirements can be discouraging. In this case keyword-based search functionality could improve the accessibility of the data. In this thesis I investigate the subject of keyword-based search in structured data. My objective in this thesis is to show that keyword-based search in the available data can yield meaningful results. I present a search application I developed that enables keyword-based search in the Lastgeld data. This data is about skippers and their cargo entering the port of Amsterdam in the period from 1744 to 1748. Answers to multiple keywords in queries are retrieved based on the following assumption: if keywords in a keyword query are related in the relational database that contains the Lastgeld data, then retrieving this relating data yields results likely to be meaningful given the keyword query. I describe how the presented search application achieves just that. Additionally given the experimental results I conclude that keyword query execution time increases if the intersection work that needs to be performed by the presented search application increases. This work is dependent on the number of keywords in the query and the amount of id-numbers associated with these keywords in the indices. I also conclude that the more SQL queries the search application proposes to be executed by the relational database, the more the overall query execution time will increase. Contents 1. INTRODUCTION .......................................................................................................................................... 1 1.1 OUTLINE ........................................................................................................................................................ 1 2. DATA REPRESENTATIONS ........................................................................................................................... 2 2.1 GRAPH REPRESENTATIONS ............................................................................................................................. 2 2.1.1 Definition ............................................................................................................................................... 3 2.2 THE LASTGELD DATABASE ............................................................................................................................. 4 3. SYSTEM DESIGN .......................................................................................................................................... 5 3.1 DEFINITION .................................................................................................................................................... 5 3.2 INDEXING ....................................................................................................................................................... 6 3.4 CREATING A GRAPH DATA STRUCTURE........................................................................................................... 8 3.5 CREATING EDGES ........................................................................................................................................... 9 3.6 FINDING AN ANSWER TO A KEYWORD QUERY ............................................................................................... 10 3.7 CREATING SQL QUERIES .............................................................................................................................. 10 3.8 RESULTS ...................................................................................................................................................... 11 4. EXPERIMENT.............................................................................................................................................. 12 4.1 EXPERIMENTAL DESIGN ............................................................................................................................... 13 4.2 EXPERIMENTAL RESULTS ............................................................................................................................. 14 4.3 ANALYSIS .................................................................................................................................................... 16 5. RELATED WORK ........................................................................................................................................ 17 5.1 GRAPH-BASED SYSTEMS............................................................................................................................... 17 5.1.1 Data representation ............................................................................................................................. 17 5.1.2 Top-k ranking ....................................................................................................................................... 18 5.2 XML-BASED SYSTEMS ................................................................................................................................. 20 5.2.2 Semantic relatedness ............................................................................................................................ 21 5.2.3 Top-k ranking ....................................................................................................................................... 22 6. IMPLEMENTATION ................................................................................................................................... 23 6.1 CREATING INDICES ....................................................................................................................................... 23 6.2 OBJECT ORIENTED DESIGN ........................................................................................................................... 24 6.3 FLOW OF CONTROL....................................................................................................................................... 25 6.4 WEIGHTED GRAPH DATA STRUCTURE ........................................................................................................... 26 5.5 ALGORITHMS ............................................................................................................................................... 27 6.5.1 Searching indices ................................................................................................................................. 27 6.5.2 Intersecting id-numbers ....................................................................................................................... 27 6.5.3 Finding combinations........................................................................................................................... 28 6.5.4 Retrieving the vertex with the most edges ............................................................................................ 28 6.6 JAVA LIBRARY CLASSES ............................................................................................................................... 29 6.7 WEB DEPLOYMENT....................................................................................................................................... 30 6.7.1 Specification ......................................................................................................................................... 30 6.7.2 User interface ....................................................................................................................................... 30 6.7.3 Web address ......................................................................................................................................... 30 6.7.4 The search application doesn’t work ................................................................................................... 30 7. EPILOGUE ................................................................................................................................................... 31 7.1 CONCLUSIONS .............................................................................................................................................. 31 7.2 DISCUSSION ................................................................................................................................................. 31 7.3 PROPOSALS FOR FUTURE WORK.................................................................................................................... 32 REFERENCES ................................................................................................................................................... 32 APPENDIX A: CREATING INDICES IN PERL ....................................................................................................... 34 APPENDIX B: SEARCH APPLICATION SOURCE CODE IN JAVA .......................................................................... 36 FIGURES 2.1 Königsberg anno 1736………………………………………………………………………………….……………………………………………..…………………. 2 2.2 Graph representation of the Königsberg problem……………………………………………..……………..………………..…………………………. 2 2.3 Graph representations……………………………………………………………………………………………………………………………………………….…... 3 2.4 A directed graph……………………………………………………………………………………………………………………………………………….………..….. 3 2.5 A weighted graph…………………………………………………………………………………………………………………………………….…………………….. 3 3.1 System design………………………………………………………………………………………………………………………………………………….…………….. 5 3.2 A fragment of an inverted index………………………………………………………………………………………………………………………….…………. 6 3.3 A directed weighted graph representation of the keyword query IJsbrand Hanning Riga 1744…..………………………..…..…… 8 3.4 Graph II IJsbrand Hanning Riga 1744......………….……………………………………………………………………………………………….…………. 10 4.1 Experimental results keyword query A……………………………………………………………………………………………………………….……..... 14 4.2 Experimental results keyword query B…………………………………………………………………………………………………………….…………… 14 4.3 Experimental results keyword query C………………………………………………………………………………………………………………….……… 15 4.4 Experimental results keyword query D………………………………………………………………………………………………………………….……… 15 5.1 A fragment of a data graph model…………………….…………………………………………………………………………………………………………. 17 5.2 Radius Steiner trees …….…………………………………..……………………………………………………………………………………………….…………. 17 5.3 Accented Steiner nodes……………………………………………………………………………………………………………..………………….…...……….. 19 5.4 A Steiner graph result………………………….……………………………………………………………………………………………………….……………… 19 5.5 An XML data representation………………..……………………………………………………………………………..…………………….………………… 21 6.1 Object oriented design………………………………………………………………………………………………………………………..….…………….…….. 24 6.2 Weighted graph data structure…………………………………………………………………………….…………………………………………...….…….. 26 6.3 User interface search application…………………………………………………………………………………………………………………………………. 30 TABLES 2.1 Graph endpoints……………………………………………………………………………………………………………………………………………………….……. 3 2.2 A fragment of the Skippers table……………………………………………………………………………………………………………………………………. 4 2.3 A fragment of the Locations table……………………………………………………………………………………………………………………………….….. 4 3.1 Fields eligible for keyword-based search………………………………………………………………………………………………….……………….……. 4 3.2 Number of intersections…………………………………………………………………………………………………………………….………………..………….9 3.3 Example queries…………………………………………………………………………………………………………………………………….…………………….. 11 4.1 Experimental queries………………………………………………………………………………………………………………………………………………….….13 5.1 Table data representation………………………………………………………………………………………………………………………………………….… 17 6.1 Implemented Java Library classes…………………………………………………………………………………………………………………………………..29 6.2 Web deployment specification……………………………………………………………………………………………………………………………………….30 LISTINGS 3.1 An index of first names…………………………………………………………..………………………………………………………………………………………. 6 3.2 An index of dates……………………………………………………………………………………………………………………………………………………………. 6 3.3 An index of harbors…………………………………………………………………………………………………………………………………………….………….. 7 3.4 An index of countries………………………………………………………………………………………………………………………………….……………………7 3.5 Retrieving occurrences of keywords from the indices given the query Jan Cornelis Nantes 1744……………….…………….........8 3.6 Intersecting vertices for the query IJsbrand Hanning Riga 1744……………………..………………………………….…………………….….. 10 5.7 System-generated SQL queries and their answers………………………………………………………………………………………………..………. 10 5.1 An XML fragment……………………………………………………………..……………………………………….…….……………………………….………….. 20 6.1 Creating indices…………………………………………………………………………………………………..…………………………………….…………………..23 1. Introduction Today‟s most widely used search engines enable users to express a search query by means of one or more keywords. This query can express a descriptive phrase or isn‟t more than a single term, coherent with a specific information need. The user can query the available data without having to know any query language or having to know how the data is stored in its internal data repository. In this thesis I investigate the subject of applying a keyword-based search approach to data stored in a relational database. More specific I focus on a database containing historical data. The database consists of two tables containing structured data. The main table contains about 13.000 records of data about skippers and their cargo entering the port of Amsterdam in the period from 1744 to 1748. My objective in this thesis is to show that keyword-based search in a relational database can yield results likely to be meaningful given the available Lastgeld data. I investigate this subject by presenting a search application I developed that enables users to perform keyword-based search in the available Lastgeld data. In general I focus on answering the following questions: 1. 2. What methods are applied in querying a relational database given on one or more keywords? How can these methods yield meaningful results in a keyword-based search application, given the available Lastgeld data and the relational database containing this data? 1.1 Outline In the second chapter I describe some of the theory associated with graph data structures, since I use a graph data structure to be able to query the data in the database. In this chapter I also describe the Lastgeld data and the way it is stored in a relational database. In chapter three, I present the application I developed and I describe the heuristics applied to achieve results. I end this chapter by showing several example queries and their resulting answers returned by the presented application. In the fourth chapter I describe the experiment I performed regarding the efficiency of the presented search application. Chapter five is about some of the work performed in the research area of keyword-based search in structured data. In this chapter I describe several systems that enable keyword-based search over structured data using distinct approaches. In chapter six I return to the subject of the presented search application and I describe which choices I made to implement it. Finally, in chapter seven I draw conclusions, discuss achievements and propose future work. The appendices contain the documented source code of the presented search application. 1 2. Data representations When a keyword-based search system receives a query, it needs to determine what this query means in terms of the data repository it is designed to search. If the query consists of a single keyword, then that single keyword can occur in many locations in the database. If there are more keywords, it also needs to determine the relation between the keywords in the underlying database. If the data in the underlying database is connected by many relations that give meaning to the data, then the search system has to be familiar with these relations to be able to meaningfully determine if the keywords entered match any of these relations. To be able to determine such relations, or to retrieve any data, the data must be transferred from a storage device to main memory. Consequently the relations existing in the database on the storage device have to be replicated in main memory also. One could say that a model of the data and the relations meaningfully connecting entities of data must be available in main memory to be useful to the search system. A way to create such a model is by means of a graph data structure. In a graph abstract data type, entities of data can be connected to one another in an unrestricted manner. Whereas tree data structures provide a useful way of representing relationships in which a hierarchy exists, a graph data structure becomes useful if relationships between data entities appear more freely. Dale et al.[3] 2.1 Graph representations Graph theory is rooted in mathematics. In 1736 graph theory was born when the King of Prussia confronted mathematician Leonard Euler with the following problem: The town of Königsberg (now Kaliningrad in Russia), is built at the point where two branches of the Pregel river come together. The river divides the town into an island and some land around the river banks. The island and the various pieces of main land are connected by seven bridges. Is it possible for a person to take a walk around town, starting and ending at the same location, and crossing each of the seven bridges exactly once? Euler‟s conclusion was that it is impossible to travel the bridges in the city on Königsberg once and only once. Euler claimed that if there are more than two landmasses with an odd number of bridges, then no such journey is possible. Second, if the number of bridges is odd for exactly two landmasses, then the journey is possible if it starts in one of the two odd numbered landmasses. Finally, Euler claims that if there are no regions with an odd number of landmasses then the journey can be accomplished starting in any region. Paoletti [12] Figure 2.1: Königsberg anno 1736 Figure 2.2: graph representation of the Königsberg problem 2 2.1.1 Definition In our time, a graph consists of three entities; a set of vertices V(G); a set of edges E(G) and an edge-endpoint function; g: E(G) → V(G) that connects each edge with a pair of vertices. Vertices can represent whatever is subject of attention; people, brain cells, cities, courses, or entities of data present in a relational database. If the vertices represent cities, then the edges might represent the roads between the cities. Because the road between Groningen and Amsterdam also runs between Amsterdam and Groningen, the edges in this representation have no direction. This is called an undirected graph. If an edge represents for instance a pipe that transports a fossil fuel like natural gas, then the gas is most probably transported in only one direction. A graph with edges directed from one vertex to another is called a directed graph. Lanzani [8] Directed graphs are often represented with arrows like for instance figure 2.4. A more formal definition of the directed graph in figure 2.4 is: Figure 2.3: graph representaions Lanzani [8] V(G) = {1, 3, 5, 7, 9, 11} E(G) = {(1,3),(3,1),(5,7)(5,9),(9,9)(9, 11)(11,1)} In an undirected graph the arrows are simply omitted since the order of the vertices in each edge is unimportant. A more formal definition of the first undirected graph in figure 2.3 is: V(G) = {1, 2, 3, 4, 5} E(G) = {(1,2),(2,3),(3,4)(4,5)(5,1) Edge e1 e2 e3 e4 e5 Endpoints {1, 2} {2, 3} {3, 4} {4, 5} {5, 1} Table 2.1: graph endpoints If two vertices in a graph are connected by an edge, then they are said to be adjacent. In figure 2.4, vertex 5 is said to be adjacent to vertices 7 and 9, while vertex 1 is said to be adjacent from vertices 3 and 11. A tree is a special case of a directed graph in which each vertex may only be adjacent from one parent vertex, except the root vertex is not adjacent from any other vertex. A path from one vertex to another consists of a sequence of vertices such that each vertex in that sequence is connected to another vertex by an edge. This sequence of vertices and edges make it possible to traverse the graph in a certain manner. A weighted graph is a graph in which edges are associated with values. Weighted graphs can be used to represent certain applications in which edges are more than a connection. For instance, Figure 2.5 depicts a graph in which the vertices are cities and the edges the roads between the cities. Additionally each edge contains a value that represents the distance in kilometers between the cities. Dale et al.[3] Figure 2.4: a directed graph Dale et al. [3] Figure 2.5: a weighted graph 3 2.2 The Lastgeld database I explained in the previous section, that a graph model of the data and the relations meaningfully connecting entities of data must be available in main memory to be useful to the search system. Now that I explained some important notions of graph theory, I continue to describe the database I used to build a graph data structure. The database of subject is called the Lastgeld database. The Lastgeld database contains data about skippers and their cargo entering the port of Amsterdam from 1744 to 1748. These skippers had to pay a toll named Lastgeld. Welling [15] Table 2.2: a fragment of the Skippers table I only use two tables. The main table (table 2.2) contains about 13.000 tuples1. Additionally I use a second table (table 2.3) that contains about 1500 tuples. This table relates to the main table by the name of the “hid” attribute. I chose this data representation because it shows two important notions within the domain of any relational database. First, keywords in Table 2.3: A fragment of the Locations the same tuple are related. For instance, a keyword query Cornelis Vos table should match the data in the first tuple. Secondly, keywords possibly span multiple relations. For instance, a keyword query Hendrik Clasen France should match data in the third tuple and should additionally match data in the Locations table, i.e. the search system should return all the tuples that contain data about a skipper named Hendrik Clasen who visited any harbor in France. The Lastgeld database is governed on a storage device by a Relational Database Management System (RDBMS). The RDBMS provides an interface to the data it retains by means of a query language. In the case of the Lastgeld database the query language is SQL (Structured Query Language). For instance, the following queries are valid when one wants to retrieve data from the Lastgeld database: SELECT date, firstname, lastname FROM skippers WHERE achternaam = 'clasen' (1) SELECT date, firstname, lastname FROM skippers INNER JOIN locations ON (skippers.hid = locations.hid) WHERE skippers.lastname = 'vos' AND locations.harbor = 'nantes' (2) The first query is directed at a single table, while the second query joins the two tables in order to get matches from both tables. If one reflects on the abstraction layer that allows us to operate the database, then one could say that the notions of relations, tuples, attributes, keys, foreign keys, SQL, etc. are all abstractions of more complicated constructs beneath the surface of the relational database. These abstractions allow us to deal with the database with relative ease. For instance, programmers need the control of a SQL query language to perform a wide variety of actions on the database. However, the abstraction layer suitable for programmers isn‟t very suitable for the people whose primary interest is the data that is stored in the database. 1 For clarity; the terms relation, attribute and tuple are more commonly referred to as table, column, and row, respectively. 4 3. System design In chapter 2 I describe some of the properties of the Lastgeld database. I also explain that the data and relations residing in the database are eventually stored on a storage device. To be able to perform any kind of search operation, it is necessary to retrieve all the data from the database and have it available in main memory. This can be done in several ways, for instance: 1. 2. 3. Replicate all the data available in the database as a graph data structure that is loaded into main memory, and traverse this data graph to obtain answers to keyword queries; Aditya et al.[1] Replicate only the data scheme of the database as a graph data structure that is loaded into main memory and traverse this scheme graph to obtain answers to keyword queries; Hristidis et al.[6] Export all data as XML and use an extended XML query language to obtain answers to keyword queries. Cohen et al.[2] In chapter Five I more elaborately describe some of the methods applied in the research area of keyword-based search in structured data. Although most probably inspired by what I describe in chapter five, I have not attempted to rebuild these methods. They appear to be highly dependent on the context in which it is meant to be implemented. Considering this observation I have solely focused on the context of the Lastgeld data and its container, a relational database. 3.1 Definition I have developed a heuristic that is best described in terms of the following steps: 1. Index all table fields eligible for keyword search; 2. Search all indices based on the keywords in a keyword query; 3. For every keyword match found in an index, create a vertex in a graph data structure; 4. For every possible combination of two distinct vertices, create edges between two vertices if the idnumbers associated each vertex has at least one resemblance; 5. Find the vertex that has the most edges, since this is the vertex that relates the most keywords present in the graph, and intersect all id-numbers associated with the edges adjacent from this vertex; 6. Use the id-numbers that result from the intersection of edges to create SQL queries. keyword query search indices answer create return query graph relational database Figure 3.1 System design Figure 3.1 visualizes these steps in a more general fashion. In the following 6 sections I elaborate on each of these steps in more detail. 5 3.2 Indexing A major concept in information retrieval is indexing. Indexing is applied to gain speed in the process of retrieval. An established indexing technique is to create what is called an inverted index. The basic idea of an inverted index is depicted in figure 3.2, where the numbers denote the document id-numbers in which the terms occur. The terms occurring in the document corpus are united in what is called a dictionary. Each term in this dictionary points to what is called a postings list consisting of separate postings. For instance, given a keyword query Brutus and Ceasar. The postings lists of the terms Brutus and Ceasar are retrieved and intersected to obtain the document id-numbers that occur in both postings lists. Manning et al. [11] The term and can be handled in different ways; treat it as a phrasal element; designate it to be a stop word, hence ignoring it; or perhaps interpret it as a boolean operator.2 Figure 3.2: a fragment of an inverted index I explain all of this because I used the Manning et al. [11] concept of an inverted list within the context of the Lastgeld database, although in a very different manner. The construct is based on the following reasoning: if a term like for instance “Brutus” belongs to a document with id 1, then a term like for instance “Cornelis” belongs to a tuple with id 1. I have materialized this reasoning by creating indices of the form as shown in listing Table 3.1: fields eligible for keyword-based search 3.1 and 3.2. An advantage of the Lastgeld data is the fact that the data doesn‟t change over time. I simply pre-generated the indices and AART 1198 stored them on disk. I created these indices for ABE 4765 12726 4790 1567 1246 2 the following attributes : ABE BR 6900 12471 11380 857 Skippers - date, firstname, lastname and Locations - harbor, modcountry The dates in the second column I have reduced to only the year notation as shown in listing 3.2. It is difficult to see for instance “1744-0401” as a keyword while just “1744” is easier to grasp. However, I emphasize that this is a design choice made to reduce complexity. I will not go into which date format is the best to index. ABE BROER 5986 ABE BROERSZ 9008 ABEL 6545 ABEL BAS 3553 ABLBERT 7581 ABLBERT J 10065 ABRAHAM 3410 4434 6303 5806 9355 9064 6687 8444 9828… Listing 3.1: an index of first names 1744 1745 1746 1747 1748 1 2 3 4 5 6 7 8 9 10 11 12 13 2332 2333 2334 2335 2336 2337 5027 5028 5029 5030 5031 5032 7619 7620 7621 7622 7623 7624 10423 10424 10425 10426 10427 16 17 18 19 20... 2338 2339 2340... 5033 5034 5035… 7625 7626 7627… 10428 10429 10430… Listing 3.2: an index of dates 2 I will not go into this particular subject since the relevance of the term “and” is minimal in the Lastgeld data set. However, supporting Boolean operators in a keyword based search system can be useful, although I consider this subject to be beyond the scope of my thesis project. 3 I have decided not to use the numerical data as shown in the Skippers table (table 2.2), since numerical data will not fit the classification of a keyword. 6 I have shown the way I created indices for some of the data in the Skippers table. To be able to associate the data in the Locations table with the data in the Skippers table, I chose to ignore the relational keyforeign key concept all together. In essence this abstract concept serves the purpose of attaching relating entities of data over multiple tables within the domain of a relational database, not in the data structure I use to connect data. Listing 3.3 and 3.4 show how I span two tables by assigning the id-numbers in the Skippers table to their relating data in the locations table. This way I can deal with the data per tuple, including the data in a relating table, by a single id. I will elaborate on this benefit in section 3.7. AABO 10051 3670 12238 6360 9627 3812 6464 AAHUS 1465 4526 6088 5876 9036 1804 7579… AALBORG 4345 6911 6027 6438 8408 9311 … ALAMEIDA 10351 12755 12514 7843 12778 … ALEXANDRIEN 5544 ALICANTE 2213 506 2587 8481 12925 4544… Listing 3.3: an index of harbors BELGIUM 7659 2379 7491 7596 12956 9064 9828… DENMARK 11644 11762 1519 7549 3602 9108 10175… EGYPT 5544 ENGLAND 1000 159 6016 2225 2312 12211 7832… ESTONIA 3921 7800 1118 10851 884 11398 11622… FINLAND 10051 3670 12238 6360 9627 3812 6464… Listing 3.4: an index of countries 3.3 Searching indices Now that I explained how I created indices, I proceed explaining how I search these indices. Given a keyword query Q that consists of keywords K1,…,Kn , I search every index for every K in Q. For instance given a keyword query: Cornelis Vos France 1744, I perform the following index searches: Cornelis - Skippers.date, Skippers.firstname, Skippers.lastname, Locations.harbor, Locations.modcountry Vos - Skippers.date, Skippers.firstname, Skippers.lastname, Locations.harbor, Locations.modcountry France - Skippers.date, Skippers.firstname, Skippers.lastname, Locations.harbor, Locations.modcountry 1744 - Skippers.date, Skippers.firstname, Skippers.lastname, Locations.harbor, Locations.modcountry This seems a bit redundant at first, but I do not really know based on some keywords entered where they might occur in the indices or what their inter-keyword relationship is. Furthermore I must emphasize that I do not perform any pre-analyzing of keywords based on their appearance. For instance if a keyword entered is numerical like 1744, I still search the indices Skippers.firstname, Skippers.lastname, Locations.harbor, Locations.modcountry even though I know that no numerical values exist in these indices. I do this to keep me from extra work, at the cost of efficiency. Also I haven‟t divided the indices in partitions, for instance the index of Skippers.firstname is a continuous list from A to Z. I could have divided the indices in sub-indices that contain one letter of the alphabet, this way I could select the appropriate sub-index based on the first letter or digit of a keyword. I haven‟t done this to keep me from dealing with many separate indices, again at the cost of efficiency. For instance given a keyword query Jan Cornelis Nantes 1744, the process of searching the indices for keywords yields lists with the following contents: A)Jan - firstname[2, 19, 54, 73, 76, 88, 95, 97, 112, ..., 12999] B)Cornelis - firstname[1, 56, 130, 137, 143, 144, 176,..., 13007] C)Cornelis - lastname[8, 53, 130, 160, 211, 220, 235, 358,..., 12956] D)Nantes - harbor[1, 190, 234, 335, 379, 477, 478, 554,..., 12865] E)1744 - date[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, ..., 2331] Listing 3.5: retrieving occurrences of keywords from the indices given the query Jan Cornelis Nantes 1744 7 3.4 Creating a graph data structure Looking at listing 3.5, B, D and E seem to be related in the first tuple, however the first name A and C do not belong to the first tuple, so this isn‟t a very good match. Ideally one would want to find the tuples(s) in which all keywords are present, or tuples in which at least four of the five keywords are present. However, I realize if one wants to retrieve relating data of this kind in any way, one has to find a way to deal with the complexity of interkeyword relationships first. I decided to use a directed weighted graph data structure. This data structure enables me to handle the complexity of the data in a better way. In section 3.6 I explain why I implemented the graph as directed graph instead of an undirected graph. Recall that I described the essence of a weighted graph in section 2.1.1. Figure 2.5 depicts a weighed graph representation of the roads between cities and the distances associated with the edges between the roads. Based on this concept I came up with the following reasoning: If I declare every keyword in a keyword query, that occurs at least with one id in an index, to be a vertex, then I can connect a pair of vertices by creating edges between them, if they have at least one id in common. In doing so I can assign the id-numbers two vertices have in common to the edge between these two vertices. This way I create a weighted graph in which all keywords of the keyword query found in the indices are vertices in the graph and the id-numbers associated with a keyword in the index are associated with the vertex of that keyword. There exists an edge between any possible combination of two vertices if two vertices have at least one idnumber in common. If this is the case then an edge between that pair of vertices is created and the weight of the edge is associated with all the id-numbers these two vertices have in common. For example the query IJsbrand Hanning Riga 1744 is inserted in the directed weighted graph data structure as follows: Figure 3.3: a directed weighted graph of the keyword query IJsbrand Hanning Riga 1744 8 3.5 Creating edges In order to create edges between two vertices the id-numbers part of each of these two vertices, have to be intersected. The result of the intersection is assigned to the edge between these two vertices. I found this to be a good solution since the edge represents the relation between two data entities based on the id-numbers they have in common. If an edge is established between, for instance, firstname.ijsbrand and lastname.hanning, I chose to not also create an edge between lastname.hanning and firstname.ijsbrand. This would make the graph more complicated, while having the edges in just one direction worked out to be sufficient. I elaborate on this in the next section. First, I describe the need to determine all the pairs of vertex combinations possible to be able to intersect the id-numbers associated with vertices. Given a set of 5 vertices as depicted in figure 3.3, the following distinct combinations are possible: {0,1}{1,2}{2,3}{3,4}{0,2}{1,3}{2,4}{0,3}{1,4}{0,4}. These combinations of vertices must all be intersected because they are potentially related. For instance given the keyword query; IJsbrand Hanning Riga 1744, the following intersection sequence must be completed: ***vertex combinations*** ijsbrand firstname[64, 901, 5360, ijsbrand lastname[5039] --ijsbrand lastname[5039] hanning lastname[64] --hanning lastname[64] riga harbor[64, 65, 85, 103, 195, --riga harbor[64, 65, 85, 103, 195, 1744 date[1, 2, 3, 4, 5, 6, 7, 8, --ijsbrand firstname[64, 901, 5360, hanning lastname[64] --ijsbrand lastname[5039] riga harbor[64, 65, 85, 103, 195, --hanning lastname[64] 1744 date[1, 2, 3, 4, 5, 6, 7, 8, --ijsbrand firstname[64, 901, 5360, riga harbor[64, 65, 85, 103, 195, --ijsbrand lastname[5039] 1744 date[1, 2, 3, 4, 5, 6, 7, 8, --ijsbrand firstname[64, 901, 5360, 1744 date[1, 2, 3, 4, 5, 6, 7, 8, 7750, 7757, 11404] 198, 346, 373,...,13006] 198, 346, 373,...,13006] 9, 10, 11, 12,...,2331] 7750, 7757, 11404] 198, 346, 373,...,13006] 9, 10, 11, 12,...,2331] 7750, 7757, 11404] 198, 346, 373,...,13006] 9, 10, 11, 12,...,2331] 7750, 7757, 11404] 9, 10, 11, 12,...,2331] Listing 3.6: intersecting vertices for the query “IJsbrand Hanning Riga 1744” A consequence of intersecting all combinations of vertices this way is that the intersection work that needs to be done to create edges increases fast in relation to the number of keywords in the query. Table 3.2 shows the increase of the number of intersections as result of the increase of the number of keywords. Since some of the id-lists as shown in listing 3.6 are quite large, the retrieval time should increase in relation to the size of the id-lists together with the number of keywords in a keyword query. In chapter four I perform an experiment to further analyze these dependencies in relation to overall retrieval time. Number of keywords 4 6 8 10 12 20 Number of intersections 6 15 28 45 66 190 Table 3.2: number of intersections 9 3.6 Finding an answer to a keyword query I assume that the vertex with the most outgoing edges is the “binding” vertex. Vertex A in figure 3.4 connects to BCD, I simply intersect the numbers associated with these three edges and I obtain the final result with id 64. Because I decided to implement the graph as a directed graph this is possible. If I implemented the graph in figure 3.4 as an undirected graph each vertex would have 3 incoming and 3 outgoing edges and hence this approach wouldn‟t work I must admit that this is likely not the best solution. For instance suppose there is another vertex in the game by the name E, and A also matches this vertex, but all the other vertices A is related to do not match this vertex then the final result is an empty list. In other words the edges of the vertex with the most Figure 3.4: graph representaion of the keyword query edges must all be related. IJsbrand Hanning Riga 1744 A better solution would be to traverse the graph. There are several ways to traverse the graph. While traversing the graph the main condition that must be fulfilled is that every new edge reached must have at least one id in common with latest visited edge to be able to continue the traversal on a new edge. To find the best fit answer given the keywords of a keyword query it is evident that as many vertices possible should be visited. In the case of the graph in figure 3.4, the path ABDC would yield the id-numbers spanning the most keywords. Finding a good graph traversal algorithm is one of the subjects I suggest in chapter 8 as a possible follow-up. For now I will use the rather „naïve‟ approach of finding the vertex with the most edges. 3.7 Creating SQL queries At this stage an answer to a keyword query has or has not been found, depending on the presence of the query keywords in the indices of the relational database. If there is a result list of id-numbers then the search application possesses all it needs to efficiently retrieve all relating data from the relational database. For instance given the keyword query Cornelis Vos France, the search application creates the following SQL queries: SELECT SELECT SELECT SELECT * * * * FROM FROM FROM FROM Skippers Skippers Skippers Skippers WHERE WHERE WHERE WHERE idno idno idno idno = = = = 1 2574 5288 8102 results: id = 1 first name = CORNELIS last name = VOS harbor = NANTES toll-decimal = 2.40 weight = 128.47 guldens = 2 stuivers = 8 cargo units = 65 id = 2574 first name = CORNELIS last name = VOS harbor = LE CROISIC = 29 toll-decimal = 2.40 weight = 57.32 guldens = 2 stuivers = 8 cargo units id = 5288 units = 65 first name = CORNELIS C. last name = VOS harbor = BORDEAUX toll-decimal = 2.40 weight = 128.47 guldens = 2 stuivers = 8 cargo id = 8102 units = 38 first name = CORNELIS W. last name = VOS harbor = LIBOURNE toll-decimal = 2.40 weight = 75.10 guldens = 2 stuivers = 8 cargo Listing 3.7: system-generated SQL queries and their answers 10 3.8 Results In Information Retrieval search system effectiveness is assessed relative to an information need, not to a query. A document is classified as relevant if it is coherent with a certain information need, not because it contains all the keywords in a certain query. Manning et al.[11] However, a tuple in a relational database is not a document containing potentially many terms. When I query a document with 500 terms with just two keywords and these two keywords appear to be in that document, the relation of 2 to 500 is weak. In contrast when I query just tuples in a relational database consisting of five fields like in the Lastgeld database and two of these five fields are present in a keyword query as well in a specific tuple, it is obviously a much stronger relation. Still, the relevance of retrieved results can‟t be judged without a defined information need. An inherent characteristic of keyword queries is that they are inexact. One cannot claim that a keyword query like Jan Jansen Kleine Oost Germany 1745 is similar to for instance: SELECT * FROM skippers INNER JOIN locations ON (skippers.hid = locations.hid)WHERE skippers.date = '1745' AND skippers.firstname = 'jan' AND skippers.lastname = 'jansen' AND harbor = 'kleine oost' AND locations.harbor = 'germany' By trading a query syntax like SQL for just some keywords, control over retrieving exact answers is lost. The spaces between keywords do not per se imply a relation between keywords, but they do not imply there isn‟t any relation either. This lack of exactness of keyword queries has forced me to make an assumption of what is important when dealing with multiple keywords in a query. I assume that if the keywords in a keyword query are related in the relational database that contains the Lastgeld data, then retrieving this relating data yields results likely to be meaningful given the keyword query. Consider the following queries and their effect: Query Cornelis Cornelis Vos 1744 France Cornelis Vos Groningen Cornelis Vos Jansen Cornelis France Description If there is just one keyword present in the query, the search application retrieves all tuples in which the keyword Cornelis occurs. Note, Cornelis appears in the indices as a first name and a last name, both tuples in which Cornelis appears as a last name and a first name are retrieved. All keywords are present in the indices, and all keywords are related by several idnumbers. Based on the assumption described in the previous paragraph, the search application only retrieves the tuples in which all the keywords occur, all tuples in which they occur separately or not retrieved. Keyword order does not influence the generation of results, e.g. Vos France 1744 Cornelis yields the exact same results. The keyword Groningen doesn’t appear in the indices and thus no vertex is created for this keyword. However, Cornelis and Vos do occur in the indices and thus vertices are created for these keywords. As a result this query yields the same results as Cornelis Vos alone. This keyword query is a special case; Cornelis and Vos are related, but Cornelis and Jansen are also related. The keyword Cornelis has the most edges; it points to Vos and to Jansen. As a consequence of working with vertex that has the most edges, the id-numbers associated with the edge between Cornelis and Vos are intersected with the id-numbers associated with the edge between Cornelis and Jansen. This leads to an empty result list. I have not attempted to simulate “OR” semantics, therefore I have not merged the results of Cornelis Vos and Cornelis Jansen into one result list. In terms of the relational database Cornelis appears in the Skippers table and France appears in the Locations table, since these relations do not exist in the indices, the search application generates SQL-queries for a single table only. This query retrieves all the occurrences of Cornelis in the Skippers table in relation to France in the Locations table. Table 3.3: Example queries 11 4. Experiment In this chapter I present the results of the experiments regarding the efficiency of the presented search application. I have designed an experiment in which I measure: 1. The retrieval time of multiple keyword-based search queries posed to the search application. More specific I focus on: A) The number of keywords in the query. I state in section 3.5 that the retrieval time will increase in relation to the number of keywords in a keyword query. In table 3.2 I show that the number of intersections needed increases rapidly as the number of keywords in the query increases. B) The size of the id-list associated with a keyword. In section 3.5 I also stated that retrieval time will increase if the size of the id lists associated with the keywords increases. Given these parameters I can assess their processing weight in terms of the relative increase of retrieval time. 2. The retrieval time of the SQL-queries the search application proposes. For instance given a keyword query “Cornelis Vos France”, the search application deals with data in different tables by merging the indices of two tables within the relational database, this way the search application restricts it‟s querying to just one table like so: SELECT SELECT SELECT SELECT * * * * FROM FROM FROM FROM Skippers Skippers Skippers Skippers WHERE WHERE WHERE WHERE idno idno idno idno = = = = 1 2574 5288 8102 However, in this approach multiple queries are posed in sequence, e.g. if there are 100 result tuples, then 100 queries are formulated and posed to the relational database. Intuitively this appears to be an expensive solution in terms of processing time. On the other hand the queries are rather straightforward to process, since the queries do not span multiple tables and the „where‟ clause contains just one argument at a time. In the following sections I describe how the factors mentioned in both 1A/B and 2 influence the execution time of a search request. 12 4.1 Experimental design I formulated four search queries to measure the influence of three factors on the query execution time. The search queries are presented in table 4.1 and the three factors are: 1. 2. 3. The index size(s); The number of keywords in the query; The number of queries the search application proposes. A keyword query description number of occurrences B C D Jan Jansen Kleine Oost Germany 1745 involves relatively big lists of idnumbers Germany Nicolaas Bark Hendriks Bordeaux involves the biggest list of id-numbers involves very small lists of idnumbers Jan: 1319 Jansen: 149 Kleine Oost: 5618 Germany: 5683 1745: 2696 Germany: 5683 Nicolaas: 9 Bark: 4 involves mediumsized(relative) lists of idnumbers Hendriks: 333 Bordeaux: 264 Table 4.1: experimental queries Query A consists of 5 keywords. Each keyword occurs in only one index. As a consequence the search application needs to intersect 10 combinations of reasonable sized lists of id-numbers. The answer to the query consists of one tuple, therefore the number of queries the search application proposes to the relational database is one. Query B consists of just one keyword that occurs in only one index. As a consequence the search application doesn‟t intersect at all. The answer to the query consists of 5683 tuples, therefore the number of queries the search application proposes to the relational database is 5683. Query C consists of two keywords with small lists of id-numbers. As a consequence the search application intersects only once. The answer to the query consists of one tuple. Query D consists of two keywords that have medium-sized lists of id-numbers in the indices (relative to the available data). As a consequence the search application intersects only once. The answer to the query consists of one tuple. The answer to the query consists of 6 tuples. For each query I measured the execution time 50 times: for the speed of the execution within the search application and for the speed of the execution of the queries proposed by the search application. In between measurements I halted for 10 seconds. To add an extra element of comparison I also measured the speed of the SQL queries semantically related to A,B,C & D. For instance keyword query A could be interpreted as: SELECT * FROM skippers INNER JOIN locations ON (skippers.hid = locations.hid)WHERE skippers.date = '1745' AND skippers.firstname = 'jan' AND skippers.lastname = 'jansen' AND harbor = 'kleine oost' AND locations.harbor = 'germany' This serves as an indicator of the performance loss the presented search application causes under the assumption that keyword query expresses the same search intention as the mentioned SQL query, given the semantical relatedness of the Lastgeld data. 13 4.2 Experimental results I have performed the experiments on a Pentium 2.2 GHz processor with 4GB of RAM running on Windows Vista. The Lastgeld database as described in section 2.2, I run on the same machine. The RDBMS I use is provided free by MySQL[17]. In the next chapter I describe the implementation details of the presented search application in more detail. Figure 4.1: experimental results keyword query A Figure 4.2: experimental results keyword query B 14 Figure 4.3: experimental results keyword query C Figure 4.4: experimental results keyword query A 15 4.3 Analysis Keyword query A consists of a relative high number of keywords. The 15.500 id-numbers that are associated with the keywords in the indices cause the search application to execute many intersections and therefore the execution time of the search application increases with hundreds of milliseconds. However the SQL query the search application poses is executed in almost instantaneous. (Recall that keyword query A only yields one query). The semantically related autonomous query is somewhat slower but executes in around 10 milliseconds. Keyword query B consists of only one keyword. The search application finishes the request with a median of 44 milliseconds over 50 measurements. However the search application proposes 5683 SQL queries since the keyword Germany occurs with 5683 id-numbers. This slows down the eventual result generation considerably. The median of the execution of these SQL queries is 895 milliseconds over 50 measurements. Also the semantically related query doesn‟t appear to execute flawlessly. Keyword query C still generates about 60 ms of execution time for the keyword search application, while the number of intersections is reduced to one and the id occurrences in the indices are minimal. The search application executes keyword query D slightly slower than query C, but the overall execution times are similar to keyword query C. 16 5. Related work There has been extensive Research on the topic of keyword search over structured data. In section 5.1 I describe three keyword search systems which perform operations on data stored in graph data structures. These graph data structures serve as models of the underlying relational database. I chose to describe three systems that apply different techniques to achieve about the same goals. In section 5.2 I describe work on the subject of keyword search over XML data. Like data stored in a relational database, XML data is structured as well, although in a different sense. Techniques applied in this area are different but eventually their purpose is identical to the purpose of the techniques applied when dealing with relational databases. 5.1 Graph-based systems Discover[6], BANKS[1] and EASE[9] are systems developed in this area. In these systems a database is represented by a graph data structure where tuples are the nodes in the data structure. 5.1.1 Data representation Discover, BANKS and EASE generate query answers called „tuple trees‟ based on the keywords received as input. A tuple tree is a joining tree of tuples, i.e. a data structure that contains one or more tuples in a predefined manner. In a tuple tree each node is connected via foreign-key relationships. In BANKS and Ease a specific kind of tuple tree, called a Steiner tree, is applied to hold the answers to keyword queries A difference between these systems can be found in the way the database of subject is modeled to generate the tuple trees. The key algorithms of Discover work on a graph data structure that is modeled based on the properties of the database schema. The database schema graph is used to produce a number of SQL-queries needed to answer the keyword queries presented to the system. Unlike Discover, more advanced BANKS and EASE, model the entire database as a directed graph data structure. Figure 5.1 visualizes a fragment of the graph data model Ease constructs based on the publication database presented by tables 5.1. From here the system generates tuple trees to create answers to the keyword-queries presented to the system. In case of EASE, as mentioned before, system tuple trees are built as a Steiner graphs. For clarity a tree data structure is a kind of graph data structure. Dale et al.[3] Figure 5.2 visualizes these Steiner trees as answers to keyword queries. The circles around nodes represent a method described in EASE to reduce processing, since it can be very costly to generate Steiner trees over a large data graph. EASE propose to define a radius based on certain properties, containing the nodes necessary to produce an adequate answer to the keyword query being processed. Tables 5.1: table data representation Figure 5.1: a fragment of a data graph model Figure 5.2: radius Steiner trees 17 5.1.2 Top-k ranking An important notion in all these methods is efficiency. The described techniques possibly span very large databases. Because schema or data graphs are kept in main memory and trees are generated on the fly when the system is queried, the processing must finish almost instantaneous. According to Dalvi et al. [4] the graph data structures used in keyword search engines can potentially span very large data sets, because structured data is fairly compact compared to textual data; graphs with millions of nodes, related to hundreds of megabytes of data, can be stored in tens of megabytes of main memory. If a search system runs on dedicated servers even larger graphs such as the English Wikipedia, which contained over 1.4 million nodes and 34 million links (edges) as of October 2006, can be handled. Although efficiency becomes a very important topic when dealing with large data graphs, a certain degree of effectiveness is in most cases just as, or even more important. Discovery, BANKS and EASE all incorporate a top-K ranking mechanism to achieve a degree of effectiveness as well. Discover ranks results by the number of joins involved. The idea behind this strategy is that joins involving many tables are more difficult to grasp. This ranking strategy has a certain parallel with ranking methods used in document retrieval; documents in which keywords occur close to one another are ranked higher than documents in which keywords are far apart. However, in a follow up to Discover, Hristidis et al. [7] propose a ranking method known in the field of Information Retrieval (IR) as relevance ranking. The general idea behind relevance ranking is that according to some definition of relevance, only the few most relevant matches are generally of interest. Consequently instead of computing all matches for a keyword query, only the top-k matches are computed. This, in turn, yields a more efficient solution. The BANKS system incorporates a technique that assigns weights to tuples and assigns weights to edges between tuples. A combination of tuple weights and edge weights in a tuple tree is calculated to rank matches. Liu et al.[10] argue that, despite the methods applied to gain effectiveness, the focus of Discover and BANKS is still primarily set to obtaining efficiency by avoiding the creation of unnecessary tuple trees and by deploying algorithms to improve the time and space complexities. They state that effectiveness should be equally important. In turn they incorporate a full-fledged IR-solution on top of a system conceptually comparable to Discover, BANKS and Ease. Liu et al. define tuple trees to be super-documents and all text column values to be documents. Let T be a tuple tree and {D1, D2, …, Dm} be all text column values in T. Then to rank tuple trees they compute a similarity value between the query Q and the super-document T as shown in Equation 1. The similarity is the dot product of the query vector and the super-document vector. In contrast to the systems described earlier Liu et al. apply IR-evaluation techniques to assess the results achieved. 𝑆𝑖𝑚 ( 𝑄 , 𝑇 ) = 𝑘∈𝑄,𝑇 𝑤𝑒𝑖𝑔𝑡 𝑘, 𝑄 ∗ 𝑤𝑒𝑖𝑔𝑡 (𝑘, 𝑇) (1) In EASE, a ranking mechanism is proposed that incorporates three notions; a TF-IDF based ranking function that considers textual properties of a Steiner graph; the compactness of a Steiner graph and the keyword order in the query. The TF-IDF based ranking function assigns a weight to the Steiner graph, that is, the keywords present in the Steiner graph. Recall that Steiner graphs are an extraction of the keyword presence in the underlying data graph that is the model of the data present in a relational database. The TF-IDF based ranking function takes into account the term frequency (TF), the inverse document frequency (IDF) and the normalized document rank (NDL). TF and IDF are used to rank. In IR-literature NDL used to normalize document length, since a longer document tends to repeat the same terms, while this doesn‟t per se mean that the document should be ranked higher. Manning et al.[11] 18 Then finally these three parameters are computed as follows: (2) ntf ( k I ,G ) = 1 + ln (1 + ln (1 + tf (k i , G )) (3) idf ki = ln (N + 1) / (Nki + 1 ) (4) ndl = ( 1 – s ) + s * tlG / avgtl Where tf (ki , G ) in Equation 2 denotes the term frequency of keyword k i in the data graph G ; In Equation 3 N and Nki denote the number of Steiner graphs and the number of those Steiner graphs containing keyword k i . In Equation 4, tlG denotes the total number of terms in G and avgtl is the average number of terms among all Steiner graphs. These parameters are consequently used to compute a ranking weight between a keyword and a Steiner graph SG. In EASE however Li et al. comment that Information Retrieval ranking methods based on TF-IDF can be efficient for textual documents, but are not very efficient for semi-structured and structured data. Li et al. state that a consequence of modeling data as a graph is that the ranking of structural properties of the data graph, becomes just as, or even more important. According to Li et al., rich structural relationships should be at least as important as discovering more keywords in the data graph. To this end EASE [9] takes the structural compactness of the data into account to create an additional weight on top of the TF-IDF weight described earlier. Given a keyword query K { k1, k2 ,..., k m }, then in figure 5.3 the thickedged circles containing p5, p7 and a4 are content nodes that contain at least one keyword. (Recall that a node is represents the data of a tuple in the relational database). Node s in figure 5.5 is called a Steiner node if there exist two content nodes, u and v, and s is on the path u ↔ v (s may be u or v), where u ↔v denotes a path between u and v. Since such a path exists between p5, a4 and p7 the radius property described earlier yields a Steiner radius graph, (Figure 5.4), which serves as an input to a ranking function. The ranking is Figure 5.3: accented Steiner accordingly based on the compactness of the Steiner graph. The underlying nodes idea is that a more compact Steiner graph is more likely to be meaningful. Although the structural compactness of nodes can be important measure when generating a useful result set, it cannot be of service to evaluate interkeyword semantics. The order of the keywords can hold meaning if the query is an expression of a phrase. In EASE, the weighting function applied also takes the keyword order into account. This is done by assigning more weight to keywords that have a smaller inter-keyword distance. Figure 5.4: a Steiner graph result 19 5.2 XML-based systems A different but intrinsic related research topic is keyword search in XML-databases. This topic is related because XML is, like data in relational databases, structured by nature as well. XML queries possibly return entire XML documents, or may as well return deeply nested XML elements. Because of the immanent nested structure of XML the notion of ranking is no longer at the granularity of a document but at the granularity of an XML element. Manning et al.[11] As described in the previous section about graph-based systems, efficiency and effectiveness are very important factors in the process of developing a search system. In section 5.2.1 I describe an indexing method as employed by Florescu et al [5] . After that, I shift from the indexing technique applied by Florescu et al. to the search system XSearch [2]. First I describe the semantics derived from an XML data representation in order to meaningfully answer a keyword query. In conclusion of this chapter, I describe the ranking mechanism of the XSearch system. Florescu et al. [5] extend an XML query language for the purpose of keyword search. In their proposal the XML data is replicated in a relational database. I will not go into this particular architecture. I‟m interested in describing the index system that is employed to retrieve query answers from XML data. A common indexing approach used in traditional IR-systems is by means of an inverted file as described in section 3.2. A simple setup for an inverted file takes the following form: <word, document> This means that word can be found in document. However, when dealing with XML, retrieval is no longer at the granularity of a document but at the granularity of an XML element, as noted earlier. Consider, listing 5.1. The word „Analysis‟ appears in a title element nested in an article element. To be able to utilize the nested structure for retrieval, Florescu et al. make a distinction between keywords occurring as tags, e.g. article; as names of an <document> attribute, e.g. id or as data content of elements. <article id="1" Additionally the depth at which a keyword occurs is <author><name> taken into account. As a result an inverted file of the Adam Dingle</name></author> following form is proposed: <author><name> Peter Sturmh</name></author> <"article", elID1, 0, tag> <author><name> <"id", elID1, 1, attr> Li Zhang</name></author> ... <title>Analysis and Characterization <"name", elID1, 2, tag> of Large-Scale Web Server Access <"Adam", elID1, 2, value> Patterns and Performance</title> <year>1999</year> elIDn is associated with all elements, such that each <booktitle>World Wide Web interior node Ne is labeled with a distinct element ID, Journal</booktitle> elID. Additionally, each elID </article> </document> Listing 5.1: an XML fragment To utilize the inverted list, all data is modeled in records containing the URL of the belonging XML document, the starting and ending positions of the elements within this document, and the type of the element. As a result the following relational schema is obtained: elements(elID, docid, start_pos, end_pos, type,id_val) documents(docid, URL,...) In summary, by scanning the inverted index, which is actually a representation of the data corpus, the search system can find the desired data more efficiently. If the system must search the entire database sequentially the processing cost would be high. Florescu et al. have created an inverted list that allows search on the level of tags, attributes and data values. This way a specific fragment of an xml document can be found fast. 20 5.2.2 Semantic relatedness In XSearch Cohen et al. present a free form query language over XML documents. XML documents are modeled as trees that consist of interior nodes and leaf nodes. Each interior node is associated with a label and each leaf node is associated with one or more keywords. Figure 5.5 represents such a tree as a model of a fragment of the SIGMOD (Special Interest Group on Management of Data) publication database. Figure 5.5: an XML data representation In a sense a node in the tree can be viewed as a human being in our world; different people may have identical names. As such, two different nodes with the same label are different entities of the same type. To extend this analogy, one can say humans are related if they share the same ancestor(s). Now suppose that nodes n and n' have different ancestors, let‟s say na and n'a and these ancestors share the same label, then it is said that n and n' are not meaningfully related. This holds as long n and n' share the same relationship tree. Let T be a tree and let n1 and n2 be nodes in T, then the shortest undirected path between n1 and n2 consists of the path via the lowest common ancestor of both n1 and n2. Recall as described in chapter 2, that a path in a graph is undirected such that G = {(n1, n2), (n2, n1)}, i.e. it‟s possible to get to n2 if n1 is the present location, and vice versa. Then the sub tree consisting of the two paths is denoted as T| n1, n2 and is called a relationship tree. The overall notion of relating nodes is formalized by means of two conditional rules: 1. T| n,n' does not contain two distinct nodes with the same label. or 2. The only two distinct nodes in T| n,n' with the same label are n and n'. 21 These semantics are additionally extended with traditional information retrieval techniques to rank query answers. 5.2.3 Top-k ranking In XSearch, as described in the previous section, sub trees are generated as possible answers to a keyword query. The weights for ranking are calculated at the level of the leaf nodes of a document. Let k be a keyword and nl be a leaf node, and let occ(k, nl) denote the frequency of the occurrence k in nl.. The term frequency of k in nl is defined as: 𝑡𝑓 𝑘, 𝑛𝑙 : = max 𝑜𝑐𝑐 𝑘 ,𝑛 𝑙 𝑘 ′ ∈ 𝑤𝑜𝑟𝑑𝑠 𝑛 𝑙 (5) 𝑜𝑐𝑐 𝑘 ′ ,𝑛 𝑙 This is a variation of an IR-based approach that assigns more weight to frequent words in sparse nodes. Let N be de set of all leaf nodes in the corpus, then the inverse leaf frequency is defined as: 𝑖𝑙𝑓(𝑘): = log 1 + |𝑁| | 𝑛 ′ ∈ 𝑁 𝑘 ∈ 𝑤𝑜𝑟𝑑𝑠 𝑛 ′ | (6) Then, 𝑡𝑓𝑖𝑙𝑓 = 𝑡𝑓 𝑘, 𝑛𝑙 × 𝑖𝑙𝑓(𝑘). Note, by taking a logarithm in ilf, the importance of the tf factor is increased. The actual weight stored is a normalized version of tfilf denoted as w(k,nl), such that w is 0 if k doesn‟t appear in nl . Furthermore the labels are taken into account as a weighting factor. Recall that each interior node is associated with a label. Each label l is associated with a weight w(l) that determines its importance. These weights can either be user-defined or system-generated. The key notion using label weights is that the interior nodes, that determine the structure of the XML-data, can also be taken into account. For instance, higher weights can be assigned to less common labels. To incorporate both tfilf weights and label weights, the vector space model is utilized to determine how well an answer satisfies a query. Let L be the set of all labels and let K be the set of all keywords. Each interior node n in the data is associated with a vector Vn of size |L x K|. The vector has an entry for each pair (l,k) ∈ L x K. Then, Vn [l,k] is used to denote the entry of Vn corresponding to the pair (l,k). Let Nleaf be the set of leaf descendents of n. The values of Vn are defined as follows: 𝑉𝑛 𝑙, 𝑘 = 𝑛′ ∈𝑁𝑙𝑒𝑎𝑓 𝑤 𝑘, 𝑛′ 𝑖𝑓 𝑙𝑎𝑏𝑒𝑙 𝑛 = 𝑙 0 𝑜𝑡𝑒𝑟𝑤𝑖𝑠𝑒 (7) Note that w(k,nl) is 0 if k doesn‟t appear in nl . To be able to calculate similarities between answers and queries, there have to be vectors representing the terms in the query as well. Term t is associated with a vector of the size |L x K|, denoted by Vt , then the similarity between a Query Q and an answer N, denoted as sim(Q,N), is the sum of the cosine distances between the vectors associated with the nodes in N and the vectors associated with the matching terms in Q. As a next step the semantics described in the previous section 3.2.1 are extended by two weight factors. Let tsize(N) denote the number of nodes in the relationship tree of N. If this value is small, then the nodes are closer together, and therefore more likely to be meaningfully related. Additionally it is said that the nodes n and n' participate in an ancestor-descendent relationship if n is de ancestor of n' and vice versa. This indicates a strong relationship between n and n' . This notion is denoted by anc-des(N) , where N denotes the number of unordered pairs that participate in an ancestor-descendent relationship. Finally, given a query Q and a answer N, the factors sim(Q,N), tsize(N) and anc-des(N) are combined to determine the ranking of a query answer: 𝑠𝑖𝑚 (𝑄,𝑁)𝛼 𝑡𝑠𝑖𝑧𝑒 (𝑁)𝛽 × 1 + 𝛾 × 𝑎𝑛𝑐 − 𝑑𝑒𝑠(𝑁) (8) Experimentations were conducted with varying 𝛼, 𝛽 and 𝛾 values to gain more control over the resulting weight. 22 6. Implementation In this chapter I describe in general how I implemented the search application presented in chapter four. The first section I describe how I created the indices using the programming language Perl. After that I focus on the search application. I used the Java programming language for this task. First I show the object oriented design of the search application, followed by a description of the flow of control between functional entities. Then I describe the weighted graph data structure I implemented, followed by the most prominent algorithms. I used Object-oriented constructs to design 12 separate classes, including inner classes. I also used four classes I did not author. These are WeightedGraph, LinkedQueue, QueueInterface and ConnectionPool. The first three are published in [3] and the ConnectionPool is available in [16]. The Perl source code is documented in appendix A and the Java source code is documented in appendix B. Since I use Java I have an elaborate library of functionality at my disposal. I list the Java library classes I used and the function they perform in the search application. Finally, I describe how I implemented the application in a web-based environment. 6.1 Creating indices For every field in the Lastgeld database I wanted to index, I executed the steps as shown in listing 6.1. The Perl source code I wrote can be found in appendix A. 1. Make an alphabetically ordered SQL-dump of a field into a text file. The dump has the following form: (1198, 'AART'), (4765, 'ABE'), (12726, 'ABE'), (4790, 'ABE'),>> 2. Delete all punctuation (Perl); 1198 AART 4765 ABE 12726 ABE 4790 ABE >> 3. Place the id after the index term (Perl); AART 1198 ABE 4765 ABE 12726 ABE 4790 >> 4. Place every similar index term on the same line (Perl) AART 1198 <new line> ABE 4765 ABE 12726 ABE 4790 ABE 1567 ABE 1246 ABE BR 6900 ABE BR 11380 ABE BR 857 ABE BROER 5986 ABE BROERSZ 9008 <new line> ABEL 6545 ABEL BAS 3553 <new line> ABLBERT 7581 ABLBERT J 10065 <new line> 5. Delete all the terms except the term that starts the line (Perl. AART 1198 ABE 4765 12726 4790 1567 1246 6900 12471 11380 857 5986 9008 ABEL 6545 3553 ABLBERT 7581 10065 >> Listing 6.1: creating indices 23 12471 ABE BR 6.2 Object oriented design The UML-diagram depicted in figure 6.1 shows an overview of the classes I created to implement the search application. Figure 6.1: object oriented design 24 I declared the Firstname, Lastname, Data, Harbor and Country to be inner classes of Lastgeld because they are entities that belong to a single semantical unit, which can be thought of as a table. Grouping them together within the Lastgeld class merges them in the Lastgeld class, while still having control of instantiating them separately. I implemented the Graphable interface to make sure that every object being added to the graph has the same properties. The inner classes implement this interface and thus they must implement every method I declared in the interface. The interface construct helps to guarantee that all objects share the same properties. This can prevent inconsistencies and possible errors. All instance variables are declared private or protected. This construct guarantees that all interaction from outside the class with the variables is reserved to public methods. 6.3 Flow of control In this section I explain in general in what order control is passed between the classes depicted in figure 6.1 given a keyword query and a formulated answer. The WebFrontend class is instantiated by the Tomcat web server when it gets an URL-request from the web browser. The web browser displays an input field for a user to enter a keyword query. The web browser sends a http-request to the service method of the WebFrontend class. Here the keyword query is extracted and a database object is created. The keywords are handed over as arguments to the constructor of the database object to be created. The created Database object instantiates –for every keyword in the query- a new Lastgeld object. Every Lastgeld object instantiates five Connector objects, one for every index stored on disk, e.g. if a keyword query of three keywords is entered, three Lastgeld objects are created and thus in total 15 Connector objects are created, each Connector object is associated with one keyword and sequentially searches one index. If there is a match, then the id-numbers associated with that term in the index are stored in a list and returned to the Lastgeld object that created that Connector object. Given a match in a particular index and the id-numbers associated with that match, the Lastgeld object creates one of its inner classes; Firstname; Lastname; Country; Date or Harbor, depending on the index in which the keyword match is found. The id-numbers found in the indices are associated with these inner classes. All of the inner class objects are stored in a list and given back to the Database object. Then, the database object instantiates a Grapher object, to store these objects as vertices in a WeightedGraph object, which is instantiated by the Grapher object. Then the Grapher object retrieves all vertices present in the WeightedGraph object and intersects all the id-numbers associated with all possible unique combinations of vertices, then edges are created and id-numbers two vertices have in common are associated with the edge between them. The vertex with the most edges is retrieved and the id-numbers associated with every edge of the vertex with the most edges are intersected again. This yields a list of results. This list of results is passed back to the Database object. The database object in turn passes it back to the WebFrontend. The WebFrontend instantiates a ConnectionPool object which grants access to the MySQL database, and the WebFrontend formulates one or more SQL-queries based on the id-numbers returned by the Database object. The answers to these queries are passed back via WebFrontend‟s http-response parameter and the results are presented to the user. 25 6.4 Weighted graph data structure Recall that I explained in section 3.4 that all keywords of the keyword query found in the indices are vertices in the graph and the id-numbers associated with a keyword in the index are associated with the vertex of that keyword. There exist edges between any possible combination of two vertices if two vertices have at least one id-number in common. If this is the case then an edge between that pair of vertices is created and the weight of the edge is associated with all the id-numbers these two vertices have in common. To be able to use the weighted graph data structure as decribed in Dale et al.[3], I needed to modify the data structure in its core. The edges in the presented weighted graph data structure of Dale et al. consist of single integer values. However, I needed the weights to be lists of integer values. As a result I modified the data structure as depicted in figure 6.2. Figure 6.2: weighted graph data structure Note that the vertices are Graphable objects, they implement the Graphable interface I designed. Figure 6.1 shows that the inner classes of Lastgeld: Firstname, Lastname, Harbor, Country and Date implement this interface. 26 5.5 Algorithms The core functionality of the presented application can be summarized as; searching indices; intersecting indices; finding combinations of intersections and finding the vertex with the most edges. I describe these subjects in more detail in the next sub-sections. 6.5.1 Searching indices I decided to store the indices as text files, this appeared to be the most straightforward implementation. As a result the indices are searched sequentially. Recall that any line in an index starts with a term followed by a sequence of id-numbers. The StringTokenizer object takes two parameters the line currently read and a delimiter, which is a space in the indices I created. while (inLine != null){ tokenizer = new StringTokenizer(inLine, " "); term = tokenizer.nextToken(); if (term.equals(keyword)){ while(tokenizer.hasMoreTokens()){ String token = tokenizer.nextToken(); Integer id = Integer.valueOf(token); list.add(id); } } inLine = inFile.readLine(); } 6.5.2 Intersecting id-numbers The id-numbers associated with each graphable object are stored in an ArrayList. To be able to intersect two ArrayLists containing id-numbers I use ArrayList‟s retainall method. An ArrayList object inherits this method from the AbstractCollection class. The source code is straightforward, but like many Java classes, this method makes use of other classes to get the job done. Since ArrayList is a Collection, the “contains” method is available for the actual intersection. To be able to loop through the items in the ArrayList an iterator object is used. public boolean retainAll(Collection c) { boolean modified = false; Iterator<E> e = iterator(); while (e.hasNext()) { if (!c.contains(e.next())) { e.remove(); modified = true; } } return modified; } public boolean contains(Object o) { Iterator<E> e = iterator(); if (o==null) { while (e.hasNext()) if (e.next()==null) return true; } else { while (e.hasNext()) if (o.equals(e.next())) return true; } return false; } Comment: the contains method is shown in the right side of this box. JDK source 5.0[14] 27 6.5.3 Finding combinations All vertices are stored in an ArrayList. Each vertex is accessible via an index of the ArrayList. To be able to intersect all pairs of vertices to create edges all possible combinations of ArrayList indices must be obtained. These combinations can be obtained by the size of the ArrayList that contains the vertices. This size is the parameter of the following method I created: private int[][] calculateCombinations(int c){ int[][] combinations = new int[c*c/2][2]; int store = 0; int number = c; for (int b = 1; b < number; b++ ){ int y = b - 1; int i = 1; while (i != number-y){ combinations[store][0] = i-1; i = i + b; combinations[store][1] = i-1; int x = b - 1; i = i - x ; store++; } } return combinations; } For instance if the int 3 is passed, the method returns the combinations 12 23 and 13. 6.5.4 Retrieving the vertex with the most edges LinkedQueue edges; Graphable item; Graphable biggest = null; int biggestSize = 0; for(int i = 0; i < vertices.size();i++){ int size = 0; item = (Graphable) vertices.get(i); edges = graph.getToVertices(item); size = edges.size(); if(size > biggestSize){ biggestSize = size; biggest = item; } } return biggest; QueueInterface getToVertices(Graphable vertex){ QueueInterface adjVertices = new LinkedQueue(); int fromIndex; int toIndex; fromIndex = indexIs(vertex); for (toIndex = 0; toIndex < numVertices; toIndex++) if (edges[fromIndex][toIndex] != NULL_EDGE) adjVertices.enqueue(vertices[to Index]); return adjVertices; } Comment: Part of WeightedGraph.java Dale et al.[3] Comment: The vertices are retrieved from the graph using the getToVertices method belonging to the graph object. This method is described in the right side of this box. 28 6.6 Java library classes I used several classes available in the Java library to get several jobs done. In this section I list these classes and I explain in what way I used them. java.io.BufferedReader java.io.InputStreamReader java.io.IOException java.io.PrintWriter java.util.ArrayList java.util.Collections java.util.StringTokenizer java.net.URL java.net.URLConnection java.sql.Connection java.sql.Statement java.sql.ResultSet java.sql.SQLException javax.servlet.http.HttpServlet javax.servlet.http.HttpServletR equest javax.servlet.http.HttpServletR esponse I use the BufferedReader class to retrieve data from the text files that store the indices. A BufferedReader object has a method named readLine to read from a text file line by line. A BufferedReader object cannot read text. It needs an InputStreamReader object to bridge from a byte streams to character stream. An IOException object is created if anything goes wrong in the process of reading text files. This class prints formatted representations of objects to a text-output stream. When a servlet (Java web class) is called in a web browser a request and response object is send to that servlet. The response object has a method named getWriter which delivers a PrintWriter object to the servlet. In turn this object is necessary to print output to the web browser. I have extensively used ArrayLists when passing data from object to object. The advantage of the use of an ArrayList is that it can store any kind of object. Another advantage is that an ArrayList dynamically allocates space, i.e. it increases in size if it becomes full and decreases in size if objects are deleted from it. I use the static sort method from the Collections class to sort the idnumbers associated with a Graphable object. The sorting algorithm is a modified version of mergesort. A StringTokenizer cuts the lines read by the BufferedReader in to single tokens. I use the URL class to define where the search application can find the indices. An URLConnection object is returned when an URL object calls its method openConnection. To access the MySQL database there needs to be a connection to it. A Connection object establishes such a connection. An SQL statement is stored in a Statement object. SQL results are stored in a ResultSet object. An SQLException object is created if anything goes wrong in the process of executing a SQL statement. The WebFrontend class I wrote extends the HttpServlet class. By extending this class the WebFrontend class inherits all the methods from HttpServlet. This way I profit from the methods already written in HttpServlet. An HttpServletRequest object is created when an URL is directed at the WebFrontend class. I use this object to get the information send in the header of the WebFrontend class. For instance. I use the header like so: “web/Lastgeld?search_text=cornelis+vos” (part of the url of a search request). An httpServletResponse object is necessary to retrieve a PrintWriter object. Recall that a PrintWriter object is necessary to print output to the web browser. Table 6.1: Used Java Library classes - Java 1.4.2 API specification [13] 29 6.7 Web deployment 6.7.1 Specification Host Relational database Webserver Java Database Connectivity (JDBC) driver Connection pool Java Runtime environment Specification A slice of an 2.4 GHz processor and 512MB of RAM running on Linux 2.6.26 xen VPS MySQL 5.1 Community Server [18] Apache Tomcat 5.5 [16] Connector/J 5.1 [19] A Java class for pre-allocating, recycling, and managing JDBC connections [17] JRE 6 [20] Table 6.2: web deployment specification 6.7.2 User interface The user interface I created looks like this: Figure 6.3: user interface search application As you can see there is nothing difficult to this interface. The only function that needs a short introduction is the “verbose” option. The verbose option executes the search request while in the mean time printing the results of some important intermediate steps that lead to a final query answer. (Warning this option will flood your screen in some cases). 6.7.3 Web address I host the search application at the following web address: http://daniel.adixhosting.nl/web/lastgeld 6.7.4 The search application doesn‟t work A lot can go wrong when one runs a not-so-thoroughly-tested database-driven Java web application on an external computer with very modest resources. Therefore I post my e-mail address: [email protected] I will gladly try to solve the problem at hand. Having expressed this, I end the chapter. 30 7. Epilogue 7.1 Conclusions I presented a search application that enables keyword-based search in the available Lastgeld database. I employed several techniques to be able to retrieve meaningful answers to queries consisting of multiple keywords. These answers are based on the following assumption: if the keywords in a keyword query are related in the relational database that contains the Lastgeld data, then retrieving this relating data yields results likely to be meaningful given the keyword query. I state that my objective in this thesis is to show that keyword-based search in a relational database can yield meaningful results given the available Lastgeld data. I have presented several keyword queries and the results retrieved by the presented search application. These results show that although the control and exactness of SQL queries is lost, the results are likely to be meaningful given the available Lastgeld data. In the experiment I observed that the presented search application can be demanding in some cases. Based on the experiments I conclude that keyword query execution time increases if the intersection work that needs to be performed by the presented search application increases. This work is dependent on the number of keywords in the query and the amount of id-numbers associated with these keywords in the indices. I also conclude that the more SQL queries the search application proposes to be executed by the relational database, the more the overall query execution time will increase. 7.2 Discussion The presented search application demonstrates one of many possible solutions for obtaining keyword-based search in a relational database. However, the presented solution is mainly designed for the data available. I used only two tables to show how relations can be combined in a graph data structure. In the chapter five I describe related work it is clear that the graph data structure is employed in very different ways. It appears to be more common to designate an entire tuple to be a vertex, instead of designating a field in the tuple to be a vertex the way I did. This way edges between vertices represent relations between tables. Finding relating data is on the scale of a graph model of relating tables not on the scale of a graph model of relating data within tuples the way I have used a graph data structure. My approach seems far more difficult to scale incorporating many tables, while solutions based on an entire tuple being a vertex connecting to other vertices that are tuples in other tables, appear to be a more generalizable approach in modeling an amount of data that is stored in an undefined number of tables. I designed the presented search application to work with the available data. This data doesn‟t have a rich relational structure by means of many relating tables. The experiment indicates that the presented search application can be increasingly demanding in terms of query processing. The way id-lists are intersected to establish edges between vertices in the graph data structure can be an execution time consuming construct in some cases. Also proposing multiple queries to the relational database after the search application finished processing can be an execution time consuming construct. Although efficiency can be improved, it is unclear of these inherent demanding constructs are acceptable in practice. By trading query syntax like SQL for just some keywords, control over retrieving exact answers is lost. The spaces between keywords do not per se imply a relation between keywords, but they do not imply there isn‟t any relation either. I chose find answers to multiple keywords in a query by incorporating all the keywords as a relating phrase based on the available data. This yields meaningful results, however this remains relative to the interpretation of the meaning of spaces between keywords. 31 7.3 Proposals for future work Efficiency can be improved by revising the index structure. Currently, all indices are an alphabetical list per database field. The index first name alphabetically lists all first names from A to Z. For every letter in the alphabet, this index can be partitioned in sub-indices. Then the appropriate indices can be retrieved given the first letters of the keywords. This way the sequential search process is less demanding in terms of processing. To avoid the execution of 5600 SQL queries a keyword query “Germany” initiates, it is possible to generate just one query containing multiple AND clauses, like for instance: SELECT * FROM Skippers WHERE id=‟1‟ AND id=‟2‟ AND id=‟3‟. Still, in case of the keyword query “Germany” this approach will yield a very long query. Another solution to this problem is to take alternative steps for keyword queries that contain only one keyword, because the search application will likely propose many SQL-queries in case of one keyword. In this case a possible scenario is to avoid the search application entirely, and search the indices separately for this keyword. If it is known that the keyword occurs in index A, then a more appropriate SQL-query can be formulated given this knowledge. The retrieval of meaningful results may be improved by traversing the employed weighted graph data structure. Currently the vertex with the most edges is retrieved. This approach retrieves the tuples of data in which the keywords must occur in the same tuple. If the graph were to be traversed starting at different vertices, then different kinds of relating data can be retrieved. This way and, or and not semantics can be introduced into the application. Also there could be made an order of importance between vertices that have just one edge, vertices that have two edges, etc. However, to be able to evaluate the benefit of any kind of functional variation within the presented application, it is necessary to assess the effectiveness of the application based on actual information needs as well. References [1] B. Aditya, G. Bhalotia, S. Chakrabarti, A. Hulgeri, C. Nakhe, P. Parag, S.Sudarshan. BANKS: browsing and keyword searching in relational databases. In Proceedings of the 28 th international conference on Very Large Databases, 2002. http://delivery.acm.org.proxy-ub.rug.nl/10.1145/1290000/1287473/p1083aditya.pdf?key1=1287473&key2=0346186421&coll=ACM&dl=ACM&CFID=42949402&CFTOKEN=9 7092769, visited on may 14th, 2009. [2] S.Cohen, J. Mamou. Y. Kanza, Y. Sagiv. XSearch: a semantic search engine for XML. In Proceedings of the 29th international conference on Very large databases, vol. 29, 2003. http://delivery.acm.org.proxyub.rug.nl/10.1145/1320000/1315457/p45cohen.pdf?key1=1315457&key2=8557427421&coll=ACM&dl=ACM&CFID=43812128&CFTOKEN=5 1392105, visited on June 28th, 2009. [3] N.Dale, D.T Joyce, C.Weems. Object-oriented data structures using Java. ISBN 0-7637-1079-2. Jones and Bartlett Publishers International, London, 2002. [4] B.B. Dalvi, M. Kshirsagar, S. Sudarshan. Keyword search on external memory graphs. In Proceedings of the VLDB Endowment, Vol 1, Issue 1, 2008. http://delivery.acm.org.proxyub.rug.nl/10.1145/1460000/1453982/p1189dalvi.pdf?key1=1453982&key2=2831696421&coll=ACM&dl=ACM&CFID=42949402&CFTOKEN=97 092769, visited on July 3th, 2009. [5] D. Floresqu, D. Kossmann, I. Manolescu. Integrating keyword search into XML query processing. In Computer Networks, Vol. 33, 2000. http://www.sciencedirect.com.proxyub.rug.nl/science?_ob=MImg&_imagekey=B6VRG-40B2JGR-C11&_cdi=6234&_user=4385132&_orig=search&_coverDate=06%2F30%2F2000&_sk=999669998&vie w=c&wchp=dGLbVlW-zSkWA&md5=c122eafe4481c853b1287351178b8472&ie=/sdarticle.pdf, visited on May 17th, 2009. 32 [6] V. Hristidis, Y Papakonstantinou. Discover: keyword search in relational databases. In Proceedings of the 28th international conference on Very Large Databases, 2002. http://delivery.acm.org.proxyub.rug.nl/10.1145/1290000/1287427/p670hristidis.pdf?key1=1287427&key2=0686186421&coll=ACM&dl=ACM&CFID=42949402&CFTOKEN =97092769, visited on May 15th, 2009. [7] V. Hristidis, L. Gravano, Y, Papakonstantinou. Efficient IR-Style Keyword search over Relational databases. In Proceedings of the 29th international conference on Very large databases, Vol. 29, 2003. http://portal.acm.org.proxyub.rug.nl/citation.cfm?id=1453856.1453887&coll=ACM&dl=ACM&CFID=43719809&CFTOKEN=297 07051, visited on May 15th, 2009. [8] L.Lanzani. Discrete mathematics, chapter 11, Graph theory, 2008, http://comp.uark.edu/~lanzani/2103NOTES/11.1-11.2.pdf, visited on July 6th, 2008. [9] G. Li, B.C. Ooi, J. Feng, J. Wang, L, Zhou. EASE: an effective 3-in-1 keyword search method for unstructured, semi-structured and structured data. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data. http://delivery.acm.org.proxyub.rug.nl/10.1145/1380000/1376706/p903li.pdf?key1=1376706&key2=0314196421&coll=ACM&dl=ACM&CFID=42949402&CFTOKEN=97092 769, visited on May 21th, 2009. [10] F.Liu, C. Yu, W. Meng, A Chowdhury. Effective Keyword Search in Relational Databases. In Proceedings of the 2007 ACM SIGMOD international conference on Management of data. http://delivery.acm.org.proxy-ub.rug.nl/10.1145/1150000/1142536/p563liu.pdf?key1=1142536&key2=1057366421&coll=ACM&dl=ACM&CFID=43719809&CFTOKEN=2970 7051, visited on May 21th, 2009. [11] C.D Manning, P.Raghavan, H.Schütze. Introduction to information retrieval. ISBN 978-0-521-86571-5. Cambridge university press, Newyork, 2008. [12] L.Paoletti. Leonard Euler‟s solution to the Königsberg bridge problem, http://mathdl.maa.org/mathDL/46/?pa=content&sa=viewDocument&nodeId=1310&bodyId=1452, visited on July 6th, 2009. [13] Sun.com. Java 2 Platform, Standard Edition, v 1.4.2 API specification. htp://java.sun.com/j2se/1.4.2/docs/api/overview-summary.html, visited on July 18th [14] Sun.com. JDK Source 5.0. https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_DeveloperSite/en_US/-/USD/ViewProductDetail-Start?ProductRef=J2SE-1.5.0-OTH-G-F@CDSCDS_Developer&ProductUUID=.59IBe.oWd4AAAEZZvkZK4O9&ProductID=.59IBe.oWd4AAAEZZv kZK4O9&Origin=ViewProductDetail-Start , (SDN registration required) [15] G.M. Welling. The Prize of Neutrality. Trade relations between Amsterdam and North America 17711817, 1998. http://dissertations.ub.rug.nl/FILES/faculties/arts/1998/g.m.welling/thesis.pdf, visited on Augustus 11th, 2009. Software used in the implementation as described in section 5.2 and 5.7.1. [16] Apache Tomcat 6.0. http://tomcat.apache.org/. (open source) [17] ConnectionPool.java http://archive.coreservlets.com/coreservlets/ConnectionPool.java (freely available) [18] MySQl Community server 5.0. http://dev.mysql.com/downloads/. (open source) [19] MySQL Connector/J 5.1. http://dev.mysql.com/downloads/connector/j/5.1.html (open source) [20] Sun. Java Runtime Environment 6. http://java.sun.com/javase/downloads/index.jsp (freely available) 33 Appendix A: creating indices in Perl # # # # Step 1 Creating indices Name: step1.pl Author: Daniel Suelmann Effect: Deletes punctuation of a SQL-dump. use strict; my $readfile = shift(@ARGV); chomp($readfile); open(FILE, $readfile) or die "Cannot open $readfile: $!"; while (<FILE>) { my $line = $_; chomp($line); $line =~ s/[[:punct:]]//g; open(RESULT, ">>inputtostep2.txt") or die "Cannot open $!"; print RESULT "$line \n"; } close RESULT; ---Result: 1198 AART 4765 ABE 12726 ABE 4790 ABE 1567 ABE >> # # # # Step 2 Creating indices Name: step2.pl Author Daniel Suelmann Effect: Puts the id-numbers after the index term. use strict; my $numbers; my $readfile = shift(@ARGV); chomp($readfile); open(FILE, $readfile) or die "Cannot open $!"; while (<FILE>) { my $line = $_; chomp($line); $line =~ s/([0-9]+)//g; $numbers = $1; open(RESULT, ">>inputtostep3.txt") or die "Cannot open: $!"; print RESULT "$line $numbers\n"; } close RESULT; ---Result: AART 1198 ABE 4765 ABE 12726 ABE 4790 ABE 1567 >> # Step 3 Creating indices 34 # Name: step3.pl # Author Daniel Suelmann # Effect: Puts every similar index term on the same line. use strict; my $prevline = ""; my $string = ""; my $numbers; my $readfile = shift(@ARGV); chomp($readfile); open(FILE, $readfile) or die "Cannot open $readfile: $!"; while (<FILE>) { my $line = $_; chomp($line); $line =~ /([A-Z]+)/; if ($prevline eq $1){ open(RESULT, ">>inputfinalstep.txt") or die "Cannot open $!"; print RESULT "$line"; } else { open(RESULT, ">>inputfinalstep.txt") or die "Cannot open $!"; print RESULT "\n$line"; } $line =~ /([A-Z]+)/; $prevline = $1; } close RESULT; ---Result: AART 1198 <new line> ABE 4765 ABE 12726 ABE 4790 ABE 1567 ABE 1246 ABE BR 6900 ABE BR 12471 ABE BR 11380 ABE BR 857 ABE BROER 5986 ABE BROERSZ 9008 <new line> ABEL 6545 ABEL BAS 3553 <new line> ABLBERT 7581 ABLBERT J 10065 <new line> ABRAHAM 3410 ABRAHAM 4434 ABRAHAM 6303 ABRAHAM 5806 ABRAHAM 9355 ABRAHAM 9064 ABRAHAM 6687 ABRAHAM 8444 ABRAHAM 9828 <new line> # # # # Step 4 Creating indices. Name: step4.pl Author Daniel Suelmann Effect: Deletes all the terms except the term that starts the line. use strict; my $string; my $readfile = shift(@ARGV); chomp($readfile); open(FILE, $readfile) or die "Cannot open $readfile: $!"; while (<FILE>) { my $line = $_; chomp($line); $line =~ /([A-Z ]+)/; $string = $1; $line =~ s/([A-Z ]+)/ /g; open(RESULT, ">>resultsfinal.txt") or die "Cannot open $!"; print RESULT $string; print RESULT $line . "\n"; } close RESULT; ---Result: AART 1198 35 ABE 4765 12726 4790 1567 1246 6900 12471 11380 857 5986 9008 ABEL 6545 3553 ABLBERT 7581 10065 ABRAHAM 3410 4434 6303 5806 9355 9064 6687 8444 9828 Appendix B: search application source code in Java Note that I do not try to catch exceptions. Each search request restarts the application. If an exception occurs, the exception occurs for one search request only. The application throws the exception out to the web server, which in turn prints the exception in readable and understandable format in the web browser of the user. /* * /* * * Name: Database.java Author: Daniël Suelmann Effect: * main scenario: * 1. Put the keywords in a list; * 2. See whether it matches the table fields; * 3. Receive the matches; * 3.1 if the matches are form a single * keyword, the job is done. * 3.2 if not, continue. * 4. Put the matches in a graph. * 5. Add edges -between vertices- to the * graph-based on id resemblance. * 6. Find out which vertex has the most edges: * this is probably the most important one. * 7. Intersect the id-numbers associated with * each edge, the id-numbers are the result * id-numbers, these id-numbers are given * back to the servlet that called this method * from within the web environment. * */ package standalone; import java.io.IOException; import java.util.ArrayList; public class Database { private Grapher grapher = new Grapher(); public static ArrayList<ArrayList<Graphable>> dispatch(String[] array) throws IOException{ /* effect: creates an instance of Lastgeld * for each keyword in the arguments array. * effect: dispatches the keywords in requests * to the text files. * incoming collaboration: receives an array * of keywords from Database's main. * outgoing collaboration: sends a single * keyword to Lastgeld's request. * incoming collaboration: receives a list * with object references from Lasgeld's request. * outgoing collaboration: sends a list of lists * with object references to Database's main. */ ArrayList<ArrayList<Graphable>> list = new ArrayList<ArrayList<Graphable>>(); for(int i = 0; i < array.length; i++){ Lastgeld lg = new Lastgeld(); 36 list.add(lg.request(array[i])); } return list; } public ArrayList<Integer> getResult(String[] str) throws IOException { ArrayList <ArrayList<Graphable>> keywords; ArrayList<Integer> result = new ArrayList<Integer>(); String[] input = str; keywords = dispatch(input); /*If there's only one keyword entered there's no *need to set up a graph. *The triple loop gets lists in the list (loop 1), *these lists contain objects (loop 2), these *objects have (first name, last name, harbor, *country) and the last loop gets id-numbers from *within the objects. * */ if (input.length == 1){ if(keywords != null){ for (int i = 0; i < keywords.size(); i++){ ArrayList<Graphable> list = (ArrayList<Graphable>) keywords.get(i); if(list != null){ for(int j = 0; j < list.size(); j++){ Graphable item = (Graphable) list.get(j); if(item != null){ ArrayList<Integer> ids = (ArrayList<Integer>)item.getIDs(); if (ids !=null){ for(int k = 0; k < ids.size(); k++){ result.add(ids.get(k)); } } else{ System.out.println("No, id-numbers in the list"); } } } } else{ System.out.println("emptylist"); } } } else { System.out.println("There are no objects to start with"); } } 37 /* If there are more than one keyword * entered; a graph is created; * The graph is populated by inserting * the keyword objects(vertices) into * the graph. The edges in between two * vertices are created; The edges are * printed (for my administration * output is returned to the Tomcat server * console); And finally, the result set * is determined. */ else { if (!keywords.isEmpty()){ grapher.populate(keywords); grapher.addEdges(); grapher.printEdgesToConsole(); result = (ArrayList<Integer>) grapher.findResult(); } } return result; } public ArrayList <Graphable> getVertices(){ ArrayList <Graphable> list = grapher.getVertices(); return list; } public ArrayList <Graphable> getIntersections(){ ArrayList <Graphable> list = grapher.getIntersections(); return list; } public WeightedGraph getGraph(){ WeightedGraph graph = grapher.getGraph(); return graph; } public ArrayList<Object> getCombinedIntersections(){ ArrayList<Object> list = grapher.getCombinedIntersections(); return list; } /* Comment on the reset methods: * In between search requests all objects must * be deleted otherwise data structures are * flooded with mixed result data. */ public void resetCombinedIntersections(){ grapher.resetCombinedIntersections(); } public void resetIntersections(){ grapher.resetIntersections(); } public void resetGraph(){ grapher.resetGraph(); } public void resetVertices(){ grapher.resetVertices(); 38 } public void resetListOfLists(){ grapher.resetListOfLists(); } } /* Name: Lastgeld.java * Author: Daniël Suelmann * Effect: * Every Lastgeld object instantiates five * Connector objects, one for every index * stored on disk. The Connector objects return * lists with matching id-numbers. Per match * an inner class object is instantiated * depending on which index matched the * keyword. All object references are gathered * and send back to Database. */ package standalone; import java.io.IOException; import java.util.ArrayList; public class Lastgeld { //inner class protected class Firstname implements Graphable{ protected String content; protected final String type = "lastgeldfirstname"; protected ArrayList<Integer> ids; public String toString(){ return content; } public ArrayList<Integer> getIDs(){ return ids; } public String getType(){ return this.type; } public boolean isEqual(Graphable object){ if (object.getType().equals(this.type) && object.toString().equals(this.content)){ return true; } else return false; } } //inner class protected class Lastname implements Graphable{ protected String content; protected final String type = "lastgeldlastname"; protected ArrayList<Integer> ids; public String toString(){ return content; } public ArrayList<Integer> getIDs(){ return ids; } 39 public String getType() { return this.type; } public boolean isEqual(Graphable object){ if (object.getType().equals(this.type) && object.toString().equals(this.content)) return true; else return false; } } //inner class protected class Harbor implements Graphable{ protected String content; protected final String type = "lastgeldharbor"; protected ArrayList<Integer> ids; public String toString(){ return content; } public ArrayList<Integer> getIDs(){ return ids; } public String getType() { return this.type; } public boolean isEqual(Graphable object){ if (object.getType().equals(this.type) && object.toString().equals(this.content)) return true; else return false; } } //inner class protected class Date implements Graphable{ protected String content; protected final String type = "lastgelddate"; protected ArrayList<Integer> ids; public String toString(){ return content; } public ArrayList<Integer> getIDs(){ return ids; } public String getType() { return this.type; } public boolean isEqual(Graphable object){ if (object.getType().equals(this.type) && object.toString().equals(this.content)) return true; else return false; } } //inner class 40 protected class Country implements Graphable{ protected String content; protected final String type = "lastgeldcountry"; protected ArrayList<Integer> ids; public String toString(){ return content; } public ArrayList<Integer> getIDs(){ return ids; } public String getType() { return this.type; } public boolean isEqual(Graphable object){ if (object.getType().equals(this.type) && object.toString().equals(this.content)) return true; else return false; } } protected protected protected protected protected static static static static static Firstname firstname; Lastname lastname; Harbor harbor; Date date; Country country; // constructor of the table public Lastgeld(){ /* effect: Instantiates Lastgeld's inner * objects Firstname, Lastname, Date, * Harbor, Country and points references * to them. */ firstname = new Firstname(); lastname = new Lastname(); harbor = new Harbor(); date = new Date(); country = new Country(); } private static ArrayList<Graphable> populate(ArrayList<ArrayList<Integer>> lists, String keyword){ /* effect: attaches id-numbers to the instance * variables of Lastgeld's inner classes. * effect: attaches the keyword to the instance * variables of Lastgeld's inner classes. * incoming collaboration: receives a keyword * from Lastgeld's request. * incoming collaboration: receives a list of id * lists from Lastgeld's request. * outgoing collaboration: sends a list with * object references to the fields to Lastgeld's * request. */ ArrayList<Graphable> objectrefs = new ArrayList<Graphable>(); ArrayList<Integer> list0 = (ArrayList<Integer>) lists.get(0); ArrayList<Integer> list1 = (ArrayList<Integer>) lists.get(1); ArrayList<Integer> list2 = (ArrayList<Integer>) lists.get(2); ArrayList<Integer> list3 = (ArrayList<Integer>) lists.get(3); 41 ArrayList<Integer> list4 = (ArrayList<Integer>) lists.get(4); if(!list0.isEmpty()){ firstname.content = keyword; firstname.ids = (ArrayList<Integer>) lists.get(0); objectrefs.add(firstname); } if(!list1.isEmpty()){ lastname.content = keyword; lastname.ids = (ArrayList<Integer>) lists.get(1); objectrefs.add(lastname); } if(!list2.isEmpty()){ harbor.content = keyword; harbor.ids = (ArrayList<Integer>) lists.get(2); objectrefs.add(harbor); } if(!list3.isEmpty()){ date.content = keyword; date.ids = (ArrayList<Integer>) lists.get(3); objectrefs.add(date); } if(!list4.isEmpty()){ country.content = keyword; country.ids = (ArrayList<Integer>) lists.get(4); objectrefs.add(country); } return objectrefs; } public static ArrayList<Graphable> request(String s) throws IOException{ /* effect dispatches calls to the connector to * access the text files. * incoming collaboration: receives a keyword * form Database's dispatch. * incoming collaboration: receives lists of * id-numbers (associated with a single keyword) * from Connectors read method. * outgoing collaboration: sends a list of id-lists * to Lastgeld's populate. * outgoing collaboration: sends the list with * object references to Database's dispatch */ ArrayList<ArrayList<Integer>> lists = new ArrayList<ArrayList<Integer>>(); ArrayList<Integer> ArrayList<Integer> ArrayList<Integer> ArrayList<Integer> ArrayList<Integer> list0 list1 list2 list3 list4 = = = = = null; null; null; null; null; String keyword = s; Connector con0 = new Connector("vnfreq.txt"); list0 = (ArrayList<Integer>) con0.read(s); Connector con1 = new Connector("anfreq.txt"); list1 = (ArrayList<Integer>) con1.read(s); Connector con2 = new Connector("havenfreq.txt"); list2 = (ArrayList<Integer>) con2.read(s); Connector con3 = new Connector("datefreq.txt"); 42 list3 = (ArrayList<Integer>) con3.read(s); Connector con4 = new Connector("countryfreq.txt"); list4 = (ArrayList<Integer>) con4.read(s); lists.add(list0); lists.add(list1); lists.add(list2); lists.add(list3); lists.add(list4); return populate(lists, keyword); } } /* Name: Connector.java * Author: Daniël Suelmann * Effect: * Every Lastgeld object instantiates five Connector objects, * one for every index stored on disk, e.g. if a keyword * query of three keywords is entered, three Lastgeld objects * are created and thus in total 15 Connector objects are created, * each Connector object is associated with one keyword and * sequentially searches one index. */ package standalone; import import import import import import import import java.net.URL; java.net.URLConnection; java.io.BufferedReader; java.io.IOException; java.io.InputStreamReader; java.util.StringTokenizer; java.util.ArrayList; java.util.Collections; public class Connector { String filename; String keyword; public Connector(String fn){ this.filename = fn; } public ArrayList<Integer> read(String keyword) throws IOException{ /* effect: Opens up a file a checks line by line * if there's a match with the keyword. If there * is a match the id-numbers associated with that * match are stored in a list. This results in * a list of id-numbers of matches, accumulated * over all the lines in the files. * incoming collaboration: receives a file name * from Connector's constructor. * incoming collaboration: receives a single keyword * from Lastgeld's request. * outgoing collaboration: sends a list of id-numbers * to Lastgeld's request. * */ URL url = new URL("http://localhost:8080/web/" + filename); URLConnection urlConnection = url.openConnection(); urlConnection.connect(); 43 String inLine = null; String word; StringTokenizer tokenizer; ArrayList<Integer>list = new ArrayList<Integer>();; BufferedReader inFile; inFile = new BufferedReader(new InputStreamReader(url.openStream())); inLine = inFile.readLine(); while (inLine != null){ tokenizer = new StringTokenizer(inLine, " "); word = tokenizer.nextToken(); word = word.toLowerCase(); if (word.equals(keyword)){ while(tokenizer.hasMoreTokens()){ String token = tokenizer.nextToken(); token = token.replaceAll("\\D", ""); if (!token.equals("")){ Integer id = Integer.valueOf(token); list.add(id); } } } inLine = inFile.readLine(); } Collections.sort(list); return list; } } /* Name: Grapher.java * Author: Daniël Suelmann /* Effect: * The effect of this class is explained * in detail at the level of * each method. */ package standalone; import java.util.ArrayList; public class Grapher { private private private private private static static static static static ArrayList<ArrayList<Graphable>> list; WeightedGraph graph = new WeightedGraph(); ArrayList <Graphable> vertices; ArrayList <Graphable> intersections = null; ArrayList<Object> combinedintersections = null; public void populate(ArrayList<ArrayList<Graphable>> keywords){ /* effect: extracts the references in each list and adds * them to the graph. * incoming collaboration: receives a list of lists from * Database's main. * outgoing collaboration: sends object references to * WeightedGraph's addVertex. */ list = keywords; for (int i = 0; i < list.size(); i++){ ArrayList<Graphable> l = (ArrayList<Graphable>)list.get(i); for(int j = 0; j < l.size(); j++){ graph.addVertex((Graphable) l.get(j)); } } } 44 public void addEdges(){ /* effect: Based on the vertices currently present, * it initiates the process of finding and adding * the edges to the graph. * incoming collaboration: is called by Database's * getResults. * incoming collaboration: receives a list with match * descriptives */ findEdges(); } public static void findEdges(){ /* effect: compare the vertices based on all possible * combinations. * explanation: if there are N vertices compare the * id-numbers of the vertices in all possible * combinations. For instance: if there are three * vertices made based on keyword hits, then combinations * 1 2, 2 3 and 1 3 are possibilities. These combinations * are used as indices for the vertex list, 1 2 turns * into 0 1 etc. Then the ArrayLists of id-numbers * associated with an vertex object -based on the * combinations- is intersected to sort out all vertices * with identical id-numbers. (The intersection is done by * the ArrayList method retainAll. If identical id-numbers * are found, the ArrayList which contains matches is added * an ArrayList, so this would be a list of possible one or * more lists. It returns this list to Grapher's addEdges. * incoming collaboration: works with an ArrayList of vertices * (Grapher instance variable) from WeightedGraph's retrieves. * outgoing collaboration: passes the number of vertices to * Grapher's combinationCalculator. * incoming collaboration: receives an Two-dimensional array * with all possible combinations given the vertices as ints. * outgoing collaboration: sends a list -possible empty or * may contain one or more lists- to Grapher's addEdges */ int[][]combinations; vertices = graph.retrieve(); combinations = calculateCombinations(vertices.size()); ArrayList<Integer> idsobject0; ArrayList<Integer> idsobject1; if (intersections == null){ intersections = new ArrayList(); } if (combinedintersections == null){ combinedintersections = new ArrayList(); } for (int i = 0; i < combinations.length; i++){ if (!(combinations[i][0] == 0 && combinations[i][1] == 0)){ Graphable object0 = (Graphable) vertices.get(combinations[i][0]); Graphable object1 = (Graphable) vertices.get(combinations[i][1]); intersections.add(object0); intersections.add(object1); idsobject0 = object0.getIDs(); idsobject1 = object1.getIDs(); ArrayList<Integer> clone; clone = (ArrayList<Integer>)idsobject0.clone(); 45 clone.retainAll(idsobject1); if(!clone.isEmpty()){ if(!object0.toString().equals(object1.toString())){ graph.addEdge(object0, object1, clone); } combinedintersections.add(object0.getType()); combinedintersections.add(object0.toString()); combinedintersections.add(object1.getType()); combinedintersections.add(object1.toString()); combinedintersections.add(clone); } } } } private static int[][] calculateCombinations(int c){ /*effect finds combinations based on a fixed *value (parameter c); for instance if the *int 3 is passed it finds out that 3 consists *of combinations 12 23 and 13. To serve Grapher's *findEdges it changes these combinations in array *index format, 12 turn into 01, 13 turns into 02 etc. *outgoing collaboration: sends a two-dimensional *array with all possible combinations given the *vertices as ints to findEdges. */ int[][] combinations = new int[c*c/2][2]; int store = 0; int number = c; for (int b = 1; b < number; b++ ){ int y = b - 1; int i = 1; while (i != number-y){ combinations[store][0] = i-1; i = i + b; combinations[store][1] = i-1; int x = b - 1; i = i - x ; store++; } } return combinations; } private static void printCombinationsToConsole(ArrayList<Object> l){ /* effect: prints some additional information about * the combinations checked and their matches found. * This will not be visible to any user, printed to * the webserver's console. * incoming collaboration: receives an ArrayList with * data from grapher's addEdges. * outgoing collaboration: prints all findings to the * server console. */ int count = 0; int total = 1; for (int i = 0; i < l.size(); i++){ if (count == 5){ System.out.print("\n " + total); count = 0; total++; } System.out.print(" " + l.get(i)); count++; 46 } System.out.print("\n"); } public void printEdgesToConsole(){ /* effect: prints some additional information about the * edges present in the graph. This will not be visible * to any user, printed to the webserver's console. * incoming collaboration: uses Grapher's ArrayList * instance variable vertices to get to the vertices * present in the graph. * incoming collaboration: receives -for each vertex in * the graph- a LinkedQueue of vertices that are adjacent. * outgoing collaboration: prints all findings to the * server console. */ for (int i = 0; i < vertices.size(); i++){ Graphable object; object = (Graphable) vertices.get(i); System.out.print(object.getType()+ ", "); //type System.out.print(object.toString() + ", "); //content //System.out.println(object.getIDs()+ "\n"); //ID's QueueInterface queue; queue = graph.getToVertices(object); if (!queue.isEmpty()){ System.out.print("has edges: "); int count = 1; while(!queue.isEmpty()){ Graphable item = (Graphable) queue.dequeue(); System.out.print(" " + count++ + ": " + item.getType()); System.out.print(", " + item); } System.out.print("\n"); } else { System.out.println("no edges."); } } } private Graphable findStart(){ /* effect: finds the vertex with the most edges. * explanation: the vertex with the most edges is * the vertex with the strongest relation to the other * keywords, i.e. this is the vertex that's 'likely' relevant. * incoming collaboration: receives a queue with edges from * WeightedGraphs getToVertices. * outgoing collaboration: send the vertex with the most * edges/vertices to Grapher's findResult */ QueueInterface edges; Graphable item; Graphable biggest = null; int biggestSize = 0; for(int i = 0; i < vertices.size(); i++){ int size = 0; item = (Graphable) vertices.get(i); edges = graph.getToVertices(item); size = edges.size(); if(size > biggestSize){ biggestSize = size; biggest = item; } } return biggest; 47 } public ArrayList<Integer> findResult(){ /* effect: dispatches two functions: findstart * returns the vertex with the most vertices. * connectEdges perform the final intersection of * the id-numbers associated with the edges. * the result is a list of final result id-numbers. * incoming collaboration: is called by Database's * getResults. * incoming collaboration: receives an ArrayList with * result id-numbers from Grapher's connect edges. * outgoing collaboration: sends back the ArrayList with * result id-numbers to Database's getResult. */ ArrayList <Integer> result; Graphable startPoint = findStart(); result = connectEdges(startPoint); return result; } private static ArrayList<Integer> connectEdges(Graphable startVertex){ /* effect: * intersects all edges of the given argument * startVertex, which is the vertex with the * most edges to other vertices. * explanation: up until now there is a graph * that holds x vertices that are connected in * some way. What is done here, is intersecting * all the id-numbers that are associated with * all the edges of a particular vertex. * What makes a relation between certain vertices * is the fact that an edge between vertex a & b * holds the same or a subset of the id-numbers * between vertices a & c or a & d. * incoming collaboration: receives a Graphable * object that becomes the start of the traversal. * incoming collaboration: receives a LinkedQueue * with all the edges of startVertex (argument) * by WeightedGraph's getToVertices. * incoming collaboration: receives an ArrayList * with the id-numbers associated with an edge * between the startVertex and a vertex connected * to the startVertex. * outgoing collaboration: sends an ArrayList of * ints to Grapher's findResult. */ QueueInterface edges = graph.getToVertices(startVertex); ArrayList<ArrayList<Integer>> lists = new ArrayList<ArrayList<Integer>>(); ArrayList<Integer>sim; Graphable item; while(!edges.isEmpty()){ item = (Graphable) edges.dequeue(); sim = graph.holdSimilarities(startVertex, item); if (!sim.isEmpty()){ lists.add(sim); } } ArrayList<Integer> result = null; if (!lists.isEmpty()){ result = (ArrayList<Integer>) lists.get(0); for(int i = 1; i < lists.size();i++){ 48 result.retainAll((ArrayList<Integer>) lists.get(i)); } return result; } else { ArrayList <Integer> empty = return empty; new ArrayList<Integer>(); } } public ArrayList <Graphable> getVertices(){ return vertices; } public WeightedGraph getGraph(){ return graph; } public ArrayList<Graphable> getIntersections(){ return intersections; } public ArrayList<Object> getCombinedIntersections(){ return combinedintersections; } /* * * * The following resetters are necessary due to the fact that objects persist in between search requests of the Webfrontend class. */ public void resetIntersections(){ intersections = null; } public void resetCombinedIntersections(){ combinedintersections = null; } public void resetListOfLists(){ list = null; } public void resetVertices(){ vertices = null; } public void resetGraph(){ graph.reset(); } } 49 /* Name: Graphable.java * Author: Daniël Suelmann /* Effect: This is an interface class. * It this describes abstract methods * that must be implemented by all classes * implementing this interface. * This objects are instances of Lastgeld's * inner classes; Firstname,Lastname, Harbor, * Country and Date. * */ package standalone; import java.util.ArrayList; public interface Graphable { public abstract String toString(); //effect: implementation prints the content variable of a graphable object. public abstract ArrayList<Integer> getIDs(); //effect: implementation gets the id-numbers associated with a graphable object. public abstract String getType(); //effect: implementation gets the type of a graphable object. public abstract boolean isEqual(Graphable object); //effect: implementation compares to objects of the graphable type. } /* Name: WeightedGraph.java * This class is a slightly modified version of the * WeightedGraph data structure presented in the book * Object-oriented data structures using Java by * Dale et al.[3] * The modifications: * The values associated with the edges were initially * ints. For the application I need edges that store * multiple ints representing the id-numbers that are * shared by two vertices. Since I use ArrayLists * throughout the application, I also changed the edges * instance variable to be two-dimensional arrays of * the type ArrayList. These modifications can be found * among the instance variables and in the methods * holdSimilarties and addEdge. Also I added two methods * to retrieve information from the graph, these are * Retrieve and printNames. * */ package standalone; import java.util.ArrayList; public class WeightedGraph implements WeightedGraphInterface { public static ArrayList <Integer> NULL_EDGE = null; private int numVertices; private int maxVertices; private Graphable[] vertices; private ArrayList<Integer>[][] edges; private boolean[] marks; // marks[i] is mark for vertices[i] 50 public WeightedGraph() // Post: Arrays of size 50 are dynamically allocated for // marks and vertices, and of size 50 X 50 for edges // numVertices is set to 0; maxVertices is set to 50 { numVertices = 0; maxVertices = 50; vertices = new Graphable[50]; marks = new boolean[50]; edges = new ArrayList[50][50]; } /* Comment on the edges * Modification of the original WeightesGraph data structure: * the weights have become ArrayLists of id-numbers * Each edge contains the id-similarities between two vertices * If verticeX = {1,2,3,4,5} and verticeY = {3,4,5,6} the edge * represents the intersection result vertice{X,Y} = {3,4,5}, * these two values would be stores in a ArrayList * this ArrayList is necessary to be able to, in turn, intersect * this intersection with another intersection, * for instance vertice{P,Q) = {3,5,7,9} which would result in * vertice(P,Q,X,Y){3,5}, etc. * */ public void reset(){ vertices = new Graphable[50]; edges = new ArrayList[50][50]; numVertices = 0; } public ArrayList<Graphable> retrieve(){ ArrayList<Graphable> list = new ArrayList<Graphable>(); for (int i = 0; i < vertices.length; i++){ Graphable object = (Graphable) vertices[i]; if (object != null){ list.add(object); } } return list; } public void printNames(){ for (int i = 0; i < vertices.length; i++){ Graphable object = (Graphable) vertices[i]; if (object != null){ System.out.println(object.toString()); } } } public WeightedGraph(int maxV) // Post: Arrays of size maxV are dynamically allocated for // marks and vertices, and of size maxV X maxV for edges // numVertices is set to 0; maxVertices is set to maxV { numVertices = 0; maxVertices = maxV; vertices = new Graphable[maxV]; marks = new boolean[maxV]; edges = new ArrayList[maxV][maxV]; } public void addVertex(Graphable vertex) // Post: vertex has been stored in vertices. // Corresponding row and column of edges has been set to NULL_EDGE. 51 // numVertices has been incremented { vertices[numVertices] = vertex; for (int index = 0; index < numVertices; index++) { edges[numVertices][index] = NULL_EDGE; edges[index][numVertices] = NULL_EDGE; } numVertices++; } private int indexIs(Graphable vertex) // Post: Returns the index of vertex in vertices { int index = 0; while (vertex != vertices[index]) index++; return index; } public void addEdge(Graphable fromVertex, Graphable toVertex, ArrayList<Integer> IDs) // Post: Edge (fromVertex, toVertex) is stored in edges { int row; int column; row = indexIs(fromVertex); column = indexIs(toVertex); edges[row][column] = IDs; } public ArrayList<Integer> holdSimilarities(Graphable fromVertex, Graphable toVertex) // Post: Returns the weight associated with the edge // (fromVertex, toVertex) { int row; int column; row = indexIs(fromVertex); column = indexIs(toVertex); return edges[row][column]; } public QueueInterface getToVertices(Graphable vertex) // Returns a queue of the vertices that are adjacent from vertex. { QueueInterface adjVertices = new LinkedQueue(); int fromIndex; int toIndex; fromIndex = indexIs(vertex); for (toIndex = 0; toIndex < numVertices; toIndex++) if (edges[fromIndex][toIndex] != NULL_EDGE) adjVertices.enqueue(vertices[toIndex]); return adjVertices; } 52 /* * /* * * * * * * * * * Name: WebFrontend.java Author: Daniël Suelmann Effect: 1. Displays a search interface; 2. Receives keyword queries; 3. Instantiates a Database object, that initializes other classes to retrieve an answer to the query. 4. When the answer is retrieved, proposed SQL queries are executed. 5. The answer is returned to the web browser. 6. -Optional- Displays intermediate intersection results */ package scriptie; import import import import import import import import import import import import java.io.IOException; java.io.PrintWriter; java.sql.Connection; java.sql.ResultSet; java.sql.SQLException; java.sql.Statement; java.util.ArrayList; java.util.StringTokenizer; javax.servlet.http.HttpServlet; javax.servlet.http.HttpServletRequest; javax.servlet.http.HttpServletResponse; standalone.*; public class WebFrontend extends HttpServlet { public void service(HttpServletRequest request, HttpServletResponse response) throws IOException { Database db = new Database(); int countresults = 0; ArrayList resultstorage = new ArrayList(); String enterkeywords = null; String noresults = null; response.setContentType("text/html"); PrintWriter out = response.getWriter(); StringTokenizer tokenizer; ArrayList <Integer> result = null; long elapsedTimeMillis = 0; String input = request.getParameter("search_text"); if(input != null && input !=""){ input = input.toLowerCase(); tokenizer = new StringTokenizer(input, " "); int size = tokenizer.countTokens(); String[] feed = new String[size]; int count = 0; while(tokenizer.hasMoreTokens()){ feed[count] = tokenizer.nextToken(); count++; } long start = System.currentTimeMillis(); //measuring speed result = db.getResult(feed); if (result.isEmpty()){ noresults = "No results."; } 53 else { try { ConnectionPool pool = new ConnectionPool("com.mysql.jdbc.Driver", "jdbc:mysql://localhost:3306/scriptie", "root", "XXXXX", 10, 20, true); Connection conn = pool.getConnection(); Statement stmt; ResultSet rs; stmt = conn.createStatement(); for (int i = 0; i < result.size(); i++){ int id = (int) result.get(i); rs = stmt.executeQuery("SELECT * FROM lastgeld WHERE Idno = '" + id + "'"); while(rs.next()){ int theInt= rs.getInt("Idno"); String vn = rs.getString("voornaam"); String an = rs.getString("achternaam"); String hh = rs.getString("haven_herk"); String al = rs.getString("Aantal_lasten"); String hd = rs.getString("Heffing_decimaal"); String tn = rs.getString("tonnage"); String gl = rs.getString("guldens"); String st = rs.getString("stuivers"); String sc = rs.getString("scanfile"); countresults++; //int printcount = countresults + 1; String store = null; store = (" " + countresults + ". id = " + theInt + " first name = " + vn + " last name = " + an + " harbor = " + hh + " cargo units = " + al + " toll-decimal = " + hd + " weight = " + tn + " guldens = " + gl + " stuivers = " + st + " <a href=\"" + sc + "\" target=\"_blank\">source</a><br>"); resultstorage.add(store); } } stmt.close(); pool.free(conn); conn.close(); pool.closeAllConnections(); } catch (SQLException e){ e.printStackTrace(); } result = null; feed = null; } elapsedTimeMillis = System.currentTimeMillis()-start; // Time elapsed } //if input != null && input != "" else { enterkeywords = ("Enter one or more keywords."); } //else out.println("<html>"); out.println("<body>"); out.println("<pre>"); //Search area out.println("<center>"); out.println("<br><br>"); out.println("<form action='lastgeld' method='get'>"); 54 out.print("<input type=text name=search_text>"); out.print("<input type=submit value=Search>"); out.print("<input type='checkbox' name='checkbox' value='verbose' />"); out.print("verbose"); out.println("</form>"); out.println("<a href = '/web/experiment'>experiment</a><br>"); if (noresults != null) out.println(noresults); if (enterkeywords != null) out.println(enterkeywords); out.println("</center>"); // Creating the verbose String checkbox = request.getParameter("checkbox"); if(checkbox != null){ if (checkbox.equals("verbose")){ out.println("<table border='1' align='center'>"); out.println("<tr>"); out.println("<td align='right'>"); out.println("Available indices: <a href = '/web/datefreq.txt' target = '_blank'>date</a>, <a href = '/web/vnfreq.txt' target = '_blank'>first name</a>, <a href = '/web/anfreq.txt' target = '_blank'>last name</a>, <a href = '/web/havenfreq.txt' target = '_blank'>harbor</a>, <a href = '/web/countryfreq.txt' target = '_blank'>country</a> "); out.println("</td>"); out.println("</tr>"); out.println("<tr>"); out.println("<td>"); //intersection combinations ArrayList <Graphable> intersections = db.getIntersections(); int pair = 1; int hr = 1; int count = 0; if (intersections != null){ if(!intersections.isEmpty()){ out.println("<h2>Step 1: find intersection combinations based on the keywords entered and the available indices.</h2><br>"); for (int i = 0; i < intersections.size(); i++){ if (count == 2){ out.print("<hr size='3' color= 'gray'>"); count = 0; } Graphable object; object = intersections.get(i); if (count == 0){ out.println("<b>pair " + pair + ":</b><br>"); pair++;; } out.println("<b>"+ object.toString() + " " + object.getType()+ "</b>" + object.getIDs() + "<br>"); count++; if(hr == 1){ out.println("<hr>"); hr--; } else 55 hr++; } } } else { out.print("Verbose returns output at a minimum of two keywords."); } intersections = null; db.resetIntersections(); out.println("</td>"); out.println("</tr>"); //combined intersections ArrayList<Object> combinedintersections = db.getCombinedIntersections(); int count1 = 0; int total = 1; if (combinedintersections != null){ out.println("<tr>"); out.println("<td>"); out.println("<h2>Step 2: intersect the combinations.</h2><br>"); if(!combinedintersections.isEmpty()){ for (int i = 0; i < combinedintersections.size(); i++){ if (count1 == 2){ out.print(" <b><--></b> "); } if (count1 == 5){ out.print("<br>"); count1 = 0; total++; } if (count1 < 4){ out.print(" <b>" + combinedintersections.get(i) + "</b>"); count1++; } else{ out.print(" " + combinedintersections.get(i)); count1++; } } } out.println("</td>"); out.println("</tr>"); } combinedintersections = null; db.resetCombinedIntersections(); ArrayList <Graphable> vertices = db.getVertices() ; WeightedGraph graph = db.getGraph(); if (vertices != null && graph != null){ out.println("<tr>"); out.println("<td>"); out.println("<h2>Step 3: find the vertex with the most edges.</h2><br>"); if(!vertices.isEmpty()){ for (int i = 0; i < vertices.size(); i++){ 56 Graphable object; object = (Graphable) vertices.get(i); out.print(" <b>" + object.getType()+ "</b> "); //type out.print("<b>" + object.toString() + "</b> "); //content //System.out.println(object.getIDs()+ "\n"); //ID's QueueInterface queue; queue = graph.getToVertices(object); if (!queue.isEmpty()){ out.print("has edges: "); int count2 = 1; while(!queue.isEmpty()){ Graphable item = (Graphable) queue.dequeue(); out.print(" " + count2++ + ": " + item.getType()); out.print(", " + item); } out.print("<br>"); } else { out.println("no edges.<br>"); } } } out.println("</td>"); out.println("</tr>"); } out.println("</table>"); } } if (!resultstorage.isEmpty()){ out.print(" <b>Found "+ countresults + " records in "); if (elapsedTimeMillis != 0) out.println(elapsedTimeMillis + " milliseconds:</b><br>"); for(int i = 0; i < resultstorage.size(); i++){ out.print(resultstorage.get(i)); } } out.println("</pre>"); out.println("</body>"); out.println("</html>"); input = null; db.resetIntersections(); db.resetCombinedIntersections(); db.resetGraph(); db.resetVertices(); db.resetListOfLists(); db = null; } } 57