Download A query based approach for integrating heterogeneous data sources

A query based approach for integrating
heterogeneous data sources
Ruxandra Domenig
Klaus R. Dittrich
Department of Information Technology
University of Zurich
Switzerland
Department of Information Technology
University of Zurich
Switzerland
[email protected]
[email protected]
by translating various query languages and models of the
underlying data sources into a global query language and
data model. Regarding the process of integration, there are
three dierent approaches that can be taken:
ABSTRACT
Data available on-line toda y is spread over heterogeneous
data sources like traditional databases or sources of various forms containing unstructured and semistructured data.
Obviously , the \tec hnical" vailabilit
a
y alone is not at all sufcien t for making meaningful use of the existing information,
and thus the problem of eectively and eÆciently accessing
and querying heterogeneous data is an important research issue. One main approach is to integrate the data sources and
oer users an a priori dened global schema. Alternatively,
there are approaches which implement tools for giving users
the possibility to dene the global schema themselves. We
propose a new approach where heterogeneous sources can be
queried through a unied interface and underlying sources
are in tegrated b y means of a query language only
. In this
paper, we presen t extensions to OQL which allo w to query
structurally heterogeneous data, i.e. structured, semistructured and unstructured data alik e, and to integrate data
on-the-y. We also deal with issues of the extensive meta
information about sources that is needed for integration.
1.
The most common approach is to strictly separate in-
tegration from the querying process. First, a global
schema is manually dened, by following the usual
database design steps. Second, mappings betw een the
global and local schemas1 are dened. Only after completition of this in tegration step, the system may be
used for querying. The disadvan tage of this approach
is that the sc hema is rather rigid and cannot be easily
adjusted when, for example, new sources are connected
to the system.
A more exible approach is to oer tools whic hal-
INTRODUCTION
Progress in both, persisten t storage of all kinds of data
and computer net w orktechnology, is the main reason for
the explosive gro wth of data that is available on-line. In order to make meaningful use of it, new and innovative ways
for searching m ustbe dev eloped. One specic case is to
correlate and retrieve information - in a uniform and integrated way - scattered in a netw ork of structur ally heterogene ous data sources, i.e. sources which con tain structured,
semistructured and/or unstructured data. Generally speaking, in tegrated searc h refers to users posing queries against
a set of data sources, using a global query language. Queries
are in ternally split, sent to the respective data sources, and
later the results are combined. Query processing comprises
in this case t w o main steps:the homogenization of data and
its combination, i.e. integration. Homogenization is realized
Permission to make digital or hard copies of part or all of this work or
Permission
to make use
digital
or hard without
copies of
or part of
workare
for
personal
or classroom
is granted
feeallprovided
thatthis
copies
classroomfor
useprofit
is granted
without fee
providedand
thatthat
copies
are
notpersonal
made orordistributed
or commercial
advantage
copies
notthis
made
or distributed
profit or
advantage
that copies
bear
notice
and the fullforcitation
on commercial
the first page.
To copyand
otherwise,
to
bear this to
notice
full citation
the first page.
To copy
otherwise,
republish,
postand
onthe
servers,
or toonredistribute
to lists,
requires
priorto
republish,
to post and/or
on servers
or to redistribute to lists, requires prior specific
specific
permission
a fee.
permission and/or a fee.
CIKM
2000,
VA USA
CIKM
2000McLean,
Washington
© Copyright
ACM 20002000
1-58113-320-0/00/11
. . .$5.00..$5.00
ACM 0-89791-88-6/97/05
low to dene the global schema at run-time. While
the in tegration process is still separated from querying, it can be repeated, i.e. a new global schema can
be dened more easily when users' needs change. In
this w ay, new sources ma
y be easier connected. Before
querying, users (or their administrators) may themselves dene the global schema as w ell as mappings
betw een global and local schemas using for example a
specication language. Ho w ever, this ma
y also be regarded as a drawback, since specifying a lot of technical information (i.e. the global schema and mappings)
is not always a simple task and requires a lot of expertise. Note that compared with the previous approach,
the specication of the global schema is restricted by
the a vailable tools and thus not all desired mappings
may be denable. In addition, fewer optimizations can
be applied in this case.
The most exible approach is to let users do the in-
tegration while formulating queries, without having to
dene a schema beforehand. The system just oers
a homogenized view over the underlying data sources,
but the integration is done while querying. New sources
can be easily connected to the system, since their data
only have to be homogenized. When querying, users
1
Though there is no \schema" for unstructured data sources,
w e use this notion for such sources as well and regard the
functionality exported by an unstructured source as a function (e.g. a stored procedure in a DBMS) which has as input
parameters the query (e.g. keyw ords) and as output the result set.
453
A similar approach is used in the DIOM project [13] which
allows users to dene a personal view on the data using a
meta model, and to express queries based on it. DIOM uses
an extension of the ODMG data model [6] for representing
data on the mediation level, and an extension of OQL as
query language. As a result, more complex queries can be
formulated and more semantics of the underlying sources
can be exploited because the ODMG model is more complex
than OEM.
DISCO [16] assumes an integrated global view at the mediation layer and proposes solutions for special problems related to query processing in this context: unavailable data
sources, query rewriting, and optimization. It also oers
some solutions for the extensibility problem by dening data
which have to be exported by wrappers.
Further projects related to our work include Garlic [14],
InfoSleuth [5] and Information Manifold (IM) [12]. These
projects provide solutions to subsets of problems which we
also face. However, to the best of our knowledge, there
is no other project which aims at the integration of data
through the query language, at the time when users query
the system.
decide in which sources to search and how to integrate data. In this case, extensive information about
the sources, i.e. metadata, must be available. If users
have little or no knowledge about the structure and the
characteristics of the underlying data sources, some
\approximate" or \fuzzy" search is still possible. Alternatively, if users know the underlying data sources
and their query capabilities, they can express highly
targeted queries. Though many cases (like transactions in nancial applications) require exact queries,
there are also numerous examples where a \fuzzy" way
to query and support to connect and disconnect data
sources easily are very benecial.
As will be discussed later, research so far has focused on
the rst and second approach. The main motivation for the
third approach is twofold. First, larger numbers of diverse
data sources can be more easily made available for searching
because no (or only few) preparatory steps are needed. Second, since users have more and more diverse objectives, they
need a exible way to search. This includes \fuzzy" ways to
query, aiming at getting any information, even if it is just
loosely related to the information eectively needed. On the
other hand, precise queries also have to be supported in order to get exact information. It turns out that the absence of
an integrated schema facilitates the \fuzzy" way of querying, provided that the query language allows for querying
heterogeneous data in an integrated way.
Our system SINGAPORE (SINGle Access POint for heterogeneous data REpositories) implements this approach.
Our main goal in this paper is to present SINGAPORE's
query language, together with the appropriate infrastructure environment. Specically, we extend OQL with various features for querying structured, semistructured and
unstructured data in a exible and integrated way.
The paper is structured as follows. We start with a running example in Section 2. In Section 3 we present the user's
view of the data, and in Section 4 we present the SINGAPORE query language. We then sketch the architecture of
our system in Section 5. Section 6 deals with the meta information needed to formulate queries and to implement the
query language. In Section 7 we describe some implementation aspects of the metadata repository. Section 8 concludes
the paper.
2. A RUNNING EXAMPLE
In this section, we compare the three above-mentioned
approaches for integration by using a uniform example. To
this end, we rst consider some data sources (see Figure 1)
and second, a query against them. Structured data sources
create table Person( |
create table Institute(
ID:
NUMBER,
|
Name:
CHAR(50)
Name:
CHAR(50), |
Depart:
CHAR(50)
Surname: CHAR(50), |
Summary:
CHAR(10000)
Photo:
BLOB,
|
);
Inst:
CHAR(50) |
);
|
-----------------------------------------------------------create function idf_retrieve(CHAR(50))
RETURNS list(CHAR(10000))
EXTERNAL NAME '/home/locations/funcloc/dist'
LANGUAGE C
PARAMETER STYLE DB2SQL
NO SQL
DETERMINISTIC
NO EXTERNAL FUNCTION
NO FENCED
class Paper (
Title:
string,
Abstract: string,
Auth:
set(Author)
);
Related work.
Several approaches for integrated access to heterogeneous
sources have been proposed. Most of them (including ours)
are based on a three-tier architecture as presented in [17],
consisting of a data source layer, an integration layer and
a user or application layer. The lowest layer contains the
sources and so-called wrappers which export their functionality and data. The middle layer comprises mediators which
were introduced in [17] to \simplify, abstract, reduce, merge
and explain data". The upper layer represents the interface
to users and applications.
The TSIMMIS project, which aims at assisting users in
implementing mediators for integrating information [9], uses
a query language based on a light-weight model for semistructured data, OEM. Data of the underlying data sources are
represented in OEM, so in fact TSIMMIS oers a semistructured view of heterogeneous data. The advantage of this system is that it supports users well in specifying integration
components, which are then generated automatically.
|
|
|
|
|
class Author (
Id:
integer,
AuthorName: string,
Publications: set(Paper)
);
Literature(Text: string)
Metacrawler(Where: string, Keyword: string)
Figure 1: Data sources example
in our example include tables Person, Institute and a
stored procedure idf retrieve dened in a DB2 database,
and classes Paper and Author dened in an object-oriented
O2 database. The attribute Summary of table Institute
contains a textual description. idf retrieve is a stored
procedure which takes as argument keywords and retrieves
Summary attributes of tuples in Institute which contain the
input keywords. We also consider a search engine, Literature,
which allows to search for various information about pub-
454
lications in computer science, and the metasearch engine
Metacrawler for searching the WWW and newsgroups. We
represent the functionality exported by a search engine as a
stored procedure which takes as arguments the query (e.g. for
Literature the parameter Text and for Metacrawler the
parameter Keyword) and other information needed for answering the query (e.g. for Metacrawler the attribute Where
which species where to search, i.e. on the Web or in Newsgroups).
Let us now assume that users are interested in Which
class Person (
| class Institute (
ID:
NUMBER,
|
Name:
CHAR(50),
Name:
CHAR(50),
|
Depart:
CHAR(50),
Surname: CHAR(50),
|
Summary:
CHAR(10000)
Photo:
BLOB,
| );
Inst: CHAR(50)
|
);
|
----------------------------------------------------------class Paper (
| class Author (
Title:
string,
|
Id:
integer,
Abstract: string,
|
AuthorName: string,
Auth:
set(Author) |
Publications: set(Paper)
);
| );
----------------------------------------------------------class Literature (
| class Metacrawler (
Text: CHAR(MAX_STRING) |
Keyword: CHAR(MAX_STRING),
);
|
Where: CHAR(100)
| );
----------------------------------------------------------class Retrieval (
idf_retrieve(CHAR(50)): list(CHAR(MAX_STRING))
);
publications have been published by employees of
the ``Database Technology'' institute?
class Person (
| class Paper (
ID:
NUMBER,
|
Title:
CHAR(50),
Name:
CHAR(50), |
Abstract: CHAR(50),
Surname:
CHAR(50), |
Auth:
set(Person)
Photo:
BLOB,
| );
Inst_Name:
CHAR(50), |
publications: set(Paper) |
);
|
--------------------------------------------------------class Literature (
Text: CHAR(5000);
);
Figure 3: The global (unintegrated) schema.
Note that class Retrieval contains
just a method, idf retrieval.
stricted to certain sources. Users have to know that information about \Database Technology" may be available in structured sources and in Literature.
Figure 2: An example of an integrated schema
As mentioned before, the rst two integration approaches
oer a global integrated schema which users can query. One
example of a global schema for the given data sources is
expressed in an object-oriented model in Figure 2. In this
case, the global schema merges tables Person, Institute
and class Author into one class Person. We assume that
information delivered by Metacrawler and the functionality
oered by idf retrieve are not of importance for global
applications. Note that the generation of the global schema
is a complex process which has to be done manually (at least
in part), by a system administrator. Our example query
might then look as follows, in an OQL-like syntax [6].
SELECT
FROM
WHERE
AND
3. Select titles and author names from all sources available which contain information about publications in
the eld of \Database Technology". The search is in
this case even more precise: just titles and author
names are searched, not all information as in case
1. Obviously, users need to know that attributes like
Title and AuthName are available in some of the underlying data sources.
4. Select titles of publications from Paper and names of
employees from Person which work at the Institute
called \Database Technology". This query can only
be formulated if users have extensive knowledge about
the underlying data sources and know that it makes
sense to combine information from Person, Institute,
Paper and Author.
The challenge thus is to design a system which accepts all
such kinds of queries. First, the query language should contain various constructs to support a \fuzzy" kind of querying
(like in cases 1, 2 and 3). Second, the integration of (heterogeneous) data has to be accomplished in the last approach
at the time the system is queried. For the example in case
4, data from Paper, Person, Institute and Author must
be combined and users have to know that it makes sense to
combine it, and how to achieve the integration.
Though the \fuzzy" way of querying could also be realized
in the rst two approaches by oering a suitable query language, this has not yet been approached so far. In addition,
the two challenges complement each other, since integration
\on the y" can be more easily realized by using \fuzzy"
elements of the query language, as will be seen later.
Title
Paper, Person Pers
Paper.Auth.Name = Pers.Name
Pers.Inst_Name = ``Database Technology''.
Let us now consider the third approach. In this case,
no integration step is required before the system is queried.
Only a global view is provided (see Figure 3) which is just the
union of the homogenized schemas of the underlying sources
(and thus contains all schema elements of Figure 1).
Using this global view, we would like that users can formulate queries, ranging from \fuzzy" to precise ones, including
for example:
1. What information is available about \Database Technology"? In this case no information about the sources
is needed. Obviously, this query returns all kinds of information, from all sources and not only about publications about \Database Technology" (assuming that
publications of employees of \Database Technology"
institute are about this topic area).
3. THE DATA SPACE
2. What information in the structured sources and in the
\Literature" source is available about \Database Technology"? In this case, the information need is expressed in a more specic way, since the search is re-
In this section we present the conceptual view of data we
use in SINGAPORE which in particular denes (sets of)
data queries can be formulated against.
455
Data space
Data2
Data1
structured
DB_OO
DB_OR
DB2
Level 1
(Classification Roots)
unstructured
semistructured
HTML
Text
O2
Level 2
(Data or Content Type)
Metacrawler
D1
Institute
idf_retrieve
Paper
Literature
Addr
O2Lit
D2
Author
Level 3
(Source Location)
Address
Text
Person
street
Name Depart. Summary
ID Name Surname Photo Inst
zip_code
Keyword
Where
number
location house_name
Level 4
Title Abstract Auth. ID AuthName Publications
zip_code name
(Source Schemas)
Figure 4: The data space
We regard data stored in a connected source as class extensions (in the spirit of object-orientation) and consider
these as sets which can be queried. Furthermore, we also
want to oer more general queriable data sets, which would
typically embody more than just one data source. For this
reason, we consider unions of extensions according to classications based on various criteria like the kind of data,
the particular contents of sources, etc. The resulting structure is called the data space and is more or less a tree for
which most nodes contain the union (of extensions) of data
items of their subordinate nodes. This does not always apply for nodes which represent the contents of data sources.
In this case, a node may be the composition of its subordinate nodes. For example, a relational table is composed out
of attributes, but it is not the union of the attribute values.
This conceptual view of data allows that any node except
the leaf nodes in the data space can be used as a set which
can be queried.
The explanation above suggests that the data space comprises four levels. The lowest level contains schema denitions of data items stored in the connected sources. Each
data source is identied by a name, specied on level three.
Level two contains classications of data, and on the rst
level, entry points to classications are specied. Apart from
the structure of data which may be automatically gained
from sources, all other information of the data space is dened by the system administrator. We discuss this issue in
Section 6.
Figure 4 represents an example of a data space which contains on the rst level two entry points of classication hierarchies, Data1 and Data2. On level two, data is organized
according to its \structuredness", i.e. structured, semistructured and unstructured data. We further classify structured data sources into object-relational databases (DB OR)
and object-oriented databases (DB OO), and we assume that
structured data are stored in database management systems
DB2 and O2. Unstructured data are organized according to
the kind of data which can be queried, normal text (Text)
or HTML les (html). Level three contains the names of
sources connected to the system, and on level four we represent e.g. the schema of Figure 3 that has been dened for
these sources. Note that we also extended levels three and
four with a semistructured data source storing information
about addresses. Here, an Address is usually composed of
a street, a number, a zip code, and a location. However, an
address may also comprise a name of a house (house name),
a zip code, and a location. As a consequence, an address is
only semistructured (see below).
4. THE SINGAPORE QUERY LANGUAGE:
SOQL
Assuming the above view on the data, we can now dene
the query language of SINGAPORE. We dene it starting
from a language for structured data, in our case OQL, and
then extend it with features for unstructured and semistructured data, and for integrating data.
Querying unstructured data.
Basically, querying unstructured data means to query data
using keywords. SOQL provides an operator contains which
retrieves data from unstructured data sources. For our example, the query
SELECT * from Metacrawler, Literature WHERE
contains(``Database Technology'')
will trigger a search for the keywords \Database Technology" in the specied search engines (cf. Figure 4). A variation of contains is contains related. It takes the same
arguments, but also information pertaining to keywords similar to the given ones is retrieved. This requires further
transformations to be done on the input keywords, for ex-
456
'Meier', the exact name of the attribute is not known.
ample stemming, word decomposition or the application of
ontological knowledge2 .
A search for keywords \author, name"{ using algorithms for word decomposition, stemming and once
again applying ontological knowledge { will result in
two matches, D1.Person.Name = 'Meier' and
O2Lit.Author.AuthorName = 'Meier'.
Querying semistructured data.
As dened in [1], [7], semistructured data have structure,
but the structure is not rigid. In addition, the structure
denition or parts of it are not necessarily separated from
the data, i.e. they may be implicit. Query languages for
semistructured data (LOREL [3], StruQL [8]) deal with these
aspects. We thus extend OQL with features of those languages:
4.1 Supporting integrated access to heterogeneous data sources
Regular path expressions are used to specify the struc-
ture of data even if it is not rigid. For a path specication, we introduce two new syntactic elements: \*",
if none, one or more nodes are not specied, and \?', if
exactly one node is not specied. For example, assume
that source Addr accepts queries which specify regular paths expressions. In SOQL, a query like SELECT
Address.*.zip code FROM Addr can be formulated.
Type coercion refers to implicit type cast. For example,
the expression Address.zip-code = '8001' is allowed
in SOQL, where '8001' is a string and Address.zip-code
A last query language issue is the combination of heterogeneous data in one query, which may have various meanings
for structured and unstructured data. First, consider the
query SELECT : : : FROM D1.Person, D2.Institute, which
is supposed to combine data from D1.Person and
D2.Institute (i.e. structured data in a conventional relational database). The result of this query is a projection
(depending on the attributes specied in the SELECT part) on
the cartesian product of tables D1.Person and D2.Institute.
Second, consider metasearch engines3 which also allow the
\combination" of information from dierent search engines.
An example of a query which retrieves information from
Literature and Metacrawler (i.e. from unstructured data
sources) is SELECT : : : FROM Literature, Metacrawler.
has been specied as an integer.
User Interface
Using unstructured and semistructured query characteristics for searching in the data space.
We can now take advantage of the above-mentioned characteristics of query languages for unstructured and semistructured data and apply them to the entire data space:
user layer
Query Preprocessor
Query Processor
Query Postprocessor
Metadata
Repository
Source Registration
contains and contains related may be used for query-
ing structured or semistructured data as if it were unstructured. For example, searching for a keyword in a
database table triggers a search (exact for contains,
approximate for contains related) in each textual attribute of the table and returns the tuples which contain this string.
Query Transformer
Wrapper Manager
mediation layer
source layer
Regular path expressions can be used for specifying
paths in the entire data space. For our example in
Figure 4, structured.*.Name is matched by
structured.DB OR.DB2.D1.Person.Name and
structured.DB OR.DB2.D2.Institute.Name.
Wrapper
Wrapper
Wrapper
Search
Engine
OODBMS
RDBMS
Figure 5: The coarse architecture of SINGAPORE
As in any metasearch engine, the result of this query
would be the union of the set of documents returned by
Literature and by Metacrawler.
These examples show that we need to dierentiate between the two cases. we introduce a new syntactical feature
in SOQL: fg, used to specify the union between the result
sets of two or more data sources, and [ ], used to specify
the cartesian product between the result sets of two or more
data sources (which is also the default if no such brackets
are used).
Type coercion is also applied to structured data sources
in the data space. If for example a condition Person.ID
= '101' is specied, where '101' is a string and
Person.ID is declared as an integer in D1, a type cast
is realized and the expression Person.ID = 101 is sent
to D1, where 101 is an integer.
In analogy to [1] which proposes that data and meta-
data can be used together in queries, we implement
a similar feature in our system, but make it applicable to the whole data space. If an attribute (or a
path in the data space) has to be specied but its exact name is not known, the operator LIKE applied to
a related name triggers a keyword search in the entire data space. For example, in LIKE(author.name)=
2
Ontological knowledge is a set of concepts in a particular information domain and the relationships between them [10].
For example, \Computer Science" and \Database Technology" would be related concepts in our application domain.
5. THE SINGAPORE ARCHITECTURE
We present in this section the coarse architecture of SINGAPORE which follows the usual three-tier architecture
mentioned in Section 1 (see Figure 5).
3
A metasearch engine is a system which provides a single
access point to various search engines. It sends a query
to the underlying search engines, combines the results and
returns a single, unied hit list to the user.
457
and operations (Operation). Operations have a Signature
and a Result. The union of various properties and operations forms an Entity (for example a table in a relational
database). A data source and an entity itself may contain
other entities. Each element has a name (SName, EName,
PName, OName). The Path variable species for a source its
classication path on level 1 in the data space.
Problems related to schema integration have already been
investigated in the area of federated databases, and we use
these results in our case to determine the integration information which has to be stored in the metadata repository. As described in e.g. [4], the integration process includes
mainly two steps:
Preintegration. The database administrator determines
similarities and conicts between schemas and denes
rules for interschema correspondences. Obviously, this
has to be done manually.
Integration. The integrated schema is (automatically)
built according to the rules dened in the previous
step.
As a consequence, two issues are of importance at the moment the system is queried: Which sources contain similar
information? and Which are the conicts between sources
containing similar information?
These two problems will be discussed in the next subsections, by extending the specications shown before.
Users interact with a presentation layer and use it to submit their queries. The mediation layer has two important
tasks: to process queries and to register new sources into
the system. When a query is entered, it is passed to the
query transformer component which contains mainly two
components: the query preprocessor parses the query and
handles its \fuzzy" part, and the query processor does
further processing, including the generation of subqueries.
These are sent through the wrapper manager to the respective sources. Wrappers translate subqueries into the query
language (or the procedural query interface) of each source.
Sources return results in their particular data model and
wrappers translate them back into the SINGAPORE data
model. Wrappers may also improve the query capabilities of
sources, for example by extracting or ltering data returned
by the source. The wrapper manager sends partial results to
the query postprocessor, which combines the returned results. The source registration component is responsible
for connecting and disconnecting sources.
One of the main components is the metadata repository
which stores information about the underlying data sources.
This information is used by system components (query transformer, query postprocessor and source registration) or by
users when formulating queries. The metadata repository
will be presented in more detail in Section 6.
The mediation layer models queries and data in a common data model which is used as a logically homogenized
view of the data of the underlying sources, but also as a
means to support global query processing. We choose an
already existing one, the ODMG data model, and extend it
to meet our requirements. First, we add a composite type
ss-struct similar to the struct constructor, to represent
semistructured data. In contrast to struct, ss-struct does
not apply any type checking. Note that this type is also implemented in other systems, like [11]. Second, we implement
heterogeneous collections. This is again necessary for representing semistructured data, but also for combining data
from various sources.
6.
nonterminalPredeccessors
*
1 MetadataRepository
1..*
TreeNode
1..*
String: SName
1..* String: Description
String: Related_to
nonterminalSuccessors
ClusterTreeNode
*
1..*
* terminalSuccessors
contains 1..*
*
contains_oper.
contains_prop.
THE METADATA REPOSITORY
1..*
1..*
Property
Operation
Bool: Mandatory
String: Relation
The metadata repository contains the following information: i) structure, query capabilities and result types of data
sources, ii) descriptive information about sources, which we
call \soft" metadata, and iii) ontological knowledge.
This information is specied either manually or entered
automatically. In the former case, the system administrator
has to specify it while in the latter case, the source registration component extracts it from sources (for example, a
DBMS stores data about its schema in its data dictionary
and also allows to query it).
We present in this section examples of metadata needed
to formulate queries and to integrate the data sources at the
time the system is queried, using an XML-like notation [2].
We start from the data contained in the example data space
and extend it later on:
1..*
DataSource
*
1..*
contains_oper.
*
has_entity
1..*
*
Entity
*
*
1..*
1..*
has_underentity
contains_prop.
Figure 6: The main classes of the metadata repository
6.1 Specifying information about similarity
In the traditional case of schema integration, the system
administrator determines which sources contain similar information and whether it makes sense to integrate them.
There are dierent levels of similarities: between the topics covered in dierent sources, between entities, between
properties and between operations. In order to detect such
similarities, users need descriptive information about the
data, represented by \soft" metadata. For this reason, we
rst extend each element in a data source with a new tag
Description which contains a textual description of the content of this element. Additionally, we allow the specication
of descriptive information about which sources, properties,
operations or entities are similar, by introducing the tag
Related to. For example, the \soft" metadata of sources
D1 and O2Lit are as follows:
<! Element DataSource(SName, Entity*,
Property*, Operation*, Path*)>
<! Element Entity(EName, Type, Entity*, Property*, Operation*)>
<! Element Property(PName, Type)>
<! Element Operation(OName, Signature)>
<! Element Signature(ArgumentList, Result+)>
<! Element ArgumentList (Type*)>
<! Element Result (Type)>
Data sources contain various structural units (Property)
458
6.3 Source query capabilities
<DataSource>
<SName>D1</SName>
<Description>Database about employees of the
Computer Science Department since 1990.
</Description>
<Related_to>O2Lit</Related_to>
<Entity><SName>Person</SName>
<Related_to>O2Lit.Author</Related_to>
</Entity>
<Property><SName>Name</SName>
<Description> The name of the employee</Description>
<Related_to> O2Lit.Author.AuthName</Related_to>
</Property>
.....
</DataSource>
<DataSource>
<SName>O2Lit</SName>
<Description>Database about publications of the
Computer Science Department since 1980.
</Description>
<Related_to>D1</Related_to>
<Entity><SName>Author</SName>
<Related_to>D1.Person</Related_to></Entity>
<Property><SName>AuthName</SName>
<Description> The name of the author</Description>
<Related_to> D1.Person.Name and
D1.Person.Surname.</Related_to>
</Property>
.....
</DataSource>
Since we have made the assumption that sources are heterogeneous, they will most likely also have dierent query
capabilities which users have to know when querying.
First, it may be the case that the specication of some
properties is mandatory for some sources. This is indicated
in a tag Mandatory for a Property which can take the values
yes or (default) no. One example is Metacrawler, where the
property Keyword is mandatory, but Where is optional, since
it has \Web" as a default value.
Second, properties may have various dependencies among
them in any specic source. There are cases when a group
of properties must be specied together. For this reason,
we introduce two tags: AND op, used for properties which
have to be specied together, and XOR op, used for properties
which have to be specied alternatively. One example is the
source Address which requires that either house name, or
street and number are specied:
<DataSource>
<SName>Address</SName>
<XOR_op>house_name,<AND_op>street, number</AND_op></XOR_op>
</DataSource>
Finally, users need to know the semantics of operations
dened in data sources. The Signature tag dened before
represents just the arguments and result types. We introduce a tag Argument which contains the list of properties
of the source which must be specied as arguments for this
operation. In the example of Figure 1, idf retrieve is implemented in source D2 and takes as an argument just the
attribute Summary of table Institute.
Summarizing, the extensions for Property and Operation
are:
There are mainly two ways to determine similar information in the underlying sources. First, by using the information Description or Related to presented before. Second, one can specify similarities by using ontological knowledge. In Section 2, we have assumed that information about
publications in the eld of \Database Technology" is desired. However, the Description tag of sources D1 and
O2Lit shows that these sources contain information about
\Computer Science". The fact that \Database Technology"
is a subordinate concept of \Computer Science" is stored in
the ontology database of the SINGAPORE metadata repository.
<! Element Property(PName, Type, Description*,
Related_to*, Mandatory*, Relation*)>
<! Element Relation (XOR_cond*, AND_cond*)>
<! Element XOR_op (PName, AND_cond*)>
<! Element AND_op (PName, XOR_cond*)>
<! Element Operation(OName, Signature, Description*, Related_to*)>
<! Element Signature(ArgumentList, Result+)>
<! Element ArgumentList (Argument*)>
<! Element Argument (Property*)>
6.2 Specifying information about conflicts
As described in [15], the following kinds of conicts may
arise during the integration process:
Structural Conicts. A concept of the real world may
6.4 Using metadata for query processing
be represented dierently in two sources. A simple
example for our data space is the name of a person,
which in represented in table Person as a Name and
Surname and in class Author as one string, AuthName.
In SINGAPORE, users can search in the \soft" meta
information and learn about this structural conict.
The \soft" metadata presented in this section support
users to learn about underlying data sources in order to formulate queries which integrate data. It is obvious that this
metadata may easily become voluminous, and consequently,
suitable graphical tools need to be provided to present it to
users. However, even if such tools are available, the process
of formulating queries may get quite complex. Therefore, in
the long run, we aim at using metadata also during query
processing as far as possible, e.g. by suggesting to users
meaningful queries which might match the input query they
have formulated using fuzzy query language features. The
next section sketches some aspects of the implementation of
the metadata repository for processing these features.
Descriptive Conicts.
There are mainly two types
of descriptive conicts, name conicts (synonyms or
homonyms for attributes) and scaling conicts (e.g. different units of measurement for the same real world object are used). In SINGAPORE, users can learn about
descriptive conicts through the information stored in
the Related to tag. For scaling conicts, the necessary transformation should be available. An example
is the transformation of km into m, specied in the following way:
7. IMPLEMENTATION
We implement the metadata repository as a collection
of trees (the main classes and their relationships are represented in Figure 6). Each element in the metadata repository is a TreeNode. Class ClusterTreeNode represents the
classication nodes on levels 1 and 2. On level 3 there are
<Scale>
<From> km </From> <To> m </To>
<Through> 1 km = 1000 m </Through>
</Scale>
459
Keyword: Name
Data
Space
Index
9. ACKNOWLEDGMENTS
List of paths
D1.Person.Name
D1.Person.Surname
D2.Institute.Name
O2Lit.Author.AuthName
Addr.Address.location.name
Addr.Address.house_name
We thank Anca Vaduva for her help during the preparation of this paper.
10. REFERENCES
[1] S. Abiteboul. Querying semi-structured data. In
Proceedings of the International Conference on
Database Theory (ICDT97), 1997.
[2] S. Abiteboul, P. Buneman, and D. Suciu. Data on the
Web: From Relations to Semistructured Data and
XML. Moran Kaufmann Publishers, 1999.
[3] S. Abiteboul, D. Quass, J. McHugh, J. Widom, and
J. Wiener. The Lorel Query Language for
Semistructured Data. International Journal on Digital
Libraries, 1(1), Apr. 1997.
[4] C. Batini, M. Lenzerini, and S. Navathe. A
comparative analysis of methodologies of database
schema integration. ACM Comp. Surveys 18(4), 1986.
[5] R. J. Bayardo and W. B. et al. InfoSleuth:
Agent-based semantic integration of information in
open environments. SIGMOD Record, 1997.
[6] R. Cattell, editor. The Object Database Standard:
ODMG 3.0. Morgan Kaufmann, 2000.
[7] R. Domenig and K. R. Dittrich. An overview and
classication of mediated query systems. SIGMOD
Record, September 1999.
[8] M. Fernandez, D. Florescu, J. Kang, A. Levy, and
D. Suciu. STRUDEL: A Web site management
system. SIGMOD Record, 26(2), 1997.
[9] H. Garcia-Molina, Y. Papakonstantinou, D. Quass,
A. Rajaraman, Y. Sagiv, J. Ullman, V. Vassalos, and
J. Widom. The TSIMMIS approach to mediation:
Data models and languages. Journal of Intelligent
Information Systems, 1997.
[10] T. R. Gruber. Toward principles for the design of
ontologies used for knowledge sharing. International
Workshop on Formal Ontology, March 1993.
[11] A. Heuer and D. Priebe. Irql { yet another language
for querying semi-structured data? Preprint,
Universitat Rostock, Fachbereich Informatik, 2000.
[12] A. Levy, A. Rajaraman, and J. Ordille. Querying
heterogeneous information sources using source
descriptions. In Proceedings of the international
Conference on VLDB, India, 1996.
[13] L. Liu, C. Pu, and Y. Lee. Adaptive query mediation
across heterogeneous information sources. In
Proceedings of the International Conference on
Cooperative Information Systems (CoopIS), 1996.
[14] M. Roth, F. Ozcan, and L. Haas. Cost models do
matter: Providing cost information for diverse data
sources in a federated system. Proceedings of the
international Conference on VLDB, 1999.
[15] S. Spaccapietra, C. Parent, and Y. Dupont. Model
independent assertions for integration of
heterogeneous schemas. VLDB Journal, July 1992.
[16] A. Tomasic, L. Raschid, and P. Valduriez. Scaling
access to heterogeneous data sources with Disco. In
IEEE Transactions on Knowledge and Data
Engineering. IEEE, September/October 1998.
[17] G. Wiederhold. Mediators in the architecture of future
information systems. IEEE Computer, 25(3), 1992.
Figure 7: The data space index
DataSource nodes and on level 4 schemas of the sources
(composed of Entity, Property and Operation).
The metadata repository stores an index called Data Space
Index which supports an information retrieval kind of search
on paths in the data space4 . Paths of the data space are
transformed (indexed) by applying stemming and word decomposition algorithms. The index has as a key a keyword
and contains lists of paths in the data space which match the
keyword. For example, when processing the query SELECT
LIKE(Name) FROM : : : , the Data Space Index is checked,
which in our example contains the information shown in
Figure 7. Processing of LIKE would transform the given
query into six queries, having one of the paths specied in
Figure 7 in their SELECT part. Note that this index can also
be used by users when searching for keywords in the data
space. Additionally, the metadata repository contains an index called Soft Inform Index which allows users to search
in an information retrieval way for descriptions stored in attributes Description and Related to, for each element in
the metadata repository. These indexes are built at system
start up and updated when new sources are registered.
8.
DISCUSSION AND CONCLUSIONS
We have presented a new approach to query data from
a network of heterogeneous data sources. Our goal was to
integrate data sources when querying and to oer the possibility to formulate both, exact and fuzzy queries. For this
reason we extended OQL with special features and we have
shown that the realization of this approach is based on extensive meta information.
We believe that a system to integrate heterogeneous data
through the query language is a necessity today, because it
allows to connect new sources in an easy way, without requiring a global integrated schema. Furthermore, users can
query the system, even if they do not know all query capabilities of the sources. They can formulate queries similar
to keyword search in information retrieval systems, but in
addition, they can also formulate more targeted queries if
they use the information stored in the metadata repository.
Thus, in our approach we combine the way to search in information retrieval systems with the way to search in DBMS,
and therefore the implementation combines techniques from
both areas.
Consequently, we oer a solution for the problems of integrated search in multiple heterogeneous data sources and
of supporting various advanced search requirements. The
next step in our project is to detail the query processing
components.
4
This kind of index is called inverted le in information retrieval systems.
460