Download Digas - Oracle

Digas
Digital Archiving System
• Digas is the database program used for research and fact
checking in the Research Department (“Dokumentation”, ~
60 researchers) and by the journalists writing for DER
SPIEGEL (~ 270 journalists), manager magazin, SPIEGEL
TV and SPIEGEL Online.
• Since 1995 the majority of the incoming documents and
since 1998 all the incoming documents are digital.
Different archiving systems were used since 1991
(BRS/Search, Trip). Today we manage the document
workflow in an oracle database with a java application and
use a different application (servlet / html) for searching in
the intermedia index.
Research in dossiers vs. full text searching
• The archive in the Research Department contains more
than 25 million documents and about 4 million pictures.
These documents are traditionally organized in categories
(=dossiers regarding subjects), companies and individuals.
Documents were indexed in these categories for easy
searchability. Only relevant articles were intellectually
categorized, so by beeing indexed the documents were
automatically weighted.
• In the digital archive of DER SPIEGEL this system has
been modified, but the principal idea is still the same.
Categorizing supports fast searchability of only the
relevant articles. In the traditional archive the not indexed
part of a paper was lost. In the digital archive the full text
search in field-structured documents supports efficient
research strategies for the professional researcher. The
combination of categorization and full text search can be
used to simplify the categorization model and to reduce
this very cost-intensive work.
• The majority of our users are journalists who are not
automatically professional database researchers with
knowledge of boolean operators or complicated research
strategies. So the categorization of articles is still one of
our methods for supporting successful research, the Digas
program supports full text search and research in dossiers
in separate specialized front ends.
• Digas is a multitier client server application (thin client).
Client
http
Servlet
corba
Server
• Application Server: NT 4 (2 servers)
• Database Server: HP Superdome (16
processors, 48 GB ram (Tetragon, 6 GB
cache)
jdbc
Oracle DB
Digas - the user front end
• How does a research via Digas work
• Examples of a Dossier Search and full text search
The Digas document base – different views
• Today there are 10 million documents in our database,
about 15% in dossiers:
– 1,9 million articles in dossiers regarding individuals,
50% of these are images, not full text (1991-1995)
– 1,4 million articles in dossiers regarding subjects
– 0,2 million articles in dossiers regarding company
information
– 1 million full text articles with image data (jpg, pdf)
• The avarage size of a document is 4 KB
We import about 50 000 new documents weekly with a peak
on mondays (weekend editions)
Indexed documents per hour (week 21)
2500
2000
1500
1000
500
0
00:00
18:00
13:00
13:00
11:00
08:00
00:00
07:00
21.05.2001 21.05.2001 22.05.2001 23.05.2001 24.05.2001 25.05.2001 26.05.2001 27.05.2001
In peak times there is an index load of up to 2000 new or
changed documents per hour
sync times on monday 21.05.01 (week 21)
Users in document management and resarch
• About 30 users do work on documents
• About 60 users do research. Right now we see up to 1000
searches per day. We expect the system beeing used by
around 500 paralell users with several thousend searches
per day and in peak times up to 1000 searches per hour.
Performance
• Right now 25% of the searches are dossier-searches.
Nearly 20% of these searches take longer than 10 seconds.
• In case of full text searches 10% of the searches take
longer than 10 seconds. Wild card searches and phrases are
usually the problem.
Why Oracle
• Scalability, performance, easy support for unix / hp-ux
• Professional support and commitment for our mission
criticle application
• Integration in our document management
• Synergy effects in further developing the applications for
research and document management
• Full text features were quite limited, we expect fast
development
Intermedia index and execution of a search
• The index is built using USER_DATA_STORE. A PL/SQL
procedure creates a virtual XML document which is beeing
indexed
• „manual partitioning“: we have two sets of indices which
are divided in three parts (90 days, 270 days and rest).
These indices are kept on different columns of the
document table. This improves index performance and
manageability but searches get more complex.
• The second index set is for security: rebuilding an index
takes the whole weekend.
• All searchable document attributes are kept in the
intermedia index (performance). This results in a lot of
database triggers and complex search execution, e.g.
supporting performant searches including date ranges.
• No stoplist is used
• No substring- or prefex-indexing is used (index size)
Discussion
• The scalability we need for our application depends on a
very individual software solution for maintaining the index
and executing searches
• sync and optimize of the different indexes are scheculed by
a procedure especially created for this application
• One has to keep all searchable attributes in the intermedia
index out of performance reasons. The integration of
structured searches (joins) and fulltext searches is weak one might expect this to be different.
• Date range search, the lot of attributes due to the dossier
search and the large document base with highly structured
documents increase the number of items in the index.
• Another consequence of keeping all attributes in the
intermedia index is that search statements grow quite long.
They have to be copied and optimized for the three
seperate indexes (time slices), have to be further divided in
case they are longer than 4000 characters, the result sets
then combined and sorted for presentation.
The locking issue
• The documents beeing indexed are locked from the
beginning to the end of a sync run. Keeping a fultext index
in sync is an asynchronous process which should be done
without any locking. We believe that this is a serious
design issue.
• Even in our hardware environment a sync can run up to
two minutes, which is in itself not a bad thing. This locking
behavior is bound to be a severe problem in every large
scale environment where full text search and document
management are done on the same document base.
Together with Oracle we are currently working on
workarounds.
These are the problems we find most important to be solved in
the ongoing development of oracle intermedia:
• No locking during sync (and optimize)
• Tighter integration of fultext queries and structured
constraints (e.g. date-range)
• Archive log during ctx_dll.optimize.index: 50 to 80 GB
archive log daily
• Better performance in wildcard an phrase searches
• Support for refining a search e.g. do a search on the result
set of another search
• Better support for getting fist rows of a result set ordered by
date
• Contact:
Heiner Ulrich
DER SPIEGEL
phone +49-40-30072941
e-mail [email protected]