Download Data Management in Geodise

Data Management in Geodise
Jasmin Wason, Zhuoan Jiao and Marc Molinari
12 May 2003, Southampton
Challenges
Large amount of data generated by engineering applications at different
locations with different characteristics.
Data traditionally stored in flat files – hard to search, locate and share.
Geodise Database Toolbox
Store data with additional descriptive information:
Standard metadata
User-defined application specific metadata
Use both relational and XML databases - Oracle9i, Xindice, SQL Server.
Provide a familiar interface for engineers – Matlab functions.
© Geodise Project, University of Southampton, 2003.
http://www.geodise.org/
Database Toolbox - Services
Storage service
Archive data in file systems, sent over GridFTP.
Archive Matlab structure variables as XML documents in database.
Support datagroup concept to aggregate related data.
Location service to map logical data identities with physical storage locations.
Metadata service
Descriptive information can be added to data:
Standard technical metadata (e.g. file size, format, date): mostly auto-generated.
Application specific metadata: user defined.
Query service
Query over metadata to efficiently locate required data.
Command-line and GUI interfaces.
Authorisation service
Access rights to data can be granted to authenticated users.
© Geodise Project, University of Southampton, 2003.
http://www.geodise.org/
Database Toolbox – Client Tools
Archive data
gd_archive
Store files/ structures into archive with
metadata.
Define datagroup
gd_datagroup, gd_datagroupadd
Logically group together related data.
Query data
gd_query
Query archive from script or GUI.
Retrieve data
gd_retrieve
Retrieve archived data to local machine.
© Geodise Project, University of Southampton, 2003.
http://www.geodise.org/
Geodise Database
Toolbox
Matlab
Functions
Java
clients
XML Toolbox
Matlab Variables/Structures
Convert proprietary
format to XML and vice
versa transparently,
easy-to-use.
Benefit: XML can be
transferred, stored, and
retrieved across the
Grid.
Four functions
xml_save()
xml_load()
xml_format()
xml_parse()
XML
Matlab
(A) Generate file
File archive
(B) Archive
local file path
Data file
structure
XML
filehandle
(C) Query
Metadata
database
query string
XML
XML
XML
structure
structure
structure
filehandle
(D) Retrieve
filehandle
local file path
Data file
© Geodise Project, University of Southampton, 2003.
http://www.geodise.org/