Download Distributed Databases and Applications - DiGIR

Distributed Databases and Applications
John Wieczorek
Museum of Vertebrate Zoology, UC Berkeley
DiGIR
1
Distributed Databases –
Discipline-specific







The Species Analyst (TSA)
The Integrated Taxonomic Information System (ITIS)
FishNet
The Mammal Networked Information System (MaNIS)
HerpNET
The Ornithological Information System (ORNIS)
…
DiGIR
2
Distributed Databases –
International





European Natural History Science Information
Network (ENHSIN)
Biological Collection Access for Europe
(BioCASE)
Australia Virtual Herbarium (AVH)
Red Mundial de Información Sobre
Biodiversidad, Comisión Nacional para el
Conocimiento y Uso de la Biodiversidad
(REMIB, CONABIO)
…
DiGIR
3
Distributed Databases –
Regional







Mountain and Plains Spatio-Temporal DatabaseInformatics (MaPSTeDI)
Ocean Biogeographic Information System (OBIS)
Pacific Basin Information Node, National Biological
Information Infrastructure (PBIN, NBII)
Species Link, Centro de Referência em Informação
Ambiental (Species Link, CRIA)
A Virtual Herbarium of the Chicago Region (vPlants)
Spatial Analysis of Local Vegetation Inventories Across
Scales (SALVIAS)
…
DiGIR
4
Distributed Databases –
Intra-institutional



Berkeley Natural History Museums (BNHM)
Association of Biological Collections, UC Davis
…
DiGIR
5
Distributed Databases –
“Nodes”



LifeMapper
National Biodiversity Information
Infrastructure (NBII)
Global Biodiversity Information Facility
(GBIF)
DiGIR
6
GBIF Work Programmes





NODES
ECAT – Electronic Catalogue of Names of
Known Organisms
DIGIT – Digitisation of Natural History
Collections
OCB – Outreach and Capacity Building
DADI – Data Access and Database
Interoperability
DiGIR
7
Taxonomic Database Working Group
Standards development and maintenance





Access to Biological Collections Data
(ABCD)
Darwin Core Version 2 (DwC2)
Structure of Descriptive Data (SDD)
DiGIR
Others…
DiGIR
8
DiGIR
Distributed Generic Information Retrieval
John Wieczorek, Stan Blum, Dave Vieglais, P.J. Schwartz
DiGIR
9
Information Retrieval


Distributed - a protocol for retrieving
structured data from multiple,
heterogeneous databases across the
Internet.
Generic - a protocol independent of the
data retrieved and of the software to
retrieve it.
DiGIR
10
Project Rationale



Avoid multiple incongruous development
efforts
Pool resources and create a support
community of experts
Solve scalability problems
DiGIR
11
Design Goals




Use open protocols and standards, such as
HTTP and XML
Decouple the protocol, software and
semantics
Make new data provider installations as
easy as possible
Develop open source software with GNU
General Public Licensing (It’s free).
DiGIR
12
DiGIR Component Summary
DiGIR
13
DiGIR Architecture
Provider
DiGIR
14
Provider





Receives requests
Retrieves data from
database
Sends results to requestor
Supplies metadata to
describe content,
contacts, and capabilities
Logs requests
DiGIR
15
DiGIR Architecture
Portal
Engine
DiGIR
16
Portal Engine




The entry point for an
application
Can query a registry to
discover potential providers
Can determine, based on
provider metadata, whether a
provider should be queried
Can send requests to multiple
providers
DiGIR
17
Portal Engine, continued




Assembles responses from
providers
Returns packaged results to
the requesting application
Communicates via protocol
compliant messaging only
Logs activity
DiGIR
18
Registry




Provides a “yellow pages” to
advertise the existence and
capabilities of a provider
Provides a means to discover
potential providers of interest
May be public or private
Need not be a part of the
architecture
DiGIR
19
DiGIR Architecture
Provider
Registry
(register)
DiGIR
20
DiGIR Architecture
Portal
Engine
Registry (discover)
DiGIR
21
DiGIR Protocol

Defines request and response message
formats for communication between
provider, portal engine, and applications




Metadata requests
Search requests
Inventory requests
Remains unfettered by the structure of the
data it transfers
DiGIR
22
DiGIR Architecture
Application
DiGIR
23
DiGIR Architecture
Application
Protocol
(request)
Portal Engine
DiGIR
24
DiGIR Architecture
Application
Protocol
(request)
Portal Engine
Protocol (request)
Provider
DiGIR
25
DiGIR Architecture
Application
Protocol
(request)
Portal Engine
Protocol (response)
Provider
DiGIR
26
DiGIR Architecture
Application
Protocol
(response)
Portal Engine
DiGIR
27
Applications




Must be able to assemble and send
a request document to a portal
Must be able to receive and
interpret a response document from
the portal
Must do something incredibly
useful and interesting with the data
This is where the real fun is!
DiGIR
28
Hot topics – Interesting problems







Persistent unique identifiers
Web services (converters, translators, calculators,
transformation services)
Data validation (taxonomic and geographic thesauri,
ecological niche modeling, expedition analysis, outlier
detection,…)
Spatial query interfaces
Data quality feedback mechanisms
Automated georeferencing, event gazetteers
Concept libraries, schema extensions, schema libraries,
federated ontologies
DiGIR
29
MaNIS – The Mammal Networked
Information System
It’s more than just a pangolin…
DiGIR
30
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
UMNH-MaNIS
Presentation
Layer
MVZ-MaNIS
Presentation
Layer
UWBM-MaNIS
Presentation
Layer
DiGIR
UAM
Oracle
Database
DiGIR
Provider
31
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
UMNH-MaNIS
Presentation
Layer
MVZ-MaNIS
Presentation
Layer
UWBM-MaNIS
Presentation
Layer
DiGIR
UAM
Oracle
Database
DiGIR
Provider
32
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
UMNH-MaNIS
Presentation
Layer
MVZ-MaNIS
Presentation
Layer
UWBM-MaNIS
Presentation
Layer
DiGIR
UAM
Oracle
Database
DiGIR
Provider
33
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
UMNH-MaNIS
Presentation
Layer
MVZ-MaNIS
Presentation
Layer
UWBM-MaNIS
Presentation
Layer
DiGIR
UAM
Oracle
Database
DiGIR
Provider
34
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
UMNH-MaNIS
Presentation
Layer
MVZ-MaNIS
Presentation
Layer
UWBM-MaNIS
Presentation
Layer
DiGIR
UAM
Oracle
Database
DiGIR
Provider
35
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
UMNH-MaNIS
Presentation
Layer
MVZ-MaNIS
Presentation
Layer
UWBM-MaNIS
Presentation
Layer
DiGIR
UAM
Oracle
Database
DiGIR
Provider
36
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
UMNH-MaNIS
Presentation
Layer
MVZ-MaNIS
Presentation
Layer
UWBM-MaNIS
Presentation
Layer
DiGIR
UAM
Oracle
Database
DiGIR
Provider
37
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
UMNH-MaNIS
Presentation
Layer
MVZ-MaNIS
Presentation
Layer
UWBM-MaNIS
Presentation
Layer
DiGIR
UAM
Oracle
Database
DiGIR
Provider
38
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
MaNIS
DiGIR
Portal
MVZ-MaNIS
Presentation
Layer
DiGIR
39
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
MaNIS
DiGIR
Portal
MVZ-MaNIS
Presentation
Layer
DiGIR
40
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
MaNIS
DiGIR
Portal
MVZ-MaNIS
Presentation
Layer
DiGIR
41
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
MaNIS
DiGIR
Portal
MVZ-MaNIS
Presentation
Layer
DiGIR
42
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
MaNIS
DiGIR
Portal
MVZ-MaNIS
Presentation
Layer
DiGIR
43
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
MaNIS
DiGIR
Portal
MVZ-MaNIS
Presentation
Layer
DiGIR
44
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
MaNIS
DiGIR
Portal
MVZ-MaNIS
Presentation
Layer
DiGIR
45
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
MaNIS
DiGIR
Portal
MVZ-MaNIS
Presentation
Layer
DiGIR
46
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
MaNIS
DiGIR
Portal
MVZ-MaNIS
Presentation
Layer
DiGIR
47
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
MaNIS
DiGIR
Portal
MVZ-MaNIS
Presentation
Layer
DiGIR
48
MaNIS Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
MaNIS
DiGIR
Portal
MVZ-MaNIS
Presentation
Layer
DiGIR
49
MaNIS Network Configuration
…
…
…
…
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
MaNIS
DiGIR
Portal
UMNH-MaNIS
Presentation
Layer
MVZ-MaNIS
Presentation
Layer
UWBM-MaNIS
Presentation
Layer
DiGIR
UAM
Oracle
Database
DiGIR
Provider
…
…
…
…
50
“CalNet” Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
CalNet
DiGIR
Portal
DiGIR
51
Event Gazetteer Network
Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
CalNet
DiGIR
Portal
BioGeomancer
Web
Service
DiGIR
52
NBII Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
NBII
DiGIR
Portal
DiGIR
53
GBIF Network Configuration
CAS
SQL Server
Database
DiGIR
Provider
LACM
MS Access
Database
MVZ
Sybase
Database
LSUMZ
4D-Mac
Database
Online
MS Access
Database
Online
MS Access
Database
Online
MS Access
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Provider
UAM
Oracle
Database
DiGIR
Provider
NBII
DiGIR
Portal
GBIF
Presentation
Layers
DiGIR
54
Intra-Network Configuration
(BNMH)
Essig
Working
Database
PHMA
Working
Database
UCBG
Working
Database
UCJEPS
Working
Database
UCMP
Working
Databases (4)
Online
Database
Online
Database
Online
Database
Online
Database
Online
Database
DiGIR
Provider
BNHM
DiGIR
Portal
BNHM
Presentation
Layer
DiGIR
55
Other Network Configurations
Working
Database
Working
Database
Online
Database
Online
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Portal
Working
Database
DiGIR
Provider
DiGIR
Portal
DiGIR
Working
Database
Working
Database
Online
Database
Online
Database
DiGIR
Provider
DiGIR
Portal
56
Other Network Configurations
Working
Database
Working
Database
Online
Database
Online
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Portal
Working
Database
DiGIR
Provider
DiGIR
Portal
DiGIR
Working
Database
Working
Database
Online
Database
Online
Database
DiGIR
Provider
DiGIR
Portal
57
Other Network Configurations
Working
Database
Working
Database
Online
Database
Online
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Portal
Working
Database
DiGIR
Provider
DiGIR
Portal
DiGIR
Working
Database
Working
Database
Online
Database
Online
Database
DiGIR
Provider
DiGIR
Portal
58
Other Network Configurations
Working
Database
Working
Database
Online
Database
Online
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Portal
Working
Database
DiGIR
Provider
DiGIR
Portal
DiGIR
Working
Database
Working
Database
Online
Database
Online
Database
DiGIR
Provider
DiGIR
Portal
59
Other Network Configurations
Working
Database
Working
Database
Online
Database
Online
Database
DiGIR
Provider
DiGIR
Provider
DiGIR
Portal
Working
Database
DiGIR
Provider
DiGIR
Portal
DiGIR
Working
Database
Working
Database
Online
Database
Online
Database
DiGIR
Provider
DiGIR
Portal
60
Other Network Configurations
Working
Database
Working
Database
Working
Database
Online
Database
DiGIR
Provider
DiGIR
Portal
DiGIR
61
Project Information



DiGIR is a collaborative open source
development project on SourceForge
(https://sourceforge.net/projects/digir).
Software and documentation are available on the
DiGIR web site (http://digir.net).
MaNIS is an international network collaboration
among mammal specimen collections
(http://elib.cs.berkeley.edu/manis).
DiGIR
62
Hot topics – Interesting problems







Persistent unique identifiers
Web services (converters, translators, calculators,
transformation services)
Data validation (taxonomic and geographic thesauri,
ecological niche modeling, expedition analysis, outlier
detection,…)
Spatial query interfaces
Data quality feedback mechanisms
Automated georeferencing, event gazetteers
Concept libraries, schema extensions, schema libraries,
federated ontologies
DiGIR
63