Download SDMX Sandbox exercise

Restricted
Building a Data Portal with SDMX
The BIS SDMX Sandbox exercise
Gabriele Becker, Massimo Bruschi
Bank for International Settlements
OECD SDMX Experts Meeting
13/14 September 2012
1
Restricted
The SDMX vision
 Users need …
• good quality data, up-to-date numbers, documentation
• Single access point for data from different sources
• User interfaces: GUI and also a Web service (for automation)
 The SDMX vision …
• Data providers (originators) offer their data “in SDMX”
• Dissemination = reporting = data sharing … from a single storage
• SDMX registries help users to find data via a GUI and an SDMX
Webservice
 The BIS SDMX Sandbox exercise
• How “real” is this SDMX vision?
• How does it work with difference scenarios?
2
The Sandbox setup “in the cloud”
Restricted
 14 participating central banks: 14 Sandboxes with
• SDMX registry with user interface and SDMX web service
• SDMX data base with data browser and SDMX web service
• Space to host SDMX data files
• User interface for loading files, performing registrations etc.
• Detailed tutorials
 1 Sandbox for the BIS to experiment and participate “as central bank”
 The Data Portal (Unified Data Catalogue or “UDC”)
• “central” SDMX registry operated by the BIS
• Data browser
• Synchronises with the central and CB registries
• Accesses registered data sources
 SDMX Sandbox was developed by Metadata Technology
3
SDMX Sandbox setup
14 Sandboxes for central banks, 1 for BIS as central agency, 1 Data portal (Unified Data Catalogue)
Another CB
Sandbox
Another CB
Sandbox
Another CB
Sandbox
CB SDMX
Registy
Data browser
Another CB
Sandbox
CB SDMX
Database
SDMX Sandbox
Country A
SDMX
Registy
Another CB
Sandbox
Another CB
Sandbox
Another CB
Sandbox
Data browser
SDMX
Database
BIS SDMX
Sandbox
UDC
SDMX
Registy
Data Portal:
UDC
Unified Data
Catalogue
Browser
Restricted
Scenarios
1.
Data reporting against a “global” SDMX DSD, eg BOP
2.
Making internal data available via a “data portal”
3.
Bringing external (public) data to your users via a data
portal
 Common aspects
• data is available in SDMX files in a known DSD or …
• SDMX web service offers data in a known DSD
• We just need to know where the data is and what DSD
is follows
• An SDMX Registry is the place for this information
• A simple data browser can offer data selection based
on the known DSDs
5
Restricted
Scenario 3: “making public data available in a data portal”
 ECB publishes a lot of data as SDMX files on its
website
 IMF and BIS offer the Joint Ext. Debt Hub (JEDH)
contributions as SDMX files
 Users may want to see this data via a common data
portal
 Organisations may wish to access this data via a
SDMX 2.1 web service (that ECB, IMF and BIS
currently do NOT offer)
demo
 Load SDMX DSD into registry
demo
• In future: SDMX registries provide this as a service …
 Define Dataflow, Provision agreement and Categorisation
 Register the data files as sources for the UDC
demo
6
Restricted
Demo Movie
7
Restricted
Conclusions I
 Building blocks are working in the SDMX Sandbox
• SDMX registry and SDMX 2.1 webservice
• SDMX data base (with webservice)
• Data browser
• Unified Data Catalogue (data portal)
 Connecting different data sources …
• SDMX files from public websites
• SDMX files from other (accessible) locations
• Different SDMX versions (1.0, 2.0, 2.1)
• SDMX databases (internal and external)
• SDMX Web service
 First implementation of a federated SDMX registry
9
Restricted
Conclusions II
 Harmonisation of data access techniques was achieved…
• User sees all data through one “data portal” (the UDC)
• SDMX 2.1 web service for all data, also for those offered as
SDMX 1.0 files
• Web service supports automated data retrieval
 Works best with harmonised data structures!
• BOP example
• Key deliverable for SDMX and domain groups for 2012: DSD
for global use for BOP and National Accounts
 The SDMX Vision is real!
10
Restricted
• SDMX Sandbox is a good tool for promoting SDMX
• Tutorials, very intuitive and “hands-on”
• BOP SDMX Portal based on the BIS SDMX Sandbox experience
• Input for SDMX Technical Working Group
• Other SDMX sponsors to use it with their constituencies to promote
SDMX?
• Any other ideas?
11
Restricted
Thank you !
12
Restricted
Background slides
 Scenario 1 and 2
13
Restricted
Scenario I: “BOP” (Balance of payments)
 National agencies have to “report” their data
based on a commonly agreed data structure
 This data needs to be available at international
organisations and for the public
 This is a real life scenario for BOP, National
Accounts and other data domains.
demo
 SDMX BOP data structure “for global use” is a prerequisite
• We invented a simplified one for the purpose of this exercise …
14
SDMX Sandbox : BOP example
·
CB DB
admin
SDMX-ML file
with Central
bank BOP
data
CB data provider
Import
·
·
Central
bank BOP
data
Store on Web location
SDMX
Database
Data browser
BOP
DSD
·
Data browser
CB SDMX
Database
SDMX-ML file
with Central
bank BOP
data
·
Register as
queryable
data source
BOP
DSD
Another CB
Sandbox
CB SDMX
Registy
Another CB
Sandbox
SDMX Sandbox
Country A
Another CB
Sandbox
Another CB
Sandbox
Register file as data source
SDMX
Registy
Restricted
Data manager registers data file in UDC
Registry
SDMX data file is uploaded to a Web
location and then registered as data
source against the BOP provision
agreement
It’s immediately visible to a UDC user
Date manager imports data in the
Sandbox DB
It’s immediately visible to a user of the
Sandbox data browser
Registry synchronisation
Retrieve data from DB or file
BIS SDMX
Sandbox
provider action
user action
Registry sychronisation
BOP data flows,
provision
BOP
agreements,
DSD
data source,
registrations
UDC SDMX Registy
Retrieve
DSD info
for browsing,
registrations
for data
location
Uses UDC
Unified Data
Catalogue
UDC user can see BOP data
from all participating central banks
System interaction
Data Portal:
UDC
15
Restricted
Scenario 2: “dissemination database for BIS”
 BIS wishes to disseminate data to internal
users via a common data portal (example: BIS
Banking Statistics)
 BIS has/defines SDMX DSD for this data
demo
 BIS uses the SDMX database in the Sandbox
as “dissemination database” and loads the data
via SDMX files
demo
 Internal users use the data browser (similar to
UDC) to access the “BIS SDMX database”.
demo
 For external dissemination the BIS registers its Sandbox
database as a data source to the UDC
demo
 This also enables the UDC search service for this data
demo
16
SDMX Sandbox : dissemination data base example
·
·
·
·
·
·
Data manager creates or loads a DSD
into the registry
defines data flow and provision
agreement
loads a data file in the SDMX Database.
It is immediately visible to a user of the
Data browser attached to the Sandbox.
Data manager registers SDMX DB as
data source for the UDC.
Store on Web location
UDC users can see the data
UDC user
Uses UDC
SDMX-ML file
data for DSD
Retrieves data
from SDMX database
SDMX-ML file
data for DSD
Register as queryable data source
BIS data provider
Create DSD,
data flow etc.
Retrieves DSD info
for browsing
and registrations
for data location
SDMX
Database
with SDMX
Webservice
Load
UDC Browser
Register as
queryable
data source
Retrieves data
from SDMX database
UDC Registryy
SDMX Registry
Use data
browser
Data browser
DSD, Data flows,
Provision agreements
and data source
registrations
Registry
synchronisation
DSD, Data flows,
Provision agreements
and data source
registrations
BIS data user
BIS SDMX Sandbox
provider action
user action
System interaction
UDC: Data Portal