Download sccc_presentation_JCothran

Open Solutions to Regional
Observing Systems
Outline
Recent near real-time in-situ observations are aggregated to a
‘Xenia’ schema relational database(RDB) via its import XML
format ‘ObsKML’ schema. From this database of recent
observations a variety of file formats, web services and
applications can be driven. By suggesting a minimally
common observation oriented XML and RDB schema,
developed scripts and products can benefit from and build
around these shared schema. For more info contact
[email protected]
Keywords
Scripting languages Perl PHP
Relational Database Sqlite Postgres
Mapping Engine/Interface Mapserver OpenLayers TileCache
OpenGeospatial Consortium(OGC) services WMS/WFS/SOS
Google Earth/Maps KML/KMZ
Google Code/Analytics
Problems Xenia intended to address
• Grants for research instrumentation which will be collecting
observation data while lacking a data management/sharing
component beyond basic file storage
• Low-volume data(< 100,000 records per hour) in-situ
observational platforms or system arrays (e.g. 1 to 1000
platforms collecting 10-20 observations per hour) collecting
data at any geographic scale (local,regional,national,etc)
• Bridging the gap between raw data collection and the
organization and sharing of data using previously developed
products, services and standards(leveraging earlier work
against new data providers)
• Fostering a standardization of products and services via a
common openly shared technical infrastructure(common
database schema and product support scripts)
Problems Xenia not intended to address
High-volume data (millions of records per
hour) such as gridded model outputs, hf
radar, etc. High-volume data problems at
this time are better addressed using
traditional file processing techniques
where data management can suggest
output file formats(such as images,
shapefiles, etc) and metadata that are
conducive to search and usage needs.
Xenia Relational Database Schema
Hierarchy = organization->platform->sensor->multi_obs(observation data)
ObsKML XML schema for import/export
Data flows – general
Data flows – ObsKML
ObsKML (Observations KML)
A simple XML encoding of
observation metadata
associated with a KML
Placemark.
Default XML import/export
format for Xenia database
Instances.
Postgres or Sqlite database paths
Xenia data organized by time interval
New Data
Latest
past several hours
Julian weekly divided database files(sqlite) of all
observations are available
http://tinyurl.com/6ctz66
Recent
0-6 weeks
Archival
3+ weeks to 1-2 years
Possibly table separated by year,month,etc
Archival file
1-2+ years
Files separated by product/year/month
Xenia aggregation, replication, redundancy
Overlapping/redundant systems could aggregate/replicate via
SQL or XML
Xenia
A,B,C,D,E,F
Xenia
A,B,C
Xenia
A
Xenia Backup
A,B,C,D,E,F
Xenia
D,E,F
Xenia
B
Xenia
C
Xenia Backup
D,E,F
Xenia
D
Xenia
E
Xenia
F
Flow/event monitoring
Around 10,000 observations are collected each hour from a variety of data
providers. The data flows are monitored for both successful import and
towards provided report metrics or notifications when problems or
events occur.
Sensor count graph showing
the hourly number of
successful sensor imports
per provider
More detailed
platform/observation table
showing daily counts color
coded to highlight specific
sensor issues
Applications – National Weather Service
Latest regional platform observations from Xenia database are styled to
html tables and presented in fixed map interface alongside NWS
forecast warnings/advisories and other map layers of interest
http://forecast.weather.gov/mwp/
Applications – SECOORA interactive map
Latest regional observation maps are accessed via WMS(Web Mapping
Service) and merged into an OpenLayers(browser javascript) map
interface. Observation database requests are supported via WMS
GFI(Get Feature Info)
http://secoora.org/maps/dynamic
Applications - CarolinasRCOOS
Map image layers supported via MapServer WMS with increased map
response time via TileCache which tiles and caches image requests for
repeated reference. Site feature info supported via Xenia database
instance styled output html table accessed via WMS GFI.
http://carolinasrcoos.org
KML/KMZ – Latest obs by platform
KML (Keyhole Markup Language) which is the XML format used to visualize data
in Google Earth/Maps and potentially other globes/maps with KML support
such as NASA WorldWind and ESRI ArcExplorer
Latest observation data organized by all observations per platform
http://tinyurl.com/664wtx
KML/KMZ – latest obs by obsType
KML (Keyhole Markup Language) which is the XML format used to visualize data
in Google Earth/Maps and potentially other globes/maps with KML support
such as NASA WorldWind and ESRI ArcExplorer
Latest observation data organized by observation type for all platforms
http://tinyurl.com/664wtx
All observations carry
* observation type(obsType)
* unit of measure(uom)
* measurement value
obsType+uom = measurement
type(m_type)
Color styled low to high value
blue/green/red
Development/documentation support
Google Code provides very simple, free browser based access to the
basic project support tools such as a subversion(svn)
code/documentation sharing/versioning, wiki documentation and issue
tickets.
Google Analytics provides free browser based access to website usage
statistics via a javascript project token included in webpage access.
Lessons Learned
• Web GIS interface: php/mapscript -> openlayers/javascript/ajax,
tilecache
• Latest in-situ obs: RDB – sensor/obs hard-coded/table approach -> obs
non-specific infrastructure(xenia,obskml)
• Sensor/observation specific metadata: loosely link a variety of more
specific metadata files/schemas to common infrastructure
• Hourly gridded data: RDB quickscat, model layers ->
ogr/sql+shapefiles
• Full RDB to file-based RDB: postgresql+postgis -> sqlite+ogr,
sqlitegeo?
• Self hosted to third party hosted: third party website hosting for static
content/context, common processing/storage. Google code/analytics
for basic shared development support/documentation.
• ROA(Resource Oriented Architecture)/REST style simpler declarative
alternatives to complex query SOA(Service Oriented Architecture)
File formats and web services
• File formats (CSV,shapefile,netCDF,KML/KMZ,Sqlite DB)
http://secoora.org/data
Free desktop analysis tools
MapWindows(shapefile/GIS) http://www.mapwindow.com
ncBrowse(netCDF)
http://www.epic.noaa.gov/java/ncBrowse
ODV(ctd/csv) http://odv.awi.de
Sqlite DB http://sqlite.org
•
Web services (OGC WMS/WFS/SOS)
http://secoora.org/data
http://code.google.com/p/xenia/wiki/XeniaSOS
Credits
2C.Calloway,2J.Cleary,1J.Cothran,4J.Donovan,
3J.Dorton,1M.Fletcher,C.Galvarino,1S.King,
2S.Haines,3L.Leonard,1D.Porter,3X.Qi,
1D.Ramage,2H.Seim,4V.Subramanian,1S.Walker,4
R.Weisberg
1University
of South Carolina, 2University of North
Carolina at Chapel Hill, 3University of North Carolina at
Wilmington, 4University of South Florida