Download CUAHSIHIS2009

Development of a
Community Hydrologic
Information System
David Maidment (PI), David Tarboton,
Dan Ames, Jon Goodall, Richard Hooper,
and Ilya Zaslavsky
CUAHSI
HIS
http://his.cuahsi.org/
Sharing hydrologic data
Support
EAR 0622374
The way that data is organized can
enhance or inhibit the analysis that
can be done
I have your information
right here …
Picture from: http://initsspace.com/
Data Models
(x1,y1)
What are the basic attributes to be associated with
hydrologic data values and how can these best be
organized?
Streamflow
Precipitation
& Climate
Water Quality
Groundwater
levels
Soil moisture
data
Flux tower
data
What is HIS
• The CUAHSI Hydrologic Information System
(HIS) provides web services, tools, standards
and procedures that enhance access to more
and better data for hydrologic analysis.
http://his.cuahsi.org
Water quantity Water quality
Precipitation
Meteorology
Consortium of Universities for the
Advancement of Hydrologic Science, Inc.
• 110 US University
members
• 6 affiliate members
• 12 International
affiliate members
(as of March 2009)
An organization representing more than one hundred
United States universities, receives support from the
National Science Foundation to develop infrastructure
and services for the advancement of hydrologic
science and education in the U.S.
http://www.cuahsi.org/
HIS Team and Collaborators
• University of Texas at Austin – David Maidment (PI), Tim Whiteaker,
Ernest To, Bryan Enslein, Kate Marney
• San Diego Supercomputer Center – Ilya Zaslavsky, David Valentine,
Tom Whitenack
• Utah State University – David Tarboton, Jeff Horsburgh, Kim
Schreuders, Justin Berger
• Drexel University – Michael Piasecki
• University of South Carolina – Jon Goodall, Tony Castronova
• Idaho State University – Dan Ames, Jiří Kadlec, Ted Dunsford, Teva
Veluppillai
• CUAHSI Program Office – Rick Hooper, David Kirschtel, Conrad
Matiuk, Yoori Choi
• WATERS Network – Testbed Data Managers
• HIS Standing Committee
• USGS – Bob Hirsch, David Briar, Scott McFarlane
• NCDC – Rich Baldwin
Data Searching – What we used to have to do
Searching each data source separately
NWIS
request
return
request
return
request
return
NAWQA
request
return
NAM-12
request
return
request
return
request
return
request
return
NARR
Michael Piasecki
Drexel University
What HIS enables
Searching all data sources
collectively
GetValues
GetValues
NWIS
GetValues
generic
request
GetValues
GetValues
GetValues
NAWQA
Michael Piasecki
Drexel University
GetValues
GetValues
NARR
ODM
Services-Oriented Architecture for Water Data
HIS Central
Data Discovery and
Integration platform
Data Services
Data Publication
platform
HIS Server
Water Data Services
Spatial Data Services
Data Synthesis and
Research platform
HydroDesktop
CUAHSI Water Data Services System
Analysis
Query, Visualize, and Edit
data using ODM Tools
Streaming
Data
Loader
Base Station
Computer(s)
Telemetry
Network
Sensors
Hydroseek
GIS
Matlab
Splus
R
IDL
Java
C++
VB
WaterML
ODM
Database
Excel,
text
ODM
Data
Loader
Discovery
Access
USGS
NWIS
EPA
STORET
GetSites
GetSiteInfo
GetVariableInfo
GetValues
Hydro
Desktop
HydroExcel
Service Registry Hydrotagger
Harvester
WaterOneFlow
Web Service
NCDC
HIS Central
Water Metadata
Catalog
Others
WaterML and WaterOneFlow
WaterML is an XML language for communicating water data
WaterOneFlow is a set of web services based on WaterML
• Set of query functions
• Returns data in WaterML
Slide from David Valentine
GetVariables
GetValues
qualifier
ISO Time
value
Direct analysis from your favorite analysis
environment. e.g. Matlab
% create NWIS Class and an instance of the class
createClassFromWsdl('http://river.sdsc.edu/wateroneflow
/NWIS/DailyValues.asmx?WSDL');
WS = NWISDailyValues;
% GetValues to get the data
siteid='NWIS:02087500';
bdate='2002-09-30T00:00:00';
edate='2006-10-16T00:00:00';
variable='NWIS:00060';
valuesxml=GetValues(WS,siteid,variable,bdate,edate,'');
2.5
x 10
4
Daily Discharge NEUSE RIVER NEAR CLAYTON, NC
2
cfs
1.5
1
0.5
0
1920
1930
1940
1950
1960
1970
1980
1990
2000
2010
What are the basic attributes to be associated with each
single data value and how can these best be organized?
“When” Time, T
t
A data value
vi (s,t)
s “Where”
Space, S
Vi
“What”
Variables, V
Streamflow
Precipitation
& Climate
Water Quality
Groundwater
levels
Soil
moisture
data
Flux tower
data
Observations Data Model (ODM)
• A relational database at the single observation level (atomic model)
• Stores observation data made at points
• Metadata for unambiguous interpretation
• Traceable heritage from raw measurements to usable information
• Standard format for data sharing
• Cross dimension retrieval and analysis
CUAHSI Observations Data Model http://his.cuahsi.org/odmdatabases.html
Horsburgh, J. S., D. G. Tarboton, D. R. Maidment and I. Zaslavsky, (2008), A Relational Model for
Environmental and Water Resources Data, Water Resour. Res., 44: W05406, doi:10.1029/2007WR006392.
Stage and Streamflow Example
Lets explore the system
http://his.cuahsi.org
Hydroseek
Bora Beran, Drexel
http://www.hydroseek.org
Supports search by location and type of data across multiple observation
networks including NWIS and Storet
HydroExcel
http://his.cuahsi.org/hydroexcel.html
HIS Server
• Internet Map Server
built using ArcGIS
• Web browser client
• Combine spatial data
and observational
data
• Self contained fully
documented with
local control of data
• Launch data
visualization tools
e.g. http://icewater.usu.edu/map and http://littlebearriver.usu.edu/
Loading data into ODM
ODM Data Loader
• Interactive ODM Data Loader
– Loads data from spreadsheets and
comma separated tables in simple
format
• Streaming Data Loader (SDL)
SDL
– Loads data from datalogger files on
a prescribed schedule
– Interactive configuration
• SQL Server Integration Services
(SSIS)
– Microsoft application accompanying
SQL Server useful for programming
complex loading or data
management functions
SSIS
Reducing Semantic Heterogeneity
• ODM Controlled
Vocabulary System
– ODM CV central database
– Online submission and editing of
CV terms
– Web services for broadcasting
CVs
Variable Name
Investigator 1:
Investigator 2:
Investigator 3:
Investigator 4:
“Temperature, water”
“Water Temperature”
“Temperature”
“Temp.”
ODM VariableNameCV
Term
…
Sunshine duration
Temperature
Turbidity
…
From Jeff Horsburgh
Dynamic controlled vocabulary moderation system
ODM Data
Manager
ODM
Website
ODM
Tools
XML
Local ODM
Database
Local
Server
ODM Controlled
Vocabulary Moderator
ODM
Controlled
Vocabulary
Web Services
http://his.cuahsi.org/mastercvreg.html
Master ODM
Controlled
Vocabulary
From Jeff Horsburgh
HydroDesktop
Harvesting and analyzing data from web services
GIS
Hydrology
• Add shapefiles to map
• Change symbology and
labels
• Print and export map
• GIS toolbox
•
•
•
•
Search for data
Download data
Display time series
Export data
http://hydrodesktop.codeplex.com
HydroDesktop Building blocks
HydroDesktop
Main Application
Web Service API
Metadata Cache
Database
Database Access Layer
Actual Data
Dababase
MapWindow
GIS
Components
HIS Server
Web Services
HIS Central
Web Service
Plug-in Interface
Metadata plug-in
Graph View plug-in
Search plug-in
26
Advancement of water science is critically
dependent on integration of water information
Models
Databases: Structured data sets to
facilitate data integrity and effective
sharing and analysis.
- Standards
ODM
- Metadata
- Unambiguous interpretation
Analysis: Tools to provide windows
into the database to support
visualization, queries, analysis, and
data driven discovery.
Web Services
Databases
Analysis
Models: Numerical implementations of
hydrologic theory to integrate process
understanding, test hypotheses and
provide hydrologic forecasts.
Conclusions
• Data Storage in an Observations Data Model (ODM)
• Data Delivery through internet-based Water Data
Services using a consistent data language, called
WaterML
• Data Discovery through federated thematic keyword
search services and a National Water Metadata
Catalog
• Data Access and Analysis through a Client
Application (HydroDesktop)
The combination of these capabilities creates a
common window on water observations data for the
United States unlike any that has existed before.
Reports
• 2008 status report
summarizes the conceptual
framework, methodology,
and application tools for HIS
• Shows how to develop and
publish a CUAHSI Water
Data Service
• 2009 status report details
the expansions and updates
http://his.cuahsi.org/documents/HISOverview2009.pdf
http://his.cuahsi.org