Download The Problem: Integrating Data, Applications, and Client Devices

The Problem: Integrating Data,
Applications, and Client Devices



The key issue we try to solve is building the distributed computing infrastructure
that can connect
• Legacy data archives
• Executable codes
• Real time data sources
• Client tools for collaboration
 Audio/Video systems, whiteboard annotators, etc
Various application-specific grids can be built out of the common infrastructure
• Science Grids
• Emergency planning, crisis response
We choose certain fixed points for our foundations
• Other standards where available: GIS standards
• Web Service standards: SOAP and WSDL
• Universal messaging substrate for SOAP and other messages:
http://www.naradabrokering
OGC Standards for GIS


OpenGIS Consortium is an international group
that defines common formats for geospatial data
and service standards.
Main data format language is the XML-based
Geography Markup Language (GML).
• GML is widely accepted as the universal encoding for
geo-referenced data.

OGC defines services.
• Some example OGC services include



Web Feature Service (WFS): for retrieving GML encoded
features, like faults, roads, county boundaries, GPS station
locations,….
Web Map Service (WMS): for creating maps out of Web
Features
http://www.opengeospatial.org
What Are Web Services?
Web Services framework is a way
for doing distributed computing
with XML.
•
•
•


XML provides cross-language
support
Suitable for both human and
application clients.
We are building data grid
components in the same fashion.
•

Allows us to build general purpose
client environments like Web
portals and command line tools.
http://www.w3.org/2002/ws
Browser
Appl
Web
Server
WSDL
SOAP
WSDL
SOAP
Web
Server
WSDL

WSDL: Defines interfaces to
functions of remote components.
This is the programming interface.
SOAP: Defines the message format
that you exchange between
components. This carries the
messages over the network.
Web Services are not web pages,
CGI, servlets, applets.
WSDL

JDBC
DB
Service Oriented Architecture

Key SOA Features
(Following W3C’s WS Architecture working group.)
•
SOAs are composed of services that present
programmatic access to resources to remote
client applications.
•
Services communicate using messages.
Messages are usually encoded using SOAP.
•
SOAs are metadata rich. We must describe
service interfaces, provide descriptions of
services and publish them into registries.
Service Oriented Architecture
Service providers
- GetCapabilities
- GetFeature
- DescribeFeature
- WSDL URL
- NB Endpoint
WFS
Publish
Bind
- GetCapabilities
- GetMap
- GetFeatureInfo
Find
Information
Service
`
Web Map Client
WMS
Service requestors
Service registries
Publish/Find/Bind paradigm in a GIS Grid
NaradaBrokering (NB)

NaradaBrokering is a distributed messaging
infrastructure.

It may be thought of a as topic-based
publish/subscribe messaging system:
• interested entities can register to a NaradaBrokering
node to send and receive messages on particular topics.


It supports multiple underlying transport
protocols, intelligent message routing,
asynchronous messaging, reliable delivery, replay
of events, compression/fragmentation, discovery
etc.
http://www.naradabrokering.org
Pattern Informatics in a Grid
Environment

PI in a Grid environment:
• Hotspot forecasts are made using publicly available seismic records.
 Southern California Earthquake Data Center
 Advanced National Seismic System (ANSS) catalogs
• Code location is unimportant, can be a service through remote execution
• Results need to be stored, shared, modified
• Grid/Web Services can provide these capabilities

Problems:
• How do we provide programming interfaces (not just user interfaces) to
the above catalogs?
• How do we connect remote data sources directly to the PI code.
• How do we automate this for the entire planet?

Solutions:
• Use GIS services to provide the input data, plot the output data
• Use HPSearch tool to tie together and manage the distributed data sources
and code.
0
Extended
UDDI
5,11
3
WMS
1
WMS
Client
WFS
2
5,6: WMS starts a session, invokes HPSearch to
run workflow script for PI Code with a session id
4
http://..../..../..txt
7,8,9
6
HP Search
10
Data Filter
10: HPSearch writes the URI of the of the output
file into Context
PI Code
11: WMS polls the information from Context
Service
Data Filter
12: WMS retrieves the generated output file by
workflow script and generates a map
http://..../..../tmp.xml
Context
Information Service
7,8,9: HPSearch runs the workflow script and
generates output file in GML format (& PDF
Format) as result
<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:soap="http://www.w3...">
<soap:Header encodingStyle=“WSCTX URL"
mustUnderstand="true">
<context xmlns=“ctxt schema“ timeout="100">
<context-id>http..</context-id>
<context
xsd:type="ContextType"timeout=“100">
<context-service>
http..based
</context-service>
<context-service>http://.../HPSearch</
context-service>
The Pattern
Informatics
GIS-SOA
workflow
application
<context
<context
xsd:type="ContextType"timeout=“100">
xsd:type="ContextType"timeout=“100">
<context-manager> http.. </context-service>
<parent-context>http://../abcdef:012345<parent-context/>
<context-id>http://../abcdef:012345<context-id/>
<context-service>http://.../HPSearch</
<context-service>http://.../WMS</
context-service>
context<activity-list
<content>
shared
data
for
HPSearch
activity
</content>
service>
activity
<context-service>http://.../HPSearch</
context<activity-list
mustUnderstand="true"
mustUnderstand="true"
mustPropagate="true">
<activity-list
mustUnderstand="true"
mustPropagate="true">
<parent-context>http://../abcdef:012345<parentservice> HPSearch associated additional data
user
profile
<content>
mustPropagate="true">
shared
service
session
state
<p-service>http://../WMS</p-service>
context/>
<service>http://.../DataFilter1</service>
generated
during execution of workflow. </content>
associated
<p-service>http://../HPSearch</p-service>
SOAP header
<service>http://.../WMS</service>
<content>
profile information related WMS
<service>http://.../PICode</service>
<content>http://danube.ucs.indiana.edu:8080\x.xml</c
</activity-list>
</context>
</content>
<service>http://.../HPSearch</service>
<service>http://.../DataFilter2</service>
for Context
ontent>
</context>
</context>
</activity-list>
</soap:Header>
</activity-list>
</context>
...
</context>
</context>
The Interdependent Energy Infrastructure Simulation
System (IEISS) in a Grid Environment

IEISS in a Grid environment:
• IEISS provides modeling, simulation and analysis of interdependent
energy infrastructures. Input is data models in XML format.
 Los Alamos National Laboratory
 Argon National Laboratory
• Code location is unimportant, can be a service through remote execution
• Results need to be stored, shared, annotated
• Grid/Web Services can provide these capabilities

Problems:
• How do we provide programming interfaces (not just user interfaces) to
the input data models?
• How do we connect remote data sources directly to the IEISS code?

Solutions:
• Use a SOA with GIS services to provide the input data, plot the output
data
• Use WS-I compatible XML metadata services to support GIS workflow
applications by providing/maintaining static and dynamic information
4-5
- WFS
publishes
the
as
GML
document
into a topic
WFS
1-2-3
6
- User
and
- WMS
invokes
WMS
Client
publish
IEISS
->results
through
WMS
their
Server
WSDL
WMSFeatureCollection
->
Client
URL
UDDI
to
interface
->
the
WFS
UDDI
for the
Registry
obtained
(“/NISAC/WFS”)
in a pub/sub
based
messaging
WFS session
-> WMS in
Server
geospatial features,
and WMS
Client
starts system.
a workflow
the
(creates
map overlay) and IEISS receive this GML document. WMS Server ->
Context aService.
WMS Client (displays it)
7 - On receiving invocation message, IEISS updates the shared state data
to be “IEISS_IS_IN_PROGRES”. IEISS runs and produces an ESRI
Shape file and then invokes shp2gml tool to convert produced Shape
file to GML format. After the conversion IEISS updates shared session
state to be “IEISS_COMPLETED”. As the state changes, the Context
Service notifies all interested workflow entities such as WMS Client.
9-10
- WFS-L
publishes
the IEISSWMS
output
as amakes
GML FeatureCollection
8 – On
receiving
the notification,
Client
a request to the
document
topicoutput
‘NISAC/WFS-L’. WMS Server is subscribed to
WFS-L for to
theNB
IEISS
this topic and receives the GML file then converts it to map overlay,
and the Client displays the new model on the map.
Some Challenges





Security: These applications have interesting security requirements
• Authentication, authorization and priority for different users
• Time/event dependent security for crisis response
Performance: Are GIS services suitable for non-trivial data transfers?
• We are using streaming data transfers for higher performance
• We research fast data transfer methodologies for higher performance
Fault tolerance and scalability: We must support SOA based distributed
application with reliable information sources.
• We design and build fault tolerant and high performance decentralized
Information Services to support metadata requirements of SOA.
Adoption: We must get the tools and services to the point where science
application developers want to use them early in the development process
rather than later.
• Web Service client libraries to remote GIS data
• Develop codes to work with data streams rather than files.
Extensibility: This architecture can be enhanced by integrating various
types of distributed data sources that can provide geospatial data such as
MySQL, Oracle Spatial, ESRI SDE, Shape files etc.
Location of information and software for
dependent Grid Projects

htpp://www.naradabrokering.org provides Web service
(and JMS) compliant distributed publish-subscribe
messaging (software overlay network)

http://www.crisisgrid.org is an OGC (open geospatial
consortium) Geographical Information System (GIS)
compliant GIS and Sensor Grid.

http://www.opengrids.org has WS-Context, Extended
UDDI etc.