Download PPT 4.5M

Developing applications that harness the
Agricultural and Natural Resource Grid
(and linking your own data)
Matthew Laurenson and Seishi Ninomiya
National Agricultural Research Center
Tsukuba
Japan
Contents
• Concepts
–Remote Procedure Calls (RPC)
–Brokers
• Using a broker
–Metadata requests
–Data requests
–Issues
–Linking your own data
• SOAP
• New brokers and future
enhancements
Overall Goal: To help you make
“portable” software applications (DSS).
Current Situation: Web-based but…
• Applications closely coupled to databases
• Limited range of applications at each site
• Duplicated development at each site
Ag and NR Grid – a “Hotel Room” for
Agricultural and Natural Resource DSS
User interface in
national language
RoomDSS
DSS
Maps
DSS
Soils
Weather
Elevation
Connectors for local data
New Zealand
Map
s
Elevation
Soils
Weather
日本
地図
地面の高さ
地
気象
中国
土壤
交通图
海拔
气象
Goal of Ag and NR Grid:
To provide this “hotel room” in your
country, so you can use models developed
by others (and they can use your models).
Distributed Computing Concepts
• Remote Procedure Calls
• Brokers to handle heterogeneous databases
Remote Procedure Calls (RPC)
• RPCs let a program execute a procedure on
a remote computer as if it were local.
Remote Procedure Call
My Computer
remote=findRemote()
A=5
B=6
C=remote.add(A,B)
Print C
...
Remote Computer
procedure add(x,y)
return x+y
}
...
Remote Procedure Calls (RPC)
• RPCs let a program execute a procedure on
a remote computer as if it were local.
• Newer RPC mechanisms let programs send
and receive data objects.
• RPC mechanisms:
CORBA, .NET, Java RMI, SOAP...
• RMI – only Java programs can use
• SOAP – used by other languages as well
RPC vs Local Calls
• Latency (delay)
– RPCs much slower than local calls
– difficult to predict how long call will take
• May fail
– network problems
– problems with other computer (“server”)
• Security issues
– firewalls...
But...
• Can distribute processing and data over
multiple machines
• Great when multiple users must share the
same dynamic data item or communicate
with each other.
Mediated Architecture – “Brokers”
Consistent data access
(using RPC mechanism)
Applications
Broker
Domain
Databases
(Heterogeneous)
MetBroker
Weather
Databases
Applications
Web Browsers
Data Retrieval
Servlet
Data Retrieval
South Africa
Korea
Crop Growth
MetBroker
Japan
Disease Risk
Weather Risk
New Zealand
USA
Contents
• Concepts
–Remote Procedure Calls (RPC)
–Brokers
• Using a broker
–Metadata requests
–Data requests
–Issues
–Linking your own data
• SOAP
• New brokers and future
enhancements
Data and Metadata Requests
Interaction between applications and brokers
uses objects – data packages
Metadata
Standard
request
request
DB-specific
query
Broker
Application
Metadata
Standard
result
DB-specific
result
Metadata
Data
Database
(various)
Example - MetBroker Services
• Weather Data
– Single station
– All stations in a region
• Metadata (data about data)
– Database details (coverage of globe, login req.)
– Regions within databases
– Station details (latitude, longitude, start date,
end date, elements recorded, resolution...)
Elements of a Station Data Request
Weather station
Username/password
Time interval
Resolution
Required weather elements
Allow summarizing
Send Request to MetBroker and Receive
Result
To MetBroker
Data Result
Data Request
Tsuchiura
amedas
Operated 1976-present
40341
1999/1/3 06:00-1999/2/1 00:00
rain, air temperature
10.1, -0.2, 0
1999/1/1-1999/2/1
12.6, 3.8, 0
daily
11.3, -1.5, 0.3
OK to summarize
...

Query Results - Single Station
Station
& query details
Station details and
data sequences
are encapsulated
in a single
result object
rain
air temp. solar rad.
Spatial Request
Client
Broker
Temperature
Rainfall
Radiation
Spatial Request
Background map
dynamically retrieved from
http://www.mapblast.com
Wakayama
Rainfall DB
Daily max/min temperatures &
rainfall for all stations AMeDAS
in region
Query results - multiple stations
Using Brokers – Implementation Issues
RMI Blocked by Firewalls
Wrapper Servlet
Client
Application
Java
Client
Application
Java
HTTP
Client-side
firewalls RMI
RMI
Broker
Server Machine
Applet Using Multiple Brokers
DEMBroker
Elevation
data
MetBroker
Meteorological
data
Resource
Server
Spatial Risk
Applet
Application
text
ChizuBroker
Background
maps
Without Multiple Threads - Sequential
DEMBroker
Elevation
data
MetBroker
Meteorological
data
Resource
Server
Spatial Risk
Applet
Application
text
ChizuBroker
Background
maps
12 seconds
With Multiple Threads – Parallel
DEMBroker
Elevation
data
MetBroker
Meteorological
data
Resource
Server
Spatial Risk
Applet
Application
text
ChizuBroker
Background
maps
5 seconds
JavaBeans for Java GUI programs
• For rapid GUI development in Java can use
existing MetBeans
• Transparently handle firewalls, latency...
• Making a simple data retrieval application
takes about 30 minutes
• Used in many MetBroker demo applications
Easy Development of New Applications
Databases Linked Using “Drivers”
Databases
Applications
Broker
Linking a Database to a Broker
• Need to write one Java class (a “driver”) with
two or three main methods
• Typically use Java Database Connectivity
(JDBC) drivers to query relational databases
• Can reuse existing classes for Oracle,
Postgres, and MySQL JDBC.
• Drivers can read data from files via HTTP, or
parse HTML tables.
• “Test harness” available for driver
development
How much work?
• Add a new database to MetBroker
– roughly working - 3 days
– tested more thoroughly – 1 week
– (assuming that database is readily accessible)
• Make a MetBroker applet
– working prototype in one day
– (assuming basic science is “done”)
Developer Resources
• Demos, documentation, downloads, pictures
– http://www.agmodel.net
• Mailing lists – subscribe at web site
• Brokers are open source under the
GNU Lesser General Public License.
– Can be used in commercial applications
– Any improvements must be released under
same license
Reminders
• Linking your data to the grid lets you use
grid tools developed by others.
• It doesn’t mean you must share your data
with everyone (password protection).
• The key to interoperability is the interface
Contents
• Concepts
–Remote Procedure Calls (RPC)
–Brokers
• Using a broker
–Metadata requests
–Data requests
–Issues
–Linking your own data
• SOAP
• New brokers and future
enhancements
SOAP
• An XML-based RPC mechanism
• Independent of programming language
• SOAP toolkits for C++, Visual Basic,
Delphi, Java, ASP, PHP, Perl, Python…
• Firewall-friendly
• Basis of Microsoft .NET architecture
• A SOAP Web service is described in
standard format using a .WSDL file
(Web Services Description Language)
Example - MetSOAP
VB function calls
MS SOAP
Toolkit
Application
code
VB objects
SOAP over HTTP
Visual Basic client application
MetSOAP
Delphi function calls
Application
code
Borland SOAP
Toolkit
Delphi objects
Delphi client application
SOAP over HTTP
RMI
MetBroker
SOAP Summary
• Positives:
– Opens brokers to a range of programming languages
– Already the standard for Web-based RPC
– Avoids most firewall issues
• Negatives
– Can’t reuse Java classes - must “mirror” in each
language
– Interoperability is still imperfect (but improving fast)
– Inefficient for large data sets
Typical Array in SOAP
•
•
•
•
•
•
•
•
•
•
•
•
•
...
<item xsi:type="xsd:float">23.43</item>
<item xsi:type="xsd:float">23.43</item>
<item xsi:type="xsd:float">23.43</item>
<item xsi:type="xsd:float">21.79</item>
<item xsi:type="xsd:float">21.79</item>
<item xsi:type="xsd:float">23.3</item>
<item xsi:type="xsd:float">21.11</item>
<item xsi:type="xsd:float">19.49</item>
<item xsi:type="xsd:float">18.48</item>
<item xsi:type="xsd:float">16.52</item>
<item xsi:type="xsd:float">14.73</item>
<item xsi:type="xsd:float">14.73</item>
Model Deployment Options - Summary
• Java applet
– flexible, responsive once loaded but requires
plug-in
• Java servlet
– simple browsers, rapid load, but clunky
interface
• SOAP
– direct access for Delphi, VB…
• Existing models (FORTAN,C,BASIC)
– Use simple Java app. to retrieve data and write
to a file
Contents
• Concepts
–Brokers
–Remote Procedure Calls (RPC)
• Using a broker
–Metadata requests
–Data requests
–Issues
–Linking your own data
• SOAP
• New brokers and future
enhancements
New Brokers
• DEM Broker - Digital Elevation Model data
– Japan 50m resolution
– World 1km resolution
• ResourceServer
– Internet-based service for software localization
• ChizuBroker – Web map services
– www.mapfan.com (Japan)
– www.mapzone.co.nz (New Zealand)
• SoilBroker – soils data
Future enhancements
• Broker-side interpolation (using DEMBroker)
– temporal
– spatial
“Virtual station”
Broker-side interpolation at single site
Client
Server
Temperature
Rainfall
Radiation
Future enhancements
• Broker-side interpolation (using DEMBroker)
– temporal
– spatial
• Other brokers
– Place names
– Crops and crop models (prototyped)
– Agrichemicals
Final Points
• The Grid provides a stable, powerful
foundation for Ag and NR software
• You can now develop grid applications in
your favourite programming language.
• NARO is keen to help you to both develop
new applications and link new databases.
• Database owners around the world will be
keen to evaluate your applications.
Thank you