Download Introduction to Parallel Programming using Basic

Global Grid Forum
Applications, Programming Models
& Environments
Area Overview
Mary Thomas
San Diego Supercomputer Center
([email protected])
Presented at the Asia Pacific Grid Workshop 2001
October 22-24, 2001
Tokyo, Japan
AP&M Area


Contains 5 active research groups:
 Advanced Collaborative Environments (ACE)
 Advanced Programming Models (APM) – Satoshi
Matsuoka
 Applications & Testbeds (APPS) -- Ed Seidel
 Grid Computing Environments (GCE)
 Grid User Services (GUS)
No Working Groups yet, some proposed
Advanced Collaborative Environments (ACE)
n
n
n
n
Research Group Chair(s):
n Rick Stevens ([email protected])
n Jason Leigh ([email protected])
n Mike Papka ([email protected])
Website: http://calder.ncsa.uiuc.edu/ACE-grid/
New RG formed last summer
Statement of purpose: The GGF Advanced Collaborative
Environment research group's charter is to investigate humancentered techniques and technologies for facilitating
interactive, collaborative, and immersive access of Grid
resources from any where and at any time.
ACE Goals





Provide a venue at which researchers can share late-breaking
ideas and results.
Encourage information and code sharing by proposing and
publishing strategies for interoperability.
Formalize human factors techniques for optimal operation of
interactive, collaborative, immersive and ubiquitous
environments.
Provide a systematic process through which researchers may
contribute their work to the community.
Identify future areas of research that will be important to the
continuing advance of the Grid.
ACE “Domains”






Collaboration Environments - Access Grid, Access Grid Augmented
Virtual Environments (AGAVE)
Tele-Immersion - CAVE, IDesks, Tiled Displays, auto stereoscopic
displays, software frameworks
Distributed Realtime Visualization - crossplatform, cross cluster scene
graph libraries, parallel rendering techniques, video and display list
streaming
Computational Steering - software frameworks
Collaborative Gaming - things we can learn from realtime online games
and communities
Ubiquitous Computing - paradigms for VR to desktop to laptop to PDA
collaboration and other human-in-the-loop Grid applications
Grid User Services (GUS)




Research Group Chair(s):
 John Towns, [email protected]
Website: http://dast.nlanr.net/GridForum/UserServ-WG/
Statement of purpose: The User Services Working Group fosters a
common understanding of user and support staff requirements in a grid
environment, acts as a venue for sharing resources, and facilitates
communication for grid activities between users, support staff, and
developers.
Recent key activities:
 Grid User Services Best Practices doc (Lead: John Towns)
 Trouble Ticket Interchange (Lead: Hank Laughlin)
 Infrastructure Requirements for Grid Sites aka the Grid
Constitution (Lead: George Myers)
 Services and Tools Requirements for Effective Grid Support
Services (Lead: Don Frederick) - Awaiting first draft.
Grid Computing Environments (GCE)




Research Group Chairs
 US: Geoffrey Fox, ([email protected])
 Mary Thomas, ([email protected])
 Dennis Gannon, ([email protected])
 EU Co-chair: Rob Allen (Univ. of Manchester)
Website: http://www.computingportals.org/
Statement of purpose: The GGF Grid Computing Environment research group is
aimed at contributing to the coherence and interoperability of frameworks,
portals, PSE's, and other Grid-based computing environments by establishing
standards that are required to integrate technology implementations and
solutions.
Recent key activities:
 Formation of new working groups
 Web Services WG
 Web Flow
 Meta Data
 UK activities added
 Testbed Project
GCE/Web Services/Testbed

Today I will focus on defining web services

Why are they useful for portals?

Why are they useful for grid services?

GCE Web Services Testbed plans
Grid Portals: the Problem

Example: portal or applications need to perform grid tasks for any arbitrary
user, on any arbitrary resource, and span all ‘layers’ of the grid
 portals must be ‘aware’ of resources (use GIS)
 What grid services are running on that resources:
 Globus/Legion/VegaGrid/SSH, etc
 GIS
 GSI/Kerberos, MyProxy
 Request syntax differs for each resource:
 GRAM/Legion/SSH/MAUI/PBS/Others
 Portal must have permission to use/access for user (GSI, MyProxy)
Grid Portals: Complexity Grows
Presents a huge complexity problem that does not scale
 Portals interact with/integrate all layers:
 GIU/Client interface
 Uses all middleware services (Globus, SRB, GSI-FTP, etc.)
 Each portal in the world must store and configure same data
 Repeated data, open to errors, variations
 Multiple programmers repeating same tasks and implementations
 Much portal software is “hard-coded” and not dynamic
The Grid is international, need for scaleable, interoperable services
 Too much ‘hard-coding’ needed at this time (big issue for Portals)

Grid Portals: Example




“Simple” example all portals face: authentication
 Portals represent user, must act as a proxy
Complex for single user:
 Each user needs an account/allocations on each machine
 Each user needs certificate signed by a CA
 Each CA must be recognized by resources as valid
 Each users DN must be placed into mapfile on each resource to be used:
 Large burden on local administrator
Portals are gateways:
 large number of resources, always changing
 large numbers of users (GryPhyN ~ 2000)
 Some grid software must be configured by local site administrator to
accept portals as proxy (e.g. MyProxy)
Complexity is one represented as a fully connected, N-dimensional, graph
Web Services: a Proposed Solution







Web services architecture provides mechanisms for
 dynamic service discovery (UDDI)
 Separation of implementation from function (WSDL)
 Know protocol (SOAP/HTTP, SOAP/RPC)
Service provider encapsulates implementation details
Client does not need to know details, just where to send the request
Challenge will be discovery  problem with Jini/CORBA
Commercial world developing web services technologies in P2P world:
 Implies funding/support
 rapid development/technology advancement
 Caution: this does NOT imply cohesiveness or standards
Note: in some ways, Globus/GRAM is a web service
Advantage: language independent, so can run on any system
 We are pursuing Perl, Python, Java, C++ at this time
Testbed Architecture



Will include the following:
 XML schemas for language/description
 SOAP Protocol (Simple Object Access Protocol)
 Over HTTP as transport protocol/RPC
 WSDL Interface for discovery (Web Services Description
Language)
 UDDI (Universal Description, Discovery and Integration)
 Go here to find who/what/where/why/how
Security model –
 Need to support single login
 Security at all levels
Adopt “anatomy of the Grid model”
 virtual organizations
 Portals be built as services in addition to applications
How will Portals use Web Services?






Portals need to worry about when to use XML, and when to use other protocols:
eg for large files, need to use gtp
 Use XML to find files, and then have suggested protocols for moving them
“Beep” low level protocol for describing way data is encoded; can carry XML,
and XMLP
 Block Extensible Exchange Protocol
 XMLP – standardization of SOAP – may subsume SOAP
 A W3C project – mainly commercial world,
Suggested GF doc: talking about XML based interoperability protocols, and
when to use/not use XML. Describe experiences using them.
Should portals create and instance of UDDI, and how does this related to
existing information service dicsovery such as GIS?
 May need a bridge between UDDI and lDAP or others.
Think about bridges (or wrappers) to data for protocols –
Separate protocol from services from data
NPACI GridPort Architecture
Portal Web Services Architecture
Portal Web Services Architecture
Environment
Management
(LaunchPad,
HotPage)
composition
frameworks
Resources
Job Submission /
Control
File Transfer
Data Management
Monitoring
Events
……
Credential
Management
Workflow
Management
other services:
•visualization
•interface builders
•collaboration tools
•numerical grid
generators
•etc.
Python, Java, Perl,
etc.,
JSPs format to html
CoG Kits implementing
Web Services in
servelets, servers, etc.
Apache SOAP,
.NET, etc.
Apache Tomcat&WebSphere
&Cold Fusion=JVM + servlet
instantiation + routing
CORBA
Condor-G
SRB/MCAT
Replica Catalog /
Management
GridFTP
Grid
Monitoring
Architecture
Grid X.509
Certification
Authority
Grid
Information
Service
Grid Web Service
Description (WSDL)
& Discovery (UDDI)
Compute
(many)
GRAM
MPI
Secure,
Reliable
Group Comm.
Grid Protocols and Grid Security Infrastructure
Problem
Solving
Environments
(AVS, SciRun,
Cactus)
Grid Services
Collective and Resource Access
Grid Protocols and Grid Security Infrastructure
http, https
Web Browser
Discipline /
Application
Specific
Portals
(e.g. SDSC
TeleScience)
Grid Web
Services
XML / SOAP over Grid Security Infrastructure
Clients
Application
Portals
Storage
(many)
Communication
Instruments
(various)
Proposed Set of Web Services




Information Services
 Jobs
 Users
 Machines
Jobs
 Job Submission (Atomic)
 Batch script builder
 Job Management
 Job Control
 Job History  uses events
 Job Status
Applications
Data Management
 FTP
 Collections (SRB)






Grid Messaging (atomic)
Events
 Monitoring
 GMA (Generic Monitoring
Framework)
Network/Performance
 NWS
 Performance
Accounts/Allocations
Scheduling
Security
 Single login env
 CA
* SC01 Demonstration
What is Required to Participate?


Goal:
 Drive development of interoperable technologies for portals and the
services needed by them
 Make portal middleware programming task easier - demonstrate that applications can use the services provided
Commitment at different levels:
 Participation:
 Agree to use WSDL’s and protocols adopted by testbed
 Development:
 Agreement to support personnel to develop web service
 $$Funding$$
 Evidence of support from Organizations, Institutions, Funding
Organizations
 Principal need: resource sharing agreements
 Informal, limited scope/user group
 Only for testbed purposes
Testbed Participants




Initial List is limited
US:
 PACI (NPACI, Alliance, PSC, DTF)
 NASA/IPG
 PNNL
 LBL/DOE
Asia:
 apGrid members
Europe:
 Daresbury (UK)
 EPCC
 Cactus
 Italy
What’s Next?






This is an experiment, not a drive towards standards (yet)
Meet at SC01 –
 BOF on Thursday, 5:30 pm
 Discuss web services and testbed
Discuss formally by GCE email list
 Testbed doc will circulate between now & GGF4 (Montreal)
Formal session at GGF4:
 New web services working group
 Testbed –
 Agree to goals, protocols, standards, etc.
Other mechanisms:
 Use AG nodes to meet monthly at start.
GCE Research Group:
 Website: http://www.computingportals.org