Download Distributed Virtual Computer (DVC): Simplifying the

Distributed Virtual Computer (DVC):
Simplifying the Development of High-Performance
Grid Applications
Nut Taesombut and Andrew A. Chien
Department of Computer Science and Engineering
University of California, San Diego
Workshop on Grids and Advanced Networks (GAN’04)
Chicago, Illinois
April 22, 2004
Outline
•
•
•
•
•
Background and Motivation
Distributed Virtual Computer (DVC)
Example Application
Related Work
Summary and Future Work
5/25/2017
2
Emerging Opportunity of Lambda Grids
•
Network Advances and Trends
– DWDM optical paths (or lambdas) enable
– Dedicated High Bandwidth
– Dynamic Configuration
•
Lambda Grid
– Distributed, shared resources interconnected by plentiful lambdas
– Configurable Connections and Capacity
– Deterministic Communication Performance
– Novel Communication Capabilities (e.g. optical multicast)
5/25/2017
3
OptIPuter Project
•
OptIPuter – Large-scale Research Project on Impact of Lambdas on System
Software and next-generation E-science
– International Testbed for Experimentation (UCSD, UIC, UCI, Amsterdam, etc.)
– Leading E-science Drivers (Neuroscience, Geophysical/Earth Sciences)
– 3-D Data Analysis, Visualization and Collaboration Applications
– Data-intensive and Real-time, Distributed data sources/sinks
– Wealth of Innovative System Software Research (protocols, DVC, storage, etc.)
Smarr, Papadopoulos,
Ellisman, Orcutt,
Chien – UCSD
DeFanti, Leigh - UIC
5/25/2017
http://www.optiputer.net
4
Motivation
•
Building Grid Applications is Difficult!
– Applications must deal with complexity of resource environment
– Resource Heterogeneity, Performance, Communication
– Multi-Organization Security, Resource Management
– Shared and Untrusted Resource Environments
– Low-Quality Networks
•
Adding Low-level Management of Network Complicates the Task
– No Uniform Interfaces (routers, switches, end nodes)
– Wildly Different Semantic Level (BELOW IP!)
– Novel Capabilities (e.g. multicast, RDMA, etc.)
•
Key Requirements
– A new abstraction which simplifies Grid environments
– A view which integrates communication as first class
5/25/2017
5
Example of Grid Complexity
http://dream.uci.edu:4010
http://intania.uic.edu:4566
Internet
http://zebra.ucsd.edu:5220
10.1.3.34
10.1.2.61
10.1.2.68
•
•
•
Access to Resources Across Multiple Namespaces
User-Controlled Configuration of Dynamic Lambda Network
Heterogeneous Communication
5/25/2017
6
Distributed Virtual Computer (DVC)
•
A Simple Execution Environment for Grid Applications
– Set of LambdaGrid resources (connections, resources)
– Naming, access, and management services
– Transparently shared amongst Applications
•
Simplify Use of Network and Grid Capabilities
– Automate compute/data resource binding
– Automate dynamic λ-configuration; expose novel λ-capabilities
– Leverage existing Grid Technologies (Globus, NWS etc.)
DVC
5/25/2017
7
DVC Design Principles
•
Separate Resource Config/Mgmt and Application Programming
– Resource Environments Configured to Spec
– Applications simply use them
•
Aggregate and Bind Grid Resources; Present as Workgroup
–
–
–
–
–
•
Central resource control
Single namespace
Unified resource access mechanisms
Trusted and secure environment
Controllable performance
Enable Collective Resource Views
– Unified naming structures (e.g. collective names)
– Collective properties (e.g. group communication, trust, access control)
5/25/2017
8
Example: Locally Simplified Grid Programming
DVC Domain
comp1
http://dream.uic.edu:4010
comp2
Internet
http://intania.uic.edu:4566
comp3
http://zebra.ucsd.edu:5220
10.1.3.34
10.1.2.61
10.1.2.68
str1
Simple View of DVC
•
•
•
Single Control Domain
Unified Naming Mechanism
Uniform Use of Different Communication Mechanisms (e.g. protocols)
5/25/2017
9
How DVC’s Simplify Application Grid Programming
•
Automate Resource Binding and Configuration
– Reduce user interaction through resource broker and manager
•
Unify Resource Naming and Access Mechanisms
– Hide heterogeneity through simple abstractions
•
Transparently Enable Security Protection
– Implement cryptographic operations at the middleware layer
•
Monitor and Control Resource and Communication Performance
– Detect asynchronous events and notify application based on subscription
5/25/2017
10
Realizing Simplified Application Grid Programming
DVC
Data Flow
Control Flow
DVC Manager
•
DVC Manager
–
–
–
–
•
Ghost Manager
Single master controller
Resource selector/negotiator/scheduler
Trust mediator and security authority
Synchronizer of global state information
Ghost Managers
–
–
–
–
Slave managers, each running at each bound resource
Job process controller at remote resource
Communication mediator
Resource status monitor and reporter
5/25/2017
11
Example: Dynamic Configuration of Lambda Grid
•
Example Application:
– BIRN (Biomedical Information Research Network)
NCMIR/UCSD
Harvard
Duke
BIRN DVC
SDSC
UNC
UCLA
UCI
•
DVC Advantages
– On-demand creation of a private Grid resource workgroup
– Transparent use of novel communication capabilities
– high-speed multi-point communication
– SAN-like storage access across geographically distributed resources
5/25/2017
12
Example: Dynamic Configuration of Lambda Grid
•
Sequence to Create a BIRN DVC
1. Create a resource configuration specification and send a request to bind
resources
2. Create resource groups (i.e. for collective data source and sink)
3. Create multipoint-to-point and point-to-multipoint communication sessions
4. Define the properties of communication sessions (e.g. security and
communication mechanisms)
NCMIR/UCSD
Harvard
harvard
Duke
duke
ucsd
Grp1
GTP + enc + auth
sdsc
TCP + Optical Multicast
SDSC
UNC
unc
UCLA
UCI
Physical-Level View of BIRN DVC
5/25/2017
uci
uci
Grp2
Application-Level View of BIRN DVC
13
Related Work
•
Abstractions of Distributed Resources
– PVM [Geist94]
•
Grid Middleware
– Globus System
•
Grid Programming Tools
– GridRPC [Nakada02], MPICH-G2 [Karonis03], Condor-G [Frey01]
– GrADS [Berman01], GridLab [Allen03],
•
Federation of Resources
– Virtual Organization (VO) [Foster01]
•
Distributed Virtual Computer
– Provide an Application-Focused Dynamic Resource Container
– Dynamic resource configuration and sharing policies
5/25/2017
14
Summary and Future Work
•
Summary
– DVC’s provide simple computing environments for Grid applications
– Locally simplified resource workgroup
– DVC’s allow on-demand instantiation and dynamic configuration of Lambda-Grid
– DVC’s enable simple use of novel communication capabilities
•
Future Work
– Develop the full implementation of the DVC Prototype
– Implement as Web Services (i.e. WS-RF specification)
– Deploy the prototype on the OptIPuter Testbed
– Demonstrate with OptIPuter applications (Bioinformatics and Geophysical)
– Explore other system technologies that can be integrated into the DVC
framework
– Real-time System
– High-Performance Distributed File System
5/25/2017
15
Thank You
•
Questions and Remarks?
•
Contact Information:
– Nut Taesombut ([email protected])
•
OptIPuter Website:
– www.optiputer.net
5/25/2017
16