Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Grid Architecture and Some Core Services US-Australia Workshop on HighPerformance Grids and Applications Sydney June 8 2004 Geoffrey Fox Community Grids Lab Indiana University [email protected] WSGA Web Service Grid Architecture • With UK e-Science program, developing list of best practices in Grids based on a pure Web Service infrastructure • Use WS-I Web Service Interoperability Profiles – WSDL, SOAP, UDDI, WS-Security (No Risk Grid Profile) • Some risk for those where IBM and Microsoft agree but not a broader consensus – WS-Addressing, WS-MessageDelivery – WS-ReliableMessaging and WS-Reliability Can federate different Standards here – Workflow: BPEL and WSCL,WSCI (No streams in BPEL but expected to win) • Higher risk where IBM and Microsoft do not agree – WSRF (No Microsoft equivalent; unclear relation to metadata services) – WS-Notification, WS-Eventing (Can be federated to each other and JMS) • Use Service not Distributed Object Paradigm • Use portlet-based portals with user interface fragments for each service • Build Grids of Grids of Simple Services • Identify and build key services (in UK OMII Open Middleware Infrastructure Institute) Typical Science Grid Service such as Research Database or simulation Campus or Enterprise Administrative Grid Learning Management or LMS Grid Digital Library Grid Science Grids Transformed by Grid Filter to form suitable for education Publisher Grid Education Grid Inservice Teachers Preservice Teachers School of Education Teacher Educator Grids Student/Parent … P2P (e.g. AOL) Grid Informal Education (Museum) Grid Education as a Grid of Grids Community Grids What is a Simple Service? • Take any system – it has multiple functionalities – We can implement each functionality as an independent distributed service – Or we can bundle multiple functionalities in a single service • Whether functionality is an independent service or one of many method calls into a “glob of software”, we can always make them as Web services by converting interface to WSDL • Simple services are gotten by taking functionalities and making as small as possible subject to “rule of millisecond” – Distributed services incur messaging overhead of one (local) to 100’s (far apart) of milliseconds to use message rather than method call – Use scripting or compiled integration of functionalities ONLY when require <1 millisecond interaction latency • Apache web site has many projects that are multiple functionalities presented as (Java) globs and NOT (Java) Simple Services – Makes it hard to integrate sharing common security, user profile, file access .. services NaradaBrokering Audio/Video Conferencing Client Computer Modem Server Web Service B Minicomputer Peers NaradaBrokering Broker Network Firewall Queues Stream P2P Messaging Server-enhanced Messaging Workstation NB role for Grid is Similar to MPI role for MPP Laptop computer Peers PDA Audio/Video Conferencing Client NB supports messages and streams 1-100 ms. v 1-10 µs Current NaradaBrokering Features Multiple transport support In publish-subscribe Paradigm with different Protocols on each link Transport protocols supported include TCP, Parallel TCP streams, UDP, Multicast, SSL, HTTP and HTTPS. Communications through authenticating proxies/firewalls & NATs. Network QoS based Routing Allows Highest performance transport Subscription Formats Subscription can be Strings, Integers, XPath queries, Regular Expressions, SQL and tag=value pairs. Reliable delivery Robust and exactly-once delivery of messages in presence of failures Ordered delivery Producer Order and Total Order over a message type Time Ordered delivery using Grid-wide NTP based absolute time Recovery and Replay Recovery from failures and disconnects. Replay of events/messages at any time. Security Message-level WS-Security compatible security Message Payload options Compression and Decompression of payloads Fragmentation a nd Coalescing of payloads Messaging Related Compliance Java Message Service (JMS) 1.0.2b compliant Support for routing P2P JXTA interactions. Grid Application Support NaradaBrokering enhanced Grid-FTP. Bridge to the Globus TK3. Web Service reliability Prototype implementation of WS-ReliableMessaging Web Service Notifications Prototype implementation of WS-Notification NaradaBrokering Service Integration Proxy Messaging Handler Messaging S1 P1 P2 S1 S2 S2 Notification S1 S? Service NB Transport S2 P? Proxy Any Transport Standard SOAP Transport Internal to Service: SOAP Handlers/Extensions/Plug-ins Java (JAXRPC) .NET Indigo and special cases: PDA's gSOAP, Axis C++ Streams and Workflow NaradaBrokering can manage streams from • Audio/Video conferences • Sensors • Inter-service communication in workflow http://www.hpsearch.org/demo/ describes scripting management interface to NaradaBrokering Grids involve streams as well as compute and data nodes Workflow and dataflow like BPEL imply streams Future/Ongoing NB Enhancements Server/Broker-free P2P version to support “immediate deployment” of NB-based Community Grids • Direct integration of P2P and Grids Rich handler and proxy support for WS-ReliableMessaging and WS-Reliability with federation between standards Federated support of WS-Eventing, WS-Notification and JMS Other Mediation/Federation to heterogeneous Grids Replicated Subscriber (Fault-tolerance/Performance) Services • NaradaBrokering will choose between several subscribing replicated services VPG Virtual Private Grid: Build standard VPN transport (PPTP, L2TP, IPSec) into NB • Could offer new Grid Security options and/or more choice on firewall/NAT tunneling Web Services and M-MVC Web Services are naturally M-MVC – Message based Model View Controller with • Model is Web Service • Controller is Portal and Messages (NaradaBrokering) • View is rendering OGCE builds M-MVC RFIO urc eso R e Facing Port Web Service Application or Model User Facing Port Input port Output port Portal Pu View ish v Ie en As Controller S Broker t ub sc rib U bl bs Su cr r ibe en d ng eri eU Ie ve Pu bli sh Aggregate WS User Facing fragments re n nt View View de rin g Model Explicit message-based Publish/Subscribe MVC model desktop handheld phone WSRP and JSR168 Portlets NSF NMI Project for Making Portal Components Robust • University of Chicago – Gregor von Laszewski • Indiana University Builds on NCSA Alliance Portal Project – Marlon Pierce, Dennis Gannon, Geoffrey Fox, and Beth Plale • University of Michigan – Charles Severance, Joseph Hardin • NCSA/UIUC – Jay Alameda, Joe Futrelle • Texas Advanced Computing Center – Mary Thomas OGCE Consortium Individual portlet for the Proxy Manager Use tabs or screen real estate to navigate through interfaces to different services OGCE Consortium LEAD Portal OGCE Consortium Aggregation Portal Architecture Clients Portlet Class: WebForm Aggregation and Rendering Clients (Pure HTML, Java Applet ..) • Supply component model for user interfaces to match component model for middeware Portlet Class: IFramePortlet Portlet Class: JspPortlet Portlet Class: VelocityPortlet Jetspeed Internal Services Portal Portlets (Jetspeed) Gateway (IU) Remote or Proxy Portlets Web/Grid service Computing Web/Grid service Data Stores Web/Grid service Instruments GridPort etc. (Java) COG Kit Local Portlets Libraries Hierarchical arrangement Services Resources • OGCE Grid Portal Components Provides Portlets for – Management of user proxy certificates – Remote file Management via Grid FTP – News/Message systems • for collaborations – Grid Event/Logging service – Access to OGSA/WSGA Grid services – Access to directory services – Specialized Application Factory access • Distributed applications • Workflow – Access to Metadata Index tools • User searchable index – Collaboration Supports distributed modular construction of Services – each with their own Interface OGCE Consortium Portlet Component and Container Technologies • Jakarta Jetspeed – Open source Java portlet project – Jetspeed is both a framework and reference implementation – Defines portlets, portal service APIs (login, authorization, customization, etc.) • CHEF from University of Michigan – Uses Jetspeed as a framework • Reimplements many of the core classes – Basis for UM CourseTools – NEESGrid portal – CMCS Portal OGCE Consortium Access Grid and Related Portlets Desktop and Web Services with MMVC OGCE uses message-based MVC as is obvious for Web Services Most desktop applications are in fact roughly MVC with controller formed by “system interrupts” with View and Model communicating by “post an event” and define a “listener” programming mode We propose to integrate desktop and Web Service approach by systematic use of MMVC and NaradaBrokering Allows easier porting to diverse clients and automatic collaboration Attractive for next generation of Linux desktop clients We have demonstrated for SVG Browser (Scalable Vector Graphics), OpenOffice and PowerPoint “Glob” programming style makes hard Global-MMCS Community Grid We are building an open source protocol independent Web Service “MCU” which will scale to an arbitrary number of users and provide collaboration support for thousands of simultaneous users The function of A/V media server is distributed using NaradaBrokering architecture. • Media Servers mix and convert A/V streams Open XGSP MCU based on the following open source projects • openh323 is basis of H323 Gateway • NIST SIP stack is basis of SIP Gateway • NaradaBrokering is open source messaging • Java Media Framework basis of Media Servers • Helix Community http://www.helixcommunity.org for Real Media http://www.globalmmcs.org open source “non advertised” release XGSP Web Service MCU Architecture Use Multiple Media servers to scale to many codecs and many versions of audio/video mixing Session Server XGSP-based Control NaradaBrokering All Messaging NB Scales as distributed Admire Web Services SIP H323 Media Servers Filters High Performance (RTP) and XML/SOAP and .. Access Grid Gateways convert to uniform XGSP Messaging NaradaBrokering Native XGSP Break up into Web Services Monolithic MCU becomes many different “Simple Services” • • • • • • • • Session Control Thumbnail “image” grabber Audio Mixer Video Mixer Codec Conversion Helix Real Streaming PDA Conversion H323/SIP Gateways As independent can replicate particular services as needed • Codec conversion might require 20 services for 20 streams spread over 5 machines 1000 simultaneous users could require: • 1 session controller, 1 audio mixer, 10 video mixers, 20 codec converters, 2 PDA converters and 20 NaradaBrokers Support with a stream optimized Grid Farm in the sky • Future billion way “Video over IP” serving 3G Phones and home media centers/TV’s could require a lot of computing GlobalMMCS and NaradaBrokering All communication – both control and “binary” codecs are handled by NaradaBrokering Control uses SOAP and codecs use RTP transport Each stream is regarded as a “topic” for NB Each RTP packet from this stream is regarded as an “event” for this topic Can use replay and persistency support in NB to support archiving and late clients Can build customized stream management to administer replay, and who gets what stream in what codec NaradaBrokering supports unicast and multicast Use firewall penetration and network monitoring services in NB to improve Q0S Some resources • Web Service Grid Architecture and best practice http://grids.ucs.indiana.edu/ptliupages/publications/ • Grid Messaging http://www.naradabrokering.org • Data Stream Management http://www.hpsearch.org/demo/crisisGrid/index.html • NSF NMI Grid Portals • http://www.collab-ogce.org/nmi/index.jsp • Web Service Colalboration Services and portlets http://www.globalmmcs.org/ • OMII UK Open Middleware Infrastructure Institute http://www.omii.ac.uk