Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Passive optical network wikipedia , lookup
Cracking of wireless networks wikipedia , lookup
Distributed firewall wikipedia , lookup
Deep packet inspection wikipedia , lookup
Zero-configuration networking wikipedia , lookup
Recursive InterNetwork Architecture (RINA) wikipedia , lookup
List of wireless community networks by region wikipedia , lookup
Airborne Networking wikipedia , lookup
Distributed operating system wikipedia , lookup
Network tap wikipedia , lookup
Service-oriented architecture implementation framework wikipedia , lookup
An Agent Based, Dynamic Service System to Monitor, Control and Optimize Distributed Systems February 2006 Iosif Legrand California Institute of Technology 1 February 2006 Iosif Legrand The MonALISA Framework MonALISA is a Dynamic, Distributed Service System capable to collect any type of information from different systems, to analyze it in near real time and to provide support for automated control decisions and global optimization of workflows in complex grid systems. The MonALISA system is designed as an ensemble of autonomous multi-threaded, self-describing agent-based subsystems which are registered as dynamic services, and are able to collaborate and cooperate in performing a wide range of monitoring tasks. These agents can analyze and process the information, in a distributed way, and to provide optimization decisions in large scale distributed applications. 2 February 2006 Iosif Legrand MonALISA is A Dynamic, Distributed Service Architecture The framework is based on a hierarchical structure of loosely coupled agents acting as distributed services which are independent & autonomous entities able to discover themselves and to cooperate using a dynamic set of proxies or self describing protocols. An agent-based architecture provides the ability to invest the system with increasing degrees of intelligence; to reduce complexity and make global systems manageable in real time. For an effective use of distributed resources, these services provide adaptability and self-organization. 3 February 2006 Iosif Legrand MonALISA service & Data Handling Client (other service) Web client WEB Service Data Stores WSDL SOAP Data Cache Service & DB Lookup Service Postgres MySQL Lookup Service Communications via the ML Proxy data MonALSIA Service Client (other service) Java Predicates & Agents Applications Configuration Control (SSL) User defined loadable Modules to write /sent data 4 February 2006 Iosif Legrand The MonALISA Discovery System & Services Fully Distributed System with no Single Point of Failure Clients , HL services repositories Proxies AGENTS MonALISA services Global Services or Clients Dynamic load balancing Scalability & Replication Security AAA for Clients Distributed System for gathering and Analyzing Information. Distributed Dynamic Network of JINI-LUSs Discovery- based on a lease Mechanism and REN Secure & Public 5 February 2006 Iosif Legrand Monitoring Internet2 backbone Network Test for a Land Speed Record ~ 7 Gb/s in a single TCP stream from Geneva to Caltech 6 February 2006 Iosif Legrand The UltraLight Network BNL ESnet IN /OUT 7 February 2006 Iosif Legrand Monitoring Network Topology Latency, Routers NETWORKS ROUTERS AS 8 February 2006 Iosif Legrand Monitoring The GLORIAD Ring 9 February 2006 Iosif Legrand Monitoring Grid sites, Running Jobs, Network Traffic, and Connectivity JOBS TOPOLOGY ACCOUNTING 10 February 2006 Iosif Legrand Monitoring OSG: Resources, Jobs & Accounting Running Jobs Accounting 42 SITES ~ 4 000 Nodes ( 10 000 CPUs) Thousands of Jobs 60 000 parameters 11 February 2006 Iosif Legrand FTP Data Transfer between GRID sites Total FTP Traffic per VO 12 February 2006 Iosif Legrand Bandwidth Challenge at SC2005 151 Gbs ~ 500 TB Total in 4h 13 February 2006 Iosif Legrand End User / Client Agent LISA- Localhost Information Service Agent Authorization Service discovery Local detection of the hardware and software configuration Complete end-system monitoring: Per-process load, I/O and network throughputs, etc. End-to-end performance measurements Will act as an active listener for all events related with the requests generated by its local applications. 14 February 2006 Iosif Legrand Host Monitoring at SC2005 Many “network” problems are actually endhost problems: misconfigured or underpowered end-systems Network Device Information TCP Settings Host/System Information The LISA application was designed to monitor the endhost and its view of the network. For SC|05 we developed we used LISA to gather the relevant host details related to network performance Information on the system information, TCP configuration and network device setup was gathered and accessible from one site. Future plans are to coordinate this with LISA and deploy this as part of OSG. The Tier-2 centers are a primary target. 15 February 2006 Iosif Legrand Available Bandwidth Measurements Embedded Pathload module. 16 February 2006 Iosif Legrand Coordination Service for Available Bandwidth Measurements Enforces measurement fairness Avoids multiple probes on shared network segments Dynamic configuration of measurements timing Logs events Provides service redundancy by using a masterslave model 17 February 2006 Iosif Legrand Monitoring the Execution of Jobs and the Time Evolution SPLIT JOBS LIFELINES for JOBS Summit a Job Job Job Job1 Job2 Job3 DAG 18 Job 31 Job 32 February 2006 Iosif Legrand ApMon – Application Monitoring Library of APIs (C, C++, Java, Perl. Python) that can be used to send any information to MonALISA services Flexibility, dynamic configuration, high communication performance APPLICATION Accounting information App. Monitoring Time;IP;procID parameter1: value parameter2: value 70 MonALISA CPU Usage (%) dynamic reloading Config Servlet Automated system monitoring UDP/XDR Monitoring Data ApMon MonALISA Service ... 60 UDP/XDR Monitoring Data APPLICATION 50 App. Monitoring Mbps_out: 0.52 Status: reading MB_inout: 562.4 40 30 20 No Lost Packages 10 0 0 1000 2000 3000 4000 Messages per second 19 5000 6000 System Monitoring load1: 0.24 processes: 97 pages_in: 83 February 2006 ApMon ApMon Config UDP/XDR Monitoring Data MonALISA Service ApMon configuration generated automatically by a servlet / CGI script Iosif Legrand MonALISA agents to create on demand on an optical path or tree Discovery & Secure Connection 2 ML Agent MonALISA ML Demon 3 Optical Switch Optical Switch 1 Control and Monitor the switch Optical Switch ML Agent MonALISA ML Agent MonALISA Runs a ML Demon >ml_path IP1 IP4 “copy file IP4” Time to create a path on demand <1s independent of the location and the number of connections 4 ML proxy services used in Agent Communication 20 February 2006 Iosif Legrand Monitoring and Controlling Optical Planes Controlling Port power monitoring 21 February 2006 Iosif Legrand Monitoring Optical Switches Agents to Create on Demand an Optical Path 22 February 2006 Iosif Legrand Communities using MonALISA Major Communities OSG CMS ALICE D0 STAR VRVS LGC RUSSIA SE Europe GRID APAC Grid UNAM Grid ABILENE ULTRALIGHT GLORIAD LHC Net RoEduNET 23 MonALISA ABILENE Demonstrated at: Running 24 X 7 SC2003 at 250 Sites Collecting 250,000 Telecom CMS-DC04 World parameters in near 2003 real-time GRID3 Update rate of 25,000 WSIS 2003 parameter updates per SC 2004 second VRVS Monitoring I2 2005 12,000 computers ALICE TERENA 2005 > 100 WAN Links Thousands of Grid IGrid 2005 jobs running concurrently SC 2005 February 2006 Iosif Legrand The MonALISA Architecture Provides: Distributed Registration and Discovery for Services and Applications. Monitoring all aspects of complex systems : System information for computer nodes and clusters Network information : WAN and LAN Monitoring the performance of Applications, Jobs or services The End User Systems, its performance Video streaming Can interact with any other services to provide in near real-time customized information based on monitoring data Secure, remote administration for services and applications Agents to supervise applications, trigger alarms, restart or reconfigure them, and to notify other services when certain conditions are detected. The MonALISA framework is used to develop higher level decision services, implemented as a distributed network of communicating agents, to perform global optimization tasks. Graphical User Interfaces to visualize complex information 24 February 2006 Iosif Legrand