Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Plan 9 from Bell Labs wikipedia , lookup
Library (computing) wikipedia , lookup
Process management (computing) wikipedia , lookup
Unix security wikipedia , lookup
Copland (operating system) wikipedia , lookup
Linux adoption wikipedia , lookup
Spring (operating system) wikipedia , lookup
Caldera OpenLinux wikipedia , lookup
FAST-OS BOF SC 04 http://www.cs.unm.edu/~fastos Follow link to subscribe to the mail list Projects • Colony Terry Jones, LLNL • Config Framework Ron Brightwell, SNL • DAiSES Pat Teller, UTEP • K42 Paul Hargrove, LBNL • MOLAR Stephen Scott, ORNL • Peta-Scale SSI Scott Studham, ORNL • Rightweight Kernels Ron Minnich, LANL • Scalable FT Jarek Nieplocha, PNNL • SmartApps L. Rauchwerger, T A&M • ZeptoOS Pete Beckman, ANL www.HPC-Colony.org Services & Interfaces For Very Large Linux Clusters Terry Jones, LLNL, Coordinating PI Laxmikant Kale, UIUC, PI Jose Moreira, IBM, PI Celso Mendes, UIUC Derek Lieber, IBM Colony Overview Collaborators Lawrence Livermore National Laboratory Title Services and Interfaces to Support Systems with Very Large Numbers of Processors Topics • University of Illinois at Urbana-Champaign International Business Machines • • • • • • Parallel Resource Instrumentation Framework Scalable Load Balancing OS mechanisms for Migration Processor Virtualization for Fault Tolerance Single system management space Parallel Awareness and Coordinated Scheduling of Services Linux OS for cellular architecture Colony Motivation • Parallel resource management Strategies for scheduling and load balancing must be improved. Difficulties in achieving a balanced partitioning and dynamically scheduling workloads can limit scaling for complex problems on large machines. • Global system management System management is inadequate. Parallel jobs require common operating system services, such as process scheduling, event notification, and job management to scale to large machines. Colony Goals • Develop infrastructure and strategies for automated parallel resource management – Today, application programmers must explicitly manage these resources. We address scaling issues and porting issues by delegating resource management tasks to a sophisticated parallel OS. – “Managing Resources” includes balancing CPU time, network utilization, and memory usage across the entire machine. • Develop a set of services to enhance the OS to improve its ability to support systems with very large numbers of processors – We will improve operating system awareness of the requirements of parallel applications. – We will enhance operating system support for parallel execution by providing coordinated scheduling and improved management services for very large machines. Colony Approach • Top Down – Our work will start from an existing full-featured OS and remove excess baggage with a “top down” approach. • Processor virtualization – One of our core techniques: the programmer divides the computation into a large number of entities, which are mapped to the available processors by an intelligent runtime system. • Leverage Advantages of Full Featured OS & Single System Image – Applications on these extreme-scale systems will benefit from extensive services and interfaces; managing these complex systems will require an improved “logical view” • Utilize Blue Gene – Suitable platform for ideas intended for very large numbers of processors Configurable OS Framework • Sandia, lead – Ron Brightwell, PI – Rolf Riesen • Caltech – Thomas Sterling, PI • UNM – Barney Maccabe, PI – Patrick Bridges Issues • Novel architectures – Lots of execution environments • Programming models – MPI, UPC, separating processing from location • Shared services – File systems, shared WAN • Usage model – Dedicated, space shared, time shared Approach • Build application specific OS – Architecture, programming model, shared resources, usage model • Develop a collection of Micro services – Compose and distribute • Compose services – Services may adapt • Kinds of services – Memory allocation, signal delivery, message receipt and handler activation The Picture Challenges • How to reason about combinations • Dependencies among services • Efficiency – Overhead associated with transfers between micro services • How many operating systems will we really need? Goals Dynamic Adaptability in Support of Extreme Scale Generalized Customized resource management Fixed Dynamically Adaptable OS/runtime services Enhanced Performance Challenges Dynamic Adaptability in Support of Extreme Scale Determining • What to adapt • When to adapt • How to adapt • How to measure effects of adaptation Deliverables Dynamic Adaptability in Support of Extreme Scale • Develop mechanisms to dynamically sense, analyze, and adjust common performance metrics, fluctuating workload situations, and overall system environment conditions • Demonstrate, via Linux prototypes and experiments, dynamic self-tuning/provisioning in HPC environments • Develop a methodology for general-purpose OS adaptation Methodology Dynamic Adaptability in Support of Extreme Scale identify adaptation targets characterize workload resource usage patterns potential adaptation targets off line (re)determine adaptation intervals off line/ run time define/adapt heuristics to trigger adaptation generate/adapt monitoring, triggering and adaptation code, and attach it to OS monitor application execution, triggering adaptation as necessary KernInst KernInst Dynamic Adaptability in Support of Extreme Scale dynamic instrumentation of the kernel IBM pSeries eServer 690 Client KernInst Daemon Instrumentation Tool KernInst Device KernInst API Linux Kernel • KernInst and Kperfmon provide the capability to perform dynamic monitoring and adaptation of commodity operating systems. • University of Wisconsin’s KernInst and Kperfmon make the problem of run-time monitoring and adaptation more tractable. Example Adaptations Dynamic Adaptability in Support of Extreme Scale Customization of • process scheduling parameters and algorithms, e.g., scheduling policy for different job types (prototype in process) • file system cache size and management • disk cache management • size of OS buffers and tables • I/O, e.g., checkpoint/restart • memory allocation and management parameters and algorithms Partners Dynamic Adaptability in Support of Extreme Scale University of Texas at El Paso Department of Computer Science Patricia J. Teller ([email protected]) University of Wisconsin — Madison Computer Sciences Department Barton P. Miller ([email protected]) International Business Machines, Inc. Linux Technology Center Bill Buros ([email protected]) U.S. Department of Energy Office of Science Fred Johnson ([email protected]) C O M P U T A T I O N A L R E S E A R C H D I V I S I O N High End Computing with K42 Paul H. Hargrove and Katherine Yelick Lawrence Berkeley National Lab Angela Demke Brown and Michael Stumm University of Toronto Patrick Bridges University of New Mexico Orran Krieger and Dilma Da Silva IBM K42 Project Motivation • The HECRTF and FastOS reports enumerate unmet needs in the area of Operating Systems for HEC, including – – – – – Availability of Research Frameworks Support for Architectural Innovation Performance Visibility Ease of Use Adaptability to Application Requirements • This project uses the K42 Operating System to address these five needs K42 K42 Background • K42 is a research OS from IBM – API/ABI compatibility with Linux – Designed for large 64-bit SMPs – Extensible object-oriented design • Features per resource-instance objects • Can change implementation/policy for individual instances at runtime – Extensive performance-monitoring – Many traditional OS functions are performed in user-space libraries K42 What Work Remains? (1 of 2) • Availability of Research Frameworks & Support for Architectural Innovation K42 is already a research platform, used by IBM for their PERCS project (DARPA HPCS) to support architectural innovation Work remains to expand K42 from SMPs to clusters • Performance Visibility Existing facilities are quite extensive Work remains to use runtime replacement of object implementations to monitoring single objects for fine-grained control K42 What Work Remains? (2 of 2) • Ease of Use Work remains to make K42 widely available, and to bring HEC user environments to K42 (e.g. MPI, batch systems, etc.) • Adaptability to Application Requirements Runtime replacement of object implementations provides extreme customizability Work remains to provide implementations appropriate to HEC, and to perform automatic dynamic adaptation MOLAR: Modular Linux and Adaptive Runtime Support for High-end Computing Operating and Runtime Systems Coordinating Principal Investigator Stephen L. Scott, ORNL [email protected] Principal Investigators J. Vetter, D.E. Bernholdt, C. Engelmann – ORNL C. Leangsuksun – Louisiana Tech University P. Sadayappan – Ohio State University F. Mueller – North Carolina State University Collaborators A.B. Maccabe – University of New Mexico C. Nuss, D. Mason – Cray Inc. OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY MOLAR MOLAR research goals • Create a modular and configurable Linux system that allows customized changes based on the requirements of the applications, runtime systems, and cluster management software. • Build runtime systems that leverage the OS modularity and configurability to improve efficiency, reliability, scalability, ease-of-use, and provide support to legacy and promising programming models. • Advance computer reliability, availability and serviceability (RAS) management systems to work cooperatively with the OS/R to identify and preemptively resolve system issues. • Explore the use of advanced monitoring and adaptation to improve application performance and predictability of system interruptions. MOLAR MOLAR: Modular Linux and Adaptive Runtime support High-end Computing OS Research Map HEC Linux OS: modular, custom, light-weight Kernel design Performance Observation Communications, IO Monitoring Extend/adapt runtime/OS Root cause analysis RAS High availability Testbeds Provided PROBLEM: • Current OSs and runtime systems (OS/R) are unable to meet the various requirements to run large applications efficiently on future ultra-scale computers. GOALS: • Development of a modular and configurable Linux framework. • Runtime systems to provide a seamless coordination between system levels. • Monitoring and adaptation of the operating system, runtime, and applications. • Reliability, availability, and serviceability (RAS) • Efficient system management tools. IMPACT: • Enhanced support and better understanding of extremely scalable architectures. • Proof-of-concept implementation open to community researchers. MOLAR MOLAR crosscut capability deployed for RAS • Monitoring Core Daemon • service monitor • resource monitor • hardware health monitor • Head nodes: active / hot standby • Services: active / hot standby • Modular Linux systems deployment & development MOLAR MOLAR Federated System Management (fSM) • fSM emphasizes simplicity • self-build • self-configuration • self-healing • simplified operation • Expand MOLAR support: • Investigate specialized architectures • Investigate other environments & OSs • Head nodes: active / active • Services: active / active Peta-Scale Single-System Image A framework for a single-system image Linux environment for 100,000+ processors and multiple architectures Coordinating Investigator R. Scott Studham, ORNL Principal Investigators Alan Cox, Rice University Bruce Walker, HP Investigators Peter Druschel, Rice University Scott Rixner, Rice University Collaborators Peter Braam, CFS Steve Reinhardt, SGI Stephen Wheat, Intel Peta-Scale SSI Project Key Objectives OpenSSI to 10,000 nodes Integration of OpenSSI with nodes with high processor counts The scalability of a shared root filesystem to 10,000 nodes Scalable booting and monitoring mechanisms Research enhancements to OpenSSI’s P2P communications The use of very large page sizes (superpages) for large address spaces Determine the proper interconnect balance as it impacts the operating system (OS) Establish system-wide tools and process management for a 100,000 processor environment OS noise (services that interrupt computation) effects Integrating a job scheduler with the OS Preemptive task migration. Peta-Scale SSI Reduce OS-Noise and increase cluster scalability via efficient compute nodes Install and sysadmin Boot and Init Devices IPC Application monitoring and restart HA Resource Mgmt and Job Scheduling MPI Boot Process load leveling CLMS Cluster Filesystem CFS MPI DLM Vproc Remote File Block Lustre client ICS Service Nodes single install; local boot (for HA); single IP (LVS) connection load balancing (LVS); single root with HA (Lustre): single file system namespace (Lustre); single IPC namespace; single process space and process load leveling; application HA strong/strict membership; CLMS Lite LVS Vproc Lustre client Remote File Block ICS Compute Nodes single install; network or local boot; not part of single IP and no connection load balance single root with caching (Lustre); single file system namespace (Lustre); no single IPC namespace (optional); single process space but no process load leveling; no HA participation; scalable (relaxed) membership; inter-node communication channels on demand only Peta-Scale SSI Researching the intersection of SSI and large kernels to get to 100,000+ processors 2048 CPUs 2) Enhance scalability of both approaches 3) Understand intersection of both methods Single Linux Kernel 1) Establish scalability baselines Continue SGIs work on single kernel scalability Test the intersection large kernels with software OpenSSI to establish sweat spot for Continue the OpenSSI’s 100,000 processor Linux work on Typical SSISSI scalability environments Stock Linux Kernel 1 CPU 1 Node Software SSI Clusters 10,000 Nodes Right-Weight Kernels The right kernel, in the right place, at the right time RWK OS effect on Parallel Applications • Simple problem: if all processors save one arrive at a join, then all wait for the laggard [Mraz SC ’94] – Mraz resolved the problem for AIX, interestingly, with purely local scheduling decisions (i.e., no global scheduler) – Sandia resolved it by getting rid of the OS entirely (i.e., creation of the “Light-Weight Kernel”) • AIX has more capability than many apps need • LWK has less capability than many apps want RWK Hence Right-Weight Kernels • Customize the kernel to the app • We’re looking at two different approaches • Customized, Modular Linux – Based on 2.6 – With some scheduling enhancements • “COTS” Secure LWK – Based, after some searching, on Plan 9 – With some performance enhancements RWK Balancing Capability and Overhead increasing per node capability RWK AIX, Tru64, Solaris, Linux, etc. RWK No OS RWK decreasing OS impact on app • We need to balance the capabilities that an full OS gives the user with the overhead of providing such services • For a given app, we want to be as close to the “optimal” balance as possible • But how do we measure what that is? RWK Measuring what is “good” • OS activity is periodic, thus we need to use techniques such as time series analysis to evaluate the measured data – Use this data to figure out what is “good” and “bad” • Caveat: you must practice good sampling hygiene [Sottile & Minnich, Cluster ’04] – Must follow rules of statistical sampling – Measuring work per unit of time leads to statistically sound data – Measuring time per unit of work leads to meaningless data RWK Conclusions • Use sound statistical measurement techniques to figure out what is “good” • Configure compute nodes on a per app basis (Right-Weight Kernel) • Rinse and repeat! • Collaborators – Sung-Eun Choi, Matt Sottile, Erik Hendriks (LANL) – Eric Grosse, Jim McKie, Vic Zandy (Bell Labs) SFT: Scalable Fault Tolerant Runtime and Operating Systems Pacific Northwest National Laboratory Los Alamos National Laboratory University of Illinois Quadrics SFT Team • Jarek Nieplocha, PNNL • Fabrizio Petrini and Kei Davis (LANL) • Josep Torrellas and Yuanyuan Zhou (UIUC) • David Addison (Quadrics) • Industrial Partner: Stephen Wheat (Intel) SFT Motivation • With the massive number of components comprising the forthcoming petascale computer systems, hardware failures will be routinely encountered during execution of large-scale applications. • Application Driver – Multidisciplinary, multiresolution, and multiscale nature of scientific problems – drive the demand for high end systems – applications place increasingly differing demands on the system resources: disk, network, memory, and CPU. • Therefore, it will not be cost-effective or practical to rely on a single fault tolerance approach for all applications. SFT Goals • Develop scalable and practical techniques for addressing fault tolerance at the Operating System and Runtime levels – Design based on requirements of DoE applications – Minimal impact on application performance SFT Petaflop Architecture ... ... processors interconnection network memories Tightly coupled node Globally addressable but non-coherent between nodes SFT Scope • We will investigate, develop, and evaluate a comprehensive range of techniques for fault tolerance. – System level incremental checkpointing approach • based on Buffered CoScheduling • temporal and spatial hybrid checkpointing • in-memory checkpointing and efficient handling of I/O – Fault awareness in communication libraries • while exploiting high performance network communication • MPI, ARMCI • scalability – Feasibility analysis of incremental checkpointing SFT Buffered CoScheduling SmartApps: Middleware for Adaptive Applications on Reconfigurable Platforms Lawrence Rauchwerger http://parasol.tamu.edu/~rwerger/ Parasol Lab, Dept of Computer Science, Texas A&M SmartApps Today: System Centric Computing System-Centric Computing Classic avenues to performance: •Parallel Algorithms •Static Compiler Optimization •OS support •Good Architecture Application (algorithm) Development, Analysis & Optimization Application Compiler Compiler (static) OS HW Execution System (OS & Arch) Input Data WHAT’s MISSING ? •Compilers are conservative No Global Optimization •OS offers generic services •No matching between Application/OS/HW •Architecture is generic •intractable for the general case SmartApps Our Approach: SmartApps Application Centric Computing Application (algorithm) Application-Centric Computing Development, Analysis & Optimization HW Compiler (static) + run-time techniques Compiler Run-time System: Execution, Analysis & Optimization SmartApp Compiler (run-time) OS Application OS (modular) Input Data Architecture (reconfigurable) Application Control Instance-specific optimization Compiler + OS + Architecture + Data + Feedback STAPL Application SmartApps SmartApps Architecture DataBase Static STAPL Compiler Predictor & Augmented with Optimizer runtime techniques Compiled code + runtime hooks Smart Application Get Runtime Information (Sample input, system information, etc.) Compute Optimal Application and RTS + OS Configuration advanced stages development stage Toolbox Large adaptation (failure, phase change) Recompute Application Configurer and/or Reconfigure RTS + OS Adaptive Software Adaptive RTS+ OS Runtime tuning (w/o recompile) Predictor & Evaluator Execute Application Continuously monitor performance and adapt as necessary Small adaptation (tuning) Predictor & Evaluator Predictor & Optimizer SmartApps SmartApps written in STAPL • STAPL (Standard Template Adaptive Parallel Library): – Collection of generic parallel algorithms, distributed containers & run-time system (RTS) – Inter-operable with Sequential Programs – Extensible, Composable by end-user – Shared Object View: No explicit communication – Distributed Objects: no replication/coherence – High Productivity Environment SmartApps The STAPL Programming Environment User Code pAlgorithms pContainers pRange RTS + Communication Library (ARMI) Interface to OS (K42) OpenMP/MPI/pthreads/native SmartApps SmartApps to RTS to OS Specialized Services from Generic OS Services – OS offers one size fits all services. – IBM K42 offers customizable services – We want customized services BUT…. we do not want to write them Interface between SmartApps(RTS) & OS(k42) • Vertical integration of Scheduling/Memory Management SmartApps Collaborative Effort: • STAPL (Amato/Rauchwerger) • STAPL Compiler (Rauchwerger/Stroustrup/Quinlan) • RTS – K42 Interface & Optimizations (Krieger/Rauchwerger) • Applications (Amato/Adams/ others) • Validation on DOE extreme HW BlueGene (Moreira) , possibly PERCS (Krieger/Sarkar) Texas A&M (Parasol, NE) + IBM + LLNL ZeptoOS Studying Petascale Operating Systems with Linux Argonne National Laboratory Pete Beckman Bill Gropp Rusty Lusk Susan Coghlan Suravee Suthikulpanit University of Oregon Al Malony Sameer Shende ZeptoOs Observations: • Extremely large systems run an “OS Suite” – BG/L and Red Storm both have at least 4 different operating system flavors • Functional Decomposition trend lends itself toward a customized, optimized point-solution OS • Hierarchical Organization requires software to manage topology, call forwarding, and collective operations QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture. ZeptoOs ZeptoOS • Investigating 4 key areas: – Linux as an ultra-lightweight kernel • Memory mgmt, scheduling efficiency, network – Collective OS calls • Explicit collective behavior may be key (DLLs?) – OS Performance monitoring for hierarchical systems – Fault tolerance ZeptoOs Linux as a Lightweight Kernel What does an OS steal from a selfish CPU application? • • Purpose: Micro benchmark measuring CPU cycles provided to benchmark application Helps understand “MPI-reduce problem” and gang scheduling issues ZeptoOs Collective OS Calls • Collective messaging passing calls have been very efficiently implemented on many architectures • Collective I/O calls permit scalable, efficient (non-Posix) file I/O • Collective OS calls, such as dynamically loading libraries, may provide scalable OS functionality ZeptoOs Scalable OS Performance Monitoring (U of Oregon) • TAU provides a framework for scalable performance analysis • Integration of TAU into hierarchical systems, such as BG/L, will all us to explore: – Instrumentation of light-weight kernels • Call forwarding, memory, etc – Intermediate, parallel aggregation of performance data at I/O nodes – Integration of data from the OS Suite ZeptoOs Exploring Faults: Faulty Towers Memory • • • Dial-a-Disaster MPI/Net Kernel Disk Middleware Modify Linux so we can selectively and predictably break things Run user code, middleware, etc at ultra scale, with faults Explore metrics for codes with good “survivability” It’s not a bug, it’s a feature! Simple Counts • OSes (4): Linux (6.5), K-42 (2), Custom (1), Plan 9 (.5) • Labs (7): ANL, LANL, ORNL, LBNL, LLNL, PNNL, SNL • Universities: Caltech, Louisiana Tech, NCSU, Rice, Ohio State, Texas A&M, Toronto, UIUC, UTEP, UNM, U of Chicago, U of Oregon, U of Wisconsin • Industry: Bell Labs, Cray, HP, IBM, Intel, CFS (Lustre), Quadrics, SGI Apple Pie • Open source • Partnerships: Labs, universities, and industry • Scope: basic research, applied research, development, prototypes, testbed systems, and deployment • Structure: “don’t choose a winner too early” – Current or near-term problems -- commonly used, open-source Oses (e.g., Linux or FreeBSD) – Prototyping work in K42 and Plan 9 – At least one wacko project (explore novel ideas that don’t fit into an existing framework) A bit more interesting • Virtualization – Colony • Adaptability – DAiSES, K42, MOLAR, SmartApps – Config, RWK • Usage model & system mgmt (OS Suites) – Colony, Config, MOLAR, Peta-scale SSI, Zepto • Metrics & Measurement – HPC Challenge (http://icl.cs.utk.edu/hpcc/) – DAiSES, K42, MOLAR, RWK, Zepto • Fault handling – Colony, MOLAR, Scalable FT, Zepto continued • Managing the memory hierarchy • Security • Common API – K42, Linux • Single System Image – Peta-scale SSI • Collective Runtime – Zepto • I/O – Peta-scale SSI • OS Noise – Colony, Peta-scale SSI, RWK, Zepto Application Driven • Meet the application developers – OS presentations – Apps people panic -- what are you doing to my machine? – OS people tell ‘em what we heard – Apps people tell us what we didn’t understand