Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Mobilab Dependability analysis of the Java Virtual Machine Salvatore Orlando Ph.D. student University of Naples Federico II [email protected] www.mobilab.unina.it [email protected] JVM Dependability Analysis 1 ::. Presentation Salvatore Orlando Ph.D. Student at the University of Naples “Federico II” MobiLab Research Group member Advisor: Prof. Stefano Russo Ph.D. Research Programme: Dependability attributes of Virtual Execution Environments Current work: Java Virtual Machine monitoring for dependability assessment www.mobilab.unina.it [email protected] JVM Dependability Analysis 2 ::. Outline • Rationale • Research work about the Java Virtual Machine • Research work about dependability issues of the JVM • Why focus on Sun Hotspot JVM • Internal architecture of the Sun Hotspot® 5.0 JVM • JVM Monitoring for dependability assessment • Failures and sources of errors in the JVM • Our Approach • Monitoring and management technologies (JSR-174 & JSR-163) • JMX (Java Management Extensions) and the platform MBeans • JVMTI (JVM Tool Interface) • The distributed monitoring system • Architecture of the monitoring system • Details of the JVM monitoring subsystem • Data Analysis methodology • Future work • Assessing JVM dependability • Augmenting the JVM with Fault Tolerance • Moving to the «mobile side» www.mobilab.unina.it [email protected] JVM Dependability Analysis 3 Advices, Hints, Questions and Criticisms are welcome! www.mobilab.unina.it [email protected] JVM Dependability Analysis 4 ::. Just a bit of history… (of the evolution of the Java Platform) Java is born, but it is only used for web applets and small digitally controlled consumer devices. Java is still at an embryonal state. Only after 4 years Java is unveiled to the community 1991 1993 www.mobilab.unina.it The Java Platform begins to be widely used, especially in embedded devices 1995 1997 Always more enterprises behave on the Java Platform for their corporate applications Java is starting to be used in missionJava becomes critical and one of the most dependable important applications platforms for cellular phones Java is now widely employed also in web applications 1999 2001 2003 The dawning of Java History Yesterday’s History Nowadays history [email protected] 2005 JVM Dependability Analysis 5 ::. Just a bit of history… (Last Milestones and near future) October 1, 2004: J2SE 5.0 Released Waiting for J2SE 6.0 release and the MVM (Multitasking Virtual Machine) www.mobilab.unina.it [email protected] JVM Dependability Analysis 6 ::. Internal architecture of the Sun Hotspot® 5.0 JVM VM Runtime •The VM Compilation Runtime includes the core components of the JVM. Unit It is in charge of: JVMTI Compilation Policy Manager Deoptimizer VM core JNI Class Loader •Dispatching andServer executing Bytecode instructions Client Interpreter Compiler Compiler runtime •Managing threads Runtime and Runtimeobjects’ locks •… Server Client (Opto) (C1) Interpreter and profiling •Collecting dataCompiler for debugging Compiler Shared Runtime OS Interface •Handling the following components OS MMU Assembler Timer Synchronizer Management Services OS Thread I/O Time •The Memory Management Unit is in charge of: •Allocating oops (i.e.: a Class, a method, an object…) •Managing objects finalization Concurrent •Managing objects generations Serial Low-Pause •Performing Garbage Collection Garbage Collectors •… Throughput Permanent Generation •The Compilation unit is in charge of: Young Generation Tenured Generation Method Area Thread-specific data (handleArea, ResourceArea) Memory Management Unit Native method stack Method stack •Discovering «Hotspots» in Java Applications frame SharedHeap •Compiling or interpreting a Java method PC Registers •Transforming intermediate bytecode representation into native assembly code JVM Heap •… www.mobilab.unina.it [email protected] JVM Dependability Analysis 7 ::. Rationale • Why focus on the Sun Hotspot® JVM Many implementations of the Java Virtual Machine specification available: Jikes RVM Kaffe MacOS RT for Java Blackdown JRockit … But… The Sun’s JVM represents the most widespread implementation The Sun’s JVM is the most complete (and complex) JVM implementation www.mobilab.unina.it [email protected] JVM Dependability Analysis 8 ::. A little snapshot of the research about the JVM Period: 2000-2005 Data from 114 articles published in IEEE and ACM conferences and reviews 6% 23% 14% 17% 9% 11% VM optimizations and technological improvements Performance measurements and improvements Real-time Fault tolerance www.mobilab.unina.it 20% Security Issues JVM-tools Hardware JVM/Java-based OS [email protected] JVM Dependability Analysis 9 ::. Research about fault tolerance features of the JVM Only a little part of the research work on the Java Virtual Machine addresses dependability issues The greater part of these works are focused on specific components of the JVM: … techniques for protecting unsafe operations in the garbage collector … techniques for protecting objects against memory errors The only works that address fault tolerance in the entire virtual machine make use of state machine replication (Hypervisor-based Fault tolerance) to build a fault-tolerant JVM and it has been carried out at University of Texas in Austin and Technion Israel Institute of Technology www.mobilab.unina.it [email protected] JVM Dependability Analysis 10 ::. Some clues about sources of errors in the JVM 2,4% 0,8% 1,6% 3,2% 4,8% 7,9% 11,1% 52,7% 11,1% 22,3% 11,1% 19,0% 15,1% 11,9% VM C ore Synchronizer Deoptimizer www.mobilab.unina.it Shared Runtime C lassloader Interpreter Runtime Unit Garbage C ollector JNI JIT C ompiler Memory Management Unit 25,0% OS Interface Shared Heap Optimizing JIT C ompiler C ompilation Unit Data taken from Sun and Jikes Bug Databases. 99 VM-related segnalations selected • Among over 500 bug reports related to the virtual machine, we selected only the ones addressing non alwaysreproducibile failures or failures reproducible only with specific workloads. • Then we furtherly refined the selection deleting reports with few details or clearly related to bugs in source code. [email protected] JVM Dependability Analysis 11 ::. At the present date… It is recognized that hardware faults longer ainto major Most of the research work related to are faultno tolerance thesource JVM of faults. greater part ofcomponents faults in modern are concerned with focusedThe only on singular of thesystems Virtual Machine, software. especially the memory management unit The JVM is becoming more and more complex,itbeing nowadays By analyzing failure reports in Bug Databases is possible to argue almost to a real operating system that thecomparable memory management unit is not the only (and not the major) source errors inwidely the JVM Java is of nowadays used in enterprise software systems Dependability issues to of the Virtual Machine have never been Java is starting be Java employed in mission-critical systems. completely fleshed out,been sinceused a thorough study its dependability For instance, Java has to develop theof ROver Sequence Editor has never been done (ROSE), a component of the Rover Sequencing on Visualization Program (RSVP), to control the Spirit robot the exploration of Summarizing, weused argue that it’s necessary to in focus on Mars the virtual machine as a whole, taking into account the role played by each of its components yet… www.mobilab.unina.it [email protected] JVM Dependability Analysis 12 ::. Failures,sources of errors and error propagation in the JVM (1/5) Java Application Objects Accumulated errors Compilers Garbage Collector Java Exception Handler VM-related errors Error manifestation Errors in Virtual Machine components Accumulated Errors www.mobilab.unina.it When an error occurs in an Internal Component of the Java Virtual Machine: Application Failures VM Core JVM layer Errors in JVM Components Applicationrelated errors Memory Subsystem Synchronizer Application Service Interface JNI Virtual Machine Service Interface Operating System Service Interface Class Loader Errors in Virtual Machine components Application Layer Sources of errors • If it is intercepted by the Java Exception Handler, The error is thrown as a Java Exception. We can observe Java exception messages such as OutOfMemory or IllegalMonitorState • If the Java Exception handler could not intercept it, the Error leads directly to an Application failure, so we get OS-level error messages (or Internal VM error messages) such as SIGSEGV or SIGBUS [email protected] JVM Dependability Analysis 13 ::. Failures,sources of errors and error propagation in the JVM (2/5) 31,7% A Java exception is thrown 18,3% The JVM dies with no messages or hangs 11,0% 39,0% JAVA_EXC EPTION Data taken from Sun and Jikes Bug Databases. 99 VM-related segnalations selected www.mobilab.unina.it OS_MESSAGE VM_ERROR HANG_SILENT_C RASH The JVM dies with an internal error message i.e.: InternalError, AssertionFailure The JVM dies with an Operating System error message i.e.: SIGSEGV, SIGBUS, Access Violation [email protected] JVM Dependability Analysis 14 ::. Failures,sources of errors and error propagation in the JVM (3/5) Java Application Objects Accumulated errors Garbage Collector Applicationrelated errors Java Exception Handler Application Failures VM-related errors Memory Subsystem Error manifestation VM Core Synchronizer Errors in Virtual Machine components Accumulated Errors JVM layer www.mobilab.unina.it • Developer’s mistakes (we assume that FileNotFoundException programs are written by skilled developers…) • Unexpected conditions in the InternalError Application Sources of errors VM related Ambiguous Compilers ArrayIndexOutOfBoundEx ception Exceptions are raised due to: Application related Errors in Virtual Machine components Application Service Interface JNI Virtual Machine Service Interface Operating System Service Interface Class Loader OutOfMemoryError Some of these exceptions are clearly application-related, others are VM (or OS) related and others ambiguous: it is IllegalMonitorStateExc not clear if the eption exception is due to errors generated at the application or at the ConcurrentModification VM layer. Exception Application Layer [email protected] JVM Dependability Analysis 15 ::. Failures, sources of errors and error propagation in the JVM (4/5) • When an ambiguous exception is raised, discriminating whether it is due to a VM-related error or an application-related one it is an hard task. • Indeed, Moreover, propagation phenomena be to: observed in the Virtual anerror ambiguous exception could becould related Machine, since errors in JVM could lead to error in other components and/or • An unexpected condition in the application (i.e.: a date has a application failures (as shown in previous example) format different from the expected one) • Errors in file system (i.e.: a required file does not exist) • Errors in hardware resources (i.e: a device cannot be opened) • Errors in the Virtual Machine (i.e.: a ConcurrentModificationException is thrown on an object even if no thread is operating on that object) •The data analysis methodology will have to deal with this phenomenon, so •We going toalgorithms work with “trusted'' i.e. applications that are errorare coalescing have to beapplications, implemented known to be bug-free and tested to work fine under stress condition. • In this way we remove ambiguous and application-related exceptions. Such exceptions are always due to an error in the VM or in the underlying OS. By analyzing OS logs we should be able to distinguish VM-related exceptions from OS-related ones. www.mobilab.unina.it [email protected] JVM Dependability Analysis 16 ::. Failures,sources of errors and error propagation in the JVM (5/5) 1 Compilers Garbage Collector Errors in Virtual Machine components Java Application Objects Accumulated errors 2 5 4 Error manifestation Errors in Virtual Machine components Accumulated Errors www.mobilab.unina.it 2. An application thread calls this method. An exception is raised since it contains bytecode for objects belonging to another class 3. The error propagates to the VM core components since the wrong VM operations are scheduled VM-related errors VM Core JVM layer 1. A fault in the compiler cauase the wrong version of a compiled method to be loaded Application Failures 3 Synchronizer A sufficiently realistic example Applicationrelated errors Java Exception Handler Memory Subsystem Application Service Interface JNI Virtual Machine Service Interface Operating System Service Interface Class Loader Sources of errors 4. Other exceptions (i.e.: InternalError) are raised 5. The user observes an application failure (the VM has crashed) Application Layer [email protected] JVM Dependability Analysis 17 ::. Approach (1/2) Monitoring the behavior of JVM components in order to gather fieldbased data to be able to ….performing dependability analisys ….identifying dependability bottlenecks ….analyzing pathological behaviors www.mobilab.unina.it [email protected] JVM Dependability Analysis 18 ::. Approach (2/2) Our first goal: Monitor Internal JVM Behavior Use built-in reporting and logging features (e.g.: -verboseGC, failure reports) Use standardized monitoring technologies (JSR-3, JSR-163) • Not enough information available • Data format strictly dependent from the particular implementation www.mobilab.unina.it Manually instrument JVM source code • Very hard task (21.336.425 bytes in 1.455 files) • High risk of bug injection • The work has to be entirely repeated to monitor another implementation of the JVM [email protected] JVM Dependability Analysis 19 ::. Monitoring and management technologies (2/4) JMX (Java Management Extensions) MBean server Management applications It isto a registry for MBeans. They connect the managed JVM It provides the services for through its connectors. manipulating MBeans In this way these application can use Agent Services MBeans and resource MBeans to retrieve Agent services information or manage Applications Control the resources and the JVM anditself make them available to remote management applications. (i.e: sending notification Connectors when the value of the Listeners for requests managed resource from remote changes) Management MBeans (Management applications. Beans) They instrumentation use RMI Provide for technology Managed resources www.mobilab.unina.it [email protected] JVM Dependability Analysis 20 ::. Monitoring and management technologies (3/4) JPPA (Java Platform Profiling Architecture) Platform MBeans (MXBeans) • • • • MBeans registered in the MBean Server at bootstrap time They provide data relative to internal VM state (aggregate and detailed values) They provide Functions for managing the virtual machine They are accessible through a JMX-compliant management application (i.e.:Jconsole) JVMTI Functions www.mobilab.unina.it Monitored JPPA-compliant JVM JVMTI events [email protected] JVMTI Agent • A JVMTI agent is a shared library that starts along with the JVM • In the JVMTI agent there are defined Callback functions for internal JVM Events • Events can be dinamycally enabled/disabled • The greater part or callbacks can use JVMTI functions for: •examining internal JVM state •Managing the JVM Callbacks JVMTI (Java Virtual Machine Tool Interface) JVM Dependability Analysis 21 ::. Distributed Monitoring System Architecture (1/2) JMX Server Connector RMI Connection JMX Client Connector Monitor MBeans Platform MBeans Notify MBeans Monitored JVM JVMTI Agent Benchmark App Used to send notification to the Data Collector and to query JVM state from the Monitored JVM XML state description processing Retrieve data JMX management App Notifications Data Collector Data Collection • State updated when a notification is received • Event notification is written to a Database • Interested JVM component(s) state is written to an XML file Collected data JVM Monitoring Subsystem • Event filtering • State change notifications are sent www.mobilab.unina.it Data Analyzer Event Database XML state descriptors • Each entry in the database represent an event notified by a JVM • Each entry is linked to an XML file describing the state of the component affected by the above mentioned event [email protected] JVM Dependability Analysis 22 ::. Distributed Monitoring System Architecture (2/2) Monitored JVM & Data Collector: Interaction diagram JVMTI Agent Monitor MBean 2. Component state updated locally 1. Event raised 3. Periodically checks for changes in component state Notify MBean 5. Query Component state (performs event filtering) Data Collector 4. Send notification to the data collector 6. Update DataBase www.mobilab.unina.it [email protected] JVM Dependability 23 ::. JVM monitoring subsystem details NotifyMBeans (JMX – agent services level) to observe MonitorMBeans and send TheJVMTI Collector queries the Monitor MBeans update theare descriptors ofevents the The Callback MonitorMBeans functions agent (JMX MonitorAgent use – JVMTI resource Function starts instrumentation along to gather with informations the level) JVM and updated handles about by the notification to the Collector when there are relevant changes in the statestate of state of the virtual Machine raised of MonitorAgent the virtual by the machine JVMTI through back-end global references in the virtual in JNI machine environment. A Monitor the MonitorMBeans, which in turns reflect the state of the various MBean is defined for each component of the JVM. components of the JVM 2) State data gathered by Callbacks in Monitor Agent 5 JVMTI Functions Notifications to Collector 4 Notify MBeans 3) MonitorMBeans updated 4) NotifyMBeans send notifications to the Collector 5) Collector Queries MonitorMBeans www.mobilab.unina.it 2 Monitored JPPA-compliant JVM Callback Functions 1) Events handled by Callbacks in Monitor Agent Queries from Collector Running Application(s) Monitor MBeans 3 JNI Environment 1 JVMTI events MonitorAgent Global references to MonitorMBeans [email protected] JVM Dependability Analysis 24 ::. Distributed Monitoring System: collected data details (1/3) Class Loader • JVMTI Events intercepted ClassLoad (Class File read) – ClassPrepare (Verification process completed) • State descriptor State of each class handled by the Class Loader: Error, Loaded, Verified, Prepared, Unloaded Compiler Unit • JVMTI Events intercepted Compiled method load (method compiled and loaded), Compiled method unload (method unloaded or moved), Method entry, Method exit • State descriptor Map of compiled/interpreted methods including: method name and, for each compiled form of the method: compiled code size, code starting address, VM –specific compiler information. www.mobilab.unina.it [email protected] JVM Dependability Analysis 25 ::. Distributed Monitoring System: collected data details (2/3) Runtime core • JVMTI Events intercepted Thread start - Thread end - Monitor enter (attempt to acquire a lock owned by another thread) - Monitor entered (lock acquired)- Monitor wait (waiting for an object) - Monitor waited (finished waiting) – Exception (Exception thrown) – Exception catch (Previously thrown exception has been catched) – Method Entry (Java or native method entered) – Method Exit (generated upon exit from a Java or native method) • State descriptor State of each Java thread including: • its stack trace • Its current lock status (locks acquired, locks which the thread is waiting for) • Its possible pending exception • Its method invocation graph (reports also the timestamp of each method entry and method exit event) • Its timeline (a timestamp is applied each time the thread changes state) www.mobilab.unina.it [email protected] JVM Dependability Analysis 26 ::. Distributed Monitoring System: collected data details (3/3) Memory Unit MemoryManagement Management Unit • JVMTI Events intercepted Garbage collection start (a serial garbage collection starts) – Garbage collection • State Descriptor finish (a serial garbage collection finishes) – VM Object Allocation (new object • Garbage collector timeline allocation) – Object Free (the Garbage collector frees an object) • Object allocation and deallocation schedule • Heap evolution history (for to both the young the tenured Huge-sized heap – It is possibile take a dump ofand the entire shared generation) heap, but it will take too long (especially if graph the dump has to be transferred over the network) • Reachable objects Data about oops in heap area are gathered • location and size of each reachable object iterating over reachable objects and sweeping the entire heap (like the GC does), obtaining in this • Access/Modification flags for each reachable object way syntethic information about objects (location, size and checksum) • Hash value describing the content of each reachable object and JNI are Stop-the-world collector – when this collector works Java thread blocked. Defer notification of events related to garbage collection until the garbage collection finishes www.mobilab.unina.it [email protected] JVM Dependability Analysis 27 ::. Benchmark applications and Workload definition Web applications: the JVM is executed on the server side to dispatch requests to a back-end architecture • Jakarta Tomcat web server with J2SE 5.0 (Tiger) • JSP+Servlets applications • Web-stress tools to impose very high workloads to the VM Client and server applications, such control or scientific ones. The JVM is deeply used for GUI design or mathematical computations • Javolution library benchmarks • Other benchmarks… In order to make the field data measurement campaign more effective, JVM is employed with two families of workloads: • Workloads designed to stress VM components separately • Workloads designed to stress the entire virtual machine as much as possible following the behavior of the above mentioned categories of applications www.mobilab.unina.it [email protected] JVM Dependability Analysis 28 ::. Data Analisys (1/2) 1. Detect failures at the application layer 2. Distinguish failures that are due to failures in the underlying layers from failure that are due to the JVM 3. Given the instant tn where the failure has been detected, analyze the evolution of the state of the JVM in the interval [tn-D;tn], in order to identify the component where the source of the failure was located. D is a thresold that we assume to be the longest time that a fault in the JVM takes to manifest as a failure at the application layer. 4. By analyzing data related to several failures identify patterns of pathological behavior in the JVM. www.mobilab.unina.it [email protected] JVM Dependability Analysis 29 ::. Future Work: Assessing JVM dependability We aim to give an answer to the following question: «Is Java ready to support dependable applications?» • The field-measurement campaign started aims to identify the most critical components in the virtual machine and evaluate their dependability attributes and bottlenecks • Our monitoring system will be applied to Sun Hotspot® and IBM JRockit® on different platforms (Solaris, Linux, win32) At the present date only Sun and IBM implementations are compliant to JSR-163 specification www.mobilab.unina.it [email protected] JVM Dependability Analysis 30 ::. Future Work: Augmenting the JVM with fault-tolerance JRE provides no direct support for fault tolerance. Hence Java applications: Completely ignore failures or… Provide application-dependent fault detection and recovery strategies or… Delegate fault tolerance to external systems such as transactional databases In previous works (Alvisi,Friedman) fault tolerant JVMs were developed using state replication We aim at building a fault-tolerant JVM applying fault tolerance techinques to internal JVM components www.mobilab.unina.it [email protected] JVM Dependability Analysis 31 ::. Future Work: Moving to the «mobile side» • Java is nowadays widely used in embedded devices such as PDAs and cellular phones. • Such devices are often shipped with «embedded JVMs» • Such JVMs are a «compact» edition of the JVMs for desktop and server systems, they include only a limited subset of the features included in those JVMs. • At present time we don’t know any JVM for embedded devices compliant to JSR-3 or JSR-163 • Our goal is to perform a dependability analysis of these JVMs To this aim we have to address the following issues: • Investigate about the architeture of these JVMs • Develop methodologies for their monitoring www.mobilab.unina.it [email protected] JVM Dependability Analysis 32 And now it’s time for questions ! Contact info: [email protected] Mobilab Research Group http://www.mobilab.unina.it/ www.mobilab.unina.it [email protected]