Download Install Zing Licensing

Zing Installation and
Administration Guide
Release 15.05.0.0
May 15, 2015
Zing Installation and Administration Guide
The Azul Systems® Zing™ platform uses the Zing Virtual Machine (ZVM) to run Java™ technologybased applications. In the Zing product, the Zing System Tools (ZST) component, installed on each
ZVM host system, manages the elastic and highly scalable shared memory resources.
Zing Vision (ZVision) and Zing Vision Robot (ZVRobot) provide ZVM process profiling and diagnostics.
Zing Documentation Overview
Zing documentation includes installation, administration, and reference information. It is not intended to
be read in its entirety. Select the only the tasks you need to perform and complete those related steps.
l
Release Notes
Release notes cover each Zing component and their associated release version.
l
System Requirements
Lists the systems, environment, hardware, and software requirements for all components.
l
Quick Start
Brief list of tasks to complete installation.
l
Installation and Configuration
l
Installing and Configuring Zing System Tools
Required for all basic Zing installations. Describes ZST installation and
configuration tasks.
l
Installing Zing Standalone Licenses
Required if you are not using Zing Pool licensing. Describes Zing License Server
installation and use for managing Pool Zing licenses.
l
Installing the Zing Virtual Machine
Required for all basic Zing installations. Describes ZVM installation tasks.
Includes installing ZVision and ZVRobot.
l
Using the Zing Virtual Machine
Provides an overview of the ZVM, and describes the command line options that
the ZVM supports.
l
Specific Use-Cases: Zing Installation
l
Using Zing through Amazon Web Services
Describes how to create an Amazon Elastic Compute Cloud (EC2) instance for
Zing.
l
Installing Zing System Tools using DKMS
Describes how to install ZST using the Dynamic Kernel Module Support (DKMS)
framework to automatically build the kernel module.
Azul Systems
Confidential and Proprietary
2
Zing Installation and Administration Guide
l
Installing and Configuring zing Using Read-Only Images
Describes how to install and configure Zing using read-only images of ZST and
ZVM for use in an Enterprise File System (EFS).
l
Installing and Using Pool Licensing
Required only if you have a Pool license contract with Azul. Describes Zing
License Server installation and use for managing Pool Zing licenses.
l
Zing Pilot Assist Kit
Describes using the Zing Pilot Assist Kit, that simplifies the Zing evaluation
process.
l
Zing Advanced Tuning
l
Zing Advanced Tuning: ReadyNow!
Describes Zing's advanced tuning option, ReadyNow! that provides a set of flags
and options to help your Java applications be ready to perform real transaction
processing at designated start times.
l
Zing Advanced Tuning: @ Contented
Describes Zing's advanced tuning use of the @sun.misc.Contended annotation
to reduce cache contention across multiple threads.
l
Starting or Stopping Zing System Tools Services
l
Upgrading or Uninstalling Zing Components
l
Zing Tools Management
l
Understanding Memory Use
Describes how memory is used to support Java applications in a environment
l
Using Zing Vision
Describes configuring and using the ZVision profiling and diagnostics tool for the
ZVM.
l
Using Zing Vision Robot
Describes configuring and using the ZVRobot profiling and diagnostic snapshot
collection tool for the ZVM.
l
Using Zing System Tools
Describes the ZST tools, zing-ps.
l
Troubleshooting Zing
Lists errors, messages, and possible solutions.
l
Appendix A - Zing Installation Prerequisites Details
Appendix B - Not utilized at this time.
Azul Systems
Confidential and Proprietary
3
Zing Installation and Administration Guide
l
Appendix C - Zing System Tools Installed Files
Azul Systems
Confidential and Proprietary
4
Table of Contents
Table of Contents
Zing Installation and Administration Guide
Zing Documentation Overview
1
2
Table of Contents
5
Zing Release Notes
26
Zing Component Versions
26
New Features and Updates
26
Zing Virtual Machine Resolved Issues
27
Advisory!
27
Zing Virtual Machine Known Issues
31
Zing System Tools Resolved Issues
32
Zing System Tools Known Issues
36
Zing System Requirements
37
Zing Components
37
Zing Machine Requirements
37
Operating System Requirements
38
DKMS Supported Operating Systems for ZST
42
CPU Requirements
42
Memory Requirements
43
Hard Disk Storage Space Requirements
43
Browser Requirements
43
Listening Connection Requirements
44
Zing Component Version Compatibility
44
Azul Systems
Confidential and Proprietary
5
Zing Installation and Administration Guide
Zing ZST and API Versions
45
Installation Quick Start
46
Ensure your environment meets the system requirements.
46
Download the ZST and ZVM installation files.
47
Install the Zing System Tools (ZST).
48
Configure Zing memory management.
48
Install Zing Licensing
48
Standalone Zing License
48
Install the Pool Server
49
Install the Zing Virtual Machine (ZVM).
49
Prepare to run Zing
49
Run your Java applications with Zing
50
Zing Installation and Configuration
Installing and Configuring Zing System Tools
51
51
Complete Prerequisites
51
Install Zing System Tools
52
Configure Zing Memory
54
Completing Zing Installation
55
Install Zing License File
56
Install Zing Virtual Machine
56
Verify Zing Installation
56
Modifying Zing Memory Configuration
57
Modify Zing Memory Allocation
58
Use System Method to Allocate Zing Memory
59
Azul Systems
Confidential and Proprietary
6
Table of Contents
Use Percentage of System Memory for Zing Memory
60
Use Exact Amount of System Memory for Zing Memory
60
Use Java Method to Allocate Zing Memory
60
Specify Contingency Memory Pool
61
Installing Zing Standalone Licenses
62
Zing Licensing Overview
62
Installing Zing Licenses
62
Renewing Zing License
63
Installing the Zing Virtual Machine
63
Install the Zing Virtual Machine
63
Completing Post ZVM Installation Tasks
66
Applying JDK Customizations
66
Setting an Unlimited ulimit
66
Ensuring Sufficient Memory
66
JDK Configuration Compatibility
67
Verifying Zing Virtual Machine Installation
Using the Zing Virtual Machine
67
68
Overview Zing Memory Management Processes
68
Generational Pauseless Garbage Collection (GPGC)
68
Zing Memory Allocation
68
Compatibility with HotSpot VM Command Line Options
69
Command Line Options for the Zing Virtual Machine
70
Zing Virtual Machine Options
70
Garbage Collection Options
71
Azul Systems
Confidential and Proprietary
7
Zing Installation and Administration Guide
ExplicitGC Options
71
Reserved Memory Command
71
Statistics Monitoring Command
72
Usage Syntax
72
Samples
72
Output Header Descriptions
73
ZVM Mini-Core File Option
75
ZVM Mini-Core File Requirements
75
Setting the ZVM Mini-Core File Option
75
Run a Java Application
76
Native Applications with Zing
76
Invoking Native Applications with Zing
76
Invoking the ZVM in Native Applications
77
Case 1: Native application dlopens libjvm.so and uses dlsym to find 'JNI_
CreateJavaVM’.
77
Case 2: Native application directly calls JNI_CreateJavaVM by linking against libjvm.
78
ZVM Instance Configuration Recommendations
78
RHEL Performance Tuning Options
78
Installing the Tuned Package
79
Running Tuned
79
Tuned Options
80
Automatically Starting Tuned
81
Disabling All Tuning
81
Specific Use-Cases: Zing Installation
Azul Systems
Confidential and Proprietary
82
8
Table of Contents
Using Zing through Amazon Web Services
82
Sign In to Amazon Web Services
82
Build an EC2 Server
84
Stop EC2 Instance
86
Start EC2 Instance
86
Prepare AWS EC2 Instance for Zing
86
Prepare Required Files
87
Start your AWS Instance
87
Login Remotely to your AWS Instance
88
Create a Non-Root User
88
Copy the Zing Installation Files to the AWS Instance
89
Install Zing Components
89
Installing Zing System Tools using DKMS
Installing and Configuring Zing System Tools
90
91
Complete Prerequisites
91
Install Zing System Tools
93
Configure Zing Memory
95
Completing Zing Installation
97
Install Zing License File
97
Install Zing Virtual Machine
97
Verify Zing Installation
97
Modifying Zing Memory Configuration
98
Modify Zing Memory Allocation
99
Use System Method to Allocate Zing Memory
Azul Systems
Confidential and Proprietary
100
9
Zing Installation and Administration Guide
Use Percentage of System Memory for Zing Memory
101
Use Exact Amount of System Memory for Zing Memory
101
Use Java Method to Allocate Zing Memory
101
Specify Contingency Memory Pool
102
Installing and Configuring Zing Using Read-Only Images
103
Installing Zing with Read Only Images Overview
103
Complete Prerequisites
104
Install Zing System Tools on the Zing Server
105
Prepare the Zing System Tools Server
106
Install Zing System Tools on the ZST Nodes
107
Configure Zing Memory on the ZST Nodes
109
Install Zing License File on the ZST Nodes
110
Install Zing Virtual Machine on the ZST Nodes
111
Verify Zing Installation on the ZST Nodes
112
Modifying Zing Memory Configuration
113
Modify Zing Memory Allocation
114
Use System Method to Allocate Zing Memory
115
Use Percentage of System Memory for Zing Memory
116
Use Exact Amount of System Memory for Zing Memory
116
Use Java Method to Allocate Zing Memory
116
Specify Contingency Memory Pool
117
Upgrading Zing System Tools in Read Only Environments
118
Uninstalling the Zing System Tools from a ZST Node
118
Uninstalling the Zing System Tools from the Zing Server
119
Azul Systems
Confidential and Proprietary
10
Table of Contents
Installing and Using Pool Licensing
120
Zing Licensing Overview
120
Pool Licensing Components
120
Zing License Locking
121
Pool Licensing Process Overview
121
Installing and Initializing the Zing License Server
122
Install the License Server
122
Connect to a Zing License Server
125
Store the Pool License Zip File
125
Add a Pool License
126
Configuring ZVM Hosts for Zing Pool Licensing
zing-licensed Command Options
127
129
Managing the Zing Licensing Server
131
Change the Listening Ports
131
Delete a Pool License from License Server
133
Backup the License Server
133
Restore the License Server
134
Configure Email Server
135
Configure Notifications
135
Managing Zing Licenses
137
Viewing Zing License Status and Activity
View Active Server Licenses
138
View Audit Log of Licensing Activity
139
Modifying Zing License Settings
Azul Systems
137
140
Confidential and Proprietary
11
Zing Installation and Administration Guide
Use an Amazon ID for ZVM Host ID
141
Require ZVM Host Authentication
141
Modify Zing License Lease Interval
141
Reserving Zing Licenses
142
Reserve Zing Licenses from the Pool
142
Add a ZVM Host Reservation
142
Remove a ZVM Host Reservation
143
Removing, Resetting, or Revoking Zing Licenses
143
View ZVM Hosts with Active Zing Licenses
143
Reset All ZVM Host Zing Licenses
144
Remove Expired Zing Licenses
145
Revoke Zing License on a ZVM Host
145
Managing ZVM Hosts
146
View ZVM Hosts in the License Server Pool
146
Disable a ZVM Host
147
Enable a ZVM Host
148
Delete a ZVM Host
148
Unconfigure a ZVM Host
148
Managing License Server Users
149
Add a User Role
149
Edit a User Role
151
Delete a User Role
151
Add a User
151
Change User Passwords
152
Azul Systems
Confidential and Proprietary
12
Table of Contents
Delete a User
152
Uninstalling the Zing Licensing Components
153
Remove the License Server
153
Remove your Pool License
153
Upgrading the Zing Licensing Components
Upgrade the Pool License Server
153
154
Zing Pilot Assist Kit
156
Zing Pilot Assist Kit Quick Start
156
Selecting and Validating Zing PAK Download Packages
158
Understanding the system-administrator-todo-list File
159
Sample system-administrator-todo-list.txt File
Understanding the Validation Profile
159
161
Sample validation File
162
Sample validation Script Output
162
Understanding the Options Profile
163
Sample options Profile
164
Configuring the Zing Pilot Assist Kit
166
Sample startup.conf File
166
Converting Zing from Evaluation to Production
167
Zing Pilot Assist Kit Operating System Requirements
167
Zing Advanced Tuning
172
Zing Advanced Tuning: ReadyNow!
172
ReadyNow! Overview
172
Run ReadyNow!
173
Azul Systems
Confidential and Proprietary
13
Zing Installation and Administration Guide
ProfileLog Use Case
173
Pre-JIT ProfileLog Use Case
174
Run a Profile Restoration
174
Display Summary Statistics Using jstat
174
Summary Statistics - Example Output A
174
Summary Statistics -- Example Output B
175
Monitoring ProfileLog Effectiveness
175
Using ProfileLog Information
175
Analyzing the Report
176
Tuning Opening Java Code
177
Ensure That Critical Methods are Compiled
178
Compilation Policy
178
Compilation Thresholds
179
Compiler API
179
Reducing Deoptimizations
180
Identifying Deoptimizations with PrintCompilation
180
Identifying Deoptimizations with TraceDeoptimization
181
Reducing Unreached Deoptimizations
183
Using DynamicBranchEliminationLevel as a Flag
183
Using DynamicBranchEliminationLevel as an Option
183
Using UseOldBranchProfileAdjustmentAtDeopt as a Flag
184
Reducing Null_Check Deoptimizations
184
Using ImplicitNullCheck as a Flag
184
Using ImplicitNullCheck as an Option
184
Azul Systems
Confidential and Proprietary
14
Table of Contents
Reducing Unloaded Deoptimizations
184
Using UseEarlyClassLoading as a Flag
185
Using UseEarlyClassLoading as an Option
185
Reducing Uninitialized Deoptimizations
185
Using UseEnhancedClassResolution as a Flag
186
Using UseEnhancedClassResolution as an Option
186
Using EagerInitializationDuringEarlyClassLoading as a Flag
186
Zing Advanced Tuning: @Contended
186
@Contended Overview
186
Using @Contended Annotation
187
Marking a Class as Contended
187
Marking an Instance Field as Contended
187
Marking Multiple Fields as Contended
187
Marking Groups as Contended
188
Zing @Contended Options
188
@Contended References
189
Starting or Stopping the Zing Services
190
Starting Zing Memory
190
Stopping Zing Memory
191
Starting Zing License Daemon
191
Stopping Zing License Daemon
191
Start the License Server
192
Stop the License Server
192
Upgrading or Uninstalling Zing Components
Azul Systems
Confidential and Proprietary
193
15
Zing Installation and Administration Guide
Upgrading Zing System Tools
193
Uninstalling the Zing System Tools
194
Upgrading the Zing Virtual Machine
195
Uninstalling the Zing Virtual Machine
196
Uninstalling the Zing Standalone License
197
Zing Tools Management
198
Understanding Memory Use
198
Understanding Memory Allocation
198
Host System Memory
199
Linux Memory
199
Linux Kernel Memory
199
Linux Process Memory
199
JNI Memory
200
JVM Internal Memory
200
Other Linux Processes Memory
200
Zing Memory
200
Estimating Zing Memory Allocation Example
201
Pause Prevention Memory
202
Contingency Memory for Elasticity
203
Contingency Memory and Generational Pauseless Garbage Collection
203
Sharing Contingency Memory and Memory Leaks
204
Reservable and Reserved Memory
Estimating the Number of Possible Concurrent ZVM Instances
Java Heap Memory
Azul Systems
204
205
205
Confidential and Proprietary
16
Table of Contents
Temporary Java Heap Memory
206
Heap Memory in 32-Bit vs. 64-Bit Systems
206
Setting Reservable Memory
207
Zing Allocation
207
Configuring -Xmx<val>
207
Configuring -XX:JavaMemMax=<val>
208
Using Zing Vision
208
Zing Vision Overview
208
Azul Customer Support
209
Statistical Tick Profiler
209
Enable or Disable Tick Profiler
209
Sorting Tables
209
Configuring ZVision
209
ZVM Command Line Options
210
Configuring the ZVM Host Port
210
Configuring the ZVision Server
211
Accessing ZVision
211
ZVision Interface
212
Overview Tab
Azul Systems
215
Configuration Window
215
Environment Variables Window
216
HotSpot Flags Window
217
Statistics Window
218
Settings Window
219
Confidential and Proprietary
17
Zing Installation and Administration Guide
Groups Window
220
Login Window
220
Features Window
220
Threads Tab
221
Threads List Window
222
Stack Trace Window
223
Deadlocks Window
224
Contention Window
225
Contention Details Window
226
CPU Tab
226
Tick Profile Window
226
Meta Ticks Window
230
Event Tracker
233
Memory Tab
233
Memory Summary Window
233
GC Summary Window
235
GC History Window
237
Browse Object Window
240
Allocated Objects Window
241
Live Objects Window
242
Type Velocity Window
243
Compilers Tab
245
Applications Tab
246
Advanced Configuration Options
Azul Systems
Confidential and Proprietary
247
18
Table of Contents
Using Zing Vision Robot
248
ZVision Overview
248
ZVRobot Benefits
248
Azul Customer Support
249
Configuring the Zing Virtual Machine to Use ZVRobot
249
Installing and Configuring ZVRobot
250
Running ZVRobot and Collecting Data
251
Enable or Disable Tick Profiler
252
Viewing ZVRobot Snapshot Data
252
ZVRobot Snapshot Data
252
Configuration Window
255
Environment Variables Window
256
HotSpot Flags Window
257
Statistics Window
258
Settings Window
259
Features Window
260
Threads List Window
261
Stack Trace Window
262
Deadlocks Window
262
Contention Window
262
Tick Profile Window
264
Memory Summary Window
265
GC Summary Window
266
GC History Window
267
Azul Systems
Confidential and Proprietary
19
Zing Installation and Administration Guide
Compilers – Task Window
270
Class List Window
271
Using Zing System Tools
271
Using the zing-ps Tool
271
Command Return Overview
272
Legend for Memory Usage Labels
273
Additional Field Descriptions
273
Summary of Total Memory Usage
274
Memory Usage for Each Running ZVM Process
274
Fund Breakdown
275
Memory Usage by Account per Process
275
Format Command Sample Options
276
Example 1: No parameters used, returns the number of ZVM processes.
276
Example 2: Comma separated.
276
Example 3: CMD column options.
277
Example 4: Units displayed options.
279
Example 5: Details of specific process by process id (PID).
286
Samples Memory Usage by Account
286
Example 6: To monitor a test run.
286
Example 7: For ZVM fault diagnosis.
286
Example 8: Memory statistics with zero ZVM processes running.
286
Example 9: Memory account details with 1 ZVM process running.
287
Example 10: Memory funding details with 1 ZVM process running.
287
Explanation of zing-ps Fund Breakdown
288
Azul Systems
Confidential and Proprietary
20
Table of Contents
Example Memory Fund Allocation for a Process
289
Use zing-ps and awk Commands to Show Available Heap Sizes in MB
291
Example 11: No JVMs have been started.
291
Example 12: An Azul Inspector 1GB JVM has been started.
292
Troubleshooting Zing
294
Core Files and Core Bundling with Zing
294
Mini-Core File Option
295
Mini-Core File Option Requirements
295
Setting the Option
296
Automatic ZVM Core Bundling
296
Enabling Automatic ZVM Core Bundling
296
Disabling Automatic ZVM Core Bundling
298
Locating Automatic ZVM Core Bundles
299
Manual ZVM Core Bundling
299
Decrypting Core Files
302
Zing License Core Files
303
Zing Installation Errors
304
/user/bin Access
304
Message:
304
Action:
304
Known Intel chip errata can cause spurious system-wide delays
304
Description:
304
Solution:
304
DKMS Installation Troubleshooting
Azul Systems
304
Confidential and Proprietary
21
Zing Installation and Administration Guide
Kernel Mismatch on a DKMS RHEL System
305
Kernel Mismatch on a DKMS SLES System
305
Kernel Mismatch on a DKMS Ubuntu or Debian System
306
Native Launch with ZVM
306
Invoking Native Launch with Zing
306
Invoking the ZVM in Native Launch Apps
306
Case 1: C-main dlopens the libjvm.so and tries to find "JNI_CreateJavaVM” using dlsym.307
Case 2: C-main app directly calls JNI_CreateJavaVM by statically linking libjvm.
Known Issues Running Native Launch Apps on Zing
307
308
Case 1: Virtual address space collisions when native launch applications allocate
memory where ZVM allocates certain internal structures like the CodeCache and the
OopTable.
308
Case 2: Virtual address space collisions when native launch applications allocate
memory where libjvm.so needs to be loaded
309
Case 3: Virtual address space collisions when native launch applications allocate
memory where ZVM allocates other VM internal data structures
309
Case 4: Native launch application uses signal handlers for signals used by the ZVM
309
Loading /usr/lib/zing/libc_hooks2.so
309
Native launch app colliding with areas used or assumed by the ZVM.
309
Case 1: Native launch app allocates/mmaps specfic areas like CodeCache/OopTable
Tick Profiler and C-Heap Leak Detection
Zing Errors
310
310
310
When abrtd running zing core_pattern cannot overwrite /proc/sys/kernel/core_pattern
310
Description:
310
Solution:
310
Azul Systems
Confidential and Proprietary
22
Table of Contents
ZVM WARNING: The Event Tick Buffer Profiling system status is: "Shutdown."
311
Reason:
311
Description:
311
Background:
311
Solution:
312
Crash When Using Tight Polling Loops
312
Description:
312
Background:
312
Solution:
312
Out of Space for OOP (Ordinary Object Pointer) Table - KlassTable
312
Description:
312
Solution:
313
pthread_exit called without calling JNI_DetachCurrentThread first
313
Description:
313
Solution:
313
%preun(zing-zst-<version>) scriptlet failed, exit status1
313
Description:
313
Solution:
313
The Zing System Tools is Not Loaded Properly
313
Description:
313
Solution:
314
The Zing System Tools Has Not Been Configured Yet
314
Description:
314
Solution:
314
Azul Systems
Confidential and Proprietary
23
Zing Installation and Administration Guide
Not Enough Free Memory During Zing Memory Configuration
314
Description 1:
314
Solution 1:
314
Description 2:
314
Solution 2:
314
Error example:
314
ZST Fails to Allocate Requested Memory
315
Description:
315
Solution:
315
Cannot find kernel config /boot/config-<kernel version>
315
Description:
315
Solution:
315
Qualification Failed Errors
316
Processor support insufficient for Zing (constant_tsc support required)
316
Description:
316
Solution:
316
UseLargePages Not Supported with Zing System Tools
Appendix A – Zing Installation Prerequisites Details
SWAP Settings
316
318
318
Persistent change:
318
Non-persistent change:
319
vm.min_free Settings
319
Persistent change:
319
Non-persistent change:
319
Azul Systems
Confidential and Proprietary
24
Table of Contents
Transparent HugePages Settings
319
Disabling the Intel Idle Driver
320
On RHEL/CentOS 7.x systems:
321
On RHEL/CentOS 5.x or 6.x systems:
322
On Ubuntu systems:
323
On SUSE systems:
323
Appendix C – Zing System Tools Installed Files
User Tools
325
325
/usr/bin/zing-ps
325
Administrator Tools
325
/usr/sbin/zing-llpmem
325
/usr/sbin/zing-configure-pmem
325
/usr/sbin/system-config-zing-memory
325
/usr/sbin/system-config-zing-core-pattern
325
Configuration Files
326
/etc/zing/pmem.conf.default
326
/etc/zing/pmem.conf
326
/etc/zing/pmem.conf.*example*
326
Man Pages
326
ZST Configuration File Relationships
326
ZST Boot Sequence File Relationships
327
ZST Update Software File Relationships
327
Legal Notice
Azul Systems
328
Confidential and Proprietary
25
Zing Installation and Administration Guide
Zing Release Notes
May 15, 2015
15.05.0.0
Zing Component Versions
The Zing 15.05.0.0 release includes the following product component versions:
l
ZVM 15.05.0.0
l
ZST 5.7.11
New Features and Updates
New for Zing 15.05.0.0 release:
l
ReadyNow! enhancements
l
Zing Pilot Assist Kit
l
Launch from native application (c-main)
l
Large heap enhancements
l
Experimental version of Zing MXBeans
l
Support for Oracle Linux versions
l
l
2.6.39-400.212.1.el5uek.x86_64
l
2.6.39-400.209.1.el5uek.x86_64
l
2.6.32-300.11.1.el5uek.x86_64
l
2.6.39-400.215.11.el6uek.x86_64
l
2.6.39-400.17.2.el5uek.x86_64
l
2.6.39-400.17.2.el6uek.x86_64
Support for RHEL/CentOS 7.1
l
3.10.0-229.1.2.el7.x86_64
l
Added flags PrintThreadDumpsCallerStacktrace and PrintFindDeadlocksCallerStacktrace
l
Added options AllowExplicitThreadDumps and AllowExplicitFindDeadlocks
l
Bug fixes
Azul Systems
Confidential and Proprietary
26
Zing Release Notes
Zing Virtual Machine Resolved Issues
Advisory!
Various hangs in the futex code have been reported by applications running on
Haswell. Please see https://access.redhat.com/solutions/1350963 for more details.
Even though the referenced link talks only about Redhat, the kernel bug is present in
other distros too. Please check with the respective OS support teams on the latest
patches available for other distros.
The following table lists known issues that are resolved as of Zing Virtual Machine 15.05.0.0. The Bug
IDs listed are Azul internal reference numbers.
Bug ID
Release
Resolved
Description
Eliminate the expiration timer check in Zing MXBeans and make the
VM option an experimental option.
4282
15.05.0.0
To use Zing MXBeans specify both of the following options, in order.
-XX:+UnlockExperimentalVMOptions
-XX:+UseZingMXBeansExperimentalVersion
Provide VM options to print stack trace for explicit calls to
ThreadDumps and FindDeadlocks.
3799
15.05.0.0
The following VM options have been added:
PrintThreadDumpsCallerStacktrace
PrintFindDeadlocksCallerStacktrace
3768
15.05.0.0
Recognize the '%p' macro in loggc param to expand the PID of the
current VM process and add it to the GC log file.
Usage: -Xloggc:verbose.gc.%p.log
3627
15.05.0.0
Failure of ImplicitNullChecks optimization causes severe performance
degradation. This manifests as high amount of CPU time spent in
handler_segv.
3501
15.05.0.0
Crash in NodeHash::hash_delete(Node const*)+0xa.
3474
15.05.0.0
VM hang in park/unpark code. This is due to a kernel bug in futex_wait.
Please see https://access.redhat.com/solutions/1350963 for more
details.
3387
15.05.0.0
CHeapLeakDetection will cause a crash in C [libjvm.so+0x3ad1f0]
Azul Systems
Confidential and Proprietary
27
Zing Installation and Administration Guide
Bug ID
Release
Resolved
Description
Thread::pop_lock(AzLock*)+0x10 in versions 14.09.0.0, 14.09.1.0 and
14.09.2.0.
3192
15.05.0.0
Have java.time.Clock constructor use linux's clock_gettime(CLOCK_
REALTIME,...) to provide nanosecond precision
Allow VM option KeepSafepointsInCountedLoops to be controlled at
the method level.
2438
15.05.0.0
Example: Add the following line to the CompileCommandFile to enable
this option on a per-method basis
option /package/Class <method>
+KeepSafepointsInCountedLoops
2051
15.05.0.0
Allow options to configure stack size for JavaThreads. -Xss and XX:ThreadStackSize=<n> are now honored. The default stack size is
2MB.
Provide VM options to disable explicit calls to ThreadDumps and
FindDeadlocks.
1255
15.05.0.0
The following VM options have been added to enable/disable the
functionalities:
AllowExplicitThreadDumps (defaults to on)
AllowExplicitFindDeadlocks (defaults to on)
3704
15.02.1.0
Using UseOptimizedCollectionsIterator shows a lot of exceptions
3557
15.02.1.0
Crash in Klass::GPGC_follow_weak_klass_links()+0x4d4
3473
15.02.0.0
Crash while freeing a methodCodeOop. (CodeCache::GPGC_unlink(),
methodCodeOopDesc::free_internals(),...)
3374
15.02.0.0
Crash in (C2) sun.security.ssl.ClientHandshaker.processMessage(BI)
V
3226
15.02.0.0
Need faster implementation of ConcurrentHashMap.values()
2411
15.02.0.0
Default number of threads (GC and compiler) now calculated based on
the physical core count rather than the logical core count
Azul Systems
Confidential and Proprietary
28
Zing Release Notes
Bug ID
Release
Resolved
Description
3444
14.09.2.0
Thread is holding jvm_lock while yield()
3373
14.09.2.0
MacroAssembler::test8il generates incorrect code
3331
14.09.2.0
Times to safepoint greater than 100ms are observed due to realloc of
certain safepoint tracking data structures
3295
14.09.2.0
NullPointerException while loading fonts on a system that has no
'fontconfig' library
3212
14.09.2.0
For Java 8, fonts from /etc/zing/java-fonts not found by ZVM
3211
14.09.2.0
Crash in frame C [libjvm.so+0x6763e1] Node::rematerialize()
const+0x1
3169
14.09.2.0
ProfileLogIn causes certain virtual call sites to be erroneously treated
as megamorphic
3139
14.09.2.0
Error: zing-zst not found when installing Zing using the shar installers.
2910
14.09.2.0
-XX:+IgnoreUnrecognizedVMOptions doesn't work in Zing
1920
14.09.2.0
Segmentation fault in GPGC_Collector::mutator_relocate_object after
incorrect code generation for java.util.
1692
14.09.2.0
WARNING: The Event Tick Buffer Profiling system status is:
"Shutdown." (reason: "Max Buffers Exceeded").
3007
14.09.1.0
Threads created in native code before the VM has been initialized will
cause a recursive SEGV when calling AttachCurrentThread.
ThreadLocalMap can cause a performance slowdown when the
hash is not optimal due to a smaller capacity.
2967
Azul Systems
14.09.1.0
Note: The initial capacity of the ThreadLocalMap can be adjusted
using the VM option XX:JavaThreadLocalMapInitialCapacity=<val> (<val>
defaults to 128).
Confidential and Proprietary
29
Zing Installation and Administration Guide
Bug ID
Release
Description
Resolved
2869
14.09.1.0
Crash in libplumbr.so while loading Plumbr agent.
2781
14.09.0.0
zing_installer.sh: Installing using the shell installer can fail.
JBoss EAP 5.1 throws one of the following messages when
serializing/deserializing the java/lang/String class with Zing VM on one
side and a VM which has a String class with the offset and count
fields.
java.lang.RuntimeException: Field hash null is not available on
current classPath for class java.lang.String
2681
14.09.0.0
or
java.io.IOException: Current classpath has lesser fields on
java.lang.String than its original version
Solution: Use -XX:+UseJDK6CompatibleStringClass to use
a version of java/lang/String that is compatible with the String class
that has the offset and count fields.
Security adjustment made to internal method
sun.reflect.Reflection.getCallerClass(depth) for JDK 6 installations
where the depth parameter was not used correctly.
Affects: This only affects JDK 6. JDK 7 and JDK 8 are unaffected by
this change.
2650
14.09.0.0
Background: sun.reflect.reflection.getCallerClass(depth) is
deprecated, not recommended, and has been disabled by default.
Calling this function results in an UnsupportedOperationException
being thrown, unless you include the command line option "Djdk.reflect.allowGetCallerClass=true”.
Solution: When enabled, the Class returned by this method now
matches behavior in JDK 7 and historical releases of JDK 6. If
possible, use the replacement function,
sun.reflect.reflection.getCallerClass(void), instead.
2426
14.09.0.0
Crash in __popcountdi2 when run with libgcc version 4.8.2
2404
14.09.0.0
Using -XX:+HeapDumpOnOutOfMemoryError does not create a
heap dump.
2359
14.09.0.0
Crash when using saved profile on restart with -XX:
Azul Systems
Confidential and Proprietary
30
Zing Release Notes
Release
Bug ID
Description
Resolved
ProfileLogIn.
2229
2106
14.09.0.0
Crash in PhaseIterGVN::transform_old
14.09.0.0
Maximum checkpoint sync time exceeded following a CTRL-C when
ZVM doing a heap dump following OutOfMemoryError (when using
option -XX:+HeapDumpOnOutOfMemoryError
-XX:AutoBoxCacheMax does not work with ZVM.
1844
14.09.0.0
Background: When using the command line option, XX:AggressiveOpts, this fix automatically increases the value of
AutoBoxCacheMax from the default value of 128 to 20000.
AutoBoxCacheMax specifies the size of the java.lang.Integer autobox
cache (property: java.lang.Integer.IntegerCache.high).
Workaround (available for JDK 1.7 only): Replace: XX:AutoBoxCacheMax=<nnn> With: Djava.lang.Integer.IntegerCache.high=<nnn>
Unable to generate a heap dump (using SIGQUIT) when ZVM forked
from an Oracle JVM process
1709
14.09.0.0
Background: The Oracle JVM masks SIGQUITs in the process and
enables the signal to be received only by the VMThread. When the
ZVM is forked it inherits the process signal mask from the Oracle JVM.
Solution: Use -XX:+ResetSignalMaskAtLaunch to clear the
signal mask to enable SIGQUITs.
Zing Virtual Machine Known Issues
The following table lists known issues that are known issues as of Zing Virtual Machine 15.05.0.0. The
Bug IDs listed are Azul internal reference numbers.
Bug Release
ID
Known
Description
For users of the technical preview release com.azul.zing.management MXBean
3552 15.02.0.0 feature, accessing the unsupported threshold and notification operations for classes
in the com.azul.zing.management package will generate an
Azul Systems
Confidential and Proprietary
31
Zing Installation and Administration Guide
Bug Release
ID
Description
Known
UnsupportedOperationExceptionNoStackTrace exception (package
com.azul.zing.impl.management and base class
java.lang.UnsupportedOperationException). When implementation of the
threshold/notifications mechanism is complete, this exception will no longer be
thrown.
Nashorn benchmark STALLS after encountering data error processing profile log
2820 14.09.0.0 error (error: 101) while running Nashorn benchmarks with ReadyNow! enabled.
ZVM running in environment with multiple scheduling policies, RR, BATCH, and
2602 14.09.0.0 OTHER, encounters checkpoint sync timeout in thread running with BATCH policy.
1980 14.09.0.0 JNI calls are faster on Oracle 1.6 vs. Zing.
Zing System Tools Resolved Issues
The following table lists known issues that are resolved as of Zing System Tools 5.7.11. The Bug IDs
listed are Azul internal reference numbers.
Release
Bug ID
Resolved
4116
5.7.11
4105
5.7.11
4081
5.7.11
3977
5.7.11
Description
Add support for Oracle Linux 5, 2.6.39-400.212.1.el5uek.x86_64, 2.6.39400.209.1.el5uek.x86_64, and 2.6.32-300.11.1.el5uek.x86_64
Add support for Oracle Linux 6 kernel, 2.6.39-400.215.11.el6uek.x86_64
Add support for Oracle Linux 5 and 6 kernels, 2.6.39-400.17.2.el5uek.x86_
64 and 2.6.39-400.17.2.el6uek.x86_64
Add support for RHEL/CentOS 7.1, kernel 3.10.0-229.1.2.el7.x86_64
Provide option to disable cstate check.
3865
5.7.11
Azul Systems
Note: To disable cstate check. Create a file, disable_cstate_check, in
/etc/zing folder and configure zing-memory.
Confidential and Proprietary
32
Zing Release Notes
Release
Bug ID
Resolved
Description
Not able to access license server on Chrome, Error code: ERR_SSL_
VERSION_OR_CIPHER_MISMATCH
3942
3842,
5.7.10
Note: If you are using a pool server. Upgrade existing pool server to
preserve existing host configurations
5.7.10
Add support for Oracle Linux 6 kernel 2.6.39-400.214.4.el6uek.x86_64 and
2.6.39-400.214.3.el6uek.x86_64
3760
5.7.10
Add support for Oracle Linux 5 kernel 2.6.39-400.246.2.el5uek.x86_64
3256
5.7.10
zing-licensed --display_system_info displays ERROR: (ioctl failed)
243
5.7.10
Verifying a UUID license sometimes fails with ERROR: getvmid binary
has invalid checksum.
5.7.9
Support for Oracle Linux Server release kernel version 2.6.32-504.el6.x86_
64, 2.6.32-300.39.2.el6uek.x86_64, and 2.6.39-200.24.1.el6uek.x86_64.
3162
5.7.8
Removed az_mmap, az_munmap, az_mremap, semaphore_create and
semaphore_destroy hooks from CHeapLeakDetection
3158
5.7.8
Crash in verify() when using custom libssl.so
3074
5.7.8
Error: Implicit declaration of function ‘preempt_enable_no_resched’ when
installing ZST
2682
5.7.7
On RHEL 7, systemctl shows failure for zing-licensed status, however
service command indicates the process is running.
4004
3585,
3650
RHEL 7 ZST uninstall displays warning messages. Ignore these
messages.
2564
5.7.7
2210
5.7.6
Azul Systems
warning: file /var/run/zing-licensed: remove failed:
No such file or directory
After a zing app is killed/terminated, /tmp/azlicdata not cleaned up on
ReiserFS
Confidential and Proprietary
33
Zing Installation and Administration Guide
Release
Bug ID
Resolved
Description
Crash in _runLocalDestructors in Tibco code when run with ZST.
1982
5.7.6
Background: On prior releases, the crash occurs in any application if
destructors are registered with pthreads and the destructors consume more
than 8k of stack.
Workaround: Do not limit the pthread stack size to PTHREAD_STACK_
MIN . It can then assume the default pthread stack size or the size that you
set in native code.
1487
5.7.6
The Zing core bundler now produces encrypted core files. A passphrase file
is produced for local decryption. Only Azul Systems and those with the
unique corresponding passphrase can decrypt the core file.
1149
5.7.6
No fonts installed by default in jre/lib. ZVM 5.10.1 or later required.
701
5.7.6
Linux kernel crash at az_pmem.c:1667.
1562
5.7.5
An incorrect error message is displayed if Zing memory is stopped and you
are using partitions when launching.
1490
5.7.4
ZST 5.7.3 system-config-zing-memory fails on SLES 11 SP2 kernel
3.0.101
1301
5.7.4
Startup crash accessing thread stack page.
1011
5.7.2
EFS - Enhance startup and shutdown scripts to accept license and
pmem.conf
992
5.7.2
Shutdown of Cassandra 2.0.1 leaves defunct process.
843
5.7.2
Warning messages seen when installing/upgrading ZST 5.7.x and port
*3824 is not in use.
643
5.7.2
zing-core-bundler does not work with Amazon Linux if hs_err is supplied.
640
5.7.2
Can't enable core pattern on Amazon Linux.
620
5.7.2
When using Amazon Linux OS possible message: libssl.so.6: cannot open
shared object file: No such file or directory.
Azul Systems
Confidential and Proprietary
34
Zing Release Notes
Release
Bug ID
Description
Resolved
789
5.7.1
Uninstalling or restarting versions 5.5.x Zing license package deleted the
crontab entries.
568
5.7.0
insmod error when installing on Amazon instances
543
5.7.0
Script name for Zing core bundling has changed to zing-core-bundler. The
.pl extension has been dropped.
512
5.7.0
zing_mm: Cannot mknod: Operation not permitted when extracting core file
511
5.7.0
Multiple nginx processes running after uninstalling ZST
464
5.7.0
Missing description for ZST package on Ubuntu
439
5.7.0
5.7 ZST does not support older (but still supported) ZVMs
389
5.7.0
Failed to enable HTTPS agent configuration! this happens if /usr/sbin is not
set in the PATH
343
5.7.0
Installation stuck during "zing-memory start”
311
5.7.0
Issues with automatic core bundling on RHEL 5.X (especially with ulimit -c
0)
256
5.7.0
Port requirements for ZST install
242
5.7.0
Installation fails because of ld: not found
225
5.7.0
ZST install requires openssl098
327
5.7.0
Core bundler script failing
282
5.7.0
ulimit -c not honored by zing-core pattern
31659
5.7.0
Blank screen displayed when session expired and user tries to login again
31763
5.7.0
User should be logged out when DB goes down
Azul Systems
Confidential and Proprietary
35
Zing Installation and Administration Guide
Zing System Tools Known Issues
The following table lists known issues that are known issues as of Zing System Tools 5.7.11. The Bug
IDs listed are Azul internal reference numbers.
Bug
Release
ID
Known
4358
5.7.11
Description
Uninstalling ZST 5.7.11 on Amazon Linux displays a message. Ignore this
message:
/bin/cat: /etc/redhat-release: No such file or directory
If ZST is installed on Ubuntu 10.04 or Ubuntu 11.04 with the ZST DKMS installers
and zing-core-pattern is turned ON, Decrypted zing cores files are zero bytes.
2943
5.7.7
Workaround: Turn off Zing Core Pattern using
# system-config-zing-core-pattern disable
2156
243
5.7.7
5.7.6
Status of zing-licensed process not displayed correctly on Sles11 SP1
Verifying a UUID license sometimes fails with ERROR: getvmid binary has invalid
checksum
Workaround: Reinstall ZST and verify license
2406
5.7.6
Zing processes still running after uninstall when both relocated binaries and regular
binaries are installed. Issue may arise under deployment control systems like EFS.
2337
5.7.6
Cannot renew license through the OEM UI.
2195
5.7.6
libpcre is not required for ZST on Ubuntu even though dpkg -p prints this information
1486
5.7.6
Can bundle Zing cores with non matching executable
Warning messages seen on MRG and Amazon Linux kernels; kernel WARNING:
at kernel/trace/ftrace.c:1583
Work around: Login as root and run the following commands:
998
5.7.6
mount -t debugfs none /sys/kernel/debug
echo 0 > /sys/kernel/debug/tracing/tracing_on
echo nop > /sys/kernel/debug/tracing/current_tracer
Azul Systems
Confidential and Proprietary
36
Zing System Requirements
Bug
Release
ID
Known
Description
1002
5.7.6
Crash with ERROR: Module zing_mm is in use,zing-memory: ERROR: rmmod
zing_mm.ko failed (error 1) Fatal error: os::safeSystem: command '/etc/init.d/zingmemory restart' failed
442
5.7.6
zing-ps shows a misleading message when given the pid of a dead process.
Zing System Requirements
This guide provides a complete list of system requirements for installing and configuring each of the
components, ZST, ZVM, ZVision, ZVRobot, and Zing licensing. This includes hardware and software
requirements.
Zing Components
Each of the Zing components are installed on machines relative to the ZVM host machine, as listed.
l
ZVM and ZST are installed on the same machine. ZVM requires ZST to use the Zing
memory management features.
The ZVM package software are packaged as standard .sh, .rpm, or .deb. The ZST
package software are packaged as standard .rpm or .deb.
l
l
Zing license file is installed on the ZVM host.
Zing Pool license server is installed on a separate machine from the ZVM host machines
that will be in its license pool. It is a Java application and requires a ZVM instance to run.
The Zing Pool license server is packaged as a .sh. A specific Pool license contract with
Azul is required to use this option.
l
ZVision and ZVRobot are the profiling and diagnostics tools for the ZVM processes. They
are automatically installed with the ZVM component.
Zing Machine Requirements
The system that components, ZVM, ZST, Zing license components, ZVision, and ZVRobot are
installed on has the following requirements:
l
Operating System Requirements
l
CPU Requirements
Azul Systems
Confidential and Proprietary
37
Zing Installation and Administration Guide
l
Memory Requirements
l
Hard Disk Storage Space Requirements
Operating System Requirements
The components ZVM, ZST, and the Zing license server can be installed on machines running the
following operating systems using .sh, .rpm, or .deb installers, as appropriate.
l
Notes
Ensure that OpenSSL 0.9.8 or later is installed.
l
On RHEL 5.x, CentOS 5x., or Oracle Linux 5 systems, also install the Development
Tools Package.
l
All RHEL installations require a registered copy of RHEL and a valid Red Hat Network
(RHN) account user name and password.
l
l
Ensure that binutils are installed.
In virtual machine environments, paravirtualization is not supported. Only HVM
(Hardware-assisted Virtual Machine) cluster instances are supported.
The following tables list the supported operating systems, kernel versions, and for Amazon EC2
Instances, the Amazon Machine Image (AMI) identification code. Where multiple kernel versions are
listed, any from the list are supported with the associated operating system.
RHEL or CentOS
Operating System
Kernel Version
Supported As Of ZST Version
RHEL/CentOS 7.1
3.10.0-229.1.2.el7.x86_64
v5.7.11
RHEL/CentOS 7
3.10.0-123
v5.7.6
RHEL/CentOS 6.6
2.6.32-504
v5.7.8
RHEL/CentOS 6.5
2.6.32-431
v5.7.3
RHEL/CentOS 6.4
2.6.32-358
v5.7.0
RHEL/CentOS 6.3
2.6.32-279
v5.7.0
RHEL/CentOS 6.2
2.6.32-220
v5.7.0
RHEL/CentOS 6.1
2.6.32-131
v5.7.0
Azul Systems
Confidential and Proprietary
38
Zing System Requirements
RHEL or CentOS
Kernel Version
Operating System
Supported As Of ZST Version
RHEL/CentOS 6.0
2.6.32-71
v5.7.0
RHEL/CentOS 5.10
2.6.18-371
v5.7.6
RHEL/CentOS 5.9
2.6.18-348
v5.7.0
RHEL/CentOS 5.8
2.6.18-308
v5.7.0
RHEL/CentOS5.7
2.6.18-274
v5.7.0
RHEL/CentOS 5.6
2.6.18-238
v5.7.0
RHEL/CentOS 5.5
2.6.18-194
v5.7.0
RHEL/CentOS 5.4
2.6.18-164
v5.7.0
RHEL/CentOS 5.3
2.6.18-128
v5.7.0
RHEL/CentOS 5.2
2.6.18-92
v5.7.0
Oracle Linux Server Operating System
Kernel Version
Supported As Of
ZST Version
Oracle Linux Server
2.6.39-400.212.1.el5uek.x86_64
v5.7.11
Oracle Linux Server
2.6.39-400.209.1.el5uek.x86_64
v5.7.11
Oracle Linux Server
2.6.32-300.11.1.el5uek.x86_64
v5.7.11
Oracle Linux Server
2.6.39-400.215.11.el6uek.x86_64
v5.7.11
Oracle Linux Server
2.6.39-400.17.2.el5uek.x86_64
v5.7.11
Azul Systems
Confidential and Proprietary
39
Zing Installation and Administration Guide
Oracle Linux Server Operating Sys-
Supported As Of
Kernel Version
tem
ZST Version
Oracle Linux Server
2.6.39-400.17.2.el6uek.x86_64
v5.7.11
Oracle Linux Server
2.6.39-400.214.3.el6uek.x86_64
v5.7.10
Oracle Linux Server
2.6.39-400.214.4.el6uek.x86_64
v5.7.10
Oracle Linux Server
2.6.39-400.246.2.el5uek.x86_64
v5.7.10
Oracle Linux Server
2.6.39-200.24.1.el6uek.x86_64
v5.7.9
Oracle Linux Server
2.6.32-300.39.2.el6uek.x86_64
v5.7.9
Oracle Linux Server
2.6.32-504.el6.x86_64.x86_64
v5.7.9
Red Hat or CentOS
Operating System
Kernel Version
Supported As Of ZST Version
Messaging Realtime Grid (MRG)
RHEL, RHEL MRG, CentOS
3.6.11.2-rt33.39.el6rt.x86_64 v5.7.3
RHEL, RHEL MRG, CentOS
3.0.36-rt57.66.el6rt.x86_64
SLES Operating System
Kernel Version
v5.7.3
Supported As Of ZST Version
SLES 11 SP3
3.0.101-0.15
v5.7.6
SLES 11 SP3
3.0.101-0.21
v5.7.5
SLES 11 SP3
3.0.93-0.8
v5.7.3
SLES 11 SP3
3.0.76-0.11
v5.7.0
Azul Systems
Confidential and Proprietary
40
Zing System Requirements
SLES Operating System
Kernel Version
Supported As Of ZST Version
SLES 11 SP2
3.0.101-0.7.17
v5.7.4
SLES 11 SP2
3.0.13-0.27
v5.7.0
SLES 11 SP1
2.6.32.59-0.7
v5.7.3
SLES11 SP1
2.6.32.12-0.7
v5.7.0
Ubuntu Operating System
Kernel Version
Supported As Of ZST Version
Ubuntu 12.04
3.5.0-34-generic
Ubuntu 12.04
3.2.0-23-generic, 3.2.0-31-generic v5.7.0
Ubuntu 11.04
2.6.38-15-generic
v5.7.0
Ubuntu 10.4
2.6.32-38, 2.6.32-38-generic
v5.7.0
Debian Operating Sys-
Kernel Ver-
tem
sion
v5.7.1
Supported As Of ZST Version
Debian Wheezy 7.6, 7.7 3.2.0-4-amd64 v5.7.8
Amazon EC2 Instances
Kernel Version
AMI ID
Supported As Of ZST
Version
Amazon Linux
3.4.37-40.44
ami-a73758ce
v5.7.0
Amazon Linux
3.4.43-43.43
ami-a73758ce
v5.7.0
Azul Systems
Confidential and Proprietary
41
Zing Installation and Administration Guide
Amazon EC2 Instances
Kernel Version
AMI ID
Supported As Of ZST
Version
Amazon Linux
3.4.48-45.46
ami-a73758ce
v5.7.0
Amazon Linux
3.4.57-48.42
ami-a73758ce
v5.7.0
3.10.0-123
ami-785bae10
v5.7.6
RHEL 6.5 - Amazon Cloud 2.6.32-431
ami-63b6910a
v5.7.3
2.6.32-358
ami-9d0b64f4
v5.7.0
3.0.101-0.8-default
ami-e572438c
v5.7.5
3.2.0-54-virtual
ami-b93264d0
v5.7.3
RHEL 7.0 – Amazon
Cloud
RHEL 6.4 - Amazon
CloudAmazonCloud
SLES 11 SP3 Amazon
Cloud
Ubuntu 12.04 Virtual
AmazonCloud
DKMS Supported Operating Systems for ZST
ZST can be installed using a Dynamic Kernel Module Support (DKMS) framework. Given that DKMS is
used to generate Linux kernel modules, Azul cannot and does not test all possible operating systems.
Azul only tests selected operating systems. However it is expected, provided all the installation prerequisites are met, that ZST can be installed on most RHEL, CentOS, Oracle Linux, SUSE, and
Ubuntu systems.
Notes
While this feature is tested and validated for a selected set of operating system versions,
there are always risks. Test thoroughly in your staging environment before going into
production.
CPU Requirements
The Zing components ZVM, ZST, and the Zing license server can be installed on machines running the
following CPU types.
l
64-bit x86 CPUs
o
Intel® Xeon® server class processors released 2009 and later.
If your are using power efficiency chips, you must disable the Intel idle driver.
Azul Systems
Confidential and Proprietary
42
Zing System Requirements
o
AMD Opteron™ server class processors released 2010 and later.
Note: Dual socket systems are preferred. The single-chip 23xx systems, which have 4
cores, and the single-chip 41xx systems, which have up to 6 cores, might not be
sufficiently scalable.
l
6+ physical cores recommended (4 minimum)
Increase the number of ZVM host systems for applications that have more threads.
Memory Requirements
The Zing components require the following amount of RAM on the ZVM machine.
l
64+ GB RAM recommended
16-32 GB RAM minimum, depending on the application memory requirements, for
demonstration purposes only. Minimum is not sufficient for performance or scalability
production testing of large or memory consuming enterprise applications.
Increase the memory as needed to accommodate the number and size of your Java
application/Zing Virtual Machine (ZVM) instances.
The Pool license server requires the following amount of RAM on the license server machine.
l
8 GB RAM recommended (4 GB minimum)
Hard Disk Storage Space Requirements
The Zingcomponents require the following amount of disk space on the ZVM machine.
l
32 GB free disk drive space recommended for your ZVM instances to run Java applications
8 GB minimum configuration is a spinning platter hard drive or enterprise quality SSD.
Installation to a USB flash device is not recommended for production use.
l
600 MB additional for JDK functions
The Pool license server requires the following amount of disk space on the license server machine.
l
10 GB free disk space recommended (5 GB minimum)
Browser Requirements
ZVision and ZVRobot are accessible from the listed supported Web browsers.
ZVision is accessible from supported Web browsers with access to the application host running the
ZVM instance. ZVRobot output is viewable from supported Web browsers with network access to the
file system where the ZVRobot files are located. Supported browsers include:
l
Microsoft® Internet Explorer 6 (or later) on Windows 2000, Windows XP, Windows Vista, or
Windows 7
Azul Systems
Confidential and Proprietary
43
Zing Installation and Administration Guide
l
Mozilla Firefox 1.0.3 (or later) on all supported operating systems
l
Apple Safari™ 2.0 (or later) for Mac OS® X
Zing license server functions are managed from the following supported Web browsers:
l
Microsoft® Internet Explorer, version 9
l
Mozilla Firefox 1.0.3 (or later) on all supported operating systems
l
Google Chrome 10.0.x (or later)
Listening Connection Requirements
ZVision and ZVRobot require ZVM host connection information:
l
ZVM host IP address and ZVM host ARTA port
The Azul Run Time Analysis (ARTA) port is the application host port specified with the XX:ARTAPort command line. See Configuring ZVision in Using Zing Vision and
Configuring the Zing Virtual Machine to Use ZVRobot in Using Zing Vision Robot.
The ARTA port provides internal API communication between ZVision and ZVM instances.
ZVision requires:
l
ZVision host IP address and ZVision browser listening port
ZVision is automatically installed on the same machine as ZVM when ZVM is installed. You
can connect a ZVision installed on one machine to a ZVM on another machine. The ZVision
listening port must be configured. See Configuring the ZVision Server in Using Zing Vision .
Zing License Server requires:
l
8443, Web port, TCP protocol to communicate with the ZVM host.
Ensure port 8443 is not used by any other service on the ZVM host. If this is not possible,
edit the standalone.xml file, <socket-binding name="https" port="8443"/>
setting or reconfigure the license server. See Installing and Using Pool Licensing.
l
l
9999 and 4447, management port and remote port, respectively. For internal communication
within the license server.
27017, database port, for license server database (mongoDB).
Zing Component Version Compatibility
This table lists the component versions that have been tested and are compatible. All component
versions listed are compatible with each other.
Azul Systems
Confidential and Proprietary
44
Zing System Requirements
ZVM Version
15.05.0.0
ZST Version
5.7.11
5.7.11, 5.7.10, 5.7.9, 5.7.8, 5.7.7, 5.7.6, 5.7.5,
15.02.1.0, 15.02.0.0
5.7.4, 5.7.3, 5.7.2, 5.7.1, 5.7.0
14.11.1.0, 14.11.0.0
5.7.11, 5.7.10, 5.7.9, 5.7.8, 5.7.7, 5.7.6
14.09.2.0, 14.09.1.0, 14.09.0.0,
5.10.1, 5.10.0,
5.9.3, 5.9.2, 5.9.1, 5.9.0,
5.7.11, 5.7.10, 5.7.9, 5.7.8, 5.7.7, 5.7.6, 5.7.5,
5.8.0,
5.7.5, 5.7.4, 5.7.3, 5.7.2, 5.7.1, 5.7.0,
5.7.4, 5.7.3, 5.7.2, 5.7.1, 5.7.0
5.6.1,
5.5.4, 5.5.1
Zing ZST and API Versions
Some ZVM and ZST features, such as DumpMiniCore and AzMemPartition, require a specific ZST API
level. The table below lists the ZST API level for each ZST version.
ZST Version
ZST API
5.7.11
6b
5.7.10, 5.7.9, 5.7.8, 5.7.7, 5.7.6
6a
5.7.5
5f
5.7.4, 5.7.3, 5.7.2, 5.7.1, 5.7.0,
5.2.7, 5.2.6
5e
5.2.5, 5.2.4, 5.2.3
5d
5.2.2, 5.2.1, 5.2.0
5c
5.1.3, 5.1.2, 5.1.1, 5.1.0
5b
Azul Systems
Confidential and Proprietary
45
Zing Installation and Administration Guide
Installation Quick Start
To configure your system to run your Java applications using the Zing platform, perform the following
steps:
1. Ensure your environment meets the system requirements.
2. Download the ZST and ZVM installation files.
3. Install the Zing System Tools (ZST).
4. Configure Zing memory management.
5. Install Zing Licensing.
6. Install the Zing Virtual Machine (ZVM).
7. Run your Java applications with Zing.
Ensure your environment meets the system requirements.
1. Review Zing System Requirements for a complete list of Zing requirements, including: supported
CPUs, memory requirements, and operating system versions.
2. Ensure the following packages are installed. This includes: OpenSSL 0.9.8 or later, development
tools, and binutils.
System
RHEL, CentOS,
supported Oracle Linux 6
RHEL 5.x, CentOS 5.x,
supported Oracle Linux 5
RHEL 7
Commands to Install Packages
# yum localinstall --nogpgcheck
libssl<version>binutils lsof curl
# yum localinstall --nogpgcheck libssl<version>
binutils lsof curl
# yum groupinstall 'development Tools'
# yum localinstall --nogpgcheck
libssl<version> binutils lsof curl
# yum install perl-Sys-Syslog.x86_64
SLES
# zypper install libssl<version> binutils lsof
curl
Ubuntu
# apt-get install libssl<version> binutils lsof
curl
Azul Systems
Confidential and Proprietary
46
Installation Quick Start
3. If you are using a Nehalem power efficiency chip set, disable the Intel idle driver. See Installing and
Configuring Zing System Tools for directions.
4. Ensure that OS settings are appropriate for performance sensitive applications.
Parameter Setting
Commands to Install Packages
1. Edit the /etc/sysctl.conf file to add
SWAP to 0
vm.swappiness=0
2. Run sysctl -p
1. Edit the /etc/sysctl.conf file to add vm.min_
vm.min_free_
kbytes
to 1 GB or more
free_kbytes=1048576
2. Run sysctl -p
1. Edit the grub configuration file to add transparent_
Disable
transparent_
hugepages
hugepage=never to the kernel boot line.
2. Reboot the server.
Download the ZST and ZVM installation files.
If you are using FTP, ensure the file transfer protocol is set to binary mode.
System
RHEL,
CentOS,
supported
Oracle
Linux, or
SLES
ZST
ZVM
zing-jdk<version>-<zvm_
version>-<platform>.rpm
or
zing-zst-<kernel_version>.<zst_
version>.<platform>.rpm
zing-zst_<kernel_version>server-<zst_version>_
<platform>.deb
Ubuntu
zing-jdk<version>_
<zvm_
version>_
<platform>.deb
or
or
zing-zst_<kernel_version>desktop-<zst_version>_
<platform>.deb
Azul Systems
zing-jdk<version>-<zvm_
version>-<platform>.sh
zing-jdk<version>-<zvm_
version>-<platform>.sh
Confidential and Proprietary
47
Zing Installation and Administration Guide
System
ZST
ZVM
If you are using Pool licenses:
1. Store the Pool license zip file, <your_company>-pool.zip you received by email.
2. Download the Pool license server, zing-license-pool-server-<version>.sh.
Install the Zing System Tools (ZST).
Run the operating system appropriate installation command, as root or use sudo.
Operating System
Installation Command
RHEL, CentOS,
supported Oracle
Linux
# sudo yum localinstall --nogpgcheck zing-zst<version>.rpm
SLES
# sudo zypper install zing-zst-<version>.rpm
Ubuntu
# sudo dpkg -i zing-zst-<version>.deb
Configure Zing memory management.
1. Run the Zing memory configuration wizard.
2. Type y to accept all defaults.
# /usr/sbin/system-config-zing-memory
Install Zing Licensing
Standalone Zing License
Copy your Zing licensing authorization file, license to /etc/zing/directory.
Azul Systems
Confidential and Proprietary
48
Installation Quick Start
Install the Pool Server
1. Install the Pool Server.
a. Run the command
$ zingserver/zing-license-pool-<version>.sh
b. Launch the Pool license server Web interface at: https://<license_server_ip_
address>:8443
c. Add the Pool license. Select Server license > Add > Choose File > Add.
2. On the ZVM host, run the listed command and confirm to proceed:
# zing-licensed –-configure –-server <server_ip> --port <port_no>
For Amazon instances, use:
# zing-licensed –-configure –-amazonId –-server <server_ip> --port
<port_no>
Install the Zing Virtual Machine (ZVM).
Install using the shar file option or choose the appropriate OS-based command.
Operating System
Command
RHEL, CentOS,
supported Oracle
Linux
$ yum localinstall –-nogpgcheck ./ zing-jdk<version><zvm_version>-<platform>.rpm
SLES
$ zypper install ./zing-jdk<version>-<zvm_version><platform>.rpm
Ubuntu
$ dpkg -i ./zing-jdk<version>_<zvm_version>_
<platform>.deb
1. Run the command:
$ ./zing-jdk<version>-<zvm_version><platform>.sh
Shar option
2. Specify the location of the Linux JDK, at the prompt.
Location of <linux_type> JDK (<path>):
Prepare to run Zing
See Installing the Zing Virtual Machine, “Completing Post ZVM Installation Tasks,” for additional
details.
Azul Systems
Confidential and Proprietary
49
Zing Installation and Administration Guide
l
l
Apply JDK customizations you made locally to the standard Linux JDK.
Ensure the ulimit is unlimited for virtual memory, resident memory, and core file size.
Run the following command or add it to a system start command file:
$ ulimit -v unlimited -m unlimited -c unlimited
l
If you are running Oracle HotSpot or OpenJDK and using XX:+UseLargePages, ensure
there is sufficient memory allocated to each: Zing, hugepages, and Linux memory.
l
If your Java application requires access to any javax.crypto unlimited strength policy
files, ensure JDK configuration compatibility.
Run your Java applications with Zing
With Zing installation and configuration completed, run Java runtime applications on ZVM instances for
your JVM just as you would run a HotSpot Java implementation. See Using the Zing Virtual Machine
for a list of Java and Zing command line parameters. See or run man zing-installation to view
common installation errors.
To verify installation:
1. Run a simple Java command, for example:
$ <zing_java_directory>/bin/java -version
For quicker access, use the fully qualified path. For example:
$ /opt/zing/zing-jdk<version>-<zing_version>-<platform>/bin/java version
2. Look for the ZVM component reference in the system response, for example:
/opt/zing/zing-jdk1.8.0-14.09.0.0-33-x86_64/bin/java -version
java version "1.8.0-zing_14.09.0.0"
Zing Runtime Environment for Java Applications (build 1.8.0-zing_
14.09.0.0-b20)
Zing 64-Bit Tiered VM (build 1.8.0-zing_14.09.0.0-b33-productazlinuxM-X86_64, mixed mode)
Azul Systems
Confidential and Proprietary
50
Zing Installation and Configuration
Zing Installation and Configuration
Installing and Configuring Zing System Tools
This section describes how to install and configure Zing System Tools (ZST).
l
Installation – The ZST installation package contains a kernel loadable module and supporting
scripts and tools. This includes:
l
l
Zing Memory
l
Zing license daemon
l
Zing tools, including zing-ps
Configuration –
l
Zing requires that a portion of host system memory is set aside and managed independently
from Linux. This is configured using the system-config-zing-memory command.
l
Usage –
l
To launch ZVM instances, the Zing Memory must be configured and running, and the
license daemon must be running.
For Ubuntu 14.04 or Debian platforms, see the Installing Zing System Tools with DKMS.
Note:
Complete Prerequisites
Prior to installing ZST, complete the following steps to ensure your system can support ZST.
Note
ZST must be installed prior to installing ZVM.
1. Ensure your environment meets the system requirements.
See Zing System Requirements, for a complete list of Zing requirements, including: supported CPUs,
memory requirements, operating system versions, supported browsers, and listening ports.
2. Ensure the following packages are installed.
System
RHEL, CentOS,
supported Oracle Linux 6
Azul Systems
Commands to Install Packages
# yum localinstall --nogpgcheck
libssl<version>binutils lsof curl
Confidential and Proprietary
51
Zing Installation and Administration Guide
System
RHEL 5.x, CentOS 5.x,
supported Oracle Linux 5
RHEL 7
Commands to Install Packages
# yum localinstall --nogpgcheck libssl<version>
binutils lsof curl
# yum groupinstall 'development Tools'
# yum localinstall --nogpgcheck
libssl<version> binutils lsof curl
# yum install perl-Sys-Syslog.x86_64
SLES
# zypper install libssl<version> binutils lsof
curl
Ubuntu
# apt-get install libssl<version> binutils lsof
curl
Depending upon the operating system the required packages include:
l
OpenSSL 0.9.8 or later
l
Development tools – Download the package from the operating system website.
l
binutils – Are required to ensure that any core files, generated when a ZVM instance fails
abnormally, include the shared libraries that were in use at the time.
3. If your systems are using power efficiency chips, such as the Nehalem chip set, disable the Intel idle
driver. See Disabling the Intel Idle Driver.
4. To ensure that OS settings are appropriate for performance sensitive applications, apply the following
configuration settings.
l
SWAP – Turn this off, either by removing swap or by setting swappiness to 0. Best practice
removes swap.
l
vm.min_free – Set this to 1 GB or more. Best practice assigns more than 1 GB.
l
Transparent Huge Pages (THP) – Turn this off.
See Appendix A -- Zing Installation Prerequisites Details for information on performing these tasks.
Install Zing System Tools
The ZST installation package contains a kernel loadable module and supporting scripts and tools.
To install ZST:
1. Ensure your system is configured to support ZST, complete the steps in Complete Prerequisites.
2. Login to your system as root administrator or use sudo.
3. Download ZST installation package to your system <zst_install> directory.
Azul Systems
Confidential and Proprietary
52
Zing Installation and Configuration
The zing-zst package contains the kernel module and various tools. Only one version of
zing-zst may be installed at a time.
The ZST installation package is specific to the operating system version, service pack, and/or
kernel on the installation machine.
System
ZST Installation Package Format
RHEL,
zing-zst-<kernel_version>.<zst_version>.<platform>.rpm
CentOS,
supported For example:
Oracle
zing-zst-5e.2.6.32-5.7.2.0.10.el6.x86_64.rp
Linux
zing-zst-<kernel_version>.<zst_version>.<platform>.rpm
SLES
For example:
zing-zst-5e.2.6.32-5.7.2.0.12.slesl1.sp2.86_64.rpm
<zst_version>_<platform>.deb
zing-zst_<kernel_version>-desktop-<zst_version>_
<platform>.deb
Ubuntu
For example:
zing-zst_5e.2.6.32.x86.64-server-5.7.2.0.10_
amd64.deb
zing-zst_5c-3.2.0.x86.64-desktop-5.7.2.0.10_
amd64.deb
Where –
<kernel_version> is 5e.2.6.32 or 5c-3.2.0
<zst_number> is 5.7.2.0.10
<platform> is el6.x86_64, sles11.sp2.86_64, or amd64
4. Install ZST using the appropriate command for your system, type:
System
RHEL, CentOS,
supported Oracle
Linux
Commands to Install Packages
# yum localinstall --nogpgcheck zing-zst<version>
For example
# sudo zing-zst-5e.2.6.32-5.7.2.0.10.el6.x86_
64.rpm
# zypper install zing-zst-<version>
SLES
Azul Systems
For example
Confidential and Proprietary
53
Zing Installation and Administration Guide
System
Commands to Install Packages
# sudo zing-zst-5e.2.6.325.7.2.0.10.sles11.sp2.86_64.rpm
# dpkg -i zing-zst_<version>.deb
Ubuntu
For example
# sudo dpkg -i zing-zst_5e.2.6.32.x86.64server-5.7.2.0.10_amd64.deb
The system responds.
You must configure Zing memory. Running 'system-config-zing-memory'
now is recommended.
5. Add /usr/sbin to any Zing executable path. For example:
To your appropriate profile file, add:
export PATH=/usr/sbin:$PATH
6. Configure Zing memory allocation. See Configure Zing Memory.
Configure Zing Memory
After ZST is installed you must configure Zing memory allocation. Zing requires a portion of host
system memory to be set aside and managed independently from Linux. The system-configzing-memory wizard assists with this.
The system-config-zing-memory creates or replaces any existing version of the
pmem.conf file.
Note:
Running the system-config-zing-memory wizard is required for first time installations.
This section describes creating a default configuration. To customize your Zing memory
configuration, see Modifying Zing Memory Configuration.
1. Complete the steps in Install Zing System Tools.
2. Configure Zing memory management using the system-config-zing-memory command.
Zing requires a portion of host system memory to be set aside and managed independently from
Linux. Use the system-config-zing-memory wizard for initial configuration.
From the command line, type:
# system-config-zing-memory
The system responds:
Welcome to the Zing memory configuration wizard by Azul Systems.
This wizard sets up the Zing memory configuration file
Azul Systems
Confidential and Proprietary
54
Zing Installation and Configuration
/etc/zing/pmem.conf.0 and initializes Zing memory. (Read the man
pages for zing, zing-zst, and zing-pmem-conf for more information
about this configuration file.)
By default, Azul recommends setting up your system with 25% Linux
memory and 75% Zing memory. Choose yes to accept this default, or no
to enter the wizard expert flow.
** accept default configuration
** (y)es or (n)o [default 'y']: Y
3. Accept default configuration settings. At the prompt, type: y.
** accept default configuration
** (y)es or (n)o [default 'y']:
Y
The system response lists the zing services that are started or restarted and memory
configuration information.
For example:
zing-memory: INFO: Restarting...
zing-memory: INFO: Stopping...
zing-memory: INFO: stop successful
zing-memory: INFO: Starting...
Info: azulPmemPages: 54454.
INFO: az_pmem_reserve_pages (num2mPages 54454) succeeded
INFO: az_pmem_fund_transfer (to 7, from 0, bytes 114198315008)
succeeded
INFO: az_pmem_fund_transfer (to 1, from 7, bytes 5708447744)
succeeded
INFO: az_pmem_fund_transfer (to 3, from 7, bytes 5708447744)
succeeded
INFO: az_pmem_fund_transfer (to 0, from 7, bytes 102781419520)
succeeded
Info: You can now run Java processes up to -Xmx98020m or -Xmx95g.
Info: Azul pmem initialized successfully.
zing-memory: INFO: start successful
zing-memory: INFO: restart successful
You have completed ZST installation!
Zing installation is complete when you install the required Zing license file and ZVM. Modifying Zing
memory usage is available if you do not want to use the default settings.
Completing Zing Installation
After you complete ZST installation, complete the following tasks to enable ZVM instances to run your
Java applications.
Azul Systems
Confidential and Proprietary
55
Zing Installation and Administration Guide
Install Zing License File
Install Zing Virtual Machine
Verify Zing Installation
Install Zing License File
After ZST is installed, install your Zing licensing.
A Zing license is required to start a Java application using a ZVM instance. Zing licenses are issued by
Azul. There are different Zing licensing options for getting the Zing license installed and ready for ZVM.
The option you use depends upon your contract with Azul.
l
Zing license (ZVM Host) – Standalone Zing licenses are installed on ZVM host machines.
These licenses are typically applied one license per ZVM host. All Zing installations require
the Zing license file. The Zing license daemon is installed with ZST.
See Installing Zing Standalone Licenses.
l
Zing Pool Server license – A Zing Pool server license allows you to use the Zing license
server to run an allocated number of Zing licenses concurrently among a set of ZVM hosts
that are configured to use Pool licenses.
See Installing and Using Pool Licensing .
Install Zing Virtual Machine
After ZST is installed and the Zing licensing is applied, install ZVM. See Installing the Zing Virtual
Machine for complete instructions.
Verify Zing Installation
After ZST is installed, configured, and started, and ZVM is installed, verify the completed installation.
1. Verify that your Zing license is installed.
Check /etc/zing/directory for your licensing authorization file, license.
l
If you are running ZVM instances on a standalone host, see Installing Zing Standalone
Licenses.
l
If you are configuring your ZVM host as a member of a Zing licensing pool, see Installing
and Using Pool Licensing.
2. Run a simple Java command, for example type:
$ <zing_java_directory>/bin/java -version
where <zing_java_directory> is the directory where the Zing /bin/java is
located. If you accept the ZVM installation defaults, this path is, for example:
$ /opt/zing/zing-<version>/bin/java -version
Azul Systems
Confidential and Proprietary
56
Zing Installation and Configuration
For quicker access, use the fully qualified path.
3. Look for the ZVM component reference in the system response, for example:
/opt/zing/zing-jdk1.8.0-14.09.0.0-33-x86_64/bin/java -version
java version "1.8.0-zing_14.09.0.0"
Zing Runtime Environment for Java Applications (build 1.8.0-zing_
14.09.0.0-b20)
Zing 64-Bit Tiered VM (build 1.8.0-zing_14.09.0.0-b33-productazlinuxM-X86_64, mixed mode)
See Troubleshooting Zing or run man zing-installation to view common
installation errors.
Modifying Zing Memory Configuration
If you completed all the steps in Installing and Configuring Zing System Tools Zing memory is
configured with default settings. Default configuration sets aside 75% of the physical memory of your
machine for use by ZVMs.
To customize the memory allocation, run the system-config-zing-memory command and do
not accept the default. Running this command automatically restarts the system.
When modifying the memory allocations for ZST, consider the following memory calculations:
/proc/meminfo.MemTotal = Linux Memory + Zing Memory
Zing Memory = Reservable + Contingency + Pause Prevention
Where –
l
Zing Memory – The ZVM uses Zing Memory to allocate the memory required for the Java
heap.
l
Reserved Memory – The amount of Reservable memory initially allocated to a specific ZVM
instance when the instance starts.
l
Contingency Memory – A backup reserve that Zing uses to temporarily allocate additional
memory to any ZVM instance running a Java application.
l
Pause Prevention Memory – A percentage of the Zing Memory set aside as part of Zing's
Generational Pauseless Garbage Collection (GPGC).
See Understanding Zing Memory Use for additional information about memory use. The following
shows an overview.
Azul Systems
Confidential and Proprietary
57
Zing Installation and Administration Guide
To modifying Zing memory allocation complete the steps in the listed sections:
1. Start the process.
Modify Zing Memory Allocation
2. Choose one of the following two allocation methods:
l
Use System Method to Allocate Zing Memory
If you use the System method, choose one of the following options:
l
l
Use Percentage of System Memory for Zing Memory
l
Use Exact Amount of System Memory for Zing Memory
Use Java Method to Allocate Zing Memory
3. Complete the process.
Modify Zing Memory Allocation
To modify Zing memory configuration using the wizard:
Note
The system-config-zing-memory creates or replaces any existing version of the
pmem.conf file.
1. Login to your RHEL system as root.
2. Ensure there are no running processes using Zing memory, type:
$ zing-ps
See “Using the zing-ps Tool” in Using Zing System Tools.
3. Run the Zing memory configuration wizard.
$ system-config-zing-memory
Azul Systems
Confidential and Proprietary
58
Zing Installation and Configuration
The system responds.
Welcome to the Zing memory configuration wizard by Azul Systems.
This wizard sets up the memory driver configuration file
/etc/zing/pmem.conf.0
and initializes the Zing memory. (Read the man pages for zing, zingzst, and zing-pmem-conf for more information about this
configuration file.)
By default, Azul recommends setting up your system with 25% Linux
memory and 75% Zing memory. Choose yes to accept this default, or no
to enter the wizard expert flow.
4. Do not accept the default configuration, type: N
** accept default configuration
** (y)es or (n)o [default 'y']: n
The system responds.
[ Current task: Sizing Zing memory / Choosing sizing method ]
Zing requires you to partition memory up front for exclusive use by
Java. (This is analogous to Linux hugepages, although Zing pages are
not interchangeable with hugepages.)
Would you like to decide how much memory to allocate to Zing based
on the total size of your system, or based on the total size of the
Java instances you expect to run?
** Enter (s)ystem or (j)ava heap size [default 's']:
5. Select a method for allocating Zing memory:
l
System – system sizing allocates memory as a percentage of the available host
system memory. The default value if 75% for Zing Fund memory.
To use this method, continue to Use System Method to Allocate Zing Memory.
l
Java – Java sizing allocates memory as a fixed value in MB based on the estimated
sum of all Xmx values that might be run concurrently.
To use this method, continue to Use Java Method to Allocate Zing Memory.
Use System Method to Allocate Zing Memory
To select the System method for allocating Zing Fund memory.
1. From the system-config-zing-memory wizard, specify System method, type: S
** Enter (s)ystem or (j)ava heap size [default 's']: s
The system responds.
[ Current task: Sizing Zing memory / Choosing system sizing method ]
Do you want to allocate a percentage of your total host system
memory to Zing, or would you prefer to specify an exact amount?
** Enter (p)ercentage or (e)xact [default 'p']:
Azul Systems
Confidential and Proprietary
59
Zing Installation and Administration Guide
2. Select a memory allocation method:
l
Percentage – allocates a percentage of the total host system memory.
To use this method, continue to Use Percentage of System Memory for Zing Memory.
l
Exact – allocates a specific value in MB of memory. This is based on the estimated maximum
Xmx values running concurrently.
To use this method, continue to Use Exact Amount of System Memory for Zing Memory.
Use Percentage of System Memory for Zing Memory
To specify the allocated amount as a percentage of the total host system memory.
1. From the system-config-zing-memory wizard, specify the allocation method, type P
** Enter (p)ercentage or (e)xact [default 'p']: p
The system responds with the next configuration prompt.
2. Specify the percentage value to use, type a percentage value. The default is %75.
** Enter percentage of total memory to dedicate to Zing [default
'75']:
The system responds with the next configuration prompt.
3. Continue to Specify Contingency Memory Pool.
Use Exact Amount of System Memory for Zing Memory
To specify allocation amount as an exact value:
1. From the system-config-zing-memory wizard, specify the exact method, type: e
** Enter (p)ercentage or (e)xact [default 'p']: e
The system responds.
[ Current task:
Sizing Zing memory / Choosing specific size ]
Please specify the amount of memory for Zing to use. (Note:
system has a total of 15g.)
examples:
your
16384m, 16g
** Enter memory to dedicate to Zing:
2. Specify the exact MB value to allocate to Zing memory.
For example:
** Enter memory to dedicate to Zing: 12g
The system responds with the next configuration prompt.
3. Continue to Specify Contingency Memory Pool.
Use Java Method to Allocate Zing Memory
To select the Java method for allocating Zing memory.
Azul Systems
Confidential and Proprietary
60
Zing Installation and Configuration
1. From the system-config-zing-memory wizard, specify the exact method, type: J
** Enter (p)ercentage or (e)xact [default 'p']: j
The system responds.
[[ Current task: Sizing Zing memory / Deriving size from Xmx ]
What is the sum of the Xmx values of all the Java processes you plan
to run concurrently?
** Enter sum of Xmx:
For example:
** Enter sum of Xmx: 2g
The system responds with the next configuration prompt.
3. Continue to Specify Contingency Memory Pool.
Specify Contingency Memory Pool
To specify the memory allocation for the Zing Contingency memory (also known as elastic memory)
pool:
1. From the system-config-zing-memory wizard, type the desired percentage. For example:
75. See Understanding Zing Memory Use for information about Contingency memory.
[ Current task: Sizing Zing memory / Sizing Contingency memory pool
]
Part of Zing memory is set aside in a common contingency memory pool
for all Azul Java processes to share. The contingency memory pool
exists as an insurance policy to allow a JVM instance to temporarily
avoid OOM behavior and grow beyond -Xmx.
What percentage of Zing memory would you like to dedicate to the
contingency memory pool?
** Enter percentage of Zing memory to be used for contingency memory
[default '5']:
2. Verify Zing Memory is restarted. View the system return from the last step to locate the following
lines. If you do not see them, see Starting or Stopping the Zing System Tools Services.
[ Current task:
driver ]
Saving configuration and initializing Zing memory
zing-memory: INFO: Restarting...
zing-memory: INFO: Stopping...
zing-memory: INFO: stop successful
zing-memory: INFO: Starting...
Info: azulPmemPages: 54454.
INFO: az_pmem_reserve_pages (num2mPages 54454) succeeded
INFO: az_pmem_fund_transfer (to 7, from 0, bytes 114198315008)
succeeded
Azul Systems
Confidential and Proprietary
61
Zing Installation and Administration Guide
INFO: az_pmem_fund_transfer (to 1, from 7, bytes 5708447744)
succeeded
INFO: az_pmem_fund_transfer (to 3, from 7, bytes 5708447744)
succeeded
INFO: az_pmem_fund_transfer (to 0, from 7, bytes 102781419520)
succeeded
Info: You can now run Java processes up to -Xmx98020m or -Xmx95g.
Info: Azul pmem initialized successfully.
zing-memory: INFO: start successful
zing-memory: INFO: restart successful
Installing Zing Standalone Licenses
This section describes the licensing required to launch ZVM, how to obtain it and install it.
Zing Licensing Overview
This section describes the Zing licensing required to allow Java runtime applications on instances of
ZVM processes, how to obtain the license and install it.
The standalone Zing licensing method requires the following:
l
Zing license file, license – Required on all ZVM hosts. It allows ZVM instances to start Java
applications. It is issued by Azul.
Installing Zing Licenses
Azul sends you a copy of the Zing license.
1. Obtain the license. Typically, the license file is emailed to your company contact.
2. Choose a method for launching the ZVM instance :
l
Copy the license file, license, to the /etc/zing/ directory of your ZVM host.
If you copy your license file to /etc/zing, you do not need to specify the Zing license in the
ZVM instance launch command.
l
Copy the license file to a location of your choice.
If you choose this option, then you must add -XX:ZingLicenseFile=<filename> to
your ZVM instance launch command.
Licenses allow ZVM instance launches up to the end of the day they are due to expire within
the local timezone specified in the license.
Depending upon your license agreement with Azul, you might see
messages showing your license is expired but the ZVM instance has
been allowed to launch. If you see these messages, contact your
Azul representative to extend your license.
Azul Systems
Confidential and Proprietary
62
Zing Installation and Configuration
Renewing Zing License
When your Zing license expires, you can request a new license. The Zing license file does not require
an upgrade process, simply replace your Zing license file, when you receive a new one.
1. When your Zing license is about to expire contact Azul to obtain a new license.
2. Follow the steps in Installing Zing Licenses.
Installing the Zing Virtual Machine
This section describes how to install the Zing Virtual Machine (ZVM) for Zing so it can be used to run
Java applications.
Install the Zing Virtual Machine
Note
The gzip and tar utilities are required to install the ZVM software packages on Linux
host platforms.
To install the ZVM software using the appropriate installer for your supported platform.
1. Ensure that the current and compatible version of ZST is installed and configured before installing
the ZVM component. See Installing and Configuring Zing System Tools.
2. Ensure that your Zing license is installed.
Check /etc/zing/directory for your licensing authorization file, license.
If you are running ZVM instances on a standalone host, see Installing Zing Standalone
Licenses.
If you are configuring your ZVM host as a member of a Zing licensing pool, see Installing and
Using Pool Licensing.
3. Download the installation package corresponding to your operating system and your version of Java.
The installer package includes the ZVM Development Kit and a Linux JDK.
a. Locate the ZVM software on the Zing download site.
Refer to the Zing download site or the release notes for specific download instructions.
b. If you are using FTP, ensure the file transfer protocol is set to binary mode.
c. Select the installation package for your supported host platform.
The ZVM self-extracting shell script is not dependent upon the operating system on the
installation machine. The ZVM .rpm and .deb installation packages are specific to the
operating system on the host machine. See Zing System Requirements for a list of
supported OS versions.
d. Verify that the file size is the same as shown on the download page.
The ZVM installation package formats are:
Azul Systems
Confidential and Proprietary
63
Zing Installation and Administration Guide
System
ZVM Installation Package Format
zing-jdk<version>-<zvm_version>.<platform>.sh
For example:
Shar
zing-jdk1.6.0_5.8.0.0-13.x86_64.sh
zing-jdk1.7.0-5.8.0.0-13.x86_64.sh
zing-jdk<version>-<zvm_version>.<platform>.rpm
RHEL or SLES
For example:
zing-jdk1.6.0-5.8.0.0-13.x86_64.rpm
zing-jdk1.7.0-5.8.0.0-13.x86_64.rpm
zing-jdk<version>_<zvm_version>_<platform>.deb
For example:
Ubuntu
zing-jdk1.6.0_5.8.0.0-13_amd64.deb
zing-jdk1.7.0_5.8.0.0-13_amd64.deb
The installer package has the following naming format:
zing-jdk<version>-<zvm_version>.<platform>
Where –
jdk<version> is 1.6.0 or 1.7.0
<zvm_version> is 5.8.0.0-13
<platform> is x86_64 or amd64
4. If you are using an .rpm or .deb installation package, login as root.
a. Enter the root shell.
b. Type the su command.
c. Enter the super-user password when prompted.
5. Run the appropriate command to install the ZVM component on the host machine where the Java
applications will be run.
Note
l
The gzip and tar utilities are required to install the ZVM software packages on Linux
host platforms.
For shar script installation:
$ ./zing-jdk<version>-<zvm_version>.<platform>.sh
For example:
$ ./zing-jdk1.6.0_5.8.0.0-13.x86_64.sh
Additional shar options include:
Azul Systems
Confidential and Proprietary
64
Zing Installation and Configuration
Option
Description
Install a full copy of the local version of the
Linux JDK along with the ZVM software
-cp
when using the full installation package.
Specify the directory to install the ZVM
software.
-dest <inst-dir>
Required. Specify the path where the
GNU tar executable is located.
Example:
-tar
./zing-jdk1.6.0_5.8.0.013.x86_64.sh -tar /opt
Optional. Specify the host compatible
version of the ZVM software to install.
<sys> options is:
linux64 – for Linux running on x86 (64bit). This is the default and the only
acceptable option.
-target <sys>
Example:
./zing-jdk1.6.0_5.8.0.013.x86_64.sh -target linux64
l
For RHEL systems:
$ yum localinstall –-nogpgcheck ./zing-jdk<version>-<zvm_
version>.<platform>.rpm
The contents of the RPM package are installed into the /opt/zing directory.
To examine the contents of the RPM package prior to installation, execute the rpm
command using the -qpl option. Using the format:
$ rpm -qpl ./zing-jdk<version>-<zvm_version>.<platform>.rpm
l
For SLES systems:
$ zypper install ./zing-jdk<version>-<zvm_
version>.<platform>.rpm
l
For Ubuntu systems:
$ dpkg -i ./zing-jdk<version>_<zvm_version>_<platform>.deb
When the installation completes, the following message displays:
Installation completed successfully.
Azul Systems
Confidential and Proprietary
65
Zing Installation and Administration Guide
Zing Virtual Machine version 5.8.0.0-13.x86_64, Java version 1.6.0
Installing VM for platform: linux64
Decompressing Zing VM...
ZVM files are located in /opt/zing, /etc/zing, and /usr/sbin.
6. For shar script installations, specify the location of the Linux JDK, at the prompt.
Location of <linux_type> JDK (<path>):
The local version of the Linux JDK must match the version of the ZVM software being installed
(such as 64-bit).
Completing Post ZVM Installation Tasks
Perform the following post installation steps as needed for your specific operating system.
l
Applying JDK Customizations
l
Setting an Unlimited ulimit
l
Ensuring Sufficient Memory
l
JDK Configuration Compatibility
Applying JDK Customizations
Install customizations made to the local version of the standard Linux JDK.
Make customizations to the ZVM JDK to match any done to the local version of the Linux JDK . For
example, if jar files were added to an ext directory of the local JDK, those files must also be added to
the ZVM JDK.
See Troubleshooting Zing or run man zing-installation to view common installation errors.
Setting an Unlimited ulimit
Ensure the ulimit is unlimited for virtual memory, resident memory, and core file size. At the
command line, type:
$ ulimit -v unlimited -m unlimited -c unlimited
Some operating systems default the ulimit to -v unlimited, others do not. The -v unlimited
value is required prior to launching a ZVM instance for the first time. The -m and -c options issue
warnings and do not prevent a ZVM instance from launching.
Set the ulimit prior to launching any ZVM instances or add it to a system start command file.
Ensuring Sufficient Memory
If you are running Oracle HotSpot or OpenJDK and using XX:+UseLargePages, you must ensure
there is sufficient memory allocated to each: Zing and Linux memory.
To ensure that OS settings are appropriate for performance sensitive applications, apply the following
settings:
Azul Systems
Confidential and Proprietary
66
Zing Installation and Configuration
l
Disable transparent_hugepages on your Linux system where you are running Zing.
l
Turn off SWAP either by removing swap or by setting swappiness to 0. Best practice
removes swap.
l
Set vm.min_free_kbytes to 1 GB or more. Best practice assigns more than 1 GB.
JDK Configuration Compatibility
If your Java application requires access to any javax.crypto unlimited strength policy files:
l
If they are included with your existing version of the JDK, copy them to the Linux JDK where the
ZVM software package is installed.
l
If you do not have an existing JDK, download the javax.crypto policy files from the Oracle
website directly and store them to the Linux JDK where the ZVM software package is installed.
This includes two jar files in the $JAVA_HOME/jre/lib/security directory:
US_export_policy.jar
local_policy.jar
If these files are not found, the Java application might experience the following exception:
java.lang.SecurityException: Unsupported keysize or algorithm
parameters at javax.crypto.Cipher.init(DashoA12275)
Verifying Zing Virtual Machine Installation
After the ZST component is installed and configured, and the Zing license and the ZVM component are
installed, verify the completed Zing installation.
1. Verify that your Zing license is installed.
Check /etc/zing/directory for your licensing authorization file, license.
l
If you are running ZVM instances on a standalone host, see Installing Zing Standalone
Licenses.
l
If you are configuring your ZVM host as a member of a Zing licensing pool, see Installing
and Using Pool Licensing .
2. Run a simple Java command, for example type:
$ <zing_java_directory>/bin/java -version
where <zing_java_directory> is the directory where the Zing /bin/java is
located. If you accept the ZVM installation defaults, this path is, for example:
$ /opt/zing/zing-jdk1.8.0_<zvm_version>/bin/java -version
For quicker access, use the fully qualified path. For example:
$ /opt/zing/zing-jdk1.8.0-14.09.0.0-33 -x86_64/bin/java -version
Azul Systems
Confidential and Proprietary
67
Zing Installation and Administration Guide
3. Look for the ZVM component reference in the system response, for example:
/opt/zing/zing-jdk1.8.0-14.09.0.0-33-x86_64/bin/java -version
java version "1.8.0-zing_14.09.0.0"
Zing Runtime Environment for Java Applications (build 1.8.0-zing_
14.09.0.0-b20)
Zing 64-Bit Tiered VM (build 1.8.0-zing_14.09.0.0-b33-productazlinuxM-X86_64, mixed mode)
See Troubleshooting Zing or run man zing-installation to view common
installation errors.
Using the Zing Virtual Machine
This section provides an overview of Azul Systems Zing component, Zing Virtual Machine (ZVM), and
describes the command line options that the ZVM supports.
Overview Zing Memory Management Processes
This section provides an overview of ZVM memory management processes. It covers the following
topics:
l
Generational Pauseless Garbage Collection (GPGC)
l
Zing Memory Allocation
l
Compatibility with HotSpot VM Command Line Options
Generational Pauseless Garbage Collection (GPGC)
The default Generational Pauseless Garbage Collector (GPGC) for the ZVM combines the benefits of
pauseless garbage collectors and generational collectors to achieve low consistent pause times
independent of Java heap size or amount of live objects.
GPGC uses the GC Pause Prevention pool to allow GC processes to allocate extra, temporary
memory to Java objects to prevent GC pauses. This is useful when the memory allocation rate for the
application exceeds the rate that the garbage collector can free memory. On GC cycle completion, the
extra memory returns to the pool and is made available for other GC processes.
See Garbage Collection Options for specific configuration information.
Zing Memory Allocation
The Azul Zing platform uses the ZVM to run Java applications, similar to a typical JVM running a Java
application. The significant difference is how memory is handled. Zing uses the ZST for managing the
memory of three components: Contingency memory, Pause Prevention memory, and the memory used
for GPGC.
Using Zing with GPGC and Contingency memory means that your Java application is no longer limited
to the maximum heap size specified on the Java command line when you started your Java application.
Azul Systems
Confidential and Proprietary
68
Zing Installation and Configuration
As an additional resource, GPGC might temporarily use Pause Prevention memory to prevent pauses
that are caused by lack of available clean pages.
See Understanding Zing Memory Use for additional information about how the memory resource is
used and assigned in a Zing environment.
Adjust the memory you configure to ensure you have sufficient memory to support ZVM instances:
l
Java heap use
l
Reserved memory
Using Azul technology, Java Application Execution uses all available memory without
consequences common to tasks such as garbage collection and heap memory
compaction. With the Zing platform you can allocate 100 GBs of memory to easily and
effectively enhance your enterprise production workloads.
Note
Compatibility with HotSpot VM Command Line Options
The ZVM recognizes the standard and non-standard command line options for the HotSpot VM that is a
core component of Java Platform, Standard Edition (J2SE) 6. See the HotSpot VM Options section, in
the Oracle developer site for detailed descriptions of the HotSpot VM command line options.
l
HotSpot command line options unsupported by ZVM
If the ZVM detects a HotSpot VM native command line option that it does not support, its
response can be set with the following option:
-Xnativevmflags:[ignore|error|warn]
The default option is ignore.
If -Xnativevmflags:error is specified and the ZVM detects an option that it does not
support, it does not launch and generates the following type of message:
HotSpot™ 64-Bit Tiered VM error: Unsupported native VM option
‘+UseParNewGC’
l
ZVM commands not implemented by HotSpot
For command line options that are not implemented by the HotSpot VM (including options
specific to the ZVM), the response is determined by the following option:
–Xflags:[ignore|error|warn]
The default setting is error.
The error setting ensures that unrecognized options are highlighted when the ZVM is used.
For example, if the incorrectly formatted option –XXMaxPermS=128m is detected, the ZVM
does not launch and generates the message:
HotSpot ™ 64-Bit Tiered VM error: Unrecognized VM option
‘MaxPermS=128m’
l
HotSpot or Open JDK XX:+UseLargePages
Azul Systems
Confidential and Proprietary
69
Zing Installation and Administration Guide
If you are running Oracle HotSpot or OpenJDK and using XX:+UseLargePages, you must
choose one of the following options to run Zing:
l
Disable the hugepages facility on your Linux system when you are running Zing.
l
Ensure there is sufficient memory allocated to each: Zing, hugepages, and Linux memory.
Command Line Options for the Zing Virtual Machine
This section describes Java command line options that are available in the ZVM. The following topics
are discussed:
l
Zing Virtual Machine Options
l
Garbage Collection Options
l
Reserved Memory Command
l
Statistics Monitoring Command
Zing Virtual Machine Options
Java command line options that control the interaction between ZVM and ZST begin with the prefix –
XX:. See Using Zing System Tools for more information about ZST. These options are summarized in
the following table.
ZVM Command Line Options1
Command Line Option
XX:JavaMemMax=<size>
See Note 2
Description
Default
Specifies the maximum amount of memory
that can be allocated to the ZVM process (the
Java heap). Units can be specified in kilobytes
(K or k), megabytes (M or m), or gigabytes (G
or g). For example:
-XX:JavaMemMax=2g
No
maximum
-XX:JavaMemMax=128M
-XX:JavaMemMax=128m
If this option is not set, the maximum is set to Xmx + Total Partition Memory.
1ZVM command line options are case sensitive.
2Contact Azul customer support for more information about this option.
Azul Systems
Confidential and Proprietary
70
Zing Installation and Configuration
Garbage Collection Options
Garbage collectors and the associated garbage collection options are described in the following table.
The default garbage collector is Generational Pauseless Garbage Collection (GPGC) that is specified
with the -XX:+UseGenPauselessGC command line option.
Garbage Collection (GC) Java Command Line Options
Command Line Option
Description
-XX:+UseGenPauselessGC
Generational Pauseless GC (GPGC) optimized for minimal
pause times.
-XX:PrintGCHistory=N
Set the number of historical GPGC cycle reports to save for
display in ZVision (default =50).
-XX:GCWarningHistory=N
Set the number of historical GPGC warnings to save for
display in the ZVision (default =50).
ExplicitGC Options
As of ZVM release 5.10.0, the default value the ZVM flag DisableExplicitGC is off. Prior to release
5.10.0, calls to System.gc() were ignored and no garbage collection was initiated unless -XX:DisableExplicitGC was set. From 5.10.0 this is no longer the case and System.gc() calls may
initiate a full GC cycle depending on the value of the GPGCOptimisticExplicitGC flag. If a
System.gc() call initiates a garbage collection, the calling thread may or may not be blocked for the
duration of the GC depending on the value of the SynchronousExplicitGC flag. The previous
behavior is achieved by turning on the DisableExplicitGC flag, using XX:+DisableExplicitGC.
l
-XX:{+/-}GPGCOptimisticExplicitGC
When enabled, a thread calling System.gc() initiates a new garbage collection if and only if
one is not already in progress or one is pending. This flag is enabled by default.
l
-XX::{+/-}SynchronousExplicitGC
When enabled, a thread that initiates a garbage collection by calling System.gc() is
blocked for the entire garbage collection cycle and resumes when the GC completes.
Otherwise a concurrent garbage collection is initiated and the calling thread continues to
execute. This flag is enabled by default.
Reserved Memory Command
When a Java application starts, heap memory is Reserved per the -Xmx value.
There is no specific ulimit pertaining to -Xmx for ZVM. Such memory is pre-allocated during the
ZST configuration which includes physical memory setup. ZST configuration requires root level
administrative access.
If -Xmx is less than 1 GB, Zing automatically increases it to 1 GB and sends a warning to stderr.
Azul Systems
Confidential and Proprietary
71
Zing Installation and Administration Guide
If you are installed on a SUSE Linux Enterprise Server (SLES) system, prior to launching ZVM for the
first time, set the ulimit. At the command line, type:
ulimit -v unlimited
Statistics Monitoring Command
To collect statistical information about the garbage collections behavior in ZVM, use the Zing -gpgc
option with Oracle's JVM statistics monitoring tool (jstat). Use the jstat distributed with Zing, not
the jstat distributed with Oracle's JVM.
Usage Syntax
jstat -gpgc [-t == show timestamps] pid [msec between samples [#
samples]]
Samples
Sample -gpgc, -t, samples, #samples options
jstat -gpgc -t 6549 10 5
Timestamp
NGU
OGU
PGU
JHMX
NGC
NGCT
OGC
OGCT
172.0 419840.0 2048.0 6144.0 1036288.0 276
265.268
170.689 123 94.579
172.1 419840.0 2048.0 6144.0 1036288.0 276
265.268
170.689 123 94.579
172.1 419840.0 2048.0 6144.0 1036288.0 276
265.268
170.689 123 94.579
172.1 419840.0 2048.0 6144.0 1036288.0 276
265.268
170.689 123 94.579
172.1 419840.0 2048.0 6144.0 1036288.0 276
265.268
170.689 123 94.579
GCT
Sample -gpgc option
jstat -gpgc 44265
NGU
OGCT
GCT
OGU
36864.0
147.621
73728.0
334.481
PGU
JHMX
36864.0
1036288.0
NGC
742
NGCT
186.860
OGC
221
Sample -class option
jstat -class 44265
Loaded
3063
Bytes
7777.3
Unloaded
1
Bytes
Time
0.0
1.14
Sample -gc option
jstat -gc 44265
Azul Systems
Confidential and Proprietary
72
Zing Installation and Configuration
NGU
OGCT
GCT
OGU
36864.0
147.621
73728.0
334.481
PGU
JHMX
NGC
36864.0
1036288.0
NGU
PGU
NGCT
742
OGC
186.860
221
Sample -gccause option
jstat -gccause 44265
JavaHeapPercentUsed
LGCC
GCC
OGU
14.23 36864.0 73728.0
147.621 No GC No GC
36864.0
JHMX
NGC
1036288.0
NGCT
742
OGC
OGCT
186.860
221
Sample -gcnew option
jstat -gcnew 44265
NGU
JHMX
NGC
OGC
NGCT
36864.0
1036288.0
742
221
186.860
Sample -gcold option
jstat -gcold 44265
PGU
OGU
JHMX
NGC
36864.0
73728.0
1036288.0
OGC
742
OGCT
221
GCT
147.621
334.481
Sample -gcutil option
jstat -gcutil 44265
JavaHeapPercentUsed
GCT
JHMX
14.23
1036288.0
NGC
742
NGCT
OGC
OGCT
186.860
221
147.621
334.481
Sample -stats option
jstat -stats 44265
Loaded
classInitTime classLinkedTime classVerifyTime
sun.cls.time applicationTime safepointSyncTime safepointTime
vmOperationTime NGC
NGCT OGC
OGCT
3063
0.759357
0.121911
0.250850
1.140906
1996.780026 0.450701
1.511731
1.663066
742
186.860000
221 147.621000
Output Header Descriptions
Header
Description
applicationTime
Time called
Bytes
Bytes consumed
Azul Systems
Confidential and Proprietary
73
Zing Installation and Administration Guide
Header
Description
classInitTime
Time called
classLinkedTime
Time called
classVerifyTime
Time called
GCC
GC cause
GCT
Both generation collectors total GC time
JavaHeapPercentUsed
Percent of memory used by Java heap
JHMX
Java heap Xmx value
Loaded
Processes loaded
LGCC
Last GC cause
NGC
New generation GC count
NGCT
New generation GC time (concurrent time the collector is
running)
OGC
Old generation GC count (at that instant)
OGCT
Old generation GC time (concurrent time the collector is
running)
NGU
New generation use (at that instant)
OGU
Old generation use (at that instant)
PGU
Permanent generation use
safepointSyncTime
Time called
safepointTime
Time called
sun.cls.time
Time called
Time
Elapsed time
Unloaded
Processes unloaded
vmOperationTime
Time called
Azul Systems
Confidential and Proprietary
74
Zing Installation and Configuration
ZVM Mini-Core File Option
There is an option, -XX:+DumpMiniCore, that creates significantly smaller Zing core files. This
option is specified through the Java application launch command and applied to the runtime Java
application. It is not applied to the ZVM host machine. When you use this option:
l
Generated Zing core files do not include application objects stored in the Java heap.
l
Time required to produce a Zing core file or core file bundle is reduced.
ZVM Mini-Core File Requirements
The mini-core file option requires ZST 5.7.5 or later and ZVM 5.10.x or later.
If the option is specified and either the ZST version or ZVM version is not compatible, a message is
issued.
l
If the ZST version supports the -XX:+DumpMiniCore option, but the ZVM version does not, an
Unrecognized VM option warning is printed to STDERR. For example:
/home/buildmaster/nightly/ZVM/5.x-zvm/released/zvm-5.9.0.07/sandbox/azlinux/jdk6/x86_64/product/bin/java -XX:+DumpMiniCore version
Zing 64-Bit Tiered VM error: Unrecognized VM option '+DumpMiniCore'
Could not create the Java virtual machine.
l
If the ZVM version supports the -XX:+DumpMiniCore option, but the ZST version does not, a
warning message is issued. For example:
/bin/java -jar -XX:+DumpMiniCore -version
Zing VM Warning: -XX:+DumpMiniCore is not supported by this ZST (found
API version 0x5e, requires API version >= 0x5f)
java version "1.6.0-internal-zing_99.99.99.99.dev"
Zing Runtime Environment for Java Applications (build 1.6.0-internalzing_99.99.99.99.dev-b25)
Zing 64-Bit Tiered VM (build 1.6.0-zing_99.99.99.99.dev-b501-productazlinuxM-X86_64, mixed mode)
Setting the ZVM Mini-Core File Option
Include the option -XX:+DumpMiniCore with your Java launch command.
Some failures cannot be effectively diagnosed with a mini-core file. In this case, Azul support may
request that you try to reproduce the failure and produce a full-sized core file, by removing XX:+DumpMiniCore from your Java launch command.
l
When -XX:+DumpMiniCore option is not specified, a full core file, including the application objects
stored in the Java heap, is created if Zing crashes.
l
When -XX:+DumpMiniCore option is specified, a core file, that does not include the application
objects stored in the Java heap, is created if Zing crashes.
Azul Systems
Confidential and Proprietary
75
Zing Installation and Administration Guide
Run a Java Application
To test ZVM installation, use the following procedure to run java -version in ZVM without
additional command line arguments.
1. Complete the installation instructions for ZST, ZVM, and the Zing License. This includes verifying
the license.
See Installing and Configuring Zing System Tools, Using the Zing Virtual Machine, and
Installing Zing Standalone Licenses or Installing and Using Pool Licensing.
2. Run a simple Java command, for example type:
$ <Zing_JVM_Directory>/bin/java -version
where <Zing_JVM_Directory> is the directory where the Zing /bin/java is located. If
you accept the ZVM installation defaults, this path is, for example:
/opt/zing/zing-jdk<version>-<zing_version>/
3. Look for the ZVM reference in the system response, for example:
java version "1.6.0-zing_5.7.0.0"
Zing Runtime Environment for Java Applications (build 1.6.0-zing_
5.7.0.0-b8)
Zing 64-Bit Tiered VM (build 1.6.0-zing_5.7.0.0-b8-productazlinuxM-X86_64, mixed mode)
See Troubleshooting Zing or run man zing-installation to view common installation
errors.
Native Applications with Zing
A native application is any application that is written in a language other than Java and calls JNI_
CreateJavaVM to create the VM and makes calls to Java code.This is usually written in C/C++ and
contains the “main” function to start execution.
For Information on known issues on native launch with Zing see Known Issues Running Native
Applications on Zing in the Troubleshooting Zing section of this guide.
Invoking Native Applications with Zing
If Native (C/C++) applications invoke ZVM, consider these issues before you execute your code.
l
Native application -- Identify how the native app creates the VM to find possible collisions with
areas used by or allocated by the ZVM.
l
Consider known limitations when running native apps on Zing. Identify the issues by the failure
messages.
l
Invoking the ZVM -- check for differences between a stock build and a Zing build.
l
Native apps should also know how to run Zing specific features such as the Tick Profiler
mechanism and C-Heap Leak detection.
Azul Systems
Confidential and Proprietary
76
Zing Installation and Configuration
Consult the cases listed below to determine your best course of action.
Invoking the ZVM in Native Applications
There are two primary ways of invoking the ZVM in native applications. Key sections of code are
highlighted.
Case 1: Native application dlopens libjvm.so and uses dlsym to find 'JNI_CreateJavaVM’.
Sample source:
……
JavaVMInitArgs vm_args;
JavaVMOption options[1];
options.optionString = "-Djava.class.path=./";
vm_args.version = JNI_VERSION_1_6;
vm_args.options = options;
vm_args.nOptions = 1;
/* Create the Java VM */
void* libjvm_handle;
jint (*createVM_handle) (JavaVM **pvm, void **penv, void *args);
libjvm_handle = dlopen("libjvm.so", RTLD_LOCAL | RTLD_LAZY);
if (!libjvm_handle) {
fprintf(stderr, "Can't load libjvm.so with error %s\n",dlerror());
exit(1);
}
createVM_handle = dlsym(libjvm_handle, "JNI_CreateJavaVM");
res = createVM_handle(&jvm, (void**)&env, &vm_args);
if (res < 0) {
fprintf(stderr, "Can't create Java VM\n");
exit(1);
}
……
Sample build/compile line:
gcc -fPIC -I${JAVA_HOME}/include -I${JAVA_HOME}/include/linux –o
hello_world hello_world.c
Sample run:
LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/amd64/server:$LD_LIBRARY_PATH
./hello_world
Azul Systems
Confidential and Proprietary
77
Zing Installation and Administration Guide
Case 2: Native application directly calls JNI_CreateJavaVM by linking against libjvm.
Sample source:
……
JavaVMInitArgs args;
JavaVMOption options;
args.version = JNI_VERSION_1_6;
args.nOptions = 1;
options.optionString = "-Djava.class.path=./";
args.options = &options;
int rv;
rv = JNI_CreateJavaVM(jvm, (void**)&env, &args);
if (rv < 0 || !env)
printf("Unable to Launch JVM %d\n",rv);
else
printf("Launched JVM! \n");
…….
Sample build/compile line:
gcc -fPIC -I${JAVA_HOME}/include -I${JAVA_HOME}/include/linux L${JAVA_HOME}/jre/lib/amd64/server –ljvm –o hello_world hello_
world.c
Sample run:
LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/amd64/server:$LD_LIBRARY_PATH
./hello_world
ZVM Instance Configuration Recommendations
Use the -Xmx command line to specify the Java heap size. The ZVM instance uses the specified
Java heap size to calculate the size of Reserved memory.
Note:
If -Xmx is less than 1 GB, Zing automatically increases it to 1 GB and sends a warning to
stderr.
Run the ZVM instance on the server running the Java processes.
RHEL Performance Tuning Options
Setting tuning options can be a complicated thing, so RedHat Enterprise Linux (RHEL) has an
installable daemon, tuned, that provides some tuning options, defined as profiles, to improve
performance. Tuned periodically collects data about hardware subsystems, such as disks, network
I/O, and switches. Then, based on the option selected, tuned dynamically adjusts system settings to
raise or lower power consumption modes for these devices on your system.
Azul Systems
Confidential and Proprietary
78
Zing Installation and Configuration
Note:
For Zing, lower power consumption modes are detrimental to performance because if
a core goes to sleep and the Garbage Collector or application has a thread that needs
to run, then the cost of 'warming' up the core is expensive in time as the cache gets
loaded and the pipeline is filled, etc.
Two tuned profiles that might be useful with your Zing installation:
l
throughput-performance
This is a server profile for typical throughput performance tuning. It disables tuned and
ktune power saving mechanisms, enables sysctl settings that improve the throughput
performance of your disk and network I/O, and switches to the deadline scheduler.
l
latency-performance
This is a server profile for typical low-latency performance tuning that can reduce RHEL6 jitter
caused by the OS. It disables tuned and ktune power saving mechanisms and enables
sysctl settings that improve the latency performance of your network I/O.
The latency-performance profile on RHEL 6 can improve the max response time,
depending upon the application running on Zing, by several (maybe 10s) of msec compared to
to the tuned default profile.
You can set and un-set these profiles with tuned-adm. The profiles are implemented with the tuned
daemon.
Installing the Tuned Package
Installing the tuned package sets up a sample configuration file at /etc/tuned.conf and activates
the default profile. The default power-saving profile has the lowest impact on power saving of the
available profiles. It only enables CPU and disk tuned plug-ins.
Installing tuned-utils adds following dependencies on a standard installation:
l
systemtap-runtime
l
kernel-devel
l
systemtap
To install tuned
1. Obtain the tuned installable package.
Refer to your RHEL distribution for the tuned package and documentation on using tuned. It
is available with several RHEL distributions, including RHEL 6, CentOS 6, and Fedora.
2. Install the tuned package and its associated systemtap scripts with the command:
# yum install tuned
Running Tuned
To use the tuned daemon:
Azul Systems
Confidential and Proprietary
79
Zing Installation and Administration Guide
1. Enable tuned. At the command prompt, type:
# chkconfig tuned on
2. Start tuned. At the command prompt, type:
# service tuned start
3. View the tuned profiles. At the command prompt, type:
# tuned-adm list
The system response is:
Available profiles:
- default
- desktop-powersave
- latency-performance
- laptop-ac-powersave
- server-powersave
- laptop-battery-powersave
- throughput-performance
- enterprise-storage
Current active profile: default
The profiles are implemented through the tuned daemon.
4. Enable the desired profile. At the command prompt, type:
# tuned-adm profile <profile_name>
For example:
# tuned-adm profile latency-performance
Calling '/etc/ktune.d/tunedadm.sh stop': [ OK ]
Stopping tuned: [ OK ]
Switching to profile 'latency-performance'
Applying ktune sysctl settings:
/etc/ktune.d/tunedadm.conf: [ OK ]
Calling '/etc/ktune.d/tunedadm.sh start': [ OK ]
Applying sysctl settings from /etc/sysctl.conf
Tuned Options
Tuned has additional options that you can use when you run it manually. The available options are:
l
-d, --daemon
Starts tuned as a daemon instead of in the foreground.
l
-c, --conffile
Uses a configuration file with the specified name and path, for example:
Azul Systems
Confidential and Proprietary
80
Zing Installation and Configuration
--conffile=/etc/tuned2.conf
The default is /etc/tuned.conf
l
-D, --debug
Uses the highest level of logging.
Automatically Starting Tuned
To start tuned every time the machine boots, at the command prompt, type:
# chkconfig tuned on
Disabling All Tuning
You can turn off the tuned profiles, use the tuned-adm command. At the command prompt, type:
# - tuned-adm off
This disables all tuned profiles.
Azul Systems
Confidential and Proprietary
81
Zing Installation and Administration Guide
Specific Use-Cases: Zing Installation
The key sections to review and complete to install and use your Zing implementation for specific use
cases are:
l
Using Zing through Amazon Web Services
l
Advanced Installation: Installing and Configuring Zing System Tools using DKMS
l
Installing and Configuring Zing Using Read-Only Images
Using Zing through Amazon Web Services
This section describes how to create an Amazon Elastic compute Cloud (EC2) instance for Zing.
Sign In to Amazon Web Services
The following directions are provided as a general reference for signing up and using Amazon EC2 with
Zing. Refer to the Amazon documentation for current and complete information about Amazon Web
Services (AWS).
1. In your web browser, enter http://aws.amazon.com/.
2. Click Sign Up.
3. In the Sign In or Create an AWS Account, enter the requested information.
Azul Systems
Confidential and Proprietary
82
Specific Use-Cases: Zing Installation
If you have an AWS account:
a. Click I am a returning user and my password is.
b. Enter your email address and password.
c. Click Sign in using our secure server.
If you do not have an Amazon Web Service (AWS) account:
a. Enter your email address.
b. Click I am a new user.
c. Click Sign in using our secure server.
d. Provide the requested information, as prompted. Required information includes your
identification, accepting the Amazon Web Services Customer Agreement, billing information,
verifying identify, and selecting a support plan.
l
An email address
If you have a personal Amazon account and use that email address, Amazon requires
that you attach the web services to your existing account.
l
A credit card
This card is billed for any time you are running one of the AWS servers. You are not
billed when the server is not running.
Azul Systems
Confidential and Proprietary
83
Zing Installation and Administration Guide
If needed, complete the Request to Increase Amazon EC2 Instance Limit form.
Contact your company's systems administrator or other authorized person to obtain
your AWS account information.
l
A phone number
Amazon calls the phone number for verification as part of the process for creating a
web services account. Use a phone number you can answer while viewing the online
forms because you are prompted to provide four-digit number displayed on the
computer screen.
Build an EC2 Server
Create the server to install Zing.
Note:
These directions create a cluster compute server with HVM based virtualization.
Currently this is the only Amazon Machine Image (AMI) with sufficient resources to
support Zing.
1. From the Amazon Web Services, aws.amazon.com, select My Account/Console > AWS
Management Console. See Sign In to Amazon Web Services.
2. From the AWS Management Console, https://console.aws.amazon.com/console/home, select EC2 in the Amazon Web Services > Compute & Networking section.
3. From your EC2 Management Console, select Region > US North (Virginia) in the upper right
corner of the Navigation section. At this time, this is the only AWS region that has servers with
Azul Systems
Confidential and Proprietary
84
Specific Use-Cases: Zing Installation
sufficient resources to support Zing. The default Region Oregon does not have a suitable image
available.
4. Click Launch Instance in the Amazon EC2 Console Dashboard > Create Instance section.
5. Select an Amazon Machine Image (AMI) from the Choose an Amazon Machine Image (AMI)
> Quick Start. See "Operating System Requirements" in Zing System Requirements for a list of
supported AMIs.
6. Select an Instance Type. Choose a compute optimized, cc2.8xlarge or larger instance.
7. Review the Instance choices. Click the numbered steps at the top and modify as needed.
l
l
l
l
Configure Instance – Specify number of instances, purchasing options, network settings,
shutdown behavior, and other advanced details.
Add Storage – Specify storage device settings.
Tag Instance – Categorize your EC2 resources by assigning a tag name to them using a key
and value you specify.
Configure Security Group – Add firewall rules for your instance.
8. Click Review and Launch. Verify your settings and choices.
9. Click Launch.
The instance begins to launch.
Usage hours on the instance start immediately. Amazon charges in hour
Note: increments. Charges continue to accumulate until you Stop or Terminate the
instance. Stopping and restarting in the same hour does not reduce costs.
10. If you did not create a key pair, you are prompted to select an existing key pair or create a new
key pair.
a. Click Choose and exiting key pair > Create a new key pair.
b. Enter a key pair name.
c. Click Download Key Pair.
A key pair file <key_pair_name>.pem is placed in your computer's default
download directory.
d. Click Launch Instances.
11. Click Close.
12. Verify your instance is running.
Refresh your ECS Management Console page. Check Amazon EC2 Console Dashboard >
Resources > Running Instance. The value should increment to include the new instance.
13. View instance information.
a. Click the Events tab under EC2 Dashboard left bar menu.
b. Click an instance check box in the Instances table.
An instance detail panel is displayed.
Azul Systems
Confidential and Proprietary
85
Zing Installation and Administration Guide
Stop EC2 Instance
Amazon charges apply the entire time your instance is running. If you are not actively using the
instance, turn it off.
1. From the EC2 Management Console > My Instances section, click the checkbox corresponding
to the instance to stop.
2. Select Instance Actions > Stop.
3. Click Yes to confirm starting the selected instance.
The state for your instance changes from running to stopping then to stopped when the stop
instance process is complete. Charges for your instance stop accumulating.
Start EC2 Instance
Once created you can return to Amazon Web Services to manage your instances. To start an instance:
1. From the EC2 Management Console > My Instances section, click the checkbox corresponding
to the instance to start.
2. Select Instance Actions > Start.
3. Click Yes to confirm starting the selected instance.
The state for your instance changes from stopped to pending then to running when the start
instance process is complete. Charges for your instance start accumulating.
Prepare AWS EC2 Instance for Zing
To install Zing onto your EC2 instance:
Azul Systems
Confidential and Proprietary
86
Specific Use-Cases: Zing Installation
l
Prepare Required Files
l
Start your AWS Instance
l
Login Remotely to your AWS Instance
l
Create a Non-Root User
l
Copy the Zing Installation Files to the AWS Instance
l
Install Zing Components
Prepare Required Files
1. Log into the local machine you used to create the AWS instance.
2. On your local machine, make a Zing directory. For example: <local_
directory>/Zing/install_files/
3. Download the Zing files.
l
Download the Zing Virtual Machine (ZVM) component.
l
Download the Zing System Tools (ZST) component.
l
Optionally, download the Zing License Server.
l
Download the Zing documentation bundle.
4. Move the <name>.pem file to the same directory where your Zing installer files are located.
This file was placed in your default download directory by AWS when you created your
instance.
Start your AWS Instance
Log into Amazon Web Services and click through to view your instances in the ECS Management
Console.
1. From a browser, go to aws.amazon.com. Select My Account/Console > My Account. Log in.
2. Select Developers > AWS Management Console. Click Sign in to the AWS Console. Click
EC2.
3. View the status of your created instances. Click Instances in the Navigation section.
4. Start the instance for the Zing installation. See Start EC2 Instance.
a. Click the checkbox corresponding to the instance to start.
b. Select Instance Actions > Start.
c. Click Yes to confirm starting the selected instance.
5. After the instance is started, click the checkbox again for your running instance.
6. Locate and copy the Public DNS. For example, ec2-107-22-31-225.compute1.amazonaws.com.
Azul Systems
Confidential and Proprietary
87
Zing Installation and Administration Guide
Login Remotely to your AWS Instance
On your local machine, start a command shell and open an SSH session to the AWS DNS address.
For example:
1. Start Cygwin.
2. CD to the directory with the .pem file. For example:
$ cd <local_directory>/Zing/install_files/
3. SSH to the AWS instance, using the instance Public DNS address.
$ ssh -i <key_pair_name>.pem root@<public_dns>
4. Confirm connection, if prompted.
Create a Non-Root User
From your shell logged into the AWS instance, create a non-root user. This is so the Zing installation is
not at the root directory.
For general example, in the Cygwin shell prompt, ip-<address>:~#
Azul Systems
Confidential and Proprietary
88
Specific Use-Cases: Zing Installation
ip-<address>:~# yast2 users add username=<new_user> password=<user_
password>
Copy the Zing Installation Files to the AWS Instance
1. Open a second command shell.
2. Change to the local directory where you have your Zing installation files. For example:
$ cd <local_directory>/Zing/install_files/
3. Set up a file transfer connection to the AWS instance as the new non-root user.
General example:
$ sftp <new_user>@<public_dns>
Password: <user_password>
Connected to <public_dns>
Literal example:
$ sftp new_user@ ec2-23-22-126-229.compute-1.amazonaws.com
Password: user_password
Connected to ec2-23-22-126-229.compute-1.amazonaws.com
4. Upload the Zing installation files to the AWS instance. If you are using a .sh package, ensure you
use secure FTP in binary mode.
l
Zing System Tools
sftp> put zing-zst-5c.3.0.13-5.2.2.0.8.sles11.sp2.x86_64.rpm
l
Zing Virtual Machine and license
sftp> put zingLX-jdk1.6.0-5.2.1.0-3.x86_64.rpm
sftp> put license
5. Logout from secure FTP.
sftp> quit
Install Zing Components
1. Open the command shell.
2. Log in as root.
3. Change to the directory where the Zing installation files are located. For example, using the <new_
user> directory created in Copy the Zing Installation Files to the AWS Instance:
$ cd <new_user>/Zing/install_files/
4. Follow the directions in the Zing installation guides.
l
Installing and Configuring Zing System Tools
Azul Systems
Confidential and Proprietary
89
Zing Installation and Administration Guide
This includes installing and configuring ZST. This must be completed prior to installing ZVM.
l
Installation – The ZST installation package contains a kernel loadable module and
supporting scripts and tools.
l
Configuration – Zing requires that a portion of host system memory is set aside and
managed independently from Linux. This is configured using the ZST system-
configzing-memory command.
l
Installing Zing Standalone Licenses
Zing licensing is required to allow Java runtime applications on instances of ZVM processes.
Installing the Zing Virtual Machine
This describes how to install the Zing Virtual Machine (ZVM) for Zing so it can be used to run
Java applications.
5. Stop the AWS instance, as needed. Start the AWS instance whenever you need to use Zing for
your runtime Java applications.
Installing Zing System Tools using DKMS
This section describes how to install ZST using the Dynamic Kernel Module Support (DKMS)
framework to automatically build the kernel module.
This guide is intended for advanced users only.
Note:
While this feature is tested and validated for a selected set of operating system
versions, there are always risks. Test thoroughly in your staging environment
before going into production.
If you are installing on standard RHEL 5, RHEL 6, or RHEL 7 kernels, use the
standard ZST installation packages.
l
Prerequisites – Ensure your system meets the prerequisites prior to installing ZST using DKMS
l
Installation – The ZST installation package contains a kernel loadable module and supporting
scripts and tools. This includes:
l
l
Zing Memory
l
Zing license daemon
l
Zing tools, including zing-ps
Configuration –
l
Zing requires that a portion of host system memory is set aside and managed independently
from Linux. This is configured using the system-config-zing-memory command.
Azul Systems
Confidential and Proprietary
90
Specific Use-Cases: Zing Installation
Installing and Configuring Zing System Tools
To install and configure Zing System Tools perform the steps in the following sections:
Complete Prerequisites
Install Zing System Tools
Configure Zing Memory
Complete Prerequisites
Prior to installing ZST, complete the following steps to ensure your system can support ZST.
Note
ZST must be installed prior to installing ZVM.
1. Ensure your environment meets the system requirements.
See Zing System Requirements, for a complete list of Zing requirements, including: supported
CPUs, memory requirements, and operating system versions.
2. Prepare the kernel and DKMS package.
Note
Ensure that your kernel header versions match your DKMS versions. Install the
kernel version matching the kernel you are running.
a. Backup your system kernels.
b. Locate and install the kernel headers and the DKMS package for the kernel version that will
be running the ZST. Run the commands as root or use sudo.
System
Commands to Install Packages
RHEL or CentOS
# sudo yum install dkms kernel-devel-$(uname r)
SLES
# sudo zypper install dkms kernel-defaultdevel-$(uname -r)
Ubuntu
# sudo apt-get install linux-headers-$(uname r) dkms
Azul Systems
Confidential and Proprietary
91
Zing Installation and Administration Guide
3. Check that the kernel headers match the kernel the installed OS kernel.
System
Commands to Install Kernel Headers and DKMS
RHEL or CentOS
# rpm -qa|grep kernel-devel-$(uname -r)
SLES
# rpm -qa|grep kernel-default-devel-$(uname -r)
Ubuntu or Debian
# dpkg -l|grep linux-headers-$(uname -r)
4. Ensure the required packages are installed.
System
RHEL, CentOS,
supported Oracle Linux
6
Commands to Install Packages
# yum localinstall --nogpgcheck
libssl<version>binutils lsof curl
RHEL 5.x, CentOS 5.x, # yum localinstall --nogpgcheck
supported Oracle Linux libssl<version> binutils lsof curl
5
# yum groupinstall 'development Tools'
RHEL 7
# yum localinstall --nogpgcheck
libssl<version> binutils lsof curl
# yum install perl-Sys-Syslog.x86_64
SLES
# zypper install libssl<version> binutils
lsof curl
Ubuntu
# apt-get install libssl<version> binutils
lsof curl
Depending upon the operating system the required packages include:
l
OpenSSL 0.9.8 or later
l
Development tools – Download the package from the operating system website.
l
binutils – Are required to ensure that any core files, generated when a ZVM
instance fails abnormally, include the shared libraries that were in use at the time.
4. If your systems are using power efficiency chips, such as the Nehalem chip set, disable the Intel
idle driver. See Disabling the Intel Idle Driver.
5. To ensure that OS settings are appropriate for performance sensitive applications, apply the
following configuration settings.
Azul Systems
Confidential and Proprietary
92
Specific Use-Cases: Zing Installation
l
SWAP – Turn this off, either by removing swap or by setting swappiness to 0. Best
practice removes swap.
l
vm.min_free – Set this to 1 GB or more. Best practice assigns more than 1 GB.
l
Transparent Huge Pages (THP) – Turn this off.
See Appendix A - Zing Installation Prerequisites Details for information on performing these tasks.
Install Zing System Tools
The ZST installation package contains a kernel loadable module and supporting scripts and tools.
To install ZST:
1. Ensure your system is configured to support ZST, complete the steps in Complete Prerequisites.
Note
Ensure that your kernel header versions match your DKMS versions. Install the
kernel version matching the kernel you are running.
2. Login to your system as root administrator or use sudo.
3. Download ZST installation package to your system <zst_install> directory.
The zing-zst package contains the kernel module and various tools. Only one version of
zing-zst may be installed at a time.
The ZST installation package is specific to the operating system version, service pack, and/or
kernel on the installation machine.
Note
If you are installing on standard RHEL 5, RHEL 6, or RHEL 7 kernels, use the
standard ZST installation packages.
System
ZST Installation Package Format
zing-zst-<dkms_package>.<kernel_version>.<zst_
version>.<platform>.rpm
RHEL
For example:
zing-zst-6a.dkms.3.10.0-5.7.7.0.10.el7.x86_64.rp
zing-zst-<dkms_package>.<kernel_version>.<zst_
version>.<platform>.rpm
SLES
For example:
zing-zst-6a.dkms.2.6.32-5.7.7.0.10.sles11.86_64.rpm
Ubuntu
or
Azul Systems
zing-zst_<dkms_package>.<kernel_version>-server-<zst_
version>_<platform>.deb
Confidential and Proprietary
93
Zing Installation and Administration Guide
System
Debian
ZST Installation Package Format
For example:
zing-zst_6a.dkms.3.2.0.x86.64-server-5.7.7.0.10_
amd64.deb
Where –
<kernel_version> is 5e.2.6.32 or 5c-3.2.0
<zst_number> is 5.7.2.0.10
<platform> is el6.x86_64, sles11.sp2.86_64, or amd64
4. Install ZST using the appropriate command for your system, type:
System
Commands to Install Packages
# rpm -ivh zing-zst-<zst-api-version>.dkms-<zst_
version>.<platform>.rpm
RHEL
For example
# sudo rpm -ivh zing-zst-6a.dkms.2.6.325.7.7.0.10..el6.x86_64.rpm
# zypper install <zst-api-version>.dkms-<zst_
version>.<platform>.rpm
For example
SLES
# sudo zypper install zing-zst-6a.dkms.2.6.325.7.7.0.10.sles11.sp2.86_64.rpm
# dpkg -i zing-zst_<zst-api-version>.dkms-<zst_
version>.<platform>.deb
Ubuntu or Debian
For example
# sudo dpkg -i zing-zst_6a.dkms.3.2.0.x86.64server-5.7.7.0.10_amd64.deb
The system responds.
Preparing...
########################################### [100%]
1:zing-zst
########################################### [100%]
Creating symlink /var/lib/dkms/zing_mm/5.7.7.0-12/source ->
/usr/src/zing_mm-5.7.7.0-12
DKMS: add completed.
Kernel preparation unnecessary for this kernel.
Skipping...
Building module:
cleaning build area...
Azul Systems
Confidential and Proprietary
94
Specific Use-Cases: Zing Installation
make KERNELRELEASE=2.6.32-71.el6.x86_64
KERNELDIR=/lib/modules/2.6.32-71.el6.x86_64/build DKMS=1 INTERNAL_
BUILD_NAME=zst-5.7.7.0 BUILD_SUFFIX= BUILD_NUMBER=12....
cleaning build area...
DKMS: build completed.
zing_mm:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/2.6.32-71.el6.x86_64/extra/
Adding any weak-modules
depmod...
DKMS: install completed.
You must configure Zing memory.
memory' now is recommended.
Running 'system-config-zing-
5. Add /usr/sbin to any Zing executable path. For example:
To your appropriate profile file, add:
export PATH=/usr/sbin:$PATH
6. Configure Zing memory allocation. See Configure Zing Memory.
Configure Zing Memory
After ZST is installed you must configure Zing memory allocation. Zing requires a portion of host
system memory to be set aside and managed independently from Linux. The system-configzing-memory wizard assists with this.
The system-config-zing-memory creates or replaces any existing version of the
pmem.conf file.
Note:
Running the system-config-zing-memory wizard is required for first time
installations.
This section describes creating a default configuration. To customize your Zing memory
configuration, see Modifying Zing Memory Configuration.
1. Complete the steps in Install Zing System Tools.
2. Configure Zing memory management using the system-config-zing-memory command.
Zing requires a portion of host system memory to be set aside and managed independently from
Linux. Use the system-config-zing-memory wizard for initial configuration.
From the command line, type:
# system-config-zing-memory
The system responds:
Azul Systems
Confidential and Proprietary
95
Zing Installation and Administration Guide
Welcome to the Zing memory configuration wizard by Azul Systems.
This wizard sets up the Zing memory configuration file
/etc/zing/pmem.conf.0 and initializes Zing memory. (Read the man
pages for zing, zing-zst, and zing-pmem-conf for more information
about this configuration file.)
By default, Azul recommends setting up your system with 25% Linux
memory and 75% Zing memory. Choose yes to accept this default, or no
to enter the wizard expert flow.
** accept default configuration
** (y)es or (n)o [default 'y']: Y
3. Accept default configuration settings. At the prompt, type:y.
** accept default configuration
** (y)es or (n)o [default 'y']:
Y
The system response lists the zing services that are started or restarted and memory
configuration information.
For example:
zing-memory: INFO: Restarting...
zing-memory: INFO: Stopping...
zing-memory: INFO: stop successful
zing-memory: INFO: Starting...
Info: azulPmemPages: 5840.
INFO: az_pmem_reserve_pages (num2mPages 5840) succeeded
INFO: az_pmem_fund_transfer (to 7, from 0, bytes 12247367680)
succeeded
INFO: az_pmem_fund_transfer (to 1, from 7, bytes 612368384)
succeeded
INFO: az_pmem_fund_transfer (to 3, from 7, bytes 612368384)
succeeded
INFO: az_pmem_fund_transfer (to 0, from 7, bytes 11022630912)
succeeded
Info: You can now run Java processes up to -Xmx10512m or -Xmx10g.
Info: Azul pmem initialized successfully.
zing-memory: INFO: start successful
zing-memory: INFO: restart successful.
You have completed ZST installation!
Zing installation is complete when you install the required Zing license file and ZVM. Modifying Zing
memory usage is available if you do not want to use the default settings.
Azul Systems
Confidential and Proprietary
96
Specific Use-Cases: Zing Installation
Completing Zing Installation
After you complete ZST installation, complete the following tasks to enable ZVM instances to run your
Java applications.
Install Zing License File
Install Zing Virtual Machine
Verify Zing Installation
Install Zing License File
After ZST is installed, install your Zing licensing.
A Zing license is required to start a Java application using a ZVM instance. Zing licenses are issued by
Azul. There are different Zing licensing options for getting the Zing license installed and ready for ZVM.
The option you use depends upon your contract with Azul.
l
Zing license (ZVM Host) – Standalone Zing licenses are installed on ZVM host machines.
These licenses are typically applied one license per ZVM host. All Zing installations require
the Zing license file. The Zing license daemon is installed with ZST.
See Installing Zing Standalone Licenses.
l
Zing Pool Server license – A Zing Pool server license allows you to use the Zing license
server to run an allocated number of Zing licenses concurrently among a set of ZVM hosts
that are configured to use Pool licenses.
See Installing and Using Pool Licensing .
Install Zing Virtual Machine
After ZST is installed and the Zing licensing is applied, install ZVM. See Installing the Zing Virtual
Machine for complete instructions.
Verify Zing Installation
After ZST is installed, configured, and started, and ZVM is installed, verify the completed installation.
1. Verify that your Zing license is installed.
Check /etc/zing/directory for your licensing authorization file, license.
l
If you are running ZVM instances on a standalone host, see Installing Zing
Standalone Licenses.
l
If you are configuring your ZVM host as a member of a Zing licensing pool, see
Installing and Using Pool Licensing .
2. Run a simple Java command, for example type:
Azul Systems
Confidential and Proprietary
97
Zing Installation and Administration Guide
$ <zing_java_directory>/bin/java -version
where <zing_java_directory> is the directory where the Zing /bin/java is
located. If you accept the ZVM installation defaults, this path is, for example:
$ /opt/zing/zing-<version>/bin/java -version
For quicker access, use the fully qualified path.
3. Look for the ZVM component reference in the system response, for example:
/opt/zing/zing-jdk1.8.0-14.09.0.0-33-x86_64/bin/java -version
java version "1.8.0-zing_14.09.0.0"
Zing Runtime Environment for Java Applications (build 1.8.0-zing_
14.09.0.0-b20)
Zing 64-Bit Tiered VM (build 1.8.0-zing_14.09.0.0-b33-productazlinuxM-X86_64, mixed mode)
See Troubleshooting Zing or run man zing-installation to view common
installation errors.
Modifying Zing Memory Configuration
If you completed all the steps in Installing and Configuring Zing System Tools Zing memory is
configured with default settings. Default configuration sets aside 75% of the physical memory of your
machine for use by ZVMs.
To customize the memory allocation, run the system-config-zing-memory command and do
not accept the default. Running this command automatically restarts the system.
When modifying the memory allocations for ZST, consider the following memory calculations:
/proc/meminfo.MemTotal = Linux Memory + Zing Memory
Zing Memory = Reservable + Contingency + Pause Prevention
Where –
l
Zing Memory – The ZVM uses Zing Memory to allocate the memory required for the Java
heap.
l
Reserved Memory – The amount of Reservable memory initially allocated to a specific ZVM
instance when the instance starts.
l
Contingency Memory – A backup reserve that Zing uses to temporarily allocate additional
memory to any ZVM instance running a Java application.
l
Pause Prevention Memory – A percentage of the Zing Memory set aside as part of Zing's
Generational Pauseless Garbage Collection (GPGC).
See Understanding Zing Memory Use for additional information about memory use. The following
shows an overview.
Azul Systems
Confidential and Proprietary
98
Specific Use-Cases: Zing Installation
To modifying Zing memory allocation complete the steps in the listed sections:
1. Start the process.
Modify Zing Memory Allocation
2. Choose one of the following two allocation methods:
l
Use System Method to Allocate Zing Memory
If you use the System method, choose one of the following options:
l
l
Use Percentage of System Memory for Zing Memory
l
Use Exact Amount of System Memory for Zing Memory
Use Java Method to Allocate Zing Memory
3. Complete the process.
Modify Zing Memory Allocation
To modify Zing memory configuration using the wizard:
Note
The system-config-zing-memory creates or replaces any existing version of the
pmem.conf file.
1. Login to your RHEL system as root.
2. Ensure there are no running processes using Zing memory, type:
$ zing-ps
See “Using the zing-ps Tool” in Using Zing System Tools.
3. Run the Zing memory configuration wizard.
$ system-config-zing-memory
Azul Systems
Confidential and Proprietary
99
Zing Installation and Administration Guide
The system responds.
Welcome to the Zing memory configuration wizard by Azul Systems.
This wizard sets up the memory driver configuration file
/etc/zing/pmem.conf.0
and initializes the Zing memory. (Read the man pages for zing, zingzst, and zing-pmem-conf for more information about this
configuration file.)
By default, Azul recommends setting up your system with 25% Linux
memory and 75% Zing memory. Choose yes to accept this default, or no
to enter the wizard expert flow.
4. Do not accept the default configuration, type: N
** accept default configuration
** (y)es or (n)o [default 'y']: n
The system responds.
[ Current task: Sizing Zing memory / Choosing sizing method ]
Zing requires you to partition memory up front for exclusive use by
Java. (This is analogous to Linux hugepages, although Zing pages are
not interchangeable with hugepages.)
Would you like to decide how much memory to allocate to Zing based
on the total size of your system, or based on the total size of the
Java instances you expect to run?
** Enter (s)ystem or (j)ava heap size [default 's']:
5. Select a method for allocating Zing memory:
l
System – system sizing allocates memory as a percentage of the available host
system memory. The default value if 75% for Zing Fund memory.
To use this method, continue to Use System Method to Allocate Zing Memory.
l
Java – Java sizing allocates memory as a fixed value in MB based on the estimated
sum of all Xmx values that might be run concurrently.
To use this method, continue to Use Java Method to Allocate Zing Memory.
Use System Method to Allocate Zing Memory
To select the System method for allocating Zing Fund memory.
1. From the system-config-zing-memory wizard, specify System method, type: S
** Enter (s)ystem or (j)ava heap size [default 's']: s
The system responds.
[ Current task: Sizing Zing memory / Choosing system sizing method ]
Do you want to allocate a percentage of your total host system
memory to Zing, or would you prefer to specify an exact amount?
** Enter (p)ercentage or (e)xact [default 'p']:
Azul Systems
Confidential and Proprietary
100
Specific Use-Cases: Zing Installation
2. Select a memory allocation method:
l
Percentage – allocates a percentage of the total host system memory.
To use this method, continue to Use Percentage of System Memory for Zing Memory.
l
Exact – allocates a specific value in MB of memory. This is based on the estimated maximum
Xmx values running concurrently.
To use this method, continue to Use Exact Amount of System Memory for Zing Memory.
Use Percentage of System Memory for Zing Memory
To specify the allocated amount as a percentage of the total host system memory.
1. From the system-config-zing-memory wizard, specify the allocation method, type P
** Enter (p)ercentage or (e)xact [default 'p']: p
The system responds with the next configuration prompt.
2. Specify the percentage value to use, type a percentage value. The default is %75.
** Enter percentage of total memory to dedicate to Zing [default
'75']:
The system responds with the next configuration prompt.
3. Continue to Specify Contingency Memory Pool.
Use Exact Amount of System Memory for Zing Memory
To specify allocation amount as an exact value:
1. From the system-config-zing-memory wizard, specify the exact method, type: e
** Enter (p)ercentage or (e)xact [default 'p']: e
The system responds.
[ Current task:
Sizing Zing memory / Choosing specific size ]
Please specify the amount of memory for Zing to use. (Note:
system has a total of 15g.)
examples:
your
16384m, 16g
** Enter memory to dedicate to Zing:
2. Specify the exact MB value to allocate to Zing memory.
For example:
** Enter memory to dedicate to Zing: 12g
The system responds with the next configuration prompt.
3. Continue to Specify Contingency Memory Pool.
Use Java Method to Allocate Zing Memory
To select the Java method for allocating Zing memory.
Azul Systems
Confidential and Proprietary
101
Zing Installation and Administration Guide
1. From the system-config-zing-memory wizard, specify the exact method, type: J
** Enter (p)ercentage or (e)xact [default 'p']: j
The system responds.
[[ Current task: Sizing Zing memory / Deriving size from Xmx ]
What is the sum of the Xmx values of all the Java processes you plan
to run concurrently?
** Enter sum of Xmx:
For example:
** Enter sum of Xmx: 2g
The system responds with the next configuration prompt.
3. Continue to Specify Contingency Memory Pool.
Specify Contingency Memory Pool
To specify the memory allocation for the Zing Contingency memory (also known as elastic memory)
pool:
1. From the system-config-zing-memory wizard, type the desired percentage. For example:
75. See Understanding Zing Memory Use for information about Contingency memory.
[ Current task: Sizing Zing memory / Sizing Contingency memory pool
]
Part of Zing memory is set aside in a common contingency memory pool
for all Azul Java processes to share. The contingency memory pool
exists as an insurance policy to allow a JVM instance to temporarily
avoid OOM behavior and grow beyond -Xmx.
What percentage of Zing memory would you like to dedicate to the
contingency memory pool?
** Enter percentage of Zing memory to be used for contingency memory
[default '5']:
2. Verify Zing Memory is restarted. View the system return from the last step to locate the following
lines. If you do not see them, see Starting or Stopping the Zing System Tools Services.
[ Current task:
driver ]
Saving configuration and initializing Zing memory
zing-memory: INFO: Restarting...
zing-memory: INFO: Stopping...
zing-memory: INFO: stop successful
zing-memory: INFO: Starting...
Info: azulPmemPages: 54454.
INFO: az_pmem_reserve_pages (num2mPages 54454) succeeded
INFO: az_pmem_fund_transfer (to 7, from 0, bytes 114198315008)
succeeded
Azul Systems
Confidential and Proprietary
102
Specific Use-Cases: Zing Installation
INFO: az_pmem_fund_transfer (to 1, from 7, bytes 5708447744)
succeeded
INFO: az_pmem_fund_transfer (to 3, from 7, bytes 5708447744)
succeeded
INFO: az_pmem_fund_transfer (to 0, from 7, bytes 102781419520)
succeeded
Info: You can now run Java processes up to -Xmx98020m or -Xmx95g.
Info: Azul pmem initialized successfully.
zing-memory: INFO: start successful
zing-memory: INFO: restart successful
Installing and Configuring Zing Using Read-Only Images
This section describes how to install and configure Zing using read-only images of ZST and ZVM for
use in an Enterprise File System (EFS). See openefs.org for additional information on EFS.
Note:
l
ZST with a read-only image is supported on RHEL/CentOS 5.2 or later, and
RHEL/CentOS 6.x. systems only.
Installation – The ZST installation package contains a kernel loadable module and supporting
scripts and tools. This includes:
l
l
Zing Memory
l
Zing license daemon
l
Zing tools, including zing-ps
Configuration –
l
Zing requires that a portion of host system memory is set aside and managed independently
from Linux. This is configured using the system-config-zing-memory command.
l
Usage –
l
To launch ZVM instances, the Zing Memory must be configured and running, and the
license daemon must be running.
Installing Zing with Read Only Images Overview
ZST includes scripts to install, start, and manage ZST on multiple nodes. The basic process includes:
l
Complete Prerequisites on the Zing server and all the ZST nodes.
l
Install Zing System Tools on the Zing server that is read only accessible by the ZST nodes.
l
Prepare the Zing System Tools Server for multiple node access.
l
Install Zing System Tools on the ZST Nodes.
l
Configure Zing Memory on the ZST Nodes.
Azul Systems
Confidential and Proprietary
103
Zing Installation and Administration Guide
l
Install Zing License File on the ZST Nodes.
l
Install Zing Virtual Machine on the ZST Nodes.
l
Verify Zing Installation on the ZST Nodes.
Complete Prerequisites
Prior to installing ZST, complete the following steps to ensure your system can support ZST.
Note
ZST must be installed prior to installing ZVM.
1. Ensure your environment meets the system requirements.
See Zing System Requirements, for a complete list of Zing requirements, including:
supported CPUs, memory requirements, operating system versions, supported
browsers, and listening ports.
2. Login to your system as root administrator.
3. Ensure the following packages are installed.
System
RHEL, CentOS,
supported Oracle Linux
6
Commands to Install Packages
# yum localinstall --nogpgcheck
libssl<version>binutils lsof curl
RHEL 5.x, CentOS 5.x, # yum localinstall --nogpgcheck
supported Oracle Linux libssl<version> binutils lsof curl
5
# yum groupinstall 'development Tools'
RHEL 7
# yum localinstall --nogpgcheck
libssl<version> binutils lsof curl
# yum install perl-Sys-Syslog.x86_64
SLES
# zypper install libssl<version> binutils
lsof curl
Ubuntu
# apt-get install libssl<version> binutils
lsof curl
Depending upon the operating system the required packages include:
Azul Systems
l
OpenSSL 0.9.8 or later
l
Development tools – Download the package from the operating system website.
Confidential and Proprietary
104
Specific Use-Cases: Zing Installation
l
binutils – Are required to ensure that any core files, generated when a ZVM
instance fails abnormally, include the shared libraries that were in use at the time.
4. If your systems are using power efficiency chips, such as the Nehalem chip set, disable the Intel
idle driver. See Disabling the Intel Idle Driver.
5. To ensure that OS settings are appropriate for performance sensitive applications, apply the
following configuration settings
l
SWAP – Turn this off, either by removing swap or by setting swappiness to 0.
Best practice removes swap.
l
vm.min_free – Set this to 1 GB or more. Best practice assigns more than 1 GB.
l
Transparent Huge Pages (THP) – Turn this off.
See for information on performing these tasks.
Install Zing System Tools on the Zing Server
Complete a standard full ZST installation on the Zing server. The ZST nodes access this server as
source for ZST installation and configuration.
The ZST installation package contains a kernel loadable module and supporting scripts and tools.
To install ZST:
1. Ensure your system is configured to support ZST, complete the steps in Prerequisites Prior to
Installing Zing System Tools.
2. Login to your system as root administrator or use sudo.
3. Download ZST installation package to your system <zst_install> directory.
zing-zst-<kernel_version>.<zst_version>.<platform>.rpm
For example:
zing-zst-5e.2.6.32-5.7.2.0.10.el6.x86_64.rpm
Where –
<kernel_version> is 5e.2.6.32
<zst_number> is 5.7.2.0.10
<platform> is el6.x86_64
The zing-zst package contains the kernel module and various tools. Only one version of
zing-zst may be installed at a time.
The ZST installation package is specific to the operating system version, service pack, and/or
kernel on the installation machine.
4. Install ZST using the following command, type:
# rpm --prefix=<zing_server_dir> -ivh <download_dir>/zing-zst<kernel_version>.<zst_version>.<platform>.rpm
Where –
Azul Systems
Confidential and Proprietary
105
Zing Installation and Administration Guide
<zing_server_dir> is the path that all ZST nodes use to access the Zing server installation.
Using the --prefix option creates the specified directory, in this example, /azul/zing_
server/zst.
<download_dir> is the location on the server where you downloaded the ZST package. In
this example, /azul.
For example:
# rpm --prefix=/azul/zing_server/zst -ivh /azul/zing-zst-5e.2.6.185.7.2.0.10.el5.x86_64.rpm
The system responds.
Preparing... ########################################### [100%]
1:zing-zst
########################################### [100%]
Zing ZST is installed at /azul/zing_server/zst
Run /azul/zing_server/zst/scripts/zing-zst.startup to start Zing
services
5. Add /usr/sbin to any Zing executable path. For example:
To your appropriate profile file, add:
export PATH=/usr/sbin:$PATH
Prepare the Zing System Tools Server
The steps in this section are typically performed by a network administrator.
The ZST read-only directory must be available to all ZST nodes.
1. On the Zing server, export the directory, /azul/zing_server/zst, to make it read-only visible to all
other ZST nodes.
For example:
# cat /etc/exports
The system responds:
/azul/zing_server/zst *(ro,fsid=0,sync)
2. Restart the network file system to apply the export.
For example:
# service nfs restart
The system responds.
Shutting down NFS mountd:
[ OK ]
Shutting down NFS daemon:
[ OK ]
Shutting down NFS quotas:
[ OK ]
Shutting down NFS services:
[ OK ]
Starting NFS services:
[ OK ]
Starting NFS quotas:
[ OK ]
Starting NFS daemon:
[ OK ]
Azul Systems
Confidential and Proprietary
106
Specific Use-Cases: Zing Installation
Starting NFS mountd:
[ OK ]
Install Zing System Tools on the ZST Nodes
Note
Install ZST on the Zing server before you install ZST on any ZST node.
The ZST installation package contains a kernel loadable module and supporting scripts and tools. This
includes:
l
Zing Memory
l
Zing license daemon
l
Zing tools, including zing-ps
To install ZST:
1. On each ZST node, login to your system as root administrator.
2. Create a ZST directory.
# mkdir /azul/zing_node/zst
3. Mount the Zing server read-only directory to a local ZST node directory.
# mount <ZST_Server_IP>:<zing_server_dir> <zst_node_dir>
For example:
# mount 10.10.163.113:/azul/zing_server/zst /azul/zing_node/zst
This command creates the local ZST node directories, as needed.
4. Change to the /script directory.
# cd /azul/zing_node/zst/scripts/
5. Install ZST using the zing-zst.startup command.
# ./zing-zst.startup
The system responds:
Installing ZST ...
Checking ZST package dependencies...
Dependency check complete...
zing-licensed: INFO: start successful [ OK ]
zing-memory: INFO: Starting...
Warning: Zing Virtual Machine support is disabled.
Warning: Run system-config-zing-memory to enable.
Warning: Try 'man zing-zst' for more information.
Warning: Azul pmem configuration disabled.
zing-memory: INFO: start successful
Installation complete!
Azul Systems
Confidential and Proprietary
107
Zing Installation and Administration Guide
You must configure Zing memory. Running '/azul/zing_
node/zst/usr/sbin/system-config-zing-memory' now is recommended.
The zing-zst.startup command has the following options:
Option
Description
--h
Displays help text.
-p, --pmem <pmem_filename>
References a pmem.conf file that
contains ZST memory configuration
information.
-l, --license <license_filename>
References a Zing license file.
For example, including the optional -l and -p options:
# /zing/57210/scripts/zing-zst.startup -l /tmp/license -p
/tmp/pmem.conf
Installing ZST ...
Checking ZST package dependencies...
Dependency check complete...
Installed: '/tmp/license'
Installed: '/tmp/pmem.conf'
zing-licensed: INFO: start successful
[
OK
]
zing-memory: INFO: Starting...
Info:
azulPmemPages: 5888.
INFO: az_pmem_reserve_pages (num2mPages 5888) succeeded
INFO: az_pmem_fund_transfer (to 7, from 0, bytes 12348030976)
succeeded
INFO: az_pmem_fund_transfer (to 1, from 7, bytes 616562688)
succeeded
INFO: az_pmem_fund_transfer (to 3, from 7, bytes 616562688)
succeeded
INFO: az_pmem_fund_transfer (to 0, from 7, bytes 11114905600)
succeeded
Info:
You can now run Java processes up to -Xmx10600m or -Xmx10g.
Info:
Azul pmem initialized successfully.
zing-memory: INFO: start successful
Installation complete!
In this example, the comment to run system-config-zing-memory is not included because
the memory configuration file was provided with the zing-zst.startup command.
Azul Systems
Confidential and Proprietary
108
Specific Use-Cases: Zing Installation
6. Configure Zing memory allocation. See Configure Zing Memory.
7. Optionally, enable zing-core-pattern for automatic ZVM core bundling. See Troubleshooting
Zing.
8. Repeat the Step 1 through Step 7 for each ZST Node.
Configure Zing Memory on the ZST Nodes
After ZST is installed you must configure Zing memory allocation. Zing requires a portion of host
system memory to be set aside and managed independently from Linux. The system-configzing-memory wizard assists with this.
The system-config-zing-memory creates or replaces any existing version of the
pmem.conf file.
Note:
Running the system-config-zing-memory wizard is required for first time
installations.
This section describes creating a default configuration. To customize your Zing memory
configuration, see Modifying Zing Memory Configuration.
1. Complete the steps in Install Zing System Tools on the ZST Nodes.
2. Configure Zing memory management using the system-config-zing-memory command.
Zing requires a portion of host system memory to be set aside and managed independently from
Linux. Use the system-config-zing-memory wizard for initial configuration.
From the command line, type:
# /azul/zing_node/zst/usr/sbin/system-config-zing-memory
The system responds:
Welcome to the Zing memory configuration wizard by Azul Systems.
This wizard sets up the Zing memory configuration file
/etc/zing/pmem.conf.0 and initializes Zing memory. (Read the man
pages for zing, zing-zst, and zing-pmem-conf for more information
about this configuration file.)
By default, Azul recommends setting up your system with 25% Linux
memory and 75% Zing memory. Choose yes to accept this default, or no
to enter the wizard expert flow.
** accept default configuration
** (y)es or (n)o [default 'y']: Y
3. Accept default configuration settings. At the prompt, type: y.
** accept default configuration
** (y)es or (n)o [default 'y']:
Azul Systems
Y
Confidential and Proprietary
109
Zing Installation and Administration Guide
The system response lists the zing services that are started or restarted and memory
configuration information.
For example:
zing-memory: INFO: Restarting...
zing-memory: INFO: Stopping...
zing-memory: INFO: stop successful
zing-memory: INFO: Starting...
Info: azulPmemPages: 54454.
INFO: az_pmem_reserve_pages (num2mPages 54454) succeeded
INFO: az_pmem_fund_transfer (to 7, from 0, bytes 114198315008)
succeeded
INFO: az_pmem_fund_transfer (to 1, from 7, bytes 5708447744)
succeeded
INFO: az_pmem_fund_transfer (to 3, from 7, bytes 5708447744)
succeeded
INFO: az_pmem_fund_transfer (to 0, from 7, bytes 102781419520)
succeeded
Info: You can now run Java processes up to -Xmx98020m or -Xmx95g.
Info: Azul pmem initialized successfully.
zing-memory: INFO: start successful
zing-memory: INFO: restart successful.
You have completed ZST installation!
Install Zing License File on the ZST Nodes
After ZST is installed, install your Zing license.
A Zing license is required to start a Java application using a ZVM instance. Zing licenses are issued by
Azul. There are different Zing licensing options for getting the Zing license installed and ready. The
option you use depends upon your contract with Azul.
l
Zing license (ZVM Host) – Standalone Zing licenses are installed on ZVM host machines.
These licenses are typically applied one license per ZVM host. All Zing installations require the
Zing license file. The Zing license daemon is installed with ZST. See Installing Zing
Standalone Licenses.
l
Zing Pool Server license – A Zing Pool server license allows you to use the Zing license server
to run an allocated number of Zing licenses concurrently among a set of ZVM hosts that are
configured to use Pool licenses. See Installing and Using Pool Licensing.
Azul Systems
Confidential and Proprietary
110
Specific Use-Cases: Zing Installation
Install Zing Virtual Machine on the ZST Nodes
After ZST is installed and the Zing licensing is applied, install ZVM. See the Zing Virtual Machine
Installation guide for complete instructions. The following is an overview of the ZVM installation when
you are using ZST with a read-only image.
1. On the Zing server, download ZVM.
The installer package uses the following naming format depending on the Java version
supported: zing-jdk<version>-<zvm_version>.<platform>.rpm
For example:
zing-jdk1.6.0-5.8.0.0-13.x86_64.rpm
zing-jdk1.7.0-5.8.0.0-13.x86_64.rpm
Where –
JDK <version> is 1.6.0 or 1.7.0
ZVM <version> is 5.8.0.0-13
platform <version> is x86_64
2. Login as root.
a. Enter the root shell.
b. Type the su command.
c. Enter the super-user password when prompted.
3. Install ZVM.
# rpm -ivh --prefix=<zing_server_dir> -ivh <download_dir>/zingjdk<version>-<zvm_version>.<platform>.rpm
Where –
<zing_server_dir> is the path that all ZVM nodes use to access the Zing server
installation. Using the --prefix option creates the specified directory, in this example,
/azul/zing_server/zvm.
<download_dir> is the location on the server where you downloaded the ZST package. In
this example, /azul.
For example:
# rpm -ivh --prefix=/azul/zing_server/zvm
5.8.0.0-13.x86_64.rpm
/azul/zing-jdk1.6.0-
Preparing...
########################################### [100%]
1:zing-jdk1.6.0
########################################### [100%]
4. Export the read-only directory, /azul/zing_server/zvm.
# more /etc/exports
The system responds:
/azul/zing_server/zvm
*(ro,fsid=0,sync)
5. Restart NFS.
Azul Systems
Confidential and Proprietary
111
Zing Installation and Administration Guide
# service nfs restart
Shutting down NFS daemon:
[
OK
]
Shutting down NFS mountd:
[
OK
]
Shutting down NFS quotas:
[
OK
]
Shutting down NFS services:
[
OK
]
Starting NFS services:
[
OK
]
Starting NFS quotas:
[
OK
]
Starting NFS mountd:
[
OK
]
Stopping RPC idmapd:
[
OK
]
Starting RPC idmapd:
[
OK
]
Starting NFS daemon:
[
OK
]
6. On each ZST node, mount the ZVM installed directory.
For example:
# mkdir /azul/zing_node/zvm
# mount 10.10.163.192:/azul/zing_server/zvm
/azul/zing_node/zvm
Verify Zing Installation on the ZST Nodes
After ZST is installed, configured, and started, and ZVM is installed, verify the completed installation.
1. Verify that your Zing license is installed.
Check /etc/zing/directory for your licensing authorization file, license.
l
If you are running ZVM instances on a standalone host, see Installing Zing Standalone
Licenses.
l
If you are configuring your ZVM host as a member of a Zing licensing pool, see Installing
and Using Pool Licensing .
2. Run a simple Java command, for example type:
$ <zing_java_directory>/bin/java -version
where <zing_java_directory> is the directory where the Zing /bin/java is
located. If you accept the ZVM installation defaults, this path is, for example:
$ /opt/zing/zing-<version>/bin/java -version
For quicker access, use the fully qualified path.
3. Look for the ZVM component reference in the system response, for example:
java version "1.8.0-zing_14.09.0.0"
Zing Runtime Environment for Java Applications (build 1.8.0-zing_
14.09.0.0-b20)
Azul Systems
Confidential and Proprietary
112
Specific Use-Cases: Zing Installation
Zing 64-Bit Tiered VM (build 1.8.0-zing_14.09.0.0-b33-productazlinuxM-X86_64, mixed mode)
See Troubleshooting Zing or run man zing-installation to view common
installation errors.
Modifying Zing Memory Configuration
If you completed all the steps in Installing and Configuring Zing System Tools Zing memory is
configured with default settings. Default configuration sets aside 75% of the physical memory of your
machine for use by ZVMs.
To customize the memory allocation, run the system-config-zing-memory command and do
not accept the default. Running this command automatically restarts the system.
When modifying the memory allocations for ZST, consider the following memory calculations:
/proc/meminfo.MemTotal = Linux Memory + Zing Memory
Zing Memory = Reservable + Contingency + Pause Prevention
Where –
l
Zing Memory – The ZVM uses Zing Memory to allocate the memory required for the Java
heap.
l
Reserved Memory – The amount of Reservable memory initially allocated to a specific ZVM
instance when the instance starts.
l
Contingency Memory – A backup reserve that Zing uses to temporarily allocate additional
memory to any ZVM instance running a Java application.
l
Pause Prevention Memory – A percentage of the Zing Memory set aside as part of Zing's
Generational Pauseless Garbage Collection (GPGC).
See Understanding Zing Memory Use for additional information about memory use. The following
shows an overview.
Azul Systems
Confidential and Proprietary
113
Zing Installation and Administration Guide
To modifying Zing memory allocation complete the steps in the listed sections:
1. Start the process.
Modify Zing Memory Allocation
2. Choose one of the following two allocation methods:
l
Use System Method to Allocate Zing Memory
If you use the System method, choose one of the following options:
l
l
Use Percentage of System Memory for Zing Memory
l
Use Exact Amount of System Memory for Zing Memory
Use Java Method to Allocate Zing Memory
3. Complete the process.
Modify Zing Memory Allocation
To modify Zing memory configuration using the wizard:
Note
The system-config-zing-memory creates or replaces any existing version of the
pmem.conf file.
1. Login to your RHEL system as root.
2. Ensure there are no running processes using Zing memory, type:
$ zing-ps
See “Using the zing-ps Tool” in Using Zing System Tools.
3. Run the Zing memory configuration wizard.
$ system-config-zing-memory
Azul Systems
Confidential and Proprietary
114
Specific Use-Cases: Zing Installation
The system responds.
Welcome to the Zing memory configuration wizard by Azul Systems.
This wizard sets up the memory driver configuration file
/etc/zing/pmem.conf.0
and initializes the Zing memory. (Read the man pages for zing, zingzst, and zing-pmem-conf for more information about this
configuration file.)
By default, Azul recommends setting up your system with 25% Linux
memory and 75% Zing memory. Choose yes to accept this default, or no
to enter the wizard expert flow.
4. Do not accept the default configuration, type: N
** accept default configuration
** (y)es or (n)o [default 'y']: n
The system responds.
[ Current task: Sizing Zing memory / Choosing sizing method ]
Zing requires you to partition memory up front for exclusive use by
Java. (This is analogous to Linux hugepages, although Zing pages are
not interchangeable with hugepages.)
Would you like to decide how much memory to allocate to Zing based
on the total size of your system, or based on the total size of the
Java instances you expect to run?
** Enter (s)ystem or (j)ava heap size [default 's']:
5. Select a method for allocating Zing memory:
l
System – system sizing allocates memory as a percentage of the available host
system memory. The default value if 75% for Zing Fund memory.
To use this method, continue to Use System Method to Allocate Zing Memory.
l
Java – Java sizing allocates memory as a fixed value in MB based on the estimated
sum of all Xmx values that might be run concurrently.
To use this method, continue to Use Java Method to Allocate Zing Memory.
Use System Method to Allocate Zing Memory
To select the System method for allocating Zing Fund memory.
1. From the system-config-zing-memory wizard, specify System method, type: S
** Enter (s)ystem or (j)ava heap size [default 's']: s
The system responds.
[ Current task: Sizing Zing memory / Choosing system sizing method ]
Do you want to allocate a percentage of your total host system
memory to Zing, or would you prefer to specify an exact amount?
** Enter (p)ercentage or (e)xact [default 'p']:
Azul Systems
Confidential and Proprietary
115
Zing Installation and Administration Guide
2. Select a memory allocation method:
l
Percentage – allocates a percentage of the total host system memory.
To use this method, continue to Use Percentage of System Memory for Zing Memory.
l
Exact – allocates a specific value in MB of memory. This is based on the estimated maximum
Xmx values running concurrently.
To use this method, continue to Use Exact Amount of System Memory for Zing Memory.
Use Percentage of System Memory for Zing Memory
To specify the allocated amount as a percentage of the total host system memory.
1. From the system-config-zing-memory wizard, specify the allocation method, type P
** Enter (p)ercentage or (e)xact [default 'p']: p
The system responds with the next configuration prompt.
2. Specify the percentage value to use, type a percentage value. The default is %75.
** Enter percentage of total memory to dedicate to Zing [default
'75']:
The system responds with the next configuration prompt.
3. Continue to Specify Contingency Memory Pool.
Use Exact Amount of System Memory for Zing Memory
To specify allocation amount as an exact value:
1. From the system-config-zing-memory wizard, specify the exact method, type: e
** Enter (p)ercentage or (e)xact [default 'p']: e
The system responds.
[ Current task:
Sizing Zing memory / Choosing specific size ]
Please specify the amount of memory for Zing to use. (Note:
system has a total of 15g.)
examples:
your
16384m, 16g
** Enter memory to dedicate to Zing:
2. Specify the exact MB value to allocate to Zing memory.
For example:
** Enter memory to dedicate to Zing: 12g
The system responds with the next configuration prompt.
3. Continue to Specify Contingency Memory Pool.
Use Java Method to Allocate Zing Memory
To select the Java method for allocating Zing memory.
Azul Systems
Confidential and Proprietary
116
Specific Use-Cases: Zing Installation
1. From the system-config-zing-memory wizard, specify the exact method, type: J
** Enter (p)ercentage or (e)xact [default 'p']: j
The system responds.
[[ Current task: Sizing Zing memory / Deriving size from Xmx ]
What is the sum of the Xmx values of all the Java processes you plan
to run concurrently?
** Enter sum of Xmx:
For example:
** Enter sum of Xmx: 2g
The system responds with the next configuration prompt.
3. Continue to Specify Contingency Memory Pool.
Specify Contingency Memory Pool
To specify the memory allocation for the Zing Contingency memory (also known as elastic memory)
pool:
1. From the system-config-zing-memory wizard, type the desired percentage. For example:
75. See Understanding Zing Memory Use for information about Contingency memory.
[ Current task: Sizing Zing memory / Sizing Contingency memory pool
]
Part of Zing memory is set aside in a common contingency memory pool
for all Azul Java processes to share. The contingency memory pool
exists as an insurance policy to allow a JVM instance to temporarily
avoid OOM behavior and grow beyond -Xmx.
What percentage of Zing memory would you like to dedicate to the
contingency memory pool?
** Enter percentage of Zing memory to be used for contingency memory
[default '5']:
2. Verify Zing Memory is restarted. View the system return from the last step to locate the following
lines. If you do not see them, see Starting or Stopping the Zing System Tools Services.
[ Current task:
driver ]
Saving configuration and initializing Zing memory
zing-memory: INFO: Restarting...
zing-memory: INFO: Stopping...
zing-memory: INFO: stop successful
zing-memory: INFO: Starting...
Info: azulPmemPages: 54454.
INFO: az_pmem_reserve_pages (num2mPages 54454) succeeded
INFO: az_pmem_fund_transfer (to 7, from 0, bytes 114198315008)
succeeded
Azul Systems
Confidential and Proprietary
117
Zing Installation and Administration Guide
INFO: az_pmem_fund_transfer (to 1, from 7, bytes 5708447744)
succeeded
INFO: az_pmem_fund_transfer (to 3, from 7, bytes 5708447744)
succeeded
INFO: az_pmem_fund_transfer (to 0, from 7, bytes 102781419520)
succeeded
Info: You can now run Java processes up to -Xmx98020m or -Xmx95g.
Info: Azul pmem initialized successfully.
zing-memory: INFO: start successful
zing-memory: INFO: restart successful
Upgrading Zing System Tools in Read Only Environments
There is no direct upgrade path for ZST using read only ZST images. You must uninstall the existing
ZST read only image and install and configure the new ZST read only image.
Uninstalling the Zing System Tools from a ZST Node
Uninstalling ZST removes the ZST elements from the ZST node.
To uninstall Zing System Tools:
1. Login to your system as root.
2. Shutdown all the Zing java processes.
3. Ensure there are no running processes using Zing memory, type:
$ zing-ps
See Using Zing System Tools.
4. Run the shutdown script, zing-zst.shutdown, and confirm at the prompt.
For example, including the optional -r option.
$ /azul/zing_node/zst/scripts/zing-zst.shutdown -r
The system responds:
Uninstalling ZST from the system...
Do you want to continue? (yes or no) y
Restoring core pattern
Removing zing-licensed
zing-licensed: INFO: Stopping...26109
zing-licensed: INFO: stop successful
Removing zing-memory
zing-httpd: INFO: Stopping...
zing-httpd: INFO: stop successful
zing-certd: INFO: Stopping...
zing-certd: INFO: stop successful
Azul Systems
Confidential and Proprietary
118
Specific Use-Cases: Zing Installation
zing-memory: INFO: Stopping...
zing-memory: INFO: stop successful
Removing directories
Keeping /var/log/zing-httpd, directory is not empty
Removing ZST Kernel Object
Removing symlinks
Removing user and groups created for ZST
Removing /etc/zing
Uninstallation complete!
If you had enabled zing-core-pattern, the zing-zst.shutdown command restores your
original core pattern.
The zing-zst.shutdown command has the following options:
-h
Displays help text.
-r, --remove-etc-zing-folder
Removes the /etc/zing folder.
5. Unmount the Zing server read-only directory from the local ZST node.
# umount -l <zst_node_dir>
For example:
# umount -l /azul/zing_node/zst
This command unmounts the local ZST node directory.
6. Optionally, remove the Zing log file directory.
The /var/log/zing-httpd directory contains log files. If you want to remove those files as
well, remove the directory manually.
Uninstalling the Zing System Tools from the Zing Server
Uninstalling ZST removes all the ZST elements, this includes the following:
l
Zing Memory
l
zing-ps
l
Zing license daemon
Before you can uninstall the ZST component, you must uninstall the ZVM component. Follow
Note: the steps below for the supported sequence.
To uninstall Zing System Tools:
1. Uninstall ZST from all the ZST nodes. See Uninstalling the Zing System Tools from a ZST Node.
2. Login to your Zing server as root.
Azul Systems
Confidential and Proprietary
119
Zing Installation and Administration Guide
3. Ensure there are no running processes using Zing memory, type:
$ zing-ps
See the Using Zing System Tools guide.
4. Uninstall the ZVM component before you uninstall ZST. Run the remove command.
See the Installing the Zing Virtual Machine guide for description of how to uninstall the ZVM software.
$ yum remove zing-jdk<version>-<zvm_version>.<platform>
5. Uninstall ZST from the Zing server. Run the remove command.
$ yum remove zing-zst-<version>
This completes the ZST uninstallation.
6. To uninstall the remaining Zing components:
l
Delete the Zing license file, see the Installing Zing Standalone Licenses guide.
l
Uninstall the Zing Pool license server, see the Installing and Using Pool Licensing guide.
Installing and Using Pool Licensing
This section describes how to install and use the Zing license server to manage Zing licenses for a pool
of ZVM hosts.
Zing Licensing Overview
A Zing license is required to start a Java application using ZVM. Zing licenses are issued by Azul.
There are different Zing licensing options for getting the Zing license installed and ready for ZVM. The
option you use depend upon your contract with Azul.
A Pool license allows you to use the Zing license server to lease an allocated number of Zing licenses
concurrently among a set of ZVM hosts that are configured to work with a Pool license server.
Pool Licensing Components
The Pool licensing method requires the following licensing components:
l
Zing license server, zing-license-pool-server-<version>.sh – The Zing license
server authenticates, issues, and renews Zing licenses on the ZVM hosts. Additional functions
include: ZVM host license lease details, host management, server license management, audit
log, license reservation, user management, role management, notifications, and email
configuration.
See Zing System Requirements for a complete list of Zing requirements, including: supported
CPUs, memory requirements, operating system versions, browser, and listening port
requirements.
Azul Systems
Confidential and Proprietary
120
Specific Use-Cases: Zing Installation
l
Pool license file, <your_company>-pool.zip – This license authorizes you to use the
license server. You need at least one Pool license file, per license server.
l
Zing license file, license – Required on all ZVM hosts. It allows ZVM to run Java runtime
applications on instances of ZVM processes. It is issued to the ZVM host by the license server.
l
Zing license daemon,This is installed with ZST. It contains elements for the licensing
processes, including the elements to configure a ZVM host to get Zing licenses from the license
server. See Installing and Configuring Zing System Tools.
Zing License Locking
Licenses are issued with a set of conditions that define selected limitations of the license, referred to as
locks. The license server issues Zing licenses to ZVM hosts with the following locks:
l
Time – Zing license files issued by the license server include a license expiration timeframe
also known as the lease interval. The default lease interval is 24 hours. The lease interval is
configurable, see Modify Zing License Lease Interval.
l
Amazon Instance ID – This lock is used if the authorized ZVM host machine is an Amazon
Elastic Compute Cloud (EC2) instance. Each time the host machine instance boots in the EC2
environment it can be assigned a different IP address. Fortunately, its Amazon Instance ID
remains the same. So, for Zing licenses in EC2 environments, an Amazon Instance ID is used
as the ZVM host identifier. Using the Amazon Instance ID must be configured, see Use an
Amazon ID for ZVM Host ID.
Pool Licensing Process Overview
The basic Pool licensing process is as follows:
1. Azul issues a Pool license. Each Pool license supports a specific number of Zing licenses.
2. Install a Zing license server. See Install the License Server and Connect to Zing License Server.
3. Add the Pool license to the license server. See Store the Pool License Zip File and Add a Pool
License.
4. Install ZST and ZVM on the ZVM hosts to be included in the Zing license pool. Installing ZST
automatically installs the pool license daemon. See Installing and Configuring Zing System
Tools and Installing the Zing Virtual Machine.
5. Configure each ZVM host to use a Zing license issued by the license server. See Configuring
ZVM Hosts for Zing Pool Licensing.
Every ZVM host connects to the license server through the license server's IP address or DNS
resolvable hostname.
6. Each configured ZVM host machine requests a Zing license from the license server.
Azul Systems
Confidential and Proprietary
121
Zing Installation and Administration Guide
l
If the license server has a Zing license available:
l
The ZVM host is issued the Zing license.
l
Having a Zing license authorizes the ZVM host to use ZVM to start Java runtime
applications on ZVM instances.
l
l
The ZVM host holds the Zing license for the duration of the license lease.
l
Periodically, the ZVM host submits Zing license renewal requests.
If the license server does not have a Zing license available:
l
The ZVM host is denied a Zing license.
l
The denied ZVM host continues to request a Zing license every 5 minutes, until it receives
one.
l
Losing a Zing license means that ZVM cannot start new Java application instances on that
ZVM host. Java instances that are already running, continue to run.
7. As needed, manage the license server, Zing licenses, and ZVM hosts. See Managing the Zing
Licensing Server, Managing Zing Licenses, and Managing License Server Users.
Installing and Initializing the Zing License Server
This section describes how to install, configure, and prepare the Zing license server for issuing Zing
licenses to ZVM hosts. See the following sections:
l
Install the License Server
l
Connect to Zing License Server
l
Store the Pool License Zip File
l
Add a Pool License
Install the License Server
To install the license server:
1. Prepare the machine where you are installing the license server.
The license server is Java software and requires that Java be available before it can run. The
server must be a standalone machine, separate from the ZVM host machines that will be in its
license pool. See Zing System Requirements.
a. On a 64 bit Linux machine, install Java 1.6 or higher, for example HotSpot.
b. Set the JAVA_HOME variable to point to your Java installation directory.
c. Make a directory for the license server.
<zing_server_path>
d. From the Azul website, download the license server package:
Azul Systems
Confidential and Proprietary
122
Specific Use-Cases: Zing Installation
zing-license-pool-server-<version>.sh
2. Run the license server package.
$ <zing_server_path>/zing-license-pool-server-<version>.sh
The license server software is installed in the directory where the command is executed. The
installer package creates a folder:
zing-license-pool/
The installer responds:
Installing Pool License Server...
Checking java 1.6 or higher runtime.
Enter Installation directory [<zing_server_path>]:
Installation directory : <zing_server_path>/zing-license-pool
Checking available disk space.
Checking ports required.
Extracting Pool License Server files...
Pool License Server installed in the folder zing-license-pool
l
If a previous version of the license server exists, the following is displayed.
The Pool License Server is already installed.
Ports used by Pool License Server : 2443 9999 4447 27017
Do you wish to shutdown Pool License Server and upgrade? y)es/
(n)o: y
Note
If you shutdown the license server now, it has no effect on ZVM hosts that already
have Zing licenses. The length of time to shutdown, upgrade, and restart the
license server does not exceed the typical ZVM host Zing license lease.
l
If you want to upgrade, type y and proceed to Upgrade the Pool License Server.
l
If you do not want to upgrade, type n, and the installation wizard exits.
l
If the ports required for the license server are already in use:
a. To continue the installation, type y.
The port <port_number> required by the Pool License Server is
in use [ tcp 0 0 0.0.0.0:<port_number> 0.0.0.0:* LISTEN ]
The ports can be reconfigured using 'zing-licensepool/bin/configureports' after install
Do you wish to continue install? (y)es/(n)o : y
If you type y, installation continues and the ports might be changed as part the
installation process.
Azul Systems
Confidential and Proprietary
123
Zing Installation and Administration Guide
This prompt is displayed for each required port that is already in use. The license server
requires four ports: 2443, 9999, 4447, and 27017.
b. To continue the installation, type y.
Extracting Pool License Server files...
Pool License Server installed in the folder zing-license-pool
Do you wish to configure Pool License Server ports? (y)es/(n)
o : y
Configuring ports for the Pool License Server...
Pool License Server Web port [8443]:
7443
Pool License Server Management port [9999]:
Pool License Server Remoting port [4447]:
Pool License Server Database port [27017]:
7999
3447
17017
The updated Pool License Server port configuration:
Web port
: 7443
Management port : 7999
Remoting port
: 3447
Database port
: 17017
c. To apply the port changes, type y.
Do you wish to apply the port changes ? (y)es/(n)o : y
Port configuration. Done
To change the port number in the configuration file later, see Change the Listening
Ports.
3. Select whether to start the license server now. At the prompt type y.
Do you wish to start Pool License Server ? (y)es/(n)o : y
The wizard continues with the installation process.
Checking ports required for the Pool License Server
Database is not running. Starting database...
Checking ports required for the Pool License Server database
...
Started Pool License Server database successfully
Starting Pool License Server...
Checking Pool License Server status...
===================================================================
======
JBoss Bootstrap Environment
...
Azul Systems
Confidential and Proprietary
124
Specific Use-Cases: Zing Installation
===================================================================
======
When the installation completes, the following message is displayed:
Started Pool License Server successfully
The license server starts automatically.
4. While logged onto the license server machine, make note of the license server IP address or DNS
resolvable hostname.
You need this identifier to access the license server Web interface and to configure your ZVM
host machines.
Connect to a Zing License Server
Access to the Zing license server is through a Web browser.
To launch the license server interface:
1. Open a Web browser.
2. Enter the URL for the license server and the listening port. For example:
https://<license_server_ip_address>:8443
3. Enter your login credentials and click Login.
The default credentials are:
l
Username: admin
l
Password: admin
4. Change the password for the admin user and create at least one additional user with a password.
See Managing License Server Users.
Store the Pool License Zip File
The Pool license authorizes you to use the license server to manage your Zing licenses. This is
different than the Zing license that allows ZVM to start Java applications to run on ZVM instances.
Once you receive the Pool license, copy it to a location that is accessible to the license server.
1. Obtain the Azul email with the license zip file, <your_company>-pool.zip.
2. Store the zip file on the license server machine.
Azul Systems
Confidential and Proprietary
125
Zing Installation and Administration Guide
Note
Do not expand or open the zip file.
3. Note location where you store the zip file, <zing_server_path>.
Add a Pool License
Before the license server can issue Zing licenses to ZVM hosts, at least one Pool license has to be
added to the license server.
To add your Pool license:
1. From the Zing license server, click Server license.
2. Click the Add button.
3. Locate your Pool license zip file. Click Choose File and browse to the file.
See Store the Pool License Zip File.
4. Optionally, add a description of the license.
5. Click Add.
The Pool license information is added to the Server license page.
The following table describes the Server license page fields.
Field
Description
Status
Current status of the Pool license.
License ID
The Azul issued Pool license.
Description
Text you entered in the Add Azul license dialog.
Azul Systems
Confidential and Proprietary
126
Specific Use-Cases: Zing Installation
Field
Description
Issue Date
Date Pool license was issued to you.
Expiry Date
Date the Pool license expires. Contact Azul to renew your server
license.
Hard Stop Date
Pool licenses are issued with a grace period. The hard stop date
indicates the end of your grace period.
Count
Per your license request and as included by Azul in your license file,
the number of licenses this server is allowed to lease out to ZVM
hosts.
Extra Count
Used
Depending upon your Azul license, this column lists the number of additional Zing licenses available for use.
The number of licenses currently leased to ZVM hosts.
Configuring ZVM Hosts for Zing Pool Licensing
ZVM hosts must be configured to request Zing licenses from a Zing license server.
Note
Installing ZST automatically installs the license daemon required on each ZVM host
to use the Zing Pool license server.
1. Install ZST on the ZVM host machine. See Installing and Configuring Zing System Tools.
2. Install ZVM on the ZVM host machine. See Installing the Zing Virtual Machine.
Locate the license server IP address or hostname. See Install the License Server.
3. Log into each ZVM host machine, as root.
Execute the license configuration command, type:
# zing-licensed –-configure
For Amazon instances, use:
# zing-licensed –-configure –-amazonId
If the ZVM host is already configured, answer the prompt:
# zing-licensed --configure
Pool license configuration or Zing license exists on system
Configuring License Manager will overwrite existing
configuration and Zing license
Do you want to proceed ? Y/N:
n ignores the current attempt to configure and retains the existing configuration.
Azul Systems
Confidential and Proprietary
127
Zing Installation and Administration Guide
y overrides the existing configuration.
The system responds:
Configuring Zing license server
Alternatively, you can specify the server and port in the single command. For example:
# zing-licensed --configure --server zst1 --port 7443
# zing-licensed --c --amazonId --server 10.148.229.124 --port 8443
4. Enter the license server IP address.
Enter IP Address of license server:
Enter either the IP address or a hostname. We recommend using a hostname if you are using
DHCP for assigning IP addresses.
Alternatively, you can specify this in the configuration command string. Add --server
<server_ip>.
5. Enter the license server listening port.
Enter Port Number of license server:
Press return to accept the default port 8443 or enter the port you specified in the pool server
configuration step. See Install the License Server.
Alternatively, you can specify this in the configuration command string. Add --port <port_
no>.
6. If the Authentication Required option is on in the license server, then at the prompt, enter the login
credentials for the license server and press Return.
See Require ZVM Host Authentication. The default login credentials are:
Enter Username: admin
Enter Password: admin
Note:
Create your own user for the license server so you are not using the default
credentials. If you have already created a new administrative user, enter that
user's credentials here. See Add User.
7. Check the configuration response messages to verify configuration is successful.
Configuring Pool License Manager
Successfully configured Pool license server <server_ip> on port
<port_no>
Trying to fetch Pool license for the host…
Successfully installed Pool license '/etc/zing/license'
8. Repeat Step 1 through Step 7 on each ZVM host.
This completes the ZVM host configuration.
Once configured, the ZVM host requests a Zing license from the license server. It sends a
request every 5 minutes until it is issued a Zing license or until it is Revoked or Unconfigured
through the license server.
Azul Systems
Confidential and Proprietary
128
Specific Use-Cases: Zing Installation
zing-licensed Command Options
The zing-licensed command provides configuration options on the ZVM host. Some ZVM host
configuration options can be modified through the license server Web interface. See the listed sections
for additional information.
The zing-licensed command syntax is:
# zing-licensed –help
The response is:
version: 5.5.2.0.-1
usage: zing-licensed options ...
--configure [--amazonId] [--overwrite] [--server <ip address>]
[--port <port number>] [--user <username>] [--password <password>]
[--reserve]
--delete
--disable
--disable_core
--display
--display_core_info
--display_system_info [--amazonId]
--enable
--enable_core
--getlicense
--install <file path>
--unconfigure [--force]
--verify [<file path>] [<jar file path>]
--help
The command options perform as follows:
--configure
Configures the license server. Configuration command options include:
--amazonId
Set this ZVM host to provide an Amazon ID for host identification. See Use an Amazon
ID for ZVM Host ID.
--overwrite
Replace the existing configuration file. This command proceeds without a confirmation.
--server <ip address>
Specifies the IP address or DNS resolvable hostname of the license server.
--port <port number>
Azul Systems
Confidential and Proprietary
129
Zing Installation and Administration Guide
Specifies a listening port of the license server during ZVM host configuration. See
Change the Listening Port.
--user <username>
Specifies a user that exists on the license server. Used to authenticate registration.
Include the --password option when this option is used.. See Add User.
--password <password>
Specifies the password for the user listed in the --user option. The --user option is
required when this option is used. See Add User.
--reserve
Marks this host as a Reserved-Host in the license server. See Reserving Zing Licenses.
--delete
Deletes the default license file at /etc/zing/license. If a ZVM instanceis running and it is
using this license, the command fails.
--disable
Deletes the license file, but retains the server configuration. If a ZVM instance is running, the
command fails. This option disables the host from fetching a license from the license server.
See Disable a ZVM Host.
--disable_core
Disables core dump for zing-licensed process. Restart zing-licensed after issuing the disablecore command.
1. Issue disable_core command zing-licensed --disable_core
2. Restart zing-licensed service: service zing-licensed restart
--display
Lists the content of the license file /etc/zing/license.
--display_core_info
Lists if core dump is enabled or disabled for zing-licensed process.
--display_system_info [--amazonId]
Displays the system parameters set to lock a license file.
--amazonId
Display the Amazon ID of the ZVM host. See Use an Amazon ID for ZVM Host ID.
--enable
Enables the host to start fetching licenses from the license server. See Enable a ZVM Host.
--enable_core
Enables core dump for zing-licensed process. Restart zing-licensed after issuing the
enable_core command.
1. Issue enable_core command zing-licensed --enable_core
2. Restart zing-licensed service: service zing-licensed restart
Azul Systems
Confidential and Proprietary
130
Specific Use-Cases: Zing Installation
Core files are stored at:
l
For RHEL or SLES, /var/crash/zing-licensed and /var/crash/zing-
licensem.
l
For Ubuntu, /var/crash/zing-licensed.
--getlicense
Requests a Zing license from license server.
--install <file path>
Given a license file, verifies the license and then install the license to /etc/zing/license.
--unconfigure [--force]
Removes the ability of the ZVM host to request a Zing license from the license server and delete
the license server configuration. See Unconfigure a ZVM Host.
--force
Unconfigure the ZVM host even if the license server is non-responsive.
--verify [<file path>] [<jar file path>]
Verifies the license file <file path>.
When a <jar file path> is provided, verify the application identified by the <jar file
path>, against the application in the application locked license specified in the <file path>.
--help
Displays the zing-licensed command options and the zing-licensed package version.
Managing the Zing Licensing Server
Managing Zing licensing server topics include:
l
Start the License Server
l
Stop the License Server
l
Change the Listening Ports
l
Delete a Pool License from License Server
l
Backup the License Server
l
Restore the License Server
l
Configure Email Server
l
Configure Notifications
Change the Listening Ports
Note
Azul Systems
If you change the listening port and you have ZVM hosts configured to connect to the
license server through the existing listening port, be sure to update the ZVM hosts to
connect to the current license server listening port.
Confidential and Proprietary
131
Zing Installation and Administration Guide
To change the listening port:
1. From the command line where the license server is installed, run:
$ <zing_server_path>/bin/configureports
Where <zing_server_path> is the license server installation directory. For example:
$ ./bin/configureports
Configuring ports for the Pool License Server...
2. Confirm shutting down the license server.
Do you wish to shutdown Pool License Server and configure ports?
(y)es/(n)o : y
Stopping Pool License Server database...
killing process with pid: 10871
Stopped Pool License Server database successfully
Stopping Pool License Server...
Stopped Pool License Server successfully
Database is not running.
3. At the prompts, enter a new portor accept the listed number, press Return.
Pool License Server Web port [8443]:
<port_number>
Pool License Server Management port [9999]:
Pool License Server Remoting port [4447]:
Pool License Server Database port [27017]:
<port_number>
<port_number>
<port_number>
The updated Pool License Server port configuration:
Web port
: 8443
Management port : 9999
Remoting port
: 4447
Database port
: 27017
4. Confirm to apply the port number changes.
Do you wish to apply the port changes ? (y)es/(n)o : y
Port configuration. Done
5. Confirm to restart the license server.
Do you wish to start Pool License Server ? (y)es/(n)o : y
Checking ports required for the Pool License Server
Database is not running. Starting database...
Checking ports required for the Pool License Server database
Starting Pool License Server database....
...
Started Pool License Server database successfully
Starting Pool License Server...
Checking Pool License Server status...
Azul Systems
Confidential and Proprietary
132
Specific Use-Cases: Zing Installation
JAVA_OPTS already set in environment; overriding default settings
with values: ...
===================================================================
======
JBoss Bootstrap Environment
...
===================================================================
======
Started Pool License Server successfully
Delete a Pool License from License Server
To remove a Pool license:
1. From the Zing license server, click Server license.
2. Ensure that no ZVM hosts are using the Pool license you plan to delete.
If the selected Pool License is a valid license then, ensure that no ZVM hosts have a license
leased by this server, before attempting to delete it. See View ZVM Hosts with Active Zing
Licenses.
3. Select the Pool license to delete. Click anywhere in the line of the Pool license.
4. Click Delete.
5. Click OK to confirm deleting the Pool license.
The Pool license is removed from the list of available Pool licenses in the license server.
Backup the License Server
Use the backupserver command to create a backup of the license server. This includes backing up
the database and configuration files.
1. From the command line where the license server is installed, run:
$ <zing_server_path>/bin/backupserver <backup_directory>
Where –
<zing_server_path> – the license server installation directory.
<backup_directory> – an optionally specified backup directory. The default directory is
<zing_server_path>/backup/server.
For example:
$ ./bin/backupserver
Azul Systems
Confidential and Proprietary
133
Zing Installation and Administration Guide
The system responds:
Date 2013-05-02_13h36m
Back up folder: /home/user1/dev/work/zing-license-dev/test/zinglicense-pool/bin/../backup/server file : zing-license-pooldbbackup-2013-05-02_13h36m
Starting Pool License Server backup...
...
Compressing Pool License Server backup file
Pool License Server backup done.
The location where the backup is stored is:
<zing_server_path>/backup/server
Where <zing_server_path> is the Zing license server installation path.
The backup filename is:
zing-license-pool-dbbackup<timestamp>.tar.gz
Where <timestamp> is in the form: yyyy-mm-dd_##h##m
Restore the License Server
Use the restoreserver command to restore the license server. The restored server is placed in the
directory you specify in the command line.
1. From the command line where the license server is installed, run:
$ <zing_server_path>/bin/restoreserver <backup_filename>
Where –
<zing_server_path> is the license server installation directory.
<backup_filename> is the filename of the backup file.
For example:
$ ./bin/restoreserver ../backup/server/zing-license-pool-dbbackup2013-05-02_13h36m.tar.gz
The system responds:
Starting Pool License Server restore...
Processing backup file ../backup/server/zing-license-pool-dbbackup2013-05-02_13h36m.tar.gz
Processing files in directory /home/user1/dev/work/zing-licensedev/test/zing-license-pool/bin/../backup/server/zing-license-pooldbbackup-2013-05-02_13h36m
...
Pool License Server restore done. Please check messages for any
error.
Azul Systems
Confidential and Proprietary
134
Specific Use-Cases: Zing Installation
Configure Email Server
The Zing license server provides facilities to send email notifications on important events detected by
the server. To configure your email settings:
1. From the Zing license server, click Configuration > Email server.
2. Enter your email server information:
l
SMTP Server – your company's SMTP server. Enter an IP address or hostname.
l
SMTP Port – the communication port used by your email server
l
From Email Address – the address the license server uses to identify itself to your
recipient. Use for example, zing-license-no-reply@your_company.com.
l
Optionally, select a security method. Click the checkbox.
l
Use SMTP Authentication
Enter a User name and Password.
l
Use SSL
3. Click Save.
The license server validates the settings.
Configure Notifications
To receive email notification of pre-defined alerts, select from the options on the Notifications tab.
1. From the Zing license server, click Configuration > Notifications.
Azul Systems
Confidential and Proprietary
135
Zing Installation and Administration Guide
2. Enter email addresses in the Email IDs field.
Use valid email addresses. Separate multiple addresses with a comma.
3. Select the notifications to send. Click the notification checkbox and enter the values, as needed.
The options are:
l
ZVM Host unable to reach license server for
This notification is generated if the ZVM host does not contact the license
server within the specified time.
Minutes – If a ZVM host cannot reach the license server within the
number of minutes specified, a notification is sent.
l
License not renewed on ZVM host
This notification is generated when the license for a ZVM host has expired. The server
expected the ZVM host to renew the license and it did not happen. The notification
includes ZVM host identification.
l
Number of available licenses is less than or equal to
This notification is generated when the number of free licenses in the
Pool fall below a configured threshold.
Count – When the number of available ZVM hosts licenses left reaches
this count, a notification is sent.
l
Pool server license expired
The Pool server can have more than one license installed on it. An email is sent out
when a Pool server license reaches its warning date.
Azul Systems
Confidential and Proprietary
136
Specific Use-Cases: Zing Installation
4. Select the notification severity. The severity is included in the notification message. The severity
options are:
l
Information
l
Critical
5. Click Save.
Managing Zing Licenses
Options for viewing and managing Zing licenses and ZVM hosts that are in Pool license server, include:
l
Viewing Zing License Status and Activity
l
Modifying Zing License Settings
l
Reserving Zing Licenses
l
Removing, Resetting, Revoking Zing Licenses
Each item (tab or node) in left panel of the license server displays the corresponding panel of current
information about the selected tab.
Viewing Zing License Status and Activity
Options for viewing the status of the license server, ZVM hosts, and Zing licenses, include:
l
View Active Server Licenses
l
View Audit Log of Licensing Activity
Azul Systems
Confidential and Proprietary
137
Zing Installation and Administration Guide
View Active Server Licenses
The Server license tab displays the Pool licenses that have been added to the license server and their
current availability for use by the license server.
1. From the Zing license server, click Server license. The following table describes the fields.
Field
Description
Status
Current status of the Pool license.
License ID
The Azul issued Pool license.
Description
Text you entered in the Add Azul license dialog.
Issue Date
Date Pool license was issued to you.
Expiry Date
Date the Pool license expires. Contact Azul when you need to renew
your server license.
Hard Stop Date
Pool licenses are issued with a grace period. The hard stop date
indicates the end of your grace period.
Count
Per your license request and as included by Azul in your license file,
the number of licenses this server is allowed to lease out to ZVM
hosts.
Extra Count
Depending upon your Azul license, this column lists the number of
additional Zing licenses available for use.
Used
The number of licenses currently leased to ZVM hosts.
2. Select activities from this tab, as needed:
Add – adds a Pool license to the license server. When a Pool license is added to the server, the
pool license also specifies the number (count) of Zing licenses that can be leased out to ZVM
hosts. The total number of Zing licenses that can be leased out from the license server is equal
to the sum of all the counts from all the Pool licenses installed on this server. See Add a Pool
License.
Azul Systems
Confidential and Proprietary
138
Specific Use-Cases: Zing Installation
Delete – removes a Pool license from the license server. Deleting a Pool license reduces the
number available Zing licenses for lease by the number (count) specified in the Pool license.
See Delete a Pool License from License Server.
View Audit Log of Licensing Activity
To see log messages between the license server and the ZVM hosts:
1. From the Zing license server, click Audit log.
When you select it, initially it is empty.
2. Set the filter for the list of items to be displayed.
Click through the status and filters fields to select the data to be displayed. This includes the
following filtering options:
l
With status as: All, License issued, License validation failed, License renewed, License
not available, License Revoked, License Denied, License Expired, License reset, Host
license renewed, Host deleted, Host registered, Host re-registered, Host unregistered,
Host enabled, Host disabled
l
in: Last 1 hour, Last 24 hours, Last 7 days, Last 30 days, Last 60 days, Custom
l
From/To: calendar dates (click in field to see calendar)
l
Filter: All, Status, Host, Date
l
Column: Contains, Matches, Starts with
3. Click Generate Report to display the list.
The following table describes the fields.
Field
Status
Description
Host or license status. Options include:
l
Azul Systems
License issued
Confidential and Proprietary
139
Zing Installation and Administration Guide
Field
Description
l
License validation failed
l
License renewed
l
License not available
l
License Revoke
l
License Denied
l
License Expired
l
License reset
l
Host license renewed
l
Host deleted
l
Host registered
l
Host re-registered
l
Host unregistered
l
Host enabled
l
Host disabled
Host
Name of the ZVM host.
Date
Timestamp for status message.
Modifying Zing License Settings
Options for modifying the Zing licenses that are issued to a ZVM host, include:
l
Use an Amazon ID for ZVM Host ID
l
Require ZVM Host Authentication
l
Modify Zing License Lease Interval
Note:
Options set through the Settings tab apply to all ZVM hosts in the license server
pool.
All the Zing license settings options are configured through the Configuration > Settings tab.
Azul Systems
Confidential and Proprietary
140
Specific Use-Cases: Zing Installation
Use an Amazon ID for ZVM Host ID
If the ZVM hosts are in the Amazon cloud, the Host IDs change each time the Amazon instance is
started. To provide a lock for the Zing license, use the Amazon ID instead of the Host ID. The default is
not to use an Amazon ID.
To have all ZVM hosts use an Amazon ID for the Host ID value:
1. From the Zing license server, click Configuration > Settings.
2. Click the check box Support Amazon ID.
3. Click Save.
Require ZVM Host Authentication
The default configuration for Pooled ZVM hosts is to request a Zing license and not require ZVM host
login credentials.
To have the ZVM hosts provide authorized user login credentials:
1. From the Zing license server, click Configuration > Settings.
2. Click the check box Authentication Required.
3. Click Save.
When a ZVM host is being configured, the license server prompts for and accepts the user login
credentials.
4. On each ZVM host, run the ZVM host configuration script and specify the user login credentials.
See Configuring ZVM Hosts for Zing Pool Licensing.
Modify Zing License Lease Interval
The license server issues Zing licenses to ZVM hosts for specified periods of time, that is a leased
period. The default is to lease Zing licenses for 24 hours.
The ZVM host requests a license renewal for its Zing license every 1/10 of the lease interval. For
example, if the lease is 10 hours, a renewal is requested every hour.
To change the Zing license expiration interval:
1. From the Zing license server, click Configuration > Settings.
2. Select the desired interval in the Lease Interval option. The options are:
Azul Systems
Confidential and Proprietary
141
Zing Installation and Administration Guide
l
0 to 30 days and/or 0 to 23 hours, in one hour increments.
l
Maximum time available to the Pool license. Selecting this option sets the Zing
license to expire when the Pool license expires.
3. Click Save.
These changes apply to all ZVM hosts in the license server pool. The Zing license expiration
interval changes at the next renewal.
Reserving Zing Licenses
To ensure that a ZVM host has access to a Zing license, set up a reservation. Complete the steps in
the following sections:
l
Reserve Zing Licenses from the Pool
l
Add a ZVM Host Reservation
To remove a ZVM host reservation, see Remove a ZVM Host Reservation.
All Reservation activity is performed on the Reservation tab.
Reserve Zing Licenses from the Pool
The number of reservations cannot exceed the number of licenses available in the pool.
1. From the Zing license server, click Reservation.
2. Reserve a number of Zing licenses. Click Reserve.
3. Enter the number of Zing licenses from your pool you want to set aside for specific ZVM hosts. Click
Save.
Add a ZVM Host Reservation
The license server allows licenses to be reserved for identified ZVM hosts.
Azul Systems
Confidential and Proprietary
142
Specific Use-Cases: Zing Installation
1. Configure the Reservation count prior to assigning specific ZVM hosts. See Reserve Zing Licenses
from the Pool.
2. From the Zing license server, click Reservation.
3. Assign a ZVM host a reserved license. Click Add.
4. Click the host name from the list of configured ZVM hosts. Click Save.
The host is added to the list of hosts displayed in the Reservation table.
Remove a ZVM Host Reservation
1. From the Zing license server, click Reservation.
2. Select the ZVM host table entry. Click anywhere in the host row to select the host.
3. Click Delete.
A confirmation message box appears.
4. Confirm removing the Zing license reservation. Click OK.
When a host reservation is removed, the host is not affected. If it has a current lease - it
continues to use the leased license. When the license expires and it requests to renew the
license, it might not receive a license if the pool is full and a license has not been reserved for it.
Removing, Resetting, or Revoking Zing Licenses
The ZVM host license tab displays all ZVM hosts with active Zing licenses. This is not a list of all
ZVM hosts that are able to run a Zing license, only those that are currently consuming a Zing license.
Options for viewing and stopping Zing license use, include:
l
View ZVM Hosts with Active Zing Licenses
l
Reset All ZVM Host Zing Licenses
l
Remove Expired Zing Licenses
l
Revoke Zing License on a ZVM Host
View ZVM Hosts with Active Zing Licenses
1. From the Zing license server, click ZVM host license.
Azul Systems
Confidential and Proprietary
143
Zing Installation and Administration Guide
The ZVM host license tab displays the ZVM hosts that are actively using Zing licenses issued
through this license server.
The following table describes the fields.
Field
Description
Current status of the Zing license on the ZVM host. Options include:
Status
Active with validation failed – The system clock is more than 24
hours out of sync. Zing license cannot be verified and is considered
invalid.
Pending Revoke – Revoke issued, waiting for a running ZVM to stop.
Active – Currently consuming Zing license.
Reserved
Indicates if the applied Zing license is reserved for this ZVM host.
Host
Name of the ZVM host.
Time Left
Remaining time on the current lease of the Zing license.
Issue Date
Date the Zing license was issued by this server.
Expiration Date
Date on which the Zing License issued by this server expires.
Hard Stop Date
Pool licenses are issued with a grace period. The hard stop date
indicates the end of your grace period.
License ID
Zing License identifier for the license issued to this host.
Reset All ZVM Host Zing Licenses
The license server can be forced to clean up all the currently leased licenses by using the Reset pool
licenses button. When this is selected all the ZVM hosts are asked to renew their licenses again. This
causes the server to re-synchronize the license state with all the registered ZVM hosts in the pool.
1. From the Zing license server, click ZVM host license.
2. Click Reset pool licenses.
A confirmation message box appears.
3. Confirm resetting all the active ZVM host Zing licenses. Click OK.
Azul Systems
Confidential and Proprietary
144
Specific Use-Cases: Zing Installation
The active Zing license count is reset to zero. The license server tells all ZVM hosts to request
new Zing licenses now. The ZVM host status changes to Renew License. This does not stop
any running ZVM instances.
Remove Expired Zing Licenses
To remove expired Zing licenses from the listing of ZVM host licenses, use the Remove expired
license button. This does not remove any active ZVM host licenses or license listings.
To remove expired Zing license listings:
1. From the Zing license server, click ZVM host license.
This page displays all ZVM hosts that have been assigned a Zing license from the license
server. This includes ZVM hosts with currently active licenses and ZVM hosts with expired
licenses.
2. Select the row of the expired Zing license. Click anywhere in the row.
3. Click Remove expired license.
4. Click OK to confirm.
Since the Zing license was already expired, this has no affect on the status of available Zing
licenses.
Revoke Zing License on a ZVM Host
To have the license server submit a request to the ZVM host to relinquish its Zing license, use the
Revoke button.
l
If the selected host is not running any ZVM instances the license is released and the license
becomes available for use by other ZVM hosts.
l
If the selected host is running a ZVM instance, the license remains assigned until all ZVM
instances on that host stop. However, the ZVM host no longer requests a license renewal.
Revoking a license is temporary, after relinquishing its license and after the standard requesting
interval passes, the ZVM host requests a new license. Revoking a license does not remove the ZVM
host from the list of ZVM hosts known to the license server. See Managing ZVM Hosts.
To revoke the Zing license on a ZVM host:
1. From the Zing license server, click ZVM host license.
2. Select the ZVM host you want to release.
3. Click Revoke.
4. Click OK to confirm.
The Zing license is canceled on the selected ZVM host.
l
If no ZVM intances are currently running on the ZVM host, then the Zing license is released
for another ZVM host to use.
l
If a ZVM instance is running, the Zing license remains active and the status is changed to
Pending Revoke. ZVM instances are allowed to start until the Zing license is released.
Azul Systems
Confidential and Proprietary
145
Zing Installation and Administration Guide
When all running ZVM instances stop, the Zing license is released for another ZVM host to
use.
Managing ZVM Hosts
The Hosts tab displays an inventory of all ZVM hosts configured to be able to run a Zing license issued
from the license server. ZVM hosts on this list can be granted a Zing license, but might not be actively
consuming a Zing license. Options for viewing and managing pooled ZVM hosts, include:
l
View ZVM Hosts in the License Server Pool
l
Disable a ZVM Host
l
Enable a ZVM Host
l
Delete a ZVM Host
l
Unconfigure a ZVM Host
View ZVM Hosts in the License Server Pool
1. From the Zing license server, click Hosts.
The following table describes the fields.
Field
Description
Current status of the Zing license on the ZVM host. Options include:
Active with license – Currently consuming Zing license.
Status
Active with validation failed – The system clock is out of sync by
more than the maximum number of leased hours. Zing license cannot
be verified and is considered invalid.
Enabled – license server will issue a Zing license to this ZVM host.
Disabled – license server will not issue a Zing license to this ZVM
host.
Azul Systems
Confidential and Proprietary
146
Specific Use-Cases: Zing Installation
Field
Description
Pending disable – User action to disable this ZVM host is pending
confirmation from ZVM host.
Pending unconfigure – User action to unconfigure this ZVM host is
pending confirmation from ZVM host.
Pending renew – When the Zing licenses have been reset, all the
ZVM hosts need to renew their Zing licenses.
License not available – The host requested a Zing license but no
licenses are available.
Reserved
Indicates if the ZVM host has a Zing license reserved for its use.
Host
Name of the ZVM host.
Version
Zing license package version installed on the ZVM host.
Registered from
Date the ZVM host was configured as a member of the license server.
Status check in
Date and time when the zing-license daemon checks its status
with the Zing license server.
Registration ID
Identification assigned to the host when it was added to the license
server.
Host lock type
Lock used to secure the Zing license. See Zing License Locking.
Host lock
Literal lock value for the Zing license. For example, the ZVM host
MAC address. See Zing License Locking.
Disable a ZVM Host
When you disable a ZVM host, the next time the ZVM host requests a Zing license, the license server
denies the request and does not issue a Zing license. If the ZVM host is running a ZVM instance, the
Zing license remains in effect until the ZVM instance stops.
Disabling a ZVM host does not remove the authenticated configuration of the ZVM host from the
license server.
To disable the Zing license on a ZVM host:
1. From the Zing license server, click Hosts.
2. Select the ZVM host you want to disable. Click anywhere in the host row to select the host.
Azul Systems
Confidential and Proprietary
147
Zing Installation and Administration Guide
3. Click Enable/Disable.
4. Click OK to confirm.
l
If there are no running ZVM instances, the host status changes to Disabled.
l
If there are running ZVM instances, the host status changes to Pending Disabled.
Enable a ZVM Host
Enabling a ZVM host allows the license server to issue a Zing license to the ZVM host, when
requested.
To enable the Zing license on a ZVM host:
1. From the Zing license server, click Hosts.
2. Select the ZVM host you want to enable. Click anywhere in the host row to select the host.
3. Click Enable/Disable.
4. Click OK to confirm.
Delete a ZVM Host
The Delete host button forcefully removes a ZVM host from the license server pool. This option is used
when the ZVM host is no longer physically available. When a ZVM host is deleted, the license on the
ZVM host is not touched.
To forcefully remove a ZVM host from the license server pool:
1. From the Zing license server, click Hosts.
2. Select the ZVM host you want to remove from the license server pool. Click anywhere in the host
row to select the host.
3. Click Delete host.
4. Click OK to confirm.
Unconfigure a ZVM Host
The Unconfigure button permanently removes a ZVM host from the license server pool and removes
the Pool license configuration from the ZVM host.
To unconfigure a ZVM host and remove it from the license server pool:
1. From the Zing license server, click Hosts.
2. Select the ZVM host you want to remove from the license server pool. Click anywhere in the host
row to select the host.
3. Click Unconfigure.
4. Click OK to confirm.
Azul Systems
Confidential and Proprietary
148
Specific Use-Cases: Zing Installation
Managing License Server Users
For best practice, create additional users for managing the license server. The default user and role for
the license server is Admin user with Administrator role. The Administrator role has permission to
view and manage all functions in all the license server.
To manage users:
1. Define custom roles.
A custom role can have any name and any assortment of access privileges. Access is granted
at tab level. All the options that are available from a tab are available to the role with that tab
selected. The Admin role cannot be edited.
2. Create users and assign them a role.
A user has a name and a login and is assigned a single role. With a role assigned the user has
access to all the tabs granted to the role. Once a new login name is created, that name cannot
be edited, but it can be deleted. The admin user can be edited.
To create users with limited permission:
1. Create a new role with limited permissions.
2. Create a new user and apply the new role.
User management activities include:
l
Add a User Role
l
Edit a User Role
l
Delete a User Role
l
Add a User
l
Change User Passwords
l
Edit User Definitions
l
Delete a User
Add a User Role
To add a Role:
1. From the Zing license server, click Configuration > Role.
Azul Systems
Confidential and Proprietary
149
Zing Installation and Administration Guide
All the roles defined for access to the license server are listed.
2. Click Add.
3. Enter a Name and optionally, a Description for the Role.
4. Select the new role privileges. Click Custom, then click the checkboxes for the tabs users with this
role can have access to.
Allows user access to <tabname> tab, where the <tabname> options are:
l
ZVM host license – Displays all ZVM hosts that have had a Zing license assigned to them
from the license server.
l
Hosts – Displays all the ZVM hosts in the license server pool.
l
Server license – Displays all the Pool licenses active with the license server.
l
Audit Log – Displays the Audit Log of messages between the hosts and license server.
l
User – Displays the users defined for the license server.
l
Role – Displays the roles defined for the license server.
l
Settings – Displays the configurable settings for ZVM hosts.
Azul Systems
Confidential and Proprietary
150
Specific Use-Cases: Zing Installation
l
Notifications – Displays the notification messages that can be sent to specified email
addresses.
A user assigned this role has permission to view and manage only the selected license server
tabs.
5. Click Save.
The new role is listed in the Role tab.
Edit a User Role
To edit a role:
1. From the Zing license server, click Configuration > Role.
2. Select the role to edit. Click Edit.
The default Admin role cannot be edited.
3. Edit the fields as needed. Click Save.
Delete a User Role
To delete a role:
1. From the license server, click Configuration > Role.
2. Select the role to delete. Click Delete.
3. Confirm to delete the selected user. Click OK.
Add a User
To add a new user:
1. From the license server, click Configuration > User.
Azul Systems
Confidential and Proprietary
151
Zing Installation and Administration Guide
All the users allowed access to the license server are listed.
2. Click Add.
3. Enter the new user information.
l
Full name – This name can be edited later.
l
Login name – This name cannot be edited, once created. Though it can be deleted.
l
Email – Must be in valid email format, including the @.
4. From the menu, select a Role for this user.
5. Enter a Password for this user.
6. Click Save.
Change User Passwords
To change a user's password:
1. From the license server, click Configuration > User.
2. Select the user to be edited. Click Change Password.
3. Enter the new password. Click Save.
Delete a User
To delete a user:
1. From the license server, click Configuration > User.
2. Select the user to delete. Click Delete.
Azul Systems
Confidential and Proprietary
152
Specific Use-Cases: Zing Installation
3. Confirm to delete the selected user. Click OK.
Uninstalling the Zing Licensing Components
To completely uninstall your Zing licensing components:
l
Remove the License Server
l
Remove your Pool License
Remove the License Server
1. Login to the license server machine using the same user ID that was used when the license server
was installed. Or login as root user.
2. Locate the license server files. Per the directions in Install the License Server, locate the
installation directory. For example:
/zingserver/zing-license-pool/
3. Stop the license server. From the command line where the license server is installed, type:
$ <path>/zing-license-pool/bin/stopserver
Where path is the license server installation directory. For example:
$ /zingserver/zing-license-pool/bin/stopserver
4. Delete the directory and all its contents.
$ cd /zingserver/zing-license-pool/
$ rm -rf zing-license-pool-server<version>
Remove your Pool License
1. Login to the license server machine using the same user ID that was used when the license server
was installed. Or login as root user.
2. Locate your Azul license server license.
This is the file Azul emailed you, <your_company>-pool.zip. Per the directions in Store
the Pool License Zip File, locate the installation directory, <zing_server_path>. For
example:
/zingtool/
3. Delete the directory and all its contents.
$ cd /zingtool/
$ rm <your_company>-pool.zip
Upgrading the Zing Licensing Components
The Zing licensing components include:
For the ZVM host:
Azul Systems
Confidential and Proprietary
153
Zing Installation and Administration Guide
l
Zing license file – The Zing license file is not upgraded. Current Zing licenses are issued to
the ZVM hosts by the Zing license server.
l
Zing license daemon – The Zing license package that is installed on every ZVM host with
ZST. Upgrading this requires that you run the ZST upgrade command appropriate to your ZVM
host operating system. See Installing and Configuring Zing System Tools.
For the license server:
l
Pool license – The Pool license file is not upgraded. If you receive a new Pool license, simply
add the new Pool license to the existing set of Pool license files. See Store the Pool License
Zip File.
l
Zing license server – Upgrading the license server is a function of the license server
installation package. To upgrade your Zing license server, follow the installation directions and
confirm the upgrade option. See Upgrade the Pool License Server.
Upgrade the Pool License Server
To upgrade the license server:
1. Prepare to upgrade.
a. Login to the machine with the license server to be upgraded.
b. Shutdown all license server processes.
c. Change to the license server directory.
<zing_server_path>
d. From the Azul website, download the license server upgrade package:
zing-license-pool-server-<version>.sh
2. Run the license server package.
$ <zing_server_path>/zing-license-pool-server-<version>.sh
The installer responds:
Installing Pool License Server...
Checking java 1.6 or higher runtime.
Enter Installation directory [</current_location>]:
Installation directory : <zing_server_path>/zing-license-pool
The Pool License Server is already installed
Ports used by Pool License Server : 8443 9999 4447 27017
If a previous version of the license server exists, prompts to upgrade are displayed.
3. Confirm to continue the upgrade.
Do you wish to shutdown Pool License Server and upgrade? (y)es/(n)o
: y
Azul Systems
Confidential and Proprietary
154
Specific Use-Cases: Zing Installation
Updating Pool License Server files...
Comparing new and installed Pool License Server version...
Installed version (5.5.1.0) is earlier than new version
(5.5.2.0.beta). Upgrading...
Stopping Pool License Server ...
Stopping Pool License Server database...
Fri Mar
8 12:53:27 BackgroundJob starting: DataFileSync
killing process with pid: 24696
Stopped Pool License Server database successfully
Stopping Pool License Server...
l
If the license server ports are busy, the upgrade is cancelled.
Another Server instance is running with Pool License Server
configured ports
Pool License Server is not running.
The port 8443 required by Pool License Server is still in use.
Upgrade canceled.
To configure ports use command 'zing-licensepool/bin/configureports'
To change the port number in the configuration file later, see Connect to License Server.
4. Confirm to start the license server.
Stopped Pool License Server successfully
Database is not running.
...
Updating zing-license-pool/...
Do you wish to start Pool License Server ? (y)es/(n)o : y
Checking ports required for the Pool License Server
...
Started Pool License Server database successfully
Starting Pool License Server...
Checking Pool License Server status...
==============================================================
JBoss Bootstrap Environment
JBOSS_HOME: <zing_server_path>/zing-license-pool/bin/../jboss
JAVA: <java_path>/bin/java
JAVA_OPTS:...
...
==============================================================
Started Pool License Server successfully
Azul Systems
Confidential and Proprietary
155
Zing Installation and Administration Guide
Your upgrade is complete.
Zing Pilot Assist Kit
The Azul Systems® Zing™ platform uses the Zing Virtual Machine (ZVM) to run Java™ technologybased applications. In the Zing product, the Zing System Tools (ZST) component, installed on each
ZVM host system, manages the elastic and highly scalable shared memory resources.
This guide is part of Azul's Zing Pilot Assist Kit (Zing PAK). It provides steps and information to help
you install Zing for evaluation, pilot, and testing purposes.
The Zing Pilot Assist Kit is designed to ensure your trial period functions smoothly. Using theZing Pilot
Assist Kit ensures a successful Zing installation, system configuration, and ZVM launch by:
l
l
Ensuring you are installing a version of Zing that is compatible with your environment.
Ensuring Zing component compatibility -- that the ZVM component version is compatible
with the ZST component version.
l
Validating the host system can support Zing.
l
Checking for required operating system settings.
l
Providing a list of settings that might need to be modified.
l
Providing a means to determine the best set of command line options to launch the ZVM
process without changing any of your pre-existing scripts.
Zing Pilot Assist Kit prepares a recipe to help system administrators configure the host system to use
the ZVM to run your Java applications.
Zing Pilot Assist Kit Quick Start
To use the Zing Pilot Assist Kit:
1. Visit the Azul pilot download website.
2. Complete the forms to order your temporary Zing license.
A temporary evaluation Zing license will be emailed to you.
3. Select a Zing bundle and download it to a <temp_directory>.
Choose the bundle that matches your Java version. You can install a version of Zing for each
Java version you wish to use. The supported Java versions are:
l
Java SE 6.0
l
Java SE 7.0
l
Java SE 8.0
See "Selecting and Validating Zing PAK Download Packages" on page 158 for a list of the Zing
Pilot Assist Kit download files.
Azul Systems
Confidential and Proprietary
156
Specific Use-Cases: Zing Installation
See "Zing Pilot Assist Kit Operating System Requirements" on page 167 for a list of supported
operating systems. Note: DKMS packages and Debian are not supported with the Zing Pilot
Assist Kit at this time.
4. Run the downloaded self-extracting Zing Pilot Assist Kit shell script to analyze your system and
validate it to ensure the ZVM can run your applications.
The script extracts installation files applicable to the operating system used and performs a
number of tests to ensure that the host system is properly configured. If needed, the script
creates a text file that contains instructions for optimally configuring your system.
a. Run the command.
$ zing-pilotkit-<version>.sh
b. Choose the profile that is most applicable to your application’s execution
requirements, low-latency or high-performance.
Use the name of this profile as the command line argument to the script you run to
validate your system.
View the Usage section that follows to help you decide which profile is most
appropriate for your application. For example:
$ ./zing-pilotkit8-XX.XX.sh
Usage:
zing-pilotkit8-XX.XX.sh --help|-h
zing-pilotkit8-XX.XX.sh [-o <output_path>] {high-performance
| low-latency}
Use the help option to view additional information
$ ./zing-pilotkit8-XX.XX.sh -h
Usage:
zing-pilotkit8-XX.XX.sh [-o <output_path>] {high-performance
| low-latency}
profile:
To obtain an application's desired performance objectives,
different sets of tuning options are recommended. Each set
of tuning options is called a profile. Choosing the profile
selects which set of tuning options to use. The recommended
value of each tuning option in the selected set is compared
with the value of that option on the machine where the Zing
VM will be run.
output_path:
When the value of an option on the machine is other than the
recommended value, this script notifies the user and
generates a file in the <output_path> directory with
instructions for fixing the problem.
A system administrator can follow the instructions in the
file to fix the problems found. If not already installed,
Azul Systems
Confidential and Proprietary
157
Zing Installation and Administration Guide
ZST and ZVM installation packages are also extracted to this
directory.
The current directory is used as <output_path> if the option
is not specified.
Important note:
Run this script as a user with the same credentials as the
user who will run the ZVM. The selected profile is copied to
the /home/<username>/.zing/profile directory.
5. Have your administrator install Zing.
a. Forward the extracted Zing installation files, the valid Zing Evaluation license, and
the generated system-administrator-todo-list.txt, to your system
administrator.
The profile files are installed in the /home/<username>/.zing/profile/
directory.
b. Have your system administrator install Zing and configure your system per the
directions in the system-administrator-todo-list.txt file.
See "Understanding the system-administrator-todo-list File" on page 159 for a sample systemadministrator-todo-list.txt file.
See "Appendix A – Zing Installation Prerequisites Details" on page 318 for directions to apply low
latency configuration settings.
6. Rerun the zing-pilotkit-<version>.sh script to ensure that all the configuration settings
are as expected.
7. Optionally, set the JAVA_HOME environment variable to point to the ZVM installation. The default
installation path is: /opt/zing/zing-jdk<version>/
8. Start your Java application using the newly installed ZVM.
Selecting and Validating Zing PAK Download Packages
1. Select the Zing Pilot Assist Kit download package that matches your Java version.
Only well-known standard kernels and operating system versions are supported with the Zing Pilot
Assist Kit. See the See "Zing Pilot Assist Kit Operating System Requirements" on page 167. DKMS
packages and Debian are not supported through the Zing Pilot Assist Kit.
The choices are:
l
Java SE 8.0
zing-pilotkit8-<version>.sh
Checksum (md5) xxxxxxxxxxxx
l
Java SE 7.0
zing-pilotkit7-<version>.sh
Checksum (md5) xxxxxxxxxxxx
l
Java SE 6.0
Azul Systems
Confidential and Proprietary
158
Specific Use-Cases: Zing Installation
zing-pilotkit6-<version>.sh
Checksum (md5) xxxxxxxxxxxx
2. Validate the checksum of your download package.
After downloading the Zing Pilot Assist Kit, at the command line, run the checksum command.
$ md5sum zing-pilotkit-<version>.sh
Sample response
xxxxxxxxxxxx zing-pilotkit-<version>.sh
3. Proceed to See "Zing Pilot Assist Kit Quick Start" on page 156 to install the Zing Pilot Assist Kit.
Understanding the system-administrator-todo-list File
When the zing-pilotkit-<version>.sh script runs, it performs an analysis on your system
settings to verify if they are compatible with Zing. It generates a system-administrator-todolist.txt file that lists any configuration changes required for Zing. The specific settings applied
depend upon the profile you selected: high-performance or low-latency. See "Appendix A – Zing
Installation Prerequisites Details" on page 318 for additional information on applying the recommended
configuration changes.
Sample system-administrator-todo-list.txt File
$ cat /tmp/pilot/system-administrator-todo-list.txt
#
# This is an auto-generated script based on <host> system validation
# results. It contains steps needed to be taken before the Zing
# installation can proceed. Please study the script carefully and follow
# the instructions provided.
#
# Hostname: <host>
# User: <username>
# Date: Wed Apr 1 10:52:23 PDT 2015
#
# OS: "CentOS release 6.4 (Final)"
# Installed memory: 129041 MB
# CPUs cores: 32
#
# *) Configure swappiness factor value
#
# It is recommended to set vm.swappiness factor value to 0 to minimise
# possible negative performance impact caused by processes swapping.
#
# Please add vm.swappiness=0 to /etc/sysctl.conf
Azul Systems
Confidential and Proprietary
159
Zing Installation and Administration Guide
#
# *) Swapping configuration
#
# Swap is enabled on the system. Make sure that it is not used when Zing
# VM is running as this could have serious performance impact.
#
# *) Increase vm.min_free_kbytes for better OOM resistance
#
# Zing requires vm.min_free_kbytes to be set to at least 1048576
#
# You can do it by adding line vm.min_free_kbytes=1048576 to
# /etc/sysctl.conf
#
# WARNING: Before changing vm.min_free_kbytes make sure that at least
# 2097152 Kb (2Gb) of free memory is available otherwise the server will
# hang.
# To display amount of free and used memory in the system use
# > free -k
#
# *) Configure processor.max_cstate kernel boot option
#
# For better performance ZVM expects processor.max_cstate to be set to 0
# Please add "processor.max_cstate=0" to the kernel boot options:
# Edit file /boot/grub/grub.conf
#
# *) Disable use of Transparent Huge Pages
#
# It is recommended that THP feature is turned off by default in Zing
# installation.
# To disable Transparent Huge Pages please add
# "transparent_hugepage=never" to the kernel boot options:
# Edit file /boot/grub/grub.conf
#
# *) Ensure your modifications of the /boot/grub/grub.conf are done
# correctly
#
# Boot arguments passed to your kernel should look like below:
#
Azul Systems
Confidential and Proprietary
160
Specific Use-Cases: Zing Installation
# kernel vmlinuz-2.6.32-358.el6.x86_64 ro root=/dev/mapper/vg_<host>-lv_
root rd_LVM_LV=vg_<host>/lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_
LV=vg_<host>/lv_swap rd_NO_MD SYSFONT=latarcyrheb-sun16
crashkernel=137M@0M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
intel_idle.max_cstate=0 processor.max_cstate=0 transparent_hugepage=never
#
# *) Reboot your system
#
# *) Install ZST
#
# > yum -y —nogpgcheck install /tmp/pilot/zing-zst-6b
#
# *) Install ZVM
#
# > yum -y —nogpgcheck install /tmp/pilot/zing-jdk1.7.0-<version>.x86
# _64.rpm
#
# *) Configure ZST memory driver
#
# > /usr/sbin/system-config-zing-memory
#
a) Type N for the defaults
#
b) Type J to specify the total Zing JVM heap requirement, 20G
# (assuming we are running the 20G JVM app, otherwise 10G)
#
c) Take the remaining defaults
#
# Ask user (<username>) to rerun pilot script in /home/<username>/
# directory:
#
# > cd "/home/<username>/tmp" && bash /home/<username>/zing-pilotkit7# <version>.sh -o /tmp/pilot low-latency
#
Understanding the Validation Profile
The validation profile is created when you run the zing-pilotkit-<version>.sh shell script.
The validation profile is a text file that contains a list of configuration validation checks. During the
pilot evaluation period, these checks are made by the ZVM each time it is started.
This file helps to ensure that your system stays properly configured during the entire pilot evaluation
period. If you provide logs to Azul, it also allows Azul Support Team to better understand the
environment in which the ZVM is running.
Azul Systems
Confidential and Proprietary
161
Zing Installation and Administration Guide
If a validation check does not pass, a warning message is printed and logged in the GC log file. The
warning does not prevent the ZVM from running. This information is also used when the systemadministrator-todo-list.txt file is created.
Pilot validation and options profile files are used during a pilot evaluation period. They are not
intended for production use. See "Converting Zing from Evaluation to Production" on page 167.
Sample validation File
#!/bin/bash
#
# Copyright 2015 Azul Systems, Inc. All Rights Reserved.
#
# Please contact Azul Systems, Inc., 1173 Borregas Avenue, Sunnyvale, CA
# 94089 USA or visit www.azulsystems.com if you need additional
# information or have any questions.
#
# Validation script for low-latency profile
#
# Available validations:
#
# verify_intel_idle # ensures that intel idle driver is switched off
# verify_processor_cstate # ensures that processor.cstate is 1
# verify_thp # ensures that Transparent Huge Pages are disabled
# verify_swappiness # ensures that vm.swappiness factor is <= 10
# verify_swap_usage # ensures that swap is not in use
# verify_min_free # ensures that vm.min_free_kbytes is at least 1G
# verify_zone_reclaim # ensures that vm.zone_reclaim is 1
# verify_zings_memory # ensures that Zing memory service was configured
#
verify_zone_reclaim
verify_swappiness
verify_swap_usage
verify_min_free
verify_intel_idle
verify_processor_cstate
verify_thp
Sample validation Script Output
$ java -jar Test.jar
Azul Systems
Confidential and Proprietary
162
Specific Use-Cases: Zing Installation
Zing VM Warning: Processing options filtering file:
/home/<username>/.zing/profile/options
ARTA: Listening for HTTP connections on 0.0.0.0:54189.
Starting /home/<username>/.zing/profile/validation interpretation...
* intel_idle.max_cstate=0 kernel boot option specified (+) ok
* processor.max_cstate=0 kernel boot option not specified (-) todo
* transparent_hugepage=never kernel boot option not specified (-) todo
* vm.zone_reclaim_mode == 0 [ actual: 0 ] (+) ok
* vm.swappiness factor is >= 0 and <= 10 [ actual: 60 ] (-) todo
* swap usage == 0 [ actual: 1136964 KB ] (-) todo
* vm.min_free_kbytes >= 1048576 Kb [ actual: 135168kb ] (-) todo
Warning:
Your system did not pass qualification tests.
Warning:
system.
Please review the list of failed validations and configure your
Warning:
Warning:
Validations definition:
/home/<user>/.zing/profile/validation
Warning: Zing startup configuration file:
HOME>/etc/startup/startup.conf
<ZING_
ZVM Startup File <ZING_HOME>/etc/startup/startup.sh execution time: 621
ms.
Understanding the Options Profile
The options profile and the contents of its corresponding options file is based on the profile option
you select when you run the zing-pilotkit-<version>.sh script. There are two options:
l
High performance
l
Low latency
The options file contains rules for altering the set of options passed to the ZVM through the Java
launcher command line and/or environment variables.
Initially this file contains a set of rules defined for the profile option you selected when the zingpilotkit-<version>.sh script is run. This file is modifiable and can be changed later.
The options file is a text file with a simple format:
[section-name]
option
...
There are three sections in the options file.
l
l
[ignore] -- lists options to be removed from the list of command line options.
[prepend] -- lists options to be added to the front of the list of command line options after
the ignore processing.
Azul Systems
Confidential and Proprietary
163
Zing Installation and Administration Guide
l
[append] -- lists options to be added to the end of the list of command line options after the
prepend and ignore processing.
The ignore list filters are applied to the those options provided on the command-line and those
specified using environment variables (_JAVA_OPTIONS/JAVA_TOOL_OPTIONS). The prepend and
append section contains options to prepend and/or append, respectively, to the command line.
This file is a convenient and recommended method for fine-tuning options used to run your application
during a Zing pilot evaluation period. Using this options file removes the need to modify any launcher
scripts that are used in your environment.
Sample options Profile
#
# Copyright 2015 Azul Systems, Inc.
All Rights Reserved.
#
# Please contact Azul Systems, Inc., 1173 Borregas Avenue, Sunnyvale,
# CA 94089 USA or visit www.azulsystems.com if you need additional
# information or have any questions.
#
#
# prepend section: enumerates which ZVM options to add to the front of
# the list of options created from those on the command-line and those
# specified using environment variables
#
[prepend]
#
# ignore section: enumerates which ZVM options to remove from the list of
# options created from those on the command-line and those specified
# using environment variables
# options added in the [prepend] and [append] sections are never removed
# from the command line
#
[ignore]
-XX:
# Ignore all –XX options (except those specified in this file)
#
# append section: enumerates which ZVM options to add to the tail of the
# list of options created from those on the command-line and those
# specified using environment variables and previously processed to
# add/ignore those options specified in the prepend and ignore sections
#
[append]
Azul Systems
Confidential and Proprietary
164
Specific Use-Cases: Zing Installation
-Xloggc:verbose.%p.gc
# Print garbage collection information and direct output to the file
# %p expands to the PID of the Zing VM process
-XX:+PrintGCDateStamps
# Print date stamps on garbage collection log lines
-XX:+PrintGCDetails
# Print detailed collector information during garbage collection
-XX:GenPauselessNewThreads=2
# Number of threads used by New Gen Collector
-XX:GenPauselessOldThreads=1
# Number of threads used by Old Gen Collector
-XX:ARTAPort=any
# Specify ARTAPort opened by ZVM (used by ZVision/ZVRobot)
-XX:+KeepSafepointsInCountedLoops
# Counted loops keep safepoint checks in their back edges
-XX:+MinimizeJNICriticalLock
# Minimize JNI_Critical lock usage in the JDK
-XX:+ScanSystemDictionaryBeforeUnload
# Use concurrent System Dictionary scan to minimize chance of long pause
# for unload phase
#-XX:+UseTickProfiler
# Generate Tick Profiler ticks (code profiling available in
# ZVision/ZVRobot)
#-XX:+CollectPollingOps
# Collect Polling Opportunity information.
# This option requires - XX:+UseTickProfiler
#-XX:PollingOpsMinThresholdMicros=50
# Keep polling opportunities that take longer than N micros
Azul Systems
Confidential and Proprietary
165
Zing Installation and Administration Guide
Configuring the Zing Pilot Assist Kit
Typically, the Zing Pilot Assist Kit does not require any additional configuration, however you have the
option to:
l
Change the location where the ZVM looks for the options and validation files
l
Change when the validation check results are reported: always or only when there is an error
l
Prevent the ZVM from running if there are validation errors
To apply Zing Pilot Assist Kit configuration changes, modify the ${ZING_JAVA_HOME}
/etc/startup/startup.conf file.
Sample startup.conf File
#
# Copyright 2015 Azul Systems, Inc. All Rights Reserved.
#
# Please contact Azul Systems, Inc., 1173 Borregas Avenue, Sunnyvale,
# CA 94089 USA or visit www.azulsystems.com if you need additional
# information or have any questions.
#
#
# validation.search_path
#
# ':' - separated list of paths to validation scripts
# The first found script is executed on VM startup
#
# validation.search_path = ${HOME}/.zing/profile/validation
# validation.search_path =
#
validation.search_path = ${HOME}/.zing/profile/validation
#
# validation.output
#
# defines a way of reporting validation script's output to user
#
# validation.output = always
# validation.output = onerror
#
validation.output = onerror
#
Azul Systems
Confidential and Proprietary
166
Specific Use-Cases: Zing Installation
# validation.terminate_on_error
#
# defines either terminate VM execution in case of non-zero exit code
# or not
#
# validation.terminate_on_error = yes
# validation.terminate_on_error = no
#
validation.terminate_on_error = no
#
# options.search_path
#
# ':' - separated list of options-filtering files to apply on VM
# startup Only first found file is processed
#
# options.search_path = ${HOME}/.zing/profile/options
# options.search_path =
options.search_path = ${HOME}/.zing/profile/options
Converting Zing from Evaluation to Production
When you have completed your pilot evaluation and are transitioning to using Zing for production use,
you must make the following changes to your environment.
1. Order and install your production license.
Zing does not continue to run on Evaluation licenses in a production environment.
2. Apply the information identified during your pilot and modify your launcher scripts and the
command line options you pass to the ZVM. Use the pilot profile options file,
~/.zing/profile/options, as the resource to guide you.
3. Remove, or optionally, move, the pilot profile directory and its files, ~/.zing/profile/. This
includes both the options and validation files.
Running the ZVM with a production license while pilot profile files are present causes a warning
on ZVM startup.
Zing Pilot Assist Kit Operating System Requirements
The components ZVM and ZST can be installed on machines running the following operating systems
using .rpm or .deb installers, as appropriate.
Notes
l
Ensure that OpenSSL 0.9.8 or later is installed.
l
On RHEL 5.x or CentOS 5.x systems, also install the Development Tools Package.
Azul Systems
Confidential and Proprietary
167
Zing Installation and Administration Guide
l
l
l
l
All RHEL installations require a registered copy of RHEL and a valid Red Hat Network
(RHN) account user name and password.
Ensure that binutils are installed.
In virtual machine environments, paravirtualization is not supported. Only HVM
(Hardware-assisted Virtual Machine) cluster instances are supported.
The Zing Pilot Assist Kit does not support DKMS or Debian installation.
The following tables list the supported operating systems, kernel versions, and for Amazon EC2
Instances, the Amazon Machine Image (AMI) identification code. Where multiple kernel versions are
listed, any from the list are supported with the associated operating system.
RHEL or CentOS Operating
System
Kernel Version
Supported As Of ZST Version
RHEL/CentOS 7
3.10.0-123
v5.7.6
RHEL/CentOS 6.6
2.6.32-504
v5.7.8
RHEL/CentOS 6.5
2.6.32-431
v5.7.3
RHEL/CentOS 6.4
2.6.32-358
v5.7.0
RHEL/CentOS 6.3
2.6.32-279
v5.7.0
RHEL/CentOS 6.2
2.6.32-220
v5.7.0
RHEL/CentOS 6.1
2.6.32-131
v5.7.0
RHEL/CentOS 6.0
2.6.32-71
v5.7.0
RHEL/CentOS 5.10
2.6.18-371
v5.7.6
RHEL/CentOS 5.9
2.6.18-348
v5.7.0
RHEL/CentOS 5.8
2.6.18-308
v5.7.0
RHEL/CentOS5.7
2.6.18-274
v5.7.0
RHEL/CentOS 5.6
2.6.18-238
v5.7.0
Azul Systems
Confidential and Proprietary
168
Specific Use-Cases: Zing Installation
RHEL or CentOS Operating
System
Kernel Version
Supported As Of ZST Version
RHEL/CentOS 5.5
2.6.18-194
v5.7.0
RHEL/CentOS 5.4
2.6.18-164
v5.7.0
RHEL/CentOS 5.3
2.6.18-128
v5.7.0
RHEL/CentOS 5.2
2.6.18-92
v5.7.0
Oracle Linux Server Operating
System
Supported As Of
Kernel Version
ZST Version
Oracle Linux Server
2.6.39-400.214.3.el6uek.x86_64
v5.7.10
Oracle Linux Server
2.6.39-400.214.4.el6uek.x86_64
v5.7.10
Oracle Linux Server
2.6.39-400.246.2.el5uek.x86_64
v5.7.10
Oracle Linux Server
2.6.39-200.24.1.el6uek.x86_64
v5.7.9
Oracle Linux Server
2.6.32-300.39.2.el6uek.x86_64
v5.7.9
Oracle Linux Server
2.6.32-504.el6.x86_64.x86_64
v5.7.9
Red Hat or CentOS
Operating System
Kernel Version
Supported As Of ZST Version
Messaging Realtime Grid (MRG)
RHEL, RHEL MRG, CentOS
3.6.11.2-rt33.39.el6rt.x86_64 v5.7.3
RHEL, RHEL MRG, CentOS
3.0.36-rt57.66.el6rt.x86_64
Azul Systems
v5.7.3
Confidential and Proprietary
169
Zing Installation and Administration Guide
SLES Operating System
Kernel Version
Supported As Of ZST Version
SLES 11 SP3
3.0.101-0.15
v5.7.6
SLES 11 SP3
3.0.101-0.21
v5.7.5
SLES 11 SP3
3.0.93-0.8
v5.7.3
SLES 11 SP3
3.0.76-0.11
v5.7.0
SLES 11 SP2
3.0.101-0.7.17
v5.7.4
SLES 11 SP2
3.0.13-0.27
v5.7.0
SLES 11 SP1
2.6.32.59-0.7
v5.7.3
SLES11 SP1
2.6.32.12-0.7
v5.7.0
Ubuntu Operating System
Kernel Version
Supported As Of ZST Version
Ubuntu 12.04
3.5.0-34-generic
Ubuntu 12.04
3.2.0-23-generic, 3.2.0-31-generic v5.7.0
Ubuntu 11.04
2.6.38-15-generic
v5.7.0
Ubuntu 10.4
2.6.32-38, 2.6.32-38-generic
v5.7.0
Amazon EC2 Instances
Kernel Version
v5.7.1
AMI ID
Supported As Of ZST
Version
Amazon Linux
3.4.37-40.44
ami-a73758ce
v5.7.0
Amazon Linux
3.4.43-43.43
ami-a73758ce
v5.7.0
Azul Systems
Confidential and Proprietary
170
Specific Use-Cases: Zing Installation
Amazon EC2 Instances
Kernel Version
AMI ID
Supported As Of ZST
Version
Amazon Linux
3.4.48-45.46
ami-a73758ce
v5.7.0
Amazon Linux
3.4.57-48.42
ami-a73758ce
v5.7.0
RHEL 7.0 – Amazon Cloud
3.10.0-123
ami-785bae10
v5.7.6
RHEL 6.5 - Amazon Cloud
2.6.32-431
ami-63b6910a
v5.7.3
RHEL 6.4 - Amazon Cloud
2.6.32-358
ami-9d0b64f4
v5.7.0
3.0.101-0.8-default
ami-e572438c
v5.7.5
3.2.0-54-virtual
ami-b93264d0
v5.7.3
SLES 11 SP3 Amazon
Cloud
Ubuntu 12.04 Virtual
Amazon Cloud
Azul Systems
Confidential and Proprietary
171
Zing Installation and Administration Guide
Zing Advanced Tuning
Zing Advanced Tuning is for expert Java programmers and experienced Zing users.
Zing Advanced Tuning: ReadyNow!
This guide provides a description of Zing's advanced tuning option, ReadyNow! that provides a set of
flags and options to help your Java applications be ready to perform real transaction processing at
designated start times.
Caution:
The suggestions presented in this guide are intended for expert users.
Applying these suggestions inappropriately can have unintended
consequences on your Java application performance.
The ReadyNow! feature is not an automated tuning tool. You must use
discretion when applying the ReadyNow! flags and options. Contact your Azul
Field Consultant or Azul Customer Support for additional information.
ReadyNow! Overview
Java applications commonly use a warm-up period to ensure that hot methods are compiled before real
transaction processing begins.
Zing now includes an enhanced version of jstat that displays summary statistics related to profile
log application. By using a profile log with jstat, you can ensure and verify that the application is
warmed up on a running system.
Note:
If your Zing release is previous to Zing 15.05.0, you must use
ProfilePrintReportrather than the enhanced version with jstat.
Unfortunately, if a condition inside the application is used to determine if a transaction is stored to a
database or a message processing system (such as an exchange), critical pieces of persistence
related code might not be compiled before real transaction processing begins.
The Zing ReadyNow! feature provides the following commands, flags and options to ensure your Java
application is ready for transaction processing:
l
CompilationPolicyChoice – Provides two cold method compilation policies: stackwalking policy (default) and a non-stack walking policy.
l
DynamicBranchEliminationLevel– Reduces unreached deoptimizations. Provides
configurable dynamic branch elimination to reduce ill-timed deoptimizations, such as rarely
taken but time-critical business logic.
Azul Systems
Confidential and Proprietary
172
Zing Advanced Tuning
l
UseEarlyClassLoading – Reduces unloaded / uninitialized deoptimizations. Loads
required classes before performing a C2 compilation.
l
EagerInitializationDuringEarlyClassLoading – Immediately initializes classes
that do not have a static initializer, which is typically 80 percent or more classes.
l
ProfileLogOut/In – Helps train your Java application to incorporate late compilations into
the entire compilation process. ProfileLogOut uses information from a log of profile
information while system is running and ProfileLogIn instructs Zing to use information
from an existing log.
Run ReadyNow!
To use the ReadyNow! flags and options:
1. Run selected options to identify possible problem areas. For example:
l
PrintCompilation, to identify late compilations.
l
TraceDeoptimization, to identify the location and reasons for deoptimizations.
2. Select and apply the Zing flags and options to your Java application runtime.
ProfileLog Use Case
Priming Java code for speed at market open is a key goal and using ProfileLog information to tune
opening code helps to achieve that goal. The practice of running simulated data to warm up the system
every morning runs the risk that the simulated data might initiate an unintended trade.
ReadyNow! ProfileLog support allows a VM to be warmed up simply by loading and initializing classes,
which is preferable to running simulated trades through a production system. You can profile the
opening code to determine which classes and methods have dependencies that slow down the opening
processing by either being unloaded or uninitialized and tune the java code.
Azul Systems
Confidential and Proprietary
173
Zing Installation and Administration Guide
Pre-JIT ProfileLog Use Case
To provide an instantaneous benefit from the ProfileLog, Zing includes a pre-JIT feature that allows the
information from the Profile Log to be loaded and initiated before main is started. This allows most
methods to be JIT-ed before the system ClassLoader starts.
Pre-JIT automatically calls classes from the bootstrap and system ClassLoaders.Currently, using
jstat with the profile log is limited to the bootstrap and system ClassLoaders.
The additional class loading, initialization, and compilation to support pre-JIT is beneficial for most
systems, but the feature does cause Zing to take longer to start. If the start-up delay is not desirable,
this functionality can be disabled using -XX:-ProfilePreloadClasses.
Run a Profile Restoration
Run a profile restoration using the -XXProfileLogIn flag. Adding jstat reports on how well the
restoration of the classes is working through loading and initialization before running main. After the
classes are ready, profile restoration can wait for the compilations to finish before running main -effectively, providing precompilation for most methods. Currently this functionality is limited to classes
loaded by the bootstrap and system ClassLoaders.
Note:
If your Zing release is older than Zing 15.05.0, you must use
ProfilePrintReport The labeling conventions in jstat are different from
those in ProfilePrintReport.
Display Summary Statistics Using jstat
Running ProfileLogIn/Out with jstat -profile {pid} {optional refresh
interval} provides enhanced jstat support that displays summary statistics related to the
profilelog feature and enables a pre-compiled JIT feature.
Using jstat with profile helps ensure and verify that the application is warmed up. Here is an
example of the code:
jstat -profile {pid} {optional refresh interval}
Summary Statistics - Example Output A
From reading the statistics, you can determine if the profile log was effectively applied and the system
is warmed up.
The following statistical values indicate a healthy system:
l
MatchedLoads is almost equal to Classes.
l
UnmatchedLoads is small relative to Classes (<5%).
l
C1Compiled is almost equal to C1Eligible (>95%).
Azul Systems
Confidential and Proprietary
174
Zing Advanced Tuning
l
C2Compiled is almost equal to C2Eligible (>90%).
l
ProfApplied is almost equal to ProfAvailable (>90%).
l
ClassErrors and MethodErrors are almost zero.
A few different scenarios interfere with using the profile log. For example, a high value for
UnmatchedLoads, ClassErrors, or MethodErrors requires further investigation. To see a break
down by error type, use jstat -profileerrors.
Summary Statistics -- Example Output B
If CollidingClasses or RepeatMatchedClassLoads is high (relative to Classes above) then...
- the same version of a class is being loaded by different ClassLoaders
OR
- the same class is being unloaded and reloaded
If UnresolvedClasses, UnresolvedMethods, or CollidingMethods is high (>10) then the
profile log is corrupt.
If UnmatchedLoads is high relative to Classes (>5%) then:
l
The original profile run did capture a significant sample of the application
or
l
There are too many classes that have changed for the profile to be useful -- this might occur
because of a recent library or JDK upgrade
For a more detail breakdown of problems use ProfilePrintReport.
Monitoring ProfileLog Effectiveness
Additional profile log information is available if you're not conducting JIT precompilation, using the
jstat command. To understand how well the information from the profile log is applied, Zing includes
reporting tools to assess the overall effectiveness and provides a detailed list of areas for improvement.
Using ProfileLog Information
In applications with a warm-up phase that does not compile all the necessary methods, using
ProfileLog can help train your Java application to incorporate late compilations into the entire
compilation process. Use the ReadyNow! ProfileLog to learn the deoptimizations encountered from
the previous run of the VM. It waits until all the necessary related classes are initialized before
compiling a method in C2.
When a previously compiled class is initialized, all of the methods declared inside the class that were
C1 compiled in the prior run will be C1 compiled. When all classes on which a method depends and
their declaring class are initialized, a profiled method will be C2 compiled.
Azul Systems
Confidential and Proprietary
175
Zing Installation and Administration Guide
To utilize ProfileLog, run your Zing Java application with the following 3 command options:
l
-XX:ProfileLogOut={file} - Tells Zing to produce a log of profile information while its
running.
l
-XX:ProfileLogIn={file} - Instructs Zing to use information from an existing log.
l
-XX:+/-ProfilePrintReport - Directs Zing to print report about how well the profile log
was applied at VM exit.
Run ProfilePrintReport carefully in a virtual environment. The virtual machine's
Note: exit routines will not be run when a process is terminated with a 'kill -9'. Using 'kill' or 'kill
-2' CTRL-C will produce the output to stdout.
The ProfileLogOut and ProfileLogIn options may be used together and can also point to the
same file. The current file information will be read in its entirety - before Zing starts to create a new log.
The ProfileLogOut is a record of the previous compilation and the deoptimization decisions from
the previous run. ProfileLogIn allows Zing to base its decisions on the information from a previous
run.
Run ProfileLog carefully in a production environment because it does incur some logging
Note: overhead in the Java threads.
Analyzing the Report
If a large number of methods or a critical method appears in the "Dependencies" list of a class in the
profile report, you may wish to manually advance the profile restoration process by loading and
initializing the class in question.
After the desired classes are loaded, run with ProfilePrintReport again to make sure that the
desired methods are no longer in any "Dependencies" list.
A method may still appear in a dependency list if it depends on more than one class. The unresolved
count next to the method indicates how many class dependencies have been and need to be satisfied.
The ProfilePrintReport Summary report:
l
l
l
Is organized by class and indicates how well the profile was able to be used. The profile
system matches classes and expects an EXACT match of the contents of the
corresponding class file.
Indicates precisely which parts of the saved profile were able to be restored and, when part
of the profile is not restored, it clearly indicates which class needs to be initialized and
loaded.
Describes the numbers of classes loaded and the types of loads that occurred. It also shows
the numbers of c1 and c2 compilable methods that were successful out of the total number
possible, and also reports on the amount of profiled methods and errors.
The ProfilePrintReport Summary example below shows that:
Azul Systems
Confidential and Proprietary
176
Zing Advanced Tuning
l
The profile log contains information about 9852 classes
l
The classes contain information on 3546 methods.
l
1435 methods have detailed profiling information, but only 95 had profiles applied.
l
2922 are eligible for c1 compilation, but only 435 were c1 compiled.
l
770 are eligible for c2 compilation, but only 95 were c2 compiled.
The ProfilePrintReport details as seen in the example below indicate what did not work:
Tuning Opening Java Code
Improve runtime performance by running a performance test using ProfileLogIn and
ProfileLogOut and develop a ProfileLog report on typical market initialization processing.
Analyze the ProfileLog Summary to interpret which classes and methods had dependencies that
caused incomplete compilation. Determine which classes and methods can be revised to prevent
common compilation errors such as unreached, null_check, or unloaded.
Azul Systems
Confidential and Proprietary
177
Zing Installation and Administration Guide
Ensure That Critical Methods are Compiled
Use a combination of the following techniques that are available to ensure critical methods are
compiled in a timely fashion:
l
Compilation Policy
l
Compilation Thresholds
l
Compiler API
Compilation Policy
The Zing ReadyNow! flag, CompilationPolicyChoice, provides two separate compilation policies
that are typically applied as a flag to the Java command:
l
Stack Walking CompilationPolicy
CompilationPolicyChoice=1 (Default)
The stack walking policy favors throughput over creating a timely (immediately beneficial)
compilation. The policy does this by walking up the stack when a hot method is identified to
find the furthest caller (direct or indirect) that is able to keep the hot method inline.
l
Non-Stack Walking CompilationPolicy
CompilationPolicyChoice=0
This Zing ReadyNow! non-stack walking policy favors timely (immediately beneficial)
compilations over throughput.
The low impact on throughput makes this policy appropriate for applications where it is not
possible to warm-up all the methods before transaction processing, even if the application
has demanding throughput requirements.
Though this policy favors timeliness over throughput, the impact on throughput compared to
the stack walking policy is usually small (-3% to +2%).
The non-stack walking policy can also increase code cache usage.
To apply the Zing CompilationPolicyChoice option:
1. Use the information from the PrintCompilation log to:
l
Determine the timing of method compilations.
l
Check how deep the average compiler queue is during warm up.
2. Set the CompilerThreads to a value high enough to support the average compiler queue. For
example:
-XX:CodeCacheOopTableSize=64m -XX:KlassTableSize=32m
XX:CIMaxCompilerThreads=12
-
See the Compilation Thresholds section below for additional options to complete your compilation
policy.
Azul Systems
Confidential and Proprietary
178
Zing Advanced Tuning
Compilation Thresholds
After the Zing CompilationPolicyChoice option has been applied, reduce the compilation
threshold, using the following steps.
1. Select one of the CompilationThreshold:
l
If critical methods are not being compiled during warm-up, then reduce
C1CompilationThreshold.
l
If appropriate for your Java application, also reduce the C2CompilationThreshold.
2. If you reduce the CompilationThreshold to compile earlier, use the Zing non-stack walking
compilation policy option, CompilationPolicyChoice=0.
Compiler API
Zing includes two separate just-in-time (JIT) compilers, C1 and C2. Prior to Zing 5.10, the JITs
occurred automatically and were controlled by a combination of policy and C1 and C2 compile
thresholds. As of Zing 5.10, JITs continue to happen automatically but they can also be controlled
manually through the compiler API. Compiles can be manually triggered using the Compiler class. The
methods provide the additional ability to select which specific JIT to trigger, using Compiler.command
to define “(c1)” or “(c2)”. Zing also accepts “(compile)”, in which case Zing compiles both C1 and C2.
Examples:
The following examples produce both C1 and C2 compiles for all the methods of
java.util.String:
Compiler.compileClass(java.util.String.class);
Compiler.compileClasses("java.util.String");
Compiler.command("{java.util.String.*}(compile)")
The following examples use Compiler.command to create just C1 or C2 compiles:
Compiler.command("{java.util.String.*}(c1)")
Compiler.command("{java.util.String.*}(c2)")
Note:
When using Compiler.compileClasses and Compiler.command:
l
The matched classes must already be loaded – and will be initialized as a result of compiling.
l
The pattern may contain leading and/or trailing wild-cards (*), but may not include a wild-card in
the middle.
The compiler API initializes the classes being compiled, but you can use
UseEnhancedClassResolution, UseEarlyClassLoading, and
EagerInitializeDuringEarlyClassLoading if using the API causes an increase
in "unloaded" and "uninitialized" deoptimizations.
Azul Systems
Confidential and Proprietary
179
Zing Installation and Administration Guide
Reducing Deoptimizations
All Java Virtual Machines use optimistic optimization to try improve throughput. Deoptimization occurs
when these optimistic optimizations need to be undone. Unfortunately, deoptimization can be a
problem for latency sensitive applications.
Even during warmup, JIT compiled code is deoptimized with some regularity. deoptimization logs
indicate that common causes are:
l
Unreached – a path is never taken during warmup
l
Unloaded/uninitialized class – a class that has not been resolved in the calling context uses a
previously untaken path
Zing's ReadyNow! provides a set of flags and options to help reduce commonly occurring types of
deoptimizations. These include:
l
DynamicBranchEliminationLevel
l
ImplicitNullChecks
l
UseEarlyClassLoading
l
EagerInitializationDuringEarlyClassLoading
l
UseEnhancedClassResolution
To apply deoptimizations:
1. Verify that deoptimizations are a problem.
l
Identifying Deoptimizations with PrintCompilation
l
Identifying Deoptimizations with TraceDeoptimization
2. Apply the appropriate adjustments as either a flag to the Java command or as an option to the
identified method.
l
Reducing Unreached Deoptimizations
l
Reducing Null_Check Deoptimizations
l
Reducing Unloaded Deoptimizations
l
Reducing Uninitialized Deoptimizations
Identifying Deoptimizations with PrintCompilation
Using the PrintCompilation flag helps identify if deoptimization is a problem.
To identify deoptimizations:
1. Review the PrintCompilation log.
2. Count the number of times each C2 method and C1 method was compiled.
3. Ensure the count is accurate.
Azul Systems
Confidential and Proprietary
180
Zing Advanced Tuning
l
Exclude OSR compiles (lines containing %) in the count.
l
Check for overloaded methods.
In the compilation log, only the name of the method is shown not the full signature. If the
number of C1 compilations exceeds the number of C2 compilations, then the method is
probably overloaded.
4. Determine if deoptimization is occurring.
If the number of C2 compilations is higher than the C2 compilations, then deoptimization is
probably occurring.
Identifying Deoptimizations with TraceDeoptimization
Running the TraceDeoptimization flag with your Java application, provides information about the
location and reason for deoptimizations.
Below is a sample output from a TraceDeoptimization log.
As shown in the output for thread[27434], each optimization may result in two or more lines in the
console log.
The first line starts with the Uncommon trap that occurred when the compiled code encountered a
trap. This line includes the top-level method being deoptimized and the reason for deoptimization.
The following one or more DEOPT REPACK lines correspond to the interpreter frames created during the
machine frame during deoptimization. Multiple interpreter frames are created if the deoptimization
occurred inside an inlined method.
To identify deoptimization types:
1. Review the TraceDeoptimization log.
2. Search for Uncommon trap entries.
a. Count the number of times each method was deoptimized. In the sample output, the Top-Level Deopted Method is
java.lang.String::equalsIgnoreCase.
Azul Systems
Confidential and Proprietary
181
Zing Installation and Administration Guide
b. Count the number of times each deoptimization reason occurs.
In the sample output, the Deopt Reason is unreached.
3. Search for occurrences where the inlining assumption is violated due to class loading. For example:
4. Identify and apply a deoptimization remedy corresponding to the deoptimization type. See the
following sections:
l
Reducing Unreached Deoptimizations
l
Reducing Null_Check Deoptimizations
l
Reducing Unloaded Deoptimizations
l
Reducing Uninitialized Deoptimizations
In Zing 5.10, the deoptimization log content and layout have a timestamp at the beginning of each line
and a compile_id at the end of the line as shown in the example below. The compile_id references
the ID from the PrintCompilation log.
Azul Systems
Confidential and Proprietary
182
Zing Advanced Tuning
Reducing Unreached Deoptimizations
The most common type of deoptimization for most applications is unreached. This deoptimization type
usually accounts for more than 80% of all deoptimizations.
This deoptimization occurs when C2’s dynamic branch elimination decides to not compile a portion of
Java code that is used later. In most applications, the majority of the unreached deoptimizations occur
at start-up and are benign.
However, some applications have rarely taken, but time critical business logic. With these types of
applications, dynamic branch elimination can result in ill-timed deoptimizations. To combat this
problem, Zing provides theDynamicBranchEliminationLevel flag and option. The
DynamicBranchEliminationLevel supported levels are:
2, conservative – Untaken branches at pointer comparison sites are eliminated. This includes
unvisited sites.
3, heuristic – Not implemented.
4, aggressive – Untaken branches at all comparison sites are eliminated. This includes unvisited
sites.
Using DynamicBranchEliminationLevel as a Flag
Zing's DynamicBranchEliminationLevel default flag is level 4, aggressive, because aggressive
branch elimination yields a significant increase in throughput.
Note:
Use caution if you reduce the elimination level to 2, conservative, across the entire
application because it might decrease throughput, from a small (1-3%) to very large
(20%) amount.
Using DynamicBranchEliminationLevel as an Option
If your applications use rarely executed, but time critical business logic, reducing the
DynamicBranchEliminationLevel on a per method basis to control compiler choices, is
recommended.
To help avoid unreached deoptimizations in business logic:
l
Use the DynamicBranchEliminationLevel option at a per method level in the
CompileCommandFile. For example:
option package/SomeClass::topLevelMethod
DynamicBranchEliminationLevel=2
l
When the DynamicBranchEliminationLevel option is used, you must also specify the
top-level method being compiled in the file. Do this even if an inlined method triggered the
deoptimization.
Azul Systems
Confidential and Proprietary
183
Zing Installation and Administration Guide
Using UseOldBranchProfileAdjustmentAtDeopt as a Flag
If an application has repeated “unreached” deoptimizations at the same BCI setting the 5.10
UseOldBranchProfileAdjustmentAtDeopt to false may allow the application to stabilize
sooner.
Reducing Null_Check Deoptimizations
A common type of deoptimization for many applications is a null_check deoptimization. This
deoptimization usually occurs when an implicit null check fails and the null pointer is handled by the
SEGV handler.
Using ImplicitNullCheck as a Flag
Zing provides an ImplicitNullCheck flag that turns on and off the null_check deoptimization.
Note:
Use caution when turning this flag off for the entire application, because it might
decrease throughput considerably.
Fortunately in most applications, the majority of null_check deoptimizations occur during warm-up
and are benign.
Using ImplicitNullCheck as an Option
If a small number of methods are experiencing post warm-up deoptimizations caused by null_check,
then it is useful to disable ImplicitNullChecks for only those specific methods. this is done in the
CompileCommandFile.
To disable ImplicitNullChecks for null_check deoptimizations:
l
Set the ImplicitNullChecks option at a per method level in the
CompileCommandFile. For example:
option package/SomeClass::topLevelMethod -ImplicitNullChecks
l
When the ImplicitNullChecks option is used, you must also specify the top-level method
being compiled in the file. Do this even if an inlined method triggered the deoptimization.
Reducing Unloaded Deoptimizations
In some applications, unloaded deoptimizations are common. This problem typically occurs in
applications with a warm-up phase that does not compile all the necessary methods.
To reduce unloaded deoptimizations, Zing provides the flag, UseEarlyClassLoading. This flag
loads required classes before performing a C2 compilation.
Reducing unloaded deoptimizations by combining UseEarlyClassLoading and
EagerInitializationDuringEarlyClassLoading can eliminate the need for using the
warmup period to run placeholder activity through the application to ensure that common paths are
compiled just in time. See Reducing Uninitialized Deoptimizations
Azul Systems
Confidential and Proprietary
184
Zing Advanced Tuning
Using UseEarlyClassLoading as a Flag
In benchmarks, using UseEarlyClassLoading had no impact on throughput. However,
UseEarlyClassLoading can simultaneously load a large number of classes.
The additional class loading can simultaneously invalidate a large number of class hierarchy analysis
(CHA) inlining decisions, that in turn, can result in a deoptimization storm. If the deoptimized methods
are still hot, the deoptimization storm is followed by compilation and does impact throughput.
If and when a deoptimization storm is occurring, look for Safepoint for Deopt patching lines
in the TraceDeoptimization log that contains a long method list. If compiling a specific method is
causing the problem, a CompileCommandFile option can be applied to that method to disable the
feature.
To disable UseEarlyClassLoading for deoptimization storms:
l
Set the UseEarlyClassLoading option at a per method level in the
CompileCommandFile. For example:
option package/SomeClass::topLevelMethod -UseEarlyClassLoading
l
When the UseEarlyClassLoading option is used, you must also specify the top-level
method being compiled in the file. Do this even if an inlined method triggered the
deoptimization.
Using UseEarlyClassLoading as an Option
If and when a deoptimization storm is occurring, look for Safepoint for Deopt patching lines
in the TraceDeoptimization log that contains a long method list. If compiling a specific method is
causing the problem, a CompileCommandFile option can be applied to that method to disable the
feature.
To disable UseEarlyClassLoading for deoptimization storms:
l
Set the UseEarlyClassLoading option at a per method level in the
CompileCommandFile. For example:
option package/SomeClass::topLevelMethod -UseEarlyClassLoading
l
When the UseEarlyClassLoading option is used, you must also specify the top-level
method being compiled in the file. Do this even if an inlined method triggered the
deoptimization.
Reducing Uninitialized Deoptimizations
In some applications, uninitialized deoptimizations are common. This problem typically occurs in
applications with a warm-up phase that does not compile all the necessary methods. Resolving
dependencies in the Java thread as compilation is being scheduled is key. It removes many trapping
situations and by combining eager initialization with eager resolution, traps can often be eliminated
completely, even for new objects and static field and method access.
Azul Systems
Confidential and Proprietary
185
Zing Installation and Administration Guide
Using UseEnhancedClassResolution as a Flag
To reduce uninitialized deoptimizations, Zing provides the flag,
UseEnhancedClassResolution. This flag reduces uninitialized deoptimizations when the
class is already loaded and initialized, but has not been resolved within a particular class
This optimization only applies to allocation sites and might not help applications that are deliberated
design to avoid garbage collection.
Using UseEnhancedClassResolution as an Option
The UseEnhancedClassResolution option can be used on per method basis in the
CompileCommandFile, but given the low impact on throughput, this is usually unnecessary.
Using EagerInitializationDuringEarlyClassLoading as a Flag
If UseEarlyClassLoading is already being used and UseEnhancedClassResolution failed to
reduce uninitialized deoptimizations, using the flag
EagerInitializationDuringEarlyClassLoading may help.
Zing Advanced Tuning: @Contended
This guide provides a description of Zing's advanced tuning use of the @sun.misc.Contended
annotation to reduce cache contention across multiple threads.
The suggestions presented in this guide are intended for expert users. Applying
these suggestions inappropriately can have unintended consequences on your
Java application performance.
Caution:
You must use discretion when applying the @Contended annotation.
Contact your Azul Field Consultant or Azul Customer Support for additional
information.
@Contended Overview
The Java SE 8 platform introduced [0] a new annotation called @Contended [1] to reduce "False
Sharing" [2],[3],[4]. False sharing is a term to describe performance degradation when several threads
are modifying independent variable(s) sharing the same cache line.
@Contended is applied as a conventional Java language annotation to fields in a class. It specifies that
some field(s) in an object are probably highly contended across processor cores.
At runtime the VM optimizes field layout. At the VM's discretion, those fields annotated as contended
are given extra padding so they do not share cache lines with other fields that are likely to be
independently accessed. This helps to avoid cache contention across multiple threads.
Applying @Contended to a class, field, or group:
Note:
l
Azul Systems
Does not force the VM to add padding or optimize the field layout in a specific way.
Confidential and Proprietary
186
Zing Advanced Tuning
Can substantially increase the size of a given object.
l
Because the increase in size can be double or more, use @Contended sparingly.
Zing provides an implementation of this feature for JDK 7.
Using @Contended Annotation
To use the @Contended annotation:
1. In your Java application, add the @Contended annotation to denote the specific classes and/or their
instance fields that are eligible for padding to avoid cache contention. See the following sections for
examples.
Marking a static field with @Contended is a non-operational (noop) action.
2. Select and apply the Zing flags and options at your Java application runtime.
Marking a Class as Contended
To mark a class as contended, precede the class with @Contended. In this example, both field1
and field2 are padded from the both sides.
@Contended
public class ContendedClass {
private int field1;
private Object field2;
}
Marking an Instance Field as Contended
To mark an instance field as contended, precede the field with @Contended. In this example, the
padding is applied to field1 only.
public class ContendedField {
@Contended
private int field1;
private Object field2;
}
Marking Multiple Fields as Contended
To mark multiple fields as contended, precede each field with @Contended. In this example, both
fields have separate padding.
public class ContendedFields {
@Contended
private int field1;
Azul Systems
Confidential and Proprietary
187
Zing Installation and Administration Guide
@Contended
private Object field2;
}
Marking Groups as Contended
To mark a group as contended, include the optional @Contended parameter in the @Contended
annotation to describe the equivalence class for contention group. In this example, The
contendedField1 and contendedField2 fields are padded from everything else, but are still
densely packed with each other.
public class ContendedGroups {
@Contended("updater1")
private Object contendedField1;
@Contended("updater1")
private Object contendedField2;
@Contended("updater2")
private Object contendedField3;
private Object plainField5;
private Object plainField6;
}
Zing @Contended Options
After applying @Contended annotation to your Java application, include the Zing launch and
configuration options in your Java command. The Zing @Contended options and flags are:
l
To enable (disable) @Contended annotation support.
-XX:+EnableContended
[<--default]
-XX:-EnableContended
l
To restrict (not restrict) @Contended to trusted classes only.
This applies to classes loaded by bootstrap or extension classloaders.
-XX:+RestrictContended
[<--default]
-XX:-RestrictContended
l
To specify the number of bytes used to pad the fields/classes marked @Contended.
Default value is 128. The value must be in between 0 and 8192 (inclusive) and be a multiple of
8.
-XX:ContendedPaddingWidth=128
Azul Systems
[<--default]
Confidential and Proprietary
188
Zing Advanced Tuning
@Contended References
The following links provide additional information about the general use of @Contended.
[0] http://openjdk.java.net/jeps/142
[1] http://gee.cs.oswego.edu/dl/jsr166/src/dl/sun/misc/sun/misc/Contended.html
[2] http://en.wikipedia.org/wiki/False_sharing
[3] http://mechanical-sympathy.blogspot.ru/2011/07/false-sharing.html
[4] https://blogs.oracle.com/dave/entry/java_contented_annotation_to_help
[5] http://mail.openjdk.java.net/pipermail/hotspot-dev/2012-November/007309.html
[6] http://shipilev.net/talks/jvmls-July2013-contended.pdf
[7] http://robsjava.blogspot.ru/2014/03/what-is-false-sharing.html
Azul Systems
Confidential and Proprietary
189
Zing Installation and Administration Guide
Starting or Stopping the Zing Services
The ZST services start automatically after installation. After configuration all ZST services are
restarted automatically. This includes Zing Memory.
It is recommended that you configure Zing memory to start with startup scripts to ensure Zing memory
can reliably allocate the required memory.
Zing Memory holds memory even after reboot. If you intend to run non-zing JDK's exclusively or rerun a
baseline, you must disable Zing to release the memory allocated to Zing.
The tasks in this section apply to standard Zing, DKMS, and Read-Only installations.
See the following:
l
Starting Zing Memory
l
Stopping Zing Memory
l
Starting Zing License Daemon
l
Stopping Zing License Daemon
Starting Zing Memory
Note
Anytime you manually modify the pmem.conf file, you must restart Zing Memory. When
you run system-config-zing-memory, the Zing Memory is automatically restarted.
To start Zing Memory:
1. Login to your system as root.
2. Ensure there are no running processes using Zing memory, type:
$ zing-ps
See Using Zing System Tools.
3. On the ZVM system, at the system prompt or through scripts, as root, execute the command:
For RHEL, SLES, or Ubuntu:
# service zing-memory start
If you receive an error indicating the service does not exist, run the following command.
l
For RHEL, SLES:
# /etc/init.d/zing-memory start
l
For Ubuntu:
# initctl start zing-memory
Azul Systems
Confidential and Proprietary
190
Starting or Stopping the Zing Services
Stopping Zing Memory
Note
ZVMs do not run without the Zing Memory.
To stop Zing Memory:
1. Login to your system as root.
2. Ensure there are no running processes using Zing memory, type:
$ zing-ps
See Using Zing System Tools .
3. On the ZVM system, at the system prompt or through scripts, as root, execute the command:
For RHEL, SLES, or Ubuntu:
# service zing-memory stop
Starting Zing License Daemon
To start Zing licensed service:
1. On the ZVM system, at the system prompt or through scripts, as root, execute the command:
For RHEL, SLES, or Ubuntu:
# service zing-licensed start
If you receive an error indicating the service does not exist, run the following command, then
reissue the start command:
l
For RHEL, SLES
# /etc/init.d/zing-licensed start
l
For Ubuntu:
# initctl start zing-licensed
Stopping Zing License Daemon
Note:
ZVMs do not run without the licensed service.
To stop Zing licensed service:
1. On the ZVM system, at the system prompt or through scripts, as root, execute the command:
For RHEL, SLES, or Ubuntu:
# service zing-licensed stop
Azul Systems
Confidential and Proprietary
191
Zing Installation and Administration Guide
Start the License Server
The license server runs continuously once started. If you need to restart it, perform the following.
1. From the command line where the license server is installed, run:
$ <zing_server_path>/bin/startserver
Where <zing_server_path> is the license server installation directory. For example:
$ ./bin/startserver
Stop the License Server
The license server runs continuously once started. If you need to stop the license server, perform the
following:
1. From the command line where the license server is installed, run:
$ <zing_server_path>/bin/stopserver
Where <zing_server_path> is the license server installation directory. For example:
$ ./bin/stopserver
Azul Systems
Confidential and Proprietary
192
Upgrading or Uninstalling Zing Components
Upgrading or Uninstalling Zing Components
This section provides the steps to uninstall or upgrade Zing components.
The tasks in this section apply to standard Zing and DKMS installations.
l
Uninstalling Zing System Tools
l
Upgrading Zing System Tools
Upgrading Zing System Tools
To upgrade ZST use the Linux service upgrade command.
1. Login to your system as root.
2. Download the latest ZST .rpm or .deb file.
3. Ensure there are no running processes using Zing memory.
All currently-running ZVM instances must be terminated before attempting to upgrade ZST. If this
is not done, the upgrade and/or memory preallocation might fail or result in incorrect
configuration.
$ zing-ps
See Using the zing-ps Tool in Using Zing System Tools.
4. Identify the installed Zing packages.
l
For RHEL or SLES:
$ rpm -qa|grep zing
Sample response:
zing-zst-5e.2.6.32-5.7.0.0.28.el6.x86_64
l
For Ubuntu:
$ dpkg -l|grep zing
Sample response:
zing-zst_5e.3.2.0.x86.64-generic-5.7.0.0.28
5. Run the upgrade command on your Zing ZST package:
l
For RHEL:
$ rpm -Uvh <packagename>
l
For SLES:
$ zypper install <packagename>
Azul Systems
Confidential and Proprietary
193
Zing Installation and Administration Guide
l
For Ubuntu:
$ dpkg -i <package name>
Uninstalling the Zing System Tools
Uninstalling ZST removes all the ZST elements, this includes the following:
l
Zing Memory
l
zing-ps
l
Zing license daemon
Note:
Before you can uninstall ZST, you must uninstall ZVM. Follow the steps below for the
supported sequence.
To uninstall Zing System Tools:
1. Login to your system as root.
2. Ensure there are no running processes using Zing memory, type:
$ zing-ps
See Using Zing System Tools.
3. Identify the installed Zing packages, use
l
For RHEL or SLES:
$ rpm -qa|grep zing
l
For Ubuntu:
$ dpkg -l|grep zing
Sample response:
zng-zst-5e.2.6.32-5.7.2.0.10.el6.x86_64
zing-jdk1.6.0-5.8.0.0-13.x86_64
4. Uninstall ZVM before you uninstall ZST. Run the appropriate remove or delete command.
See Installing the Zing Virtual Machine for description of how to uninstall ZVM software.
l
For RHEL:
$ yum remove zing-jdk<version>
l
For SLES:
$ zypper remove zing-jdk<version>
l
For Ubuntu:
$ dpkg -P zing-jdk<version>
Azul Systems
Confidential and Proprietary
194
Upgrading or Uninstalling Zing Components
5. Uninstall ZST. Run the appropriate remove or delete command.
l
For RHEL:
$ yum remove zing-zst-<version>
l
For SLES:
$ zypper remove zing-zst-<version>
l
For Ubuntu:
$ dpkg -P zing-zst-<version>
This completes the ZST uninstallation.
6. To uninstall the remaining Zing components:
l
Delete the Zing license file.
l
Uninstall the Zing Pool license server.
Upgrading the Zing Virtual Machine
To upgrade ZVM, uninstall all the existing Zing components, then perform a fresh ZVM install.
1. Login to your RHEL system as root.
2. Download the latest ZVM software .rpm, .deb, or .sh file.
3. Ensure there are no running processes using Zing memory.
All currently-running ZVM instances must be terminated before attempting to upgrade. If this is
not done, the upgrade and/or memory preallocation might fail or result in incorrect configuration.
Type:
$ zing-ps
See Using Zing System Tools.
4. Identify the installed ZVM packages.
l
For RHEL or SLES:
$ rpm -qa|grep zing
l
For Ubuntu:
$ dpkg -l|grep zing
Sample response:
zing-zst-<version>
zing-jdk1.6.0-<version>
5. Uninstall ZVM. Run the appropriate remove or delete command.
l
For RHEL:
$ yum remove zing-jdk<version>-<zvm_version>.<platform>
Azul Systems
Confidential and Proprietary
195
Zing Installation and Administration Guide
l
For SLES:
$ zypper remove zing-jdk<version>-<zvm_version>.<platform>
l
For Ubuntu:
$ dpkg -P zing-jdk<version>-<zvm_version>.<platform>
Alternatively, delete the ZVM installation directory.
6. Uninstall ZST. See Installing and Configuring Zing System Tools.
7. Install a compatible version of ZST. See Installing and Configuring Zing System Tools and the
Zing Release Notes.
8. Install the downloaded ZVM installation file. Use the appropriate operating system installation
command:
l
For RHEL:
$ yum localinstall –-nogpgcheck ./zing-jdk<version>-<zvm_
version>.<platform>.rpm
The contents of the RPM package are installed into the /opt/zing directory.
To examine the contents of the RPM package prior to installation, execute the rpm
command using the -qpl option. For example:
$ rpm -qpl ./zing-jdk<version>-<zvm_version>.<platform>.rpm
l
For SLES:
$ zypper install ./zing-jdk<version>-<zvm_
version>.<platform>.rpm
l
For Ubuntu:
$ dpkg -i ./zing-jdk<version>_<zvm_version>_<platform>.deb
Uninstalling the Zing Virtual Machine
To uninstall ZVM:
1. Login to your RHEL system as root administrator. Change to the /root directory.
2. Ensure there are no running processes using Zing memory, type:
$ zing-ps
See Using Zing System Tools.
3. Identify the installed Zing packages, use
l
For RHEL or SLES:
$ rpm -qa|grep zing
l
For Ubuntu:
$ dpkg -l|grep zing
Azul Systems
Confidential and Proprietary
196
Upgrading or Uninstalling Zing Components
Sample response:
zing-zst-<version>
zing-jdk1.6.0-<version>
4. Uninstall ZVM before you uninstall ZST. Run the appropriate remove or delete command.
See Installing the Zing Virtual Machine for description of how to uninstall ZVM software.
l
For RHEL:
$ yum remove zing-jdk<version>-<zvm_version>.<platform>
l
For SLES:
$ zypper remove zing-jdk<version>-<zvm_version>.<platform>
l
For Ubuntu:
$ dpkg -P zing-jdk<version>_<zvm_version>_<platform>
5. See Installing Zing Standalone Licenses or Installing and Using Pool Licensing for information on
removing your Zing Licenses and ZST.
Uninstalling the Zing Standalone License
To uninstall the Zing license package and license file:
1. Login to your system as root.
2. Ensure there are no running processes using Zing memory, type:
$ zing-ps
See Using the zing-ps Tool in Using Zing System Tools.
3. Delete the Zing license file from /etc/zing/directory.
$ cd /etc/zing
$ rm license
Azul Systems
Confidential and Proprietary
197
Zing Installation and Administration Guide
Zing Tools Management
Understanding Memory Use
For the purposes of this document the term ZVM instance refers to a ZVM that is running a
Java application.
Note:
This section describes how the memory resource is used and assigned in a environment.
Understanding Memory Allocation
The Azul platform uses the ZVM to run Java applications, similar to a typical JVM running a Java
application. The significant difference is how memory is handled. uses the ZST for managing the
memory of three components: Contingency memory, Pause Prevention memory, and the memory used
for Generational Pauseless Garbage Collection (GPGC).
Using with GPGC and Contingency memory means that your Java application is no longer limited to
the maximum heap size specified on the Java command line when you started your Java application.
As an additional resource, GPGC might temporarily use Pause Prevention memory to prevent pauses
that are caused by lack of available clean pages.
The following graphic shows the distribution of memory among the subsystem components.
Memory Use on System Running ZVM Instances
See the following topics for descriptions of memory:
l
Host System Memory
Azul Systems
Confidential and Proprietary
198
Zing Tools Management
l
l
Linux Kernel Memory
l
Linux Process Memory
l
JNI Memory
l
JVM Internal Memory
l
Other Linux Processes Memory
Zing Memory
l
Pause Prevention Memory
l
Contingency Memory for Elasticity
l
Reservable and Reserved Memory
l
Java Heap Memory
l
Temporary Java Heap Memory
Host System Memory
A Linux system,where the ZVM and ZST components are installed, has a certain amount of physical
memory (RAM). The operating system’s virtual memory system manages the underlying RAM and
satisfies requests made for memory when a process is launched and, as necessary, when the process
is running. The host system memory is used for running all the processes on the machine. This
includes the Linux kernel, Linux processes, and the ZVM instances running Java applications.
To avoid page swapping of Linux memory, ensure the amount of the operating system's virtual memory
in use is less than or equal to the size of the RAM. Should swapping occur during a Java heap garbage
collection, potentially long delays can occur as pages are swapped.
Linux Memory
Linux memory is the portion of the host system memory allocated for the Linux kernel, related Linux
system processes, and user processes. Default ZST installation allocates 25% of the total physical
memory to Linux memory.
Linux Kernel Memory
Linux kernel memory is memory required to run the Linux operating system. Linux kernel memory is
allocated from Linux memory.
Linux Process Memory
Linux process memory is memory required by the Linux system to support all the applications running
on the system. This includes other Linux processes and ZVMs running Java applications. Linux
process memory is allocated from Linux memory.
Azul Systems
Confidential and Proprietary
199
Zing Installation and Administration Guide
JNI Memory
Java applications use the Java Native Interface (JNI) to access native code. If the Java application
uses JNI, any requests for non-Java heap memory from the native code allocates memory from the
Linux process memory.
There are two ways an application can allocate JNI memory. One is through JDK Java classes that call
JNI functions. The other is through user-defined application classes that call user-defined JNI
functions.
JVM Internal Memory
JVM Internal memory is the portion of the Linux process memory used to support the ZVM instances
running Java applications. JVM internal memory contains the ZVM executable code, thread stacks,
and data structures for Zing's Generational Pauseless Garbage Collection. The amount of memory
being consumed at any moment varies depending upon the ZVM instance and Java application’s
requirements.
Other Linux Processes Memory
If any other processes are running on the system, such as MySQL or Apache, these processes use
Linux process memory. These other Linux processes running on the system compete for memory and
reduce the amount of Linux memory available to the ZVMs running Java applications.
Zing Memory
The ZVM instance uses Zing memory to allocate the memory required for the Java heap. In addition,
Zing memory provides resiliency in the event of out-of-memory conditions that can occur when a ZVM
instance running a Java application temporarily requires more memory than it was initially allocated. An
example of this requirement might occur when a Java application’s workload spikes in an enterprise
environment. The Zing is distributed between three memory pools:
l
Reservable memory
Includes memory for each ZVM instance's Generational Pauseless Garbage Collection Java
heap
l
Contingency memory for elasticity
l
Pause Prevention memory
Contingency memory and Pause Prevention memory are each given a percentage of the total
Zing .
The total amount of Zing is the amount assigned during ZST configuration. This can be allocated in one
of two ways:
l
As a percentage of the total system memory. The default is 75%.
Azul Systems
Confidential and Proprietary
200
Zing Tools Management
l
As a specific amount of memory in MB, where the value is the sum of the Java heap sizes for
the Java instances you expect to run plus any additional Contingency and Pause Prevention
memory you would like to allocate.
Estimating Zing Memory Allocation Example
The percentages used in this example are based on the default values in the configuration file
generated by the system-config-zing-memory wizard. If you want to allocate different
percentages, you can edit and customize the values in the configuration file. The key relationship to
remember is that the amount of Reservable memory in the system must always be greater than or
equal to the sum of the –Xmx values of all concurrently running ZVMs on the system.
When estimating the amount of Zing to allocate:
l
Consider:
Zing = Reservable memory + Contingency memory + Pause Prevention memory
Reservable memory = Zing – (Contingency memory + Pause Prevention memory)
Note:
Reservable memory can be used for a single ZVM instance or multiple ZVM
instances.
Maximum ZVM instance heap memory (-Xmx) = Available Reservable memory
Available Reservable memory = Reservable memory – Sum of Java VM heap memory (Xmx) for all currently running ZVM instances
l
Generally:
To estimate the amount of Zing needed, for each ZVM instance that will be running
concurrently, add the amount of Java heap memory (-Xmx) required; then add the memory
required for Pause Prevention and Contingency memory (10% of total Zing ).
l
For example:
If you typically run 5 ZVM instances and each is allocated 20 GB of memory using the –
Xmx command line option then you need a minimum of 111 GB of Zing . 100 GB is used by all
the ZVM instances and 11 GB is used for the Pause Prevention and Contingency memory.
The following is an approximate breakdown of how the system's memory is used when there is a single
ZVM instance running.
100% host system memory is shared between:
25% to Linux memory
75% to Zing
100% Zing is shared between:
5% Pause Prevention memory for GPGC
Azul Systems
Confidential and Proprietary
201
Zing Installation and Administration Guide
5% Contingency memory
90% Reservable for ZVM instances which is shared between:
Every Java heap (–Xmx) allocation for each ZVM instance running a Java application
For example:
If total host system memory is 100 GB.
25 GB is allocated to Linux memory
75 GB is allocated to Zing
7.5 GB (10%) is allocated to Pause Prevention and Contingency memory
3.75 GB for Pause Prevention memory
3.75 GB for Contingency memory
67.5 GB is available as Reservable memory
Every ZVM instance running a Java application is allocated memory from Reservable
memory based on the value of the -Xmx command line option.
Pause Prevention Memory
For other JVMs, when object allocation rates suddenly increase, the garbage collector can have a
difficult time keeping up with the increased rate of object creation. The result is that the other JVMs
need to pause all work in the process to do a garbage collection. This causes an increase in Java
application pause times. A common work-around to minimize the length of garbage collection pauses is
to specify a small Java heap size – the theory being that a smaller Java heap requires less time to
collect. Unfortunately, when the rate of work goes up, more live objects are in the heap when the heap
fills up and the amount of time required for the garbage collection is proportional to the number of live
objects in the Java heap. The pauses are still long.
As part of Zing's Generational Pauseless Garbage Collection (GPGC), Zing sets aside a percentage of
the Zing for Pause Prevention memory. When there is a sudden increase in the ZVM instance can use
the Pause Prevention memory to efficiently keep up with and sustain the throughput during the heavy,
unexpected workloads. Using memory from Pause Prevention memory gives the ZVM instance's
GPGC additional time to adjust to and keep up with suddenly increased allocation rates. This ensures
that the garbage collector does not stop or pause a ZVM instance running a Java application.
Having Pause Prevention memory ensures that:
l
Zing delivers consistent response times even under sudden high loads.
l
Zing eliminates garbage collection pauses.
Note:
Azul Systems
Pause Prevention memory is used seamlessly by the Generational Pauseless
Garbage Collection algorithm. There are no settings you need to change or adjust.
Confidential and Proprietary
202
Zing Tools Management
Using Pause Prevention memory is not a warning or error condition, and is not
reported to or visible to you or the application.
Contingency Memory for Elasticity
The -Xmx command line option of the Java application launch command specifies the amount of Java
heap memory made available to a ZVM running a Java application. ZVM instances can require
additional Java heap memory for assorted reasons, including unplanned events and incremental
memory needs.
To help ensure that ZVM instances survive peak loads and to prevent out of memory errors, Zing sets
aside a percentage of the Zing for Contingency memory. Contingency memory is a backup reserve that
Zing uses to temporarily allocate additional memory to any ZVM instance running a Java application.
The ZVM instance uses Contingency memory to manage memory requests for space in the ZVM
instance’s Java heap.
Memory from Contingency memory is elastically allocated, growing and shrinking as the needs of ZVM
instances fluctuate:
l
Using Contingency memory allows ZVM instances to be launched without extensive, up-front
JVM or garbage collector tuning.
l
ZVM instances are allowed to grow or shrink based on actual needs.
l
If sufficient Contingency memory is available, ZVM instances are allocated extra memory,
allowing them to continue processing without triggering an out of memory failure.
l
The ZST tracks application memory usage and, as soon as the application decreases its live set
Java heap object requirements, the ZST returns unused memory back to Contingency memory.
l
Applications can run with larger heap sizes to improve application performance. The heap size
is no longer limited in an attempt to prevent potential garbage collection pauses.
l
For information purposes only, Zing Vision and ZVRobot provide visibility to the actual resource
usage.
Contingency Memory and Generational Pauseless Garbage Collection
The ZVM’s ability to temporarily increase the Java heap memory assigned to a process beyond the
originally specified -Xmx value is unique to the Zing Java platform. The Contingency memory is only
available with a ZVM instance that has Generational Pauseless Garbage Collection (GPGC) enabled
on a host running ZST. The only limits of Contingency memory use are:
l
How much unused memory remains in Contingency memory
When a ZVM instance running a Java application uses memory from Contingency memory,
the ZVM instance automatically increases the frequency of garbage collections to ensure
memory is available for object allocation. If a ZVM instance constantly uses Contingency
Azul Systems
Confidential and Proprietary
203
Zing Installation and Administration Guide
memory, adjust the heap size when you restart the ZVM instance. This adjustment can
prevent spending excessive time in garbage collection.
l
The ZVM instance’s maximum Java heap memory limit
Use the -XX:JavaMemMax command line option to specify the Java heap memory limit.
To prevent use of Contingency memory for a ZVM instance, set the -XX:JavaMemMax value
equal to the -Xmx value.
Sharing Contingency Memory and Memory Leaks
Contingency memory is a reserve memory of memory for use in temporary high load situations. Do not
depend on Contingency memory when calculating the -Xmx value for ZVM instances running Java
applications.
Contingency memory is shared among all Java processes on the system. It can only be effective in
preventing processes from terminating with an out of memory error if the increases in memory
requested by each of the ZVM instances are kept to a minimum.
Once a ZVM instance is using memory from Contingency memory, the application is at some risk
because there are no guarantees that additional Contingency memory will be available when needed.
Applications that experience memory leaks can eventually exhaust the Contingency memory,
potentially affecting other Java processes running on the same system that might also need
Contingency memory.
Reservable and Reserved Memory
The Reservable memory is the memory from which specific allotments are assigned to each ZVM
instance running a Java application. Allotments of Reservable memory are assigned to new ZVM
instances until the available Reservable memory is insufficient to start another ZVM instance.
Reservable memory comprises the bulk of the Zing .
Reserved memory is the amount of memory initially allocated to a specific ZVM instance when the
instance starts. This is the Java heap memory. The Reserved memory is reserved for each JVM
instance. It guarantees space to store a minimum live set for each ZVM instance plus the space to
support the object creation required to support the work done in the application. The live set is the set of
objects in the Java heap referenced by the Java application. The live set can also be described as the
amount of memory remaining in the heap after a simultaneous new and old generation garbage
collection.
Multiple ZVM instances can be started on a system, each being assigned its allocated amount of
Reservable memory. When a ZVM instance exits, its Reservable memory is released back to the
system’s Reservable memory and is available for other ZVM instances.
The amount of memory allocated to a ZVM instance when it starts is determined by the value of the –
Xmx Java heap size command line option.
Azul Systems
Confidential and Proprietary
204
Zing Tools Management
Estimating the Number of Possible Concurrent ZVM Instances
When estimating the number of ZVM instances running Java applications that can run concurrently on
a system, determine the amount of Zing available, subtract the percentage of Zing allocated to Pause
Prevention and Contingency memory, and then divide the remaining amount of Zing by the typical Java
heap (-Xmx) memory that would be set for your ZVM instances.
For example:
The allocated Zing is 100 GB. The memory allocated to the Pause Prevention and Contingency
memory is 10 GB. The remaining available Zing , the Reservable memory, is 90 GB.
Total the -Xmx values you will use for each of the Java applications you want to run.
Remember, in a Zing environment, the -Xmx value can be large because there is no concern for
garbage collection pauses.
So, for this example, in a Zing where you want to maximize your Java application performance,
set -Xmx to 20 GB for each ZVM instance. This enables running 4 concurrent ZVM instances
(4 ZVM instances x 20 GB in -Xmx = 80 GB consumed, with 10 GB left over from the 90
available).
Note:
In addition to the number of concurrent ZVM instances, account for the number of
available cores. Do not expect multiple busy ZVM instances to run well on a limited
number of cores. Distribute ZVM instances over the available host systems.
Java Heap Memory
Java heap memory is the amount of memory dynamically allocated to a ZVM instance running on the
system. Objects in Java heap can be shared between threads.
The Java heap is the working memory for the instance running the Java application and its size is
specified by the -Xmx command line option. It is the amount of memory the ZVM instance needs to
run the Java application, not includingnot including Pause Prevention and Contingency memory.
The -XX:JavaMemMax options allows you to set a maximum amount of memory usable for the Java
heap. If this option is not set, the maximum is unlimited.
When determining Java heap settings, consider the following:
l
l
A larger Java heap
l
Allows more objects to be created
l
Takes longer to fill
l
Allows the ZVM instance to run longer between garbage collection (GC) cycles
l
Consumes less CPU for garbage collection
A smaller Java heap
Azul Systems
Confidential and Proprietary
205
Zing Installation and Administration Guide
l
Holds fewer objects
l
Fills more quickly
l
Garbage is collected more frequently
l
May lead to out-of-memory errors
l
Consumes more CPU for garbage collection over time
Temporary Java Heap Memory
When needed to support the Java application, Zing temporarily allocates memory from Contingency
memory and/or Pause Prevention memory. See Contingency Memory for Elasticity and Pause
Prevention Memory.
Heap Memory in 32-Bit vs. 64-Bit Systems
Zing requires 64-bit systems. So, when determining the amount of Java heap memory to allocate, remember there is a difference in memory use in a 32-bit or 64-bit system.
Java heap memory requirements for 64-bit systems are generally 1.5 times greater than 32-bit
systems. Though primitives such as integers use 32-bit words on either a 32-bit or 64-bit system, Java
object headers and pointers use 32-bits on a 32-bit system and 64-bits on a 64-bit system. This applies
to all Java environments, whether they are running HotSpot JVMs or JVMs through a ZVM instance..
The result is that proportionally more memory is used to accommodate the same number of objects on
a 64-bit system. Increasing the efficiency of the garbage collector on a 64-bit system requires additional
space relative to a 32-bit system.
The drawing below shows the relative scale for the cost of 32-bit and 64-bit words in Java objects.
Notice, in the brown label area, how the same number of words consumes more memory for the 64-bit
system. If no additional Java heap memory is allocated when using a 64-bit system then the available
Java heap memory and the number of objects that can reside in the Java heap are reduced.
Azul Systems
Confidential and Proprietary
206
Zing Tools Management
Memory Word Consumption in 32-bit and 64-bit Systems
Setting Reservable Memory
When you are configuring your Zing platform to run your ZVM instances, you can use the default
options or modify memory allocation. The following is a list of considerations when you modify the
memory allocation from default values. See Installing and Configuring Zing System Tools for
information on how to change the default memory configuration.
Zing Allocation
Apart from Reservable memory for the ZVM instances, consider the memory required for Pause
Prevention and Contingency memory. These typically constitute 10% of the total memory available for
the Java heaps of ZVM instances; Java heaps. These percentages can be modified. Refer to Installing
and Configuring Zing System Tools for information on how to configure and modify these percentages.
Configuring -Xmx<val>
l
As described in the previous section, Heap Memory in 32-Bit vs. 64-Bit Systems, running Java
applications on 64-bit machines requires more Java heap.
l
Concurrent garbage collectors need larger heaps than stop-the-world garbage collectors
because concurrent garbage collectors allow application threads to actively allocate objects
while the garbage collector is reclaiming pages.
Azul Systems
Confidential and Proprietary
207
Zing Installation and Administration Guide
l
In general, specifying 2.5x the -Xmx value used with other garbage collectors is a good
starting point. Using Zing, you no longer need to worry about GC pauses that result from
garbage collecting very large Java heaps.
l
The ZVM component supports Java heap sizes up to 338 GB.
Configuring -XX:JavaMemMax=<val>
-XX:JavaMemMax controls the maximum amount of memory that can be allocated from Reservable
and Contingency memory. Allocation from Pause Prevention memory is not limited by this option.
As displayed by the command zing-ps –acct, the -XX:JavaMemMax sets the maximum for
account 2 (heap). Pause Prevention memory is allocated from account 3 (pause_prevention), which
has a different maximum. Setting the maximum for account 3 is not currently supported.
Note:
When -XX:JavaMemMax=<val> equals -Xmx<val>, then the ZVM instance is
not allowed to use any Contingency memory.
Refer to Using the Zing Virtual Machine for more information on using the command line options.
Using Zing Vision
Zing Vision (ZVision) and Zing Vision Robot (ZVRobot) provide ZVM process profiling and diagnostics.
This section describes configuring and using the Azul Systems Zing, Zing Vision (ZVision) profiling and
diagnostics tool for the Zing Virtual Machine (ZVM).
Note
The use of ZVision should be taken under careful consideration since access to
sensitive performance data should be restricted following company policy and
security recommendations.
See Using the Zing Virtual Machine for more information about running Java applications using the
ZVM.
Zing Vision Overview
ZVision provides programmers and deployment engineers with information about both a running
application and the underlying ZVM. It also helps users tune command line options and code to
optimize performance using the following capabilities:
l
Complete internal and environment settings inventory
l
Statistical, flat profiling of processor core usage from the thread level to run-time tasks down to
the method level
Azul Systems
Confidential and Proprietary
208
Zing Tools Management
Take two data samples within a period of time (for example, in a 20 sec. interval) to calculate
performance counter increments.
Azul Customer Support
If providing data for problem resolution for a case with Azul customer support, capture the requested
data in its entirety and send it by e-mail or FTP upload for analysis, including the timestamp (if
available).:
Statistical Tick Profiler
The statistical tick profiler displays how processor core time is spent between various software
execution units (for example, threads, methods, and run-time tasks). The Zing tick profiler differs from
other profilers as follows:
l
It is a statistical, flat (not tree-based) profiler. It does not capture every processor core cycle.
Instead, for every n processor core cycles, it records where in the ZVM the processor core was
running (this is the program counter). It then uses this data to build a flat, non-nested profile.
l
The captured data represents a specific time period.
l
It profiles both within and outside the Java code. Most Java profilers use byte code
instrumentation for Java code.
l
It does not profile when threads are blocked on locks. To retrieve information on monitors, go to
the Monitors window.
The statistical tick profiler is displayed in the Tick profile window accessed from the CPU tab (see
CPU Tab for detailed information about the Tick profile window).
Enable or Disable Tick Profiler
The Tick Profiler is enabled and disabled using the Java command line option, UseTickProfiler.
Default for UseTickProfiler is false, which means the Tick Profiler will not collect profiling
information.
l
To enable the Tick Profiler, use -XX:+UseTickProfiler. (set to true)
l
To disable the Tick Profiler, use -XX:-UseTickProfiler. (default, set to false)
Sorting Tables
Variable size tables can be sorted by column. Click the column head to sort in ascending order. Click
the column head again to resort in descending order. Fixed tables (for example, the GC Summary
Window) cannot be sorted.
Configuring ZVision
This section describes how to configure ZVision. Topics include:
Azul Systems
Confidential and Proprietary
209
Zing Installation and Administration Guide
l
ZVM Command Line Options
l
Configuring the ZVM Host Port
l
Configuring the ZVision Server
ZVM Command Line Options
The ZVM command line options listed in the table below are used to configure ZVision to provide
profiling and diagnostics information when running a Java application on a ZVM application host.
ZVM Configuration Command Line Options
Java Command Option
Description
Enables ZVision and allows the ZVision server to
listen on the specified port of the host where the ZVM
is running. Valid options include:
l
off – disables ZVision (default).
l
<port> – port number for ZVision to use on the
host.
-XX:ARTAPort=
{off|<port>|any}
l
any – ZVision uses any free port. This setting is
useful when starting multiple instances of ZVision
for multiple VMs.
Supply a port number or set this option to any to
enable ZVision.
The ARTA (Azul Run Time Analysis) port provides
internal API communication between ZVision and
ZVM.
-XX:PrintGCHistory=<number
of historical PGC cycle
reports>
Set the number of historical PGC cycle reports to
save for display in ZVision (default =50).
Set the number of historical PGC warnings to save
XX:GCWarningHistory=<number
for display in ZVision (default =50).
of historical PGC warnings>
Configuring the ZVM Host Port
To configure ZVision on a ZVM application host:
1. Log on to the ZVM application host where the Java workload will be run.
2. Add the following Java command line option to your Java application launch command:
-XX:ARTAPort=<port>
Azul Systems
Confidential and Proprietary
210
Zing Tools Management
where:
<port> is the port ZVision server use to connect on the ZVM.
The Azul Real Time Analysis port (ARTA port) provides internal API communication between
ZVision and ZVM.
See Zing Virtual Machine Options in Installing the Zing Virtual Machine for more information on ZVM
command line options.
Configuring the ZVision Server
The ZVision server is included with the ZVM software. The ZVision server does not need to be run on
the same host where the application is running, it only requires a network connection to the host where
the application is running.
To configure the ZVision server:
1. Log in to a host where the ZVision will run.
2. Set the JAVA_HOME variable (or equivalent) so that the Java call points to the parent directory of
the ZVM software. For example:
JAVA_HOME=<ZVM install location>/<ZVM top-level directory>
3. Type the following Java command line:
$JAVA_HOME/bin/zvision -p <port>
where:
<port> is port number that is used by a browser to connect to ZVision (default: 8088)
When the ZVision server starts successfully, the port number that is used by the browser to connect to
ZVision is displayed on the command line. For example:
ZVision server listening on port 8088
See Accessing ZVision for details about accessing ZVision from a browser session.
Note
See Advanced Configuration Options for more information about ZVision server command line options.
Accessing ZVision
Before you can use ZVision to view ZVM data:
l
Configure the ZVM ARTA port. See Configuring the ZVM Host Port
l
Configure the ZVision listening port. See Configuring the ZVision Server
To access ZVision:
1. Start a browser session and enter the URL of the ZVision server:
http://<ZVisionserver>:<port>
where:
Azul Systems
Confidential and Proprietary
211
Zing Installation and Administration Guide
l
<ZVision_server_IP_address> is the host name or IP address of the ZVision
server
l
<port> is the ZVision server port number that accepts connections from a Web
browser
For example: http://host123:8088
See Configuring the ZVision Server for detailed configuration information.
2. On successful connection to the ZVision server, the ZVision login window appears.
Enter the following information:
l
ARTA host – ZVM server host name or IP address
See Configuring the ZVision Server.
l
ARTA port – ZVM application host port specified with the -XX:ARTAPort command
line
See Configuring ZVision for a Host.
A ZVision session window is displayed.
ZVision Interface
The ZVision interface consists of a series of tabbed pages and corresponding windows. Select a tab to
access additional windows for a given component.
Azul Systems
Confidential and Proprietary
212
Zing Tools Management
The following table describes the organization of ZVision tabs. The rest of the section provides more
information about the windows that comprise each ZVision tab.
ZVision Tabs
Tab
Overview Tab
Azul Systems
Window
Summary
Configuration
Displays information for all process settings for the
ZVM.
Environment
Variables
Displays information for all environmental variables set
by the shell that launched the ZVM front end.
HotSpot Flags
Displays information for all of the HotSpot flags that
the ZVM uses.
Statistics
Displays information about running applications.
Settings
Displays configuration to enable GC logging, control
level of detail logged, and destination of the output.
Groups
This feature is not supported at this time
Login
is not supported at this time.
Features
Enables or disables user access to restricted ZVision
Confidential and Proprietary
213
Zing Installation and Administration Guide
ZVision Tabs
Tab
Window
Summary
features.
List
Displays all threads with links to detailed information
for each thread.
Stack Trace
Lists all threads with information whether each one is
executing or blocked.
Deadlocks
Lists detected deadlock cycles.
Contention
Displays statistics for each monitor (sorted by total
time held).
Tick Profile
Displays information that shows how percentage of
processor core time used for an interval of recorded
time.
Meta Ticks
Lists the time it takes to complete logical tasks.
Event Tracker
This feature is not supported at this time.
Summary
Displays a summary of total memory usage and
Contingency memory usage.
GC Summary
Displays a cumulative summary of recent GC cycle
history.
GC History
Displays a detailed summary of recent GC cycle
history (one row per cycle).
Browse Object
Displays information about object, instance, method,
or class.
Allocated
Objects
Displays information on allocated objects by object
type.
Live Objects
Displays information on class instances.
Compilers Tab
Tasks
Displays the status of Server and Client compiler
tasks.
Applications
Tab
Class List
Displays the application classes and Java class
libraries loaded in the ZVM.
Threads Tab
CPU Tab
Memory Tab
Azul Systems
Confidential and Proprietary
214
Zing Tools Management
Overview Tab
The Overview tab of ZVision contains the following windows:
l
Configuration
l
Environmental Variables
l
HotSpot Flags
l
Statistics
l
Settings
l
Groups
l
Login
l
Features
Configuration Window
The Configuration window provides a detailed view of all the process settings for the ZVM.
The following table describes the Configuration window elements.
Configuration Window Elements
Element
Description
Version
The ZVM version.
Command line
The list of classes and command line options used to launch a Java
Azul Systems
Confidential and Proprietary
215
Zing Installation and Administration Guide
Configuration Window Elements
Element
Description
application.
Java Home
The Java Home path.
Class path
The path to the class.
Boot class path
The path to the boot class.
Extensions
directories
The directories for extensions.
Endorsed directories
The directories for classes implementing newer versions of endorsed
standards.
Library path
The native library path.
Boot library path
The boot library path.
ARTA Version
The version of the instrumentation in the ZVM that provides data to
the ZVision server.
Open file descriptors
The limit in bytes for open file descriptors for the ZVM front end.
Core file size limit
The maximum size in bytes for a core dump file.
Environment Variables Window
The Environment Variables window provides a detailed view of all environmental variables set by the
shell that launched the ZVM. The Environment Variables window is displayed only if it is enabled in
the Features window (see Features Window for more information).
Azul Systems
Confidential and Proprietary
216
Zing Tools Management
The following table describes the Environment Variables window elements.
Environment Variables Window Elements
Element
Description
Name
Name of the environment variable.
Value
Current setting or value for the environment variable.
HotSpot Flags Window
The HotSpot Flags window displays all -XX flags that the ZVM uses, their description, and current
values. This screen is for advanced diagnostics and is not intended for beginning users. Azul customer
support may instruct users to look up flag values in this window.
Azul Systems
Confidential and Proprietary
217
Zing Installation and Administration Guide
The following table describes the HotSpot Flags window elements.
HotSpot Flags Window Elements
Element
Description
The type of flag. Possible values are:
Type
l
Boolean – the flag requires a Boolean value of true (+) or false (-).
l
Integer or Unsigned Integer – the flag requires an integer value.
l
String – the flag requires a string.
Name
The name of the flag.
Value
The value configured for the flag.
Description
The description of the flag.
Statistics Window
The Statistics window displays information about running applications.
Azul Systems
Confidential and Proprietary
218
Zing Tools Management
The following table describes the Statistics window elements
Element
Description
SystemDictionary Stats
Displays the number of Java classes loaded in the ZVM.
Thread Count Stats
Displays the number of running threads.
Settings Window
The Settings window provides controls for enabling GC logging, choosing the level of detail logged,
and whether the output is sent to standard out or saved to a new or existing file.
The following table describes the Settings window elements.
Settings Window Elements
Element
Description
Off
Disables GC logging for this ZVM instance. This is the default setting.
Print GC
Enables GC logging with summarized output. Output is equivalent to
using the command line option -XX:+PrintGC.
Print GC details
Enables GC logging with detailed output. Output is equivalent to using
the command line option -XX:+PrintGCDetails.
Azul Systems
Confidential and Proprietary
219
Zing Installation and Administration Guide
Settings Window Elements
Element
Options: File name
Description
Type a file name to send GC log output instead of sending it to
standard out. Select the check box and type an existing file name to
append the output to the end of an existing file.
Groups Window
This feature is not supported.
Login Window
This feature is not supported.
Features Window
The Features window is displayed to enable user access to certain ZVision features for the currently
running ZVM.
Warning
Enabling user access to any of these features may interfere with the ZVM,
cause system delays, or expose sensitive information.
The following table describes the Features window elements.
Azul Systems
Confidential and Proprietary
220
Zing Tools Management
Element
Description
Azul Support
Enables user access to Azul Customer Support Options. Use at
your risk, as some of these diagnostic features can interfere with
the proper functioning of the ZVM, or cause the system to crash.
Browse Instance
Enables users to browse a class instance. Turning this feature on
could expose sensitive data within the object.
Browse Class
Enables users to browse the static class data. Turning this feature
on could expose sensitive data within the object.
Browse Method
Enables users to browse a class method. Turning this feature on
could expose sensitive data within the object.
Environment variables
Enables users to view the Environment Variables. Turning this
feature on could expose sensitive data.
Stack trace
Enables users to use the Stack Trace window. Turning this feature
on could expose sensitive data.
MetaTicks
Enables users to view metaticks. Turning this feature on could
interfere with the proper functioning of the VM and cause the system
to crash. This tab should ONLY be enabled if advised by Azul
Support.
Polling Opportunities
Enables users to view the Polling Opportunities entries. Turning this
feature on could interfere with the proper functioning of the VM and
cause the system to crash. This tab should ONLY be enabled if
advised by Azul Support.
Weblogic Muxer
Enables users to use the Web Logic Muxer Application. Turning this
feature on could interfere with the proper functioning of the VM and
cause the system to crash. This tab should ONLY be enabled if
advised by Azul Support.
Threads Tab
The Threads tab contains the following windows:
l
Threads List Window
l
Stack Trace Window
l
Deadlocks window
l
Contention Window
l
Contention Details Window
Azul Systems
Confidential and Proprietary
221
Zing Installation and Administration Guide
Threads List Window
The Threads List window displays all threads, status of each thread, and links to detailed information
about a thread. View and action options include:
l
Refresh this window to view thread progress.
l
Click the Name and State column titles to sort the table.
l
Click the Details link item display the status of a thread and where it is blocked. Refer to Stack
Trace Window.
l
Set values in the Name, State, Group, and/or Per page fields to filter the displayed list
The following table describes the Threads List window elements.
Threads List Window Elements
Elements
Name
Description
Lists the name of the thread as assigned by the application.
The most common states for threads are:
l
running – the thread is executing on a processor core.
l
I/O wait – the thread is waiting for socket or file I/O.
l
waiting on monitor – the thread is blocked in java.lang.Object.wait
State
() awaiting notification.
l
low memory detection –
l
acquiring monitor – the thread is waiting to acquire a synchronization
monitor.
l
Azul Systems
acquiring and releasing monitor – the thread a requires and releases a
Confidential and Proprietary
222
Zing Tools Management
Threads List Window Elements
Elements
Description
synchronization monitor. This typically happens when the critical section of
the monitor is extremely small.
l
sleeping – the thread executed java.lang.Thread.sleep().
l
Other states, such as idle, semaphore wait, waiting on ZVM monitor
Threads_lock, or Waiting on Monitor SLT lock indicate threads that are
internal to the ZVM. These states are not normally useful for analysis of
application performance. Also, during start up, compiler threads may be
running. The time (in seconds) that the thread has spent since entering the
state is provided in parenthesis after the waiting state indicators.
Provides links to a stack trace, tick profile, and object profile for the selected
thread. These functions are the same as the regular tick and profile features,
except that only information on the selected thread displays. The following
links appear inline with threads listed in this window:
l
Details
Stack – displays a thread execution details window. Displays the page
Threads > Stack Trace.
l
Ticks – displays a per-thread profile. Displays the page CPU > Tick
Profiler.
l
System calls, RPC out, RPC in, FDC, and Transport are not supported at
this time.
Stack Trace Window
The Stack Trace window lists all threads and displays whether a thread is executing or is blocked. This
can help to determine changes to make in the application code.
To perform a search by thread name, type a string in the Name field to display all thread names that
include that text string (case sensitive).
Click the thread links in the Stack Trace window to display a thread execution details window. Use the
State drop-down menu to only display threads in a certain state (for example, running, I/O wait,
acquiring monitor, and so on). Use the Group drop-down menu to select whether to list system, nonsystem, or all group types. Use the Per Page drop-down menu to determine the number of items to
display on this page.
Toggle between low or high display of detailed information.
Azul Systems
Confidential and Proprietary
223
Zing Installation and Administration Guide
Refresh this screen to view the progress of the thread. Relevant information displays at the top of the
window:
l
Address
l
Priority
l
Object blocked
l
Object wait
l
CPU wait
l
I/O wait
The Stack Trace window is displayed only if it is enabled in the Features window (see Features
Window for more information).
Deadlocks Window
The Deadlocks window lists detected deadlock cycles. Threads involved for each cycle of locks are
listed with the monitor type the thread is trying to acquire and the thread that is the current owner. Click
the thread links to view additional information.
Azul Systems
Confidential and Proprietary
224
Zing Tools Management
Contention Window
The Contention window provides statistical information for each monitor and sorts monitors by total
acquire time. The following types of monitors appear in the Contention window:
l
Java-level monitors – identified by their class name.
l
Internal JVM monitors – identified with the extension _lock. If there are a high number of
internal JVM monitors, examine the verbose garbage collection (GC) logs. Observe if there are
a large number of GC pauses. Contact Azul customer support if this becomes a problem.
The total acquire time parameter is the time, in milliseconds, for acquires to occur for a given internal or
Java-level monitor.
The following table describes the Contention window elements. Click column titles to sort by
relevance.
Contention Window Elements
Elements
Name
Description
The name of the monitor.
l
Acquire Time
(ms)
Total – the time, in milliseconds, for acquires to occur for a given
internal or Java-level monitor (not tracked for lightly contended “thin”
monitors).
l
Max (ms) – the maximum time, in milliseconds, to perform an acquire.
Blocking acquires
l
Count – the number of blocking acquires.
Waits
l
Count – the number of wait cycles on the thread.
Azul Systems
Confidential and Proprietary
225
Zing Installation and Administration Guide
Contention Window Elements
Elements
Description
l
Max (ms) – the maximum time, in milliseconds, of wait cycles that are
allowed.
l
Total (ms) – Cumulative wait cycle in milliseconds
Contention Details Window
The Contention Detail window provides additional information about the selected monitor in
contention. Detail information is displayed through the Azul Support tab.
Note
If Azul Support is enabled, links to related Monitor details are displayed. Only enable
and use the Azul Support tab as directed by Azul Support team members.
CPU Tab
The CPU tab provides information about how much CPU time is being used by threads. This is
measured by recording the length of time it takes a thread to complete a tick, how often a thread
accesses sensitive areas of the application code, and the length of time a thread takes to achieve a
safepoint.
The CPU tab contains the following windows:
l
Tick Profile
l
Meta Ticks
l
Event Tracker
Tick Profile Window
The Tick Profile window provides information to allow calculation of how processor resources are
used for an interval of recorded time. The recorded time interval depends on the number of threads and
Azul Systems
Confidential and Proprietary
226
Zing Tools Management
how busy the threads are. The busier threads are, the greater the number of ticks generated. See
Statistical Tick Profiler for more information the statistical tick profiler.
Note
Method names in the tick profiler are compiled methods and are no longer being
interpreted. All time spent by methods being interpreted is summed as one entry
named “Interpreter.”
Check the Tick Profile window for functions such as VM_C2 Compiler or Interpreter after launching an
application. If these functions are high on the list, then the application is still starting up (a warning
message may also appear in the Compiler tab). To determine if the application is warmed up, wait a few
seconds, click Reset Tick Profile, then refresh the Web browser until these two threads are not high in
the list.
The following table describes the Tick Profile window elements.
Azul Systems
Confidential and Proprietary
227
Zing Installation and Administration Guide
Tick Profile Window Elements
Element
Description
Click Pause The Tick Collection to stop refreshing the list of ticks.
Pause/Resume the This is useful when you need to locate a specific item in a rapidly
Tick Collection
changing list. Click Resume The Tick Collection and refresh the
screen to continue updating the list of ticks.
Reset Tick Profile
Clears the existing list of ticks and restarts tick collection.
Cutoff
Filters the list of ticks to list only those values that are greater than or
equal to the set cutoff value. Enter value. Click Submit.
Below Cutoff
At the bottom of the list of ticks, it lists the percentage and number of
ticks that are not included in the displayed list as defined by the Cutoff
percent value and Threads value.
Threads
Display the ticks for only the threads listed in this field. Enter value.
Click Submit.
The three radio buttons filter the tick data by a tag assigned to it.
Whenever a thread is running, it has a current tag assigned to it. That
value can change at any time, as the thread does different things. For
instance, if you have a thread that is a garbage collection thread, it's tag
might be "garbage collection" most of the time, but when that thread
was running some specific subtask of GC, it might say GC marking
phase. This feature allows collates and displays the data from the
aspect of the tag value instead of the method name.
The options are:
l
None – don't filter the list by tags.
l
JVM -- group all the JVM compiler ticks into one entry, so that
None, JVM, All
everything is lumped together. Useful if you are trying to get a sense of
what overhead the compiler is having.
l
All - only display ticks by tag type for all threads. Instead of showing
ticks entries by their method name, group and display them by their tag
type.
To apply tag filter:
Click a radio button. Click Submit.
Azul Systems
Confidential and Proprietary
228
Zing Tools Management
Tick Profile Window Elements
Element
Percent
Description
The percentage of CPU resources (% of the total number of ticks
collected during the collection period) the function used during the time
interval.
The number of program counters (“ticks”) collected in the time interval.
This includes:
l
Ticks
Timer ticks – collect for each tick, the thread ID, CPU, stack trace and
last eight calling functions when the tick occurs. The time reported, in
milliseconds, is thread time, not clock time.
l
Time to safe point ticks – collect the time required to complete a save
point process.
A description of the function being counted. The following are
definitions for specific functions:
l
VM_GC Task – the garbage collector for the ZVM.
l
ARTA_Performance_Monitor – the function that collects and displays
Source
ZVision data.
l
VM_C2 Compiler – the ZVM JIT compiler compiling methods
indicating that an application is still warming up .
l
generate_monitor enter – the amount of time spent busy-waiting to
acquire monitors.
Statistical information about the total list of ticks. This includes:
l
Ticks within a Buffer – number of ticks a standard tick buffer can hold.
There is a fixed maximum of 2K.
l
Available Buffers – number of buffers the user has allocated, as
determined by the -XX:EventTickBufferDataBufferMargin=N value,
Event Profiling
Tick Buffer
Statistics
plus the high water mark number of active threads. For example, if you
specify 100, and the application has 400 threads simultaneously
active, then there are 500 available buffers. This total value is
dynamic, but only grows. If suddenly the application drops 300
threads, there continues to be 500 buffers.
l
Azul Systems
Theoretical Maximum Ticks (Available Buffer * Ticks/Buffer) – per the
Confidential and Proprietary
229
Zing Installation and Administration Guide
Tick Profile Window Elements
Element
Description
listed formula, for example using Ticks within a Buffer maximum
(2000) X Available Buffers (500) = 1,000,000.
l
Count of Ticks processed – not all buffers are completely filled at any
moment in time. The tick buffers are collecting ticks. Some have not
been used yet. This value is the count of all ticks that are in buffers,
which is some value less than the theoretical maximum.
l
Profiler Ticks applicable (matched filter criteria) – number of ticks that
match the search criteria
l
Other Ticks not applicable (Failed filter criteria) – number of ticks that
do not match the search criteria. Profiler Ticks and Other Ticks equal
the total number of ticks processed.
Meta Ticks Window
Meta Ticks measure how long it takes to perform a function. This is not a measure of each thread. Meta
Ticks are like timer ticks in nature. They are a discreet packet of information about the state of a thread,
but unlike the Profile Ticks, which are created based on some periodic timer, Meta Ticks are created as
the thread enters certain specific code paths.
Azul Systems
Confidential and Proprietary
230
Zing Tools Management
The following table describes the Meta Ticks window elements.
Meta Ticks Window Elements
Element
Description
Pause/Resume
the Tick
Collection
Click Pause The Tick Collection to stop refreshing the list of ticks. This
is useful when you need to locate a specific item in a rapidly changing
list. Click Resume The Tick Collection and refresh the screen to
continue updating the list of ticks.
Reset Tick
Profile
Clears the existing list of ticks and restarts tick collection.
Search Criteria:
Single Value
Entries
l
Entries Displayed – number of entries displayed on a given page.
l
Tick Start – a search field which determines the earliest tick that is
included in the display, based on the elapsed tick counter.
l
Tick End – a search field which determines the latest tick that is
l
included in the display, based on the elapsed tick counter.
These items are all search criteria for metaticks. The five fields in this
row can all use comma separated lists as a search criteria. That is, you
can search for more than one term simultaneously, by listing those terms
with commas separating them. For example, CPU field could contain
"2,5,9", which would list all metaticks that came from CPU 2, 5 or 9.
Search Criteria:
Comma
Separated Lists
l
MetaTick # – value that can be used for searches. The values are
preassigned by Azul.
l
User Info # – value assigned dynamically by the application that can be
searched on.
l
Tag # – The current Virtual Machine Process Tag of the thread when the
metatick occurred. This is a pre-defined value we assign to a thread that
can be changed dynamically, to indicate what kind of work the thread is
Azul Systems
Confidential and Proprietary
231
Zing Installation and Administration Guide
Meta Ticks Window Elements
Element
Description
doing. The values are preassigned by Azul.
l
CPU # – the CPU that the thread was executing on when the metatick
occurred.
l
Thread # – thread ID for Azul internal use.
List of actual metatick data. See below listed field descriptions.
l
Line item – last 100 meta ticks recorded.
l
Tick Counter – elapsed ticks since startup.
l
Thread ID – thread ID for Azul internal use.
l
CPU – CPU that the thread was executing on when the metatick
occurred.
Data Table
l
MetaTick Type – value for Azul internal use. Value indicates the tag
value at the time the tick occurred.
l
MetaTick Type # – string translation of the stored bit value to the
metatick type.
l
Tag Name – string human readable translation of the tag number.
l
Tag # – the VM Tag currently assigned when the metatick occurred
l
Stack Frame Address – address of the stack frame at the time the tick
occurred.
The following are diagnostics related to capturing tick statistics.
l
Ticks within a Buffer – number of ticks a standard tick buffer can hold.
There is a fixed maximum of 2K.
l
Available Buffers – number of buffers the user has allocated, as
determined by the -XX:EventTickBufferDataBufferMargin=N value, plus
Event Profiling
Tick Buffer
Statistics
the high water mark number of active threads. For example, if you
specify 100, and the application has 400 threads simultaneously active,
then there are 500 available buffers. This total value is dynamic, but only
grows. If suddenly the application drops 300 threads, there continues to
be 500 buffers.
l
Azul Systems
Theoretical Maximum Ticks (Available Buffer * Ticks/Buf) – per the
Confidential and Proprietary
232
Zing Tools Management
Meta Ticks Window Elements
Element
Description
listed formula, for example using Ticks within a Buffer maximum (2000)
X Available Buffers (500) = 1,000,000.
l
Count of Ticks processed – not all buffers are completely filled at any
moment in time. The tick buffers are collecting ticks. Some have not
been used yet. This value is the count of all ticks that are in buffers,
which is some value less than the theoretical maximum.
l
Profiler Ticks applicable (matched filter criteria) – number of ticks that
match the search criteria
l
Other Ticks not applicable (Failed filter criteria) – number of ticks that do
not match the search criteria. Profiler Ticks and Other Ticks equal the
total number of ticks processed.
Event Tracker
This feature is not supported.
Memory Tab
The Memory tab contains the following windows:
l
Memory Summary Window
l
GC Summary Window
l
GC History Window
l
Browse Object Window
l
Allocated Objects Window
l
Live Objects Window
l
Type Velocity Window
Memory Summary Window
The Memory Summary window provides a summary of total memory usage and Contingency memory
usage. See “Cooperative Memory Management and Memory Resiliency” in Using the Zing Virtual
Machine for detailed information about Contingency memory.
Azul Systems
Confidential and Proprietary
233
Zing Installation and Administration Guide
The following tables describe the Memory Summary window elements.
Memory Summary Window Elements – Java Heap Usage
Element
Description
Used
The amount of memory currently used by the ZVM.
Capacity
The amount of memory allocated for use by the ZVM.
Requested capacity
The maximum amount of memory requested for use by the ZVM (Xmx).
GC cycle count
The number of garbage collection cycles.
Memory Summary Window Elements – Memory Accounts
Element
Description
VM internal
Shows allocated, balance, Reserved, and Contingency memory usage for the
ZVM
Java heap
Shows allocated, balance, Reserved, and Contingency memory usage for the
Java heap.
Java Pause
Prevention
Shows allocated, balance, Reserved, and Contingency memory usage for the
Java Pause Prevention pool.
Azul Systems
Confidential and Proprietary
234
Zing Tools Management
GC Summary Window
The GC Summary window provides a cumulative summary of the recent GC cycle history. See
“Generational Pauseless Garbage Collection (GPGC)” in Using the Zing Virtual Machine for more
information about the GPGC garbage collector.
The following table lists the GC Summary window elements. The screen lists the garbage collection
information for both New generation and Old generation cycles in two parallel tables.
GC Summary Window Elements
Element
Header
Azul Systems
Description
l
Category – element being measured
l
Statistic – aspect of the element being measured
Confidential and Proprietary
235
Zing Installation and Administration Guide
GC Summary Window Elements
Element
Cycle
Description
l
Mean – statistical calculation of collected values
l
Stddev – standard deviation calculated for collected values
l
Min – minimum value in collected sample
l
Max – maximum value in collected sample
l
Interval (sec) – elapsed time between successive GC cycle starts.
l
Pause ratio – ratio of time application spent in safepoint pauses
during the cycle to cycle interval time
Reserved
l
Peak used (MB) – the peak size of the Java heap recorded during the
GC cycle
Pause
These values are used for internal Zing diagnostic purposes only.
l
New used (MB) – the amount of memory used by all young objects
not known to be dead during the GC cycle.
l
Old used (MB) – the amount of memory used by all old objects not
known to be dead during the GC cycle.
Generations
l
Perm used (MB) – the amount of memory used by all permanent
objects not known to be dead during the GC cycle.
l
Live (MB) – the live memory in the generation(s) being collected.
l
Fragmentation (MB) – the fragmentation in the generation(s) being
collected.
l
Found (MB) – the garbage found in the generation(s) being
collected.
Garbage
l
Collected (MB) – the garbage freed in the generation(s) being
collected.
l
Sideband limited (MB) – used for internal Zing diagnostics purposes
only.
Pages
These values are used for internal Zing diagnostic purposes only.
Pauses
l
Azul Systems
Pause 1 duration (ms) – the duration time of the pause 1 safepoint.
Confidential and Proprietary
236
Zing Tools Management
GC Summary Window Elements
Element
Description
l
Pause 2 duration (ms) – the duration of the pause 2 safepoint
l
Pause 3 duration (ms) – the duration of the pause 3 safepoint
l
Pause 4 duration (ms) – the total duration of all pause 4 safepoints
for this GC cycle.
l
Duration (sec.) – the elapsed time from the end of the last GC cycle
to the start of this GC cycle.
l
Intercycle
Allocation rate (MB/s) – the object allocation rate in the generation
being collected in the period between the last GC cycle to the start of
this GC cycle.
l
Perm allocation rate (MB/s) – the object allocation rate in the
permanent generation during this GC cycle.
l
Duration (sec.) – the GC cycle elapsed time; the duration of time
from the beginning to the end of this GC cycle.
Intracycle
l
Allocation rate (MB/s) – the object allocation rate in the generation
being collected during this GC cycle.
l
Perm allocation rate (MB/s) – the object allocation rate in the
permanent generation during this GC cycle.
l
Total threads – the total count of Java threads in the ZVM.
l
Threads delayed – the number of threads blocked trying to allocate
memory during the GC cycle. When the number of delayed threads
is greater than 0, application performance is affected due to lack of
App threads
available memory.
l
Average thread delay (sec) – the average delay for threads blocked
trying to allocate during the GC cycle.
l
Max thread delay (sec) – the maximum delay for threads blocked
trying to allocate.
GC History Window
The GC History window provides a detailed summary of the recent GC cycle history with a row per
cycle. Click column titles to sort by relevance.
Azul Systems
Confidential and Proprietary
237
Zing Installation and Administration Guide
The following table lists the GC History window elements.
GC History Window Elements
Element
Description
Indicates GC cycle, as follows:
Cycle Type
l
Old – old generation cycle
l
New – new generation cycle
l
NTO – new generation cycle that immediately precedes an Old
Gen cycle
Cycle #
Mode
Azul Systems
For each New or Old Gen cycle, the number of that collection in the
sequence of all collections of that type (New or Old Gen).
l
p – indicates if allocation from ause pool is
l
g – indicates if allocation from Contingency pool is .
Confidential and Proprietary
238
Zing Tools Management
GC History Window Elements
Element
Memory Heap
Description
l
Max – maximum amount of heap memory guaranteed to the
application. This is generally set by Xmx option.
l
Peak – the peak size of the Java heap recorded during the GC
cycle.
Memory Reserved
l
Used – the amount of memory used by all objects not known to be
dead during the GC cycle.
Memory Pause
These values are used for internal Zing diagnostic purposes only.
l
New – the amount of memory used by all young objects not
known to be dead during the GC cycle.
l
Old – the amount of memory used by all old objects not known to
be dead during the GC cycle.
Memory Generations
l
Perm – the amount of memory used by all permanent objects not
known to be dead during the GC cycle.
Memory Garbage
l
Live – the live memory in the generation(s) being collected.
l
Frag – the fragmentation in the generation(s) being collected.
l
Found – the garbage found in the generation(s) being collected.
l
Freed – the garbage freed in the generation(s) being collected.
l
Sideband limited – used for internal Zing diagnostics purposes
only.
Memory Pages
These values are used for internal Zing diagnostic purposes only.
l
Dur (sec.) – the elapsed time from the end of the last GC cycle to
the start of this GC cycle.
l
Intercycle
Alloc rate (MB/s) – the object allocation rate in the generation
being collected in the period between the last GC cycle to the
start of this GC cycle.
l
Perm alloc rate (MB/s) – the object allocation rate in the
permanent generation during this GC cycle.
Azul Systems
Confidential and Proprietary
239
Zing Installation and Administration Guide
GC History Window Elements
Element
Description
l
Dur (sec.) – the GC cycle elapsed time; the duration of time from
the beginning to the end of this GC cycle.
Intracycle
l
Alloc rate (MB/s) – the object allocation rate in the generation
being collected during this GC cycle.
l
Perm alloc rate (MB/s) – the object allocation rate in the
permanent generation during this GC cycle.
App threads #
l
# – the total count of Java threads in the ZVM.
l
# – the number of threads blocked trying to allocate memory
during the GC cycle. When the number of delayed threads is
greater than 0, application performance is affected due to lack of
available memory.
App threads Delay
l
Ave (sec) – the average delay for threads blocked trying to
allocate during the GC cycle.
l
Max (sec) – the maximum delay for threads blocked trying to
allocate.
Browse Object Window
Do not select this page directly, instead select a highlighted object, instance, method or class.
Azul Systems
Confidential and Proprietary
240
Zing Tools Management
The following table describes the Browse Object tab window elements.
Browse Object Tab Window Elements
Element
Class Property and
Value
Description
If applicable, list of class properties associated with the selected
object. Includes values for the listed properties.
If applicable, list of static information associated with the selected
object. This includes descriptive information:
Static Fields
l
Access
l
Name
l
Type
l
Value
Methods
If applicable, list of methods associated with the selected object.
Transitive Interfaces
If applicable, list of transitive interfaces associated with the
selected object.
Allocated Objects Window
Data collected for allocated objects only when the -XX:ProfileAllocatedObjects option is
included in the JVM command.
Azul Systems
Confidential and Proprietary
241
Zing Installation and Administration Guide
This feature is not supported.
Live Objects Window
The Live Objects window displays a current list of classes in the old generation heap that are being kept
alive. Use this information to identify possible memory leaks in your Java application.
Each listed row [with a plus (+)] is a referent class. Expanded rows show a list of referrer classes to the
referent class. A class is listed as a referrer, when all the following conditions are met:
l
An object of that class directly points to an object of the referent class.
l
The referent object was seen by the GC through the referrer object.
Only the first referrer seen by the GC is counted. This means that there could be other referrer objects
that point to the referent objects, but they are not included.
To display Live Object data, include the -XX:ProfileLiveObjects option in the JVM command.
To view current data, refresh the screen. If a GC has not occurred yet, a message to refresh is
displayed.
The following table describes the Live Object tab window elements. Note, all the descriptions are
about the old generation heap.
Azul Systems
Confidential and Proprietary
242
Zing Tools Management
Live Object Tab Window Elements
Element
Description
Total
Total number of old generation live objects.
Class Name
Name of classes whose objects reside in old generation memory.
Click the plus (+) to see referring classes.
Count
The number of objects of this class that are alive in the old
generation heap.
Size (bytes)
The total amount of memory used by objects of this class.
Avg Size (bytes)
The average amount of memory used by objects of this class.
Percentage of Live Set
The percentage of the live set memory consumed by object of this
class.
For referrer rows this is a percentage of referrer count.
Type Velocity Window
The Type Velocity window displays a current list of classes in the old generation heap and their rate of
memory growth. Where velocity for a class is the rate of heap growth in bytes over a selected time
interval. Use this information to identify possible memory leaks in your Java application.
To gather Type Velocity data, include the -XX:ProfileLiveObjects option in the JVM command
and allocate extra memory. To view current data, refresh the screen.
All time listings are in format: [dd] hr:min:sec. Number of days are listed only if it is non-zero.
Only the top 1000 classes, sorted by Growth Rate, are shown.
Azul Systems
Confidential and Proprietary
243
Zing Installation and Administration Guide
The following table describes the Type Velocity tab window elements.
Type Velocity Tab Window Elements
Element
Description
Interval
Options in increments ranging: 1 – 45 mins, 1-18 hrs, 1-7 days
Requested time interval
Selected time interval by the user. See options in Interval.
Time interval starting from the last GC cycle that most closely
matches the selected interval.
Best available match
interval
For example, if GC occurred at 10 minutes, 15 minutes, 20
minutes, and 45 minutes from the start of the VM, and the selected
interval is 10, then the data from the GC at 20 minutes (intervalstart) and 45 minutes (interval-end) are used for calculating the
type velocity.
Interval-start
Interval-end
The starting time and ending time of the best available match
interval. Time is not clock time. It is time from start of VM.
VM uptime
Amount of time passed since the start of the VM.
Azul Systems
Confidential and Proprietary
244
Zing Tools Management
Type Velocity Tab Window Elements
Element
Description
Class Name
Name of classes whose objects reside in old generation memory.
Latest Count
The number of objects of this class that are alive at interval-end
time.
Latest Size (bytes)
The total amount of memory used by objects of this class that are
alive at interval-end time.
Prior Count
The number of objects of this class that are alive at interval-start
time.
Prior Size (bytes)
The total amount of memory used by objects of this class that are
alive at interval-start time.
Delta Count
The difference between the number of objects of this class that are
alive between interval-start and interval-end time.
Delta Size (bytes)
The difference between the memory used by objects of this class
between interval-start and interval-end time.
Growth Rate
(bytes/min)
The rate of heap growth for a class over Best available match
interval.
Growth Rate = Delta Size / Best available match interval.
Compilers Tab
The Compiler tab of ZVision displays the status of Server and Client compiler tasks.
Azul Systems
Confidential and Proprietary
245
Zing Installation and Administration Guide
The following table describes the Compiler tab window elements.
Compiler Tab Window Elements
Element
Description
Compiling
Status of threads being compiled.
Waiting
Status of threads waiting to be compiled
Applications Tab
The Application tab displays a list of application classes and Java class libraries that are loaded in the
ZVM. If the Browse Class feature is enabled for a user in the Features tab, the hyperlink for a specific
class can be selected to browse the static class data in the Browse Object window (see Features
Window for more information about enabling the Browse Class feature).
Azul Systems
Confidential and Proprietary
246
Zing Tools Management
Advanced Configuration Options
The ZVision server can be invoked from the command line as follows:
zvision [options]
The following command line options are supported:
Command Line Option
Description
-h or --help
Display a list of supported command line options
-p PORT, --port=PORT
Specify the port number port number that browsers listen
to the ZVision server
-i INTERFACE, -interface=INTERFACE
Specify the network interface that the ZVM accepts
HTTP connections.
-t ARTA_TIMEOUT, -arta_
timeout=ARTA_ TIMEOUT
Specify the timeout interval (in seconds) when
communicating with the ZVM.
-1 DEBUG_LEVEL-logging=DEBUG_LEVEL
Specify the debug logging level: debug, info, warning,
error, or critical
--document_
root=DOCUMENT_ ROOT
Specify the path to static HTML files used by the ZVision
server.
Azul Systems
Confidential and Proprietary
247
Zing Installation and Administration Guide
Using Zing Vision Robot
This section describes configuring and using the Azul Systems Zing, Zing Vision Robot (ZVRobot)
profiling and diagnostic snapshot collection tool for the Zing Virtual Machine (ZVM).
See Using Zing Vision for more information about using the Zing Vision real-time profiling and
diagnostic tool. See Using the Zing Virtual Machine for more information about running Java
applications using the ZVM.
ZVision Overview
ZVRobot provides a method to collect metrics about the state of a running application and the
underlying ZVM. It uses the same data collection mechanisms as Zing Vision (ZVision) but stores each
data snapshot persistently to a file. ZVRobot automates collecting and persistently storing the data
displayed with a manual Zing Vision page refresh. The data stored in the files can be viewed through a
browser, simulating the ZVision environment. You can set which Zing Vision pages to collect and, how
often and how many times you want to capture the page data.
Use the ZVRobot data to help you determine how to tune command line options and optimize your code
performance. ZVRobot data includes:
l
Complete internal and environment settings inventory
l
Sample-based method profiling
l
ZVM internal subsystem behavior, including GC and the runtime compiler
l
Lock and monitor usage metrics
l
Thread behavior
ZVRobot is a simple http client that collects ZVision data from the ARTAPort according to the criteria
specified in the ZVRobot.prop file and outputs it to a specified directory as a set of time stamped
XML files. It has a very low impact on performance. The ZVRobot client can be started and stopped at
any time during a ZVM session.
See “ZVision Interface” in Using Zing Vision for detailed information about the Zing Vision interface.
ZVRobot Benefits
ZVRobot can be used in a few ways, think of ZVRobot as a flight recorder, persisting away data that
can be reviewed later for offline analysis.
ZVRobot data includes:
l
Complete internal and environment settings inventory
l
Sample-based method profiling
l
ZVM internal subsystem behaviour, including GC and the runtime compiler
l
Lock and monitor usage metrics
l
Thread behavior
Use ZVRobot
Azul Systems
Confidential and Proprietary
248
Zing Tools Management
l
In development, use the ZVRobot data to help you optimize your code performance by
monitoring locking behavior, finding hot methods, and monitoring other behaviors such as: per
thread stack trace; tick profile; I/O wait; object size and count profile; live objects type velocity;
deadlocks; monitor contention; memory usage; GC; Compilation; etc.
l
In the Operations Center, when an alert is triggered, use a script in the management tool to
start an instance of ZVRobot to record what is going on for a set period (controllable in the
ZVRobot.prop file). This ensures that ZVRobot has data from when the incident occurred.
l
In preproduction, use ZVRobot to take samples of the application before implementing major
changes or code changes. Compare the ZVRobot data with samples taken with previous
application code versions. This helps to identify if code changes have changes the code profile,
locking or allocation rates.
l
Periodically (weekly) take a few minutes of samples as a reference point, these reference
points can be used to compare peak request loads as well as be used for long term capacity
planning with your other tools.
Azul Customer Support
If you are providing data for problem resolution for a case with Azul customer support:
1. Capture the requested data in its entirety using ZVRobot.
2. Compress the data into zip or gzipped tar file.
3. Send it by e-mail or FTP upload for analysis.
Configuring the Zing Virtual Machine to Use ZVRobot
ZVRobot must be connected to the ZVM you are monitoring. The following command enables ZVRobot
to communicate with the ZVM on the specified port of the host where the ZVM is running.
Add the following command line option to your Java command string that starts the ZVM and
your Java application.
-XX:ARTAPort={off|<port>|any}
Supply a port number or set this option to any to enable the communication between ZVM and
ZVRobot. The ARTA (Azul Run Time Analysis) port provides internal API communication
between ZVision, ZVRobot, and ZVM.
Valid options include:
l
off: Default. ZVM does not open a communication port and ZVRobot cannot communicate.
l
<port>: port number for ZVRobot to use on the host.
Azul Systems
Confidential and Proprietary
249
Zing Installation and Administration Guide
l
any: ZVRobot uses any free port.
The port number opened by the ZVM is echoed to stderr at startup. This setting is useful
when starting multiple ZVMs on the same host.
For complete instructions on using ZVM command line options, see “ZVM Options” in Using the Zing
Virtual Machine.
Installing and Configuring ZVRobot
Install ZVRobot and modify the ZVRobot properties file to specify the connection to the ZVM and select
the type of data to collect.
1. Locate the ZVRobot zip file.
The ZVRobot.zip file is installed with ZVM at, /<zvm_home>/etc/ZVRobot.zip.
For example:
/opt/zing/zing-jdk1.7.0-5.9.3.0-1-x86_64/etc/ZVRobot.zip
2. Copy the ZVRobot.zip file to a host machine where either a Java Development Kit (JDK) or Java
Runtime Environment (JRE) is installed.
For efficient use, locate the ZVRobot.zip file on a different machine than the ZVM
Note:
host or Java application you are monitoring. The ZVRobot machine must be networked
to the ZVM host machine it is monitoring.
3. Locate and open the ZVRobot.prop file.
The file is in the top level ZVRobot directory. The file has two sections:
l
Configuration – contains required information you must provide for ZVRobot to
communicate with the ZVM and options that control collection behavior. See Step 4.
l
Page Types to Collect – contains a list of Zing Vision pages. Some pages are already
configured to be collected. You can turn on or off the collection of any Zing Vision page.
4. Provide the configuration information.
In the top section of the file, enter appropriate values for the following variables:
l
ZVRobotVars.HOST: IP address of the application host where the ZVM is running.
l
ZVRobotVars.PORT: Port on the host machine that was specified with the XX::ARTAPort command line option when the ZVM was started. See Configuring the
Zing Virtual Machine to Use ZVRobot.
l
ZVRobotVars.THSIZE: The number of top-level items to request for display in a page.
For example, for:
Azul Systems
Confidential and Proprietary
250
Zing Tools Management
l
IO_OPEN_SOCKETS, this is the number of sockets.
l
THREADS_LIST and THREADS_STACK_TRACE, this is the number of threads to list
in a single page.
l
LEAK_SITES, this is the per page number of location in the code identified as sites
where memory is being leaked.
l
ZVRobotVars.SECONDS: The interval in seconds between sequential requests by
ZVRobot to the ZVM.
l
ZVRobotVars.COUNT: The number of snapshots to gather. A negative number is
interpreted as a request for an infinite number of snapshots. If you use this option, stop the
ZVRobot process with CTRL-C from the command line.
5. Select the metrics to collect.
In the third section of the file, remove or add the pound (#) character to enable or disable,
respectively, collecting a page's data.
For example:
l
To enable a <metric_name> metric, the line in the file should be:
<metric_name>=http://${HOST}:${PORT} ${SECONDS} ${COUNT}
l
To disable the <metric_name> metric, the line in the file should be:
# <metric_name>=http://${HOST}:${PORT} ${SECONDS} ${COUNT}
To disable the metric, comment out the line by adding the pound sign (#) in the line.
Running ZVRobot and Collecting Data
ZVRobot communicates with the ZVM using the host:port combination specified in the
ZVRobot.prop file. The collected metric snapshots are written to a directory suffixed with the date
and time when ZVRobot starts.
In the command line for the machine where ZVRobot is installed, type the following command
to run ZVRobot:
java -jar ZVRobot-<version>.jar <directory> ZVRobot.prop
where:
l
ZVRobot-<version>.jar is the JAR file that contains the ZVRobot classes
l
<directory> is the directory in which ZVRobot creates the metric collection directory.
You must have write permission for this top level directory.
Use any as the directory and ZVRobot creates the metric collection directory in the
current working directory.
l
Azul Systems
ZVRobot.prop is the edited ZVRobot property file
Confidential and Proprietary
251
Zing Installation and Administration Guide
A directory is created that contains the XML-formatted metric snapshot files as well as
manifest files that contain a list of hyperlinks to these metric files.
Note
A sample start up script startsample.sh is provided to run ZVRobot. The script
uses the provided sample ZVRobot.prop file that has a set of default metrics
selected. To run this test script, modify the configuration variables in the sample
ZVRobot.prop file, particularly the host and port name. See Installing and
Configuring ZVRobot.
Enable or Disable Tick Profiler
The Tick Profiler is enabled and disabled using the Java command line option, UseTickProfiler.
Default for UseTickProfiler is false, which means the Tick Profiler will not collect profiling
information.
l
To enable the Tick Profiler, use -XX:+UseTickProfiler. (set to true)
l
To disable the Tick Profiler, use -XX:-UseTickProfiler. (default, set to false)
Viewing ZVRobot Snapshot Data
ZVRobot outputs two sets of files: manifest .html files and page .xml files. View the files through a
browser.
1. Navigate through your directories to the ZVRobot directory.
2. Select a date/time stamp directory
3. Select a manifest or page file to view.
4. Select Open with a browser.
See ZVRobot Snapshot Data for a description of each output manifest or page file.
ZVRobot Snapshot Data
The ZVRobot interface consists of a series of windows that format the collected information for a given
metric in the same manner as the equivalent metric that is collected in ZVision. The rest of the section
describes the ZVRobot windows.
Azul Systems
Confidential and Proprietary
252
Zing Tools Management
Sample AzulZVRobotIndex.html file
The ZVRobot manifest, AzulZVRobotIndex.html, lists hyperlinks to the manifest files specified by
the appropriate metric for the corresponding ZVision window in the ZVRobot properties file,
ZVRobot.prop. Click a hyperlink to display the manifest file for a specific ZVRobot metric. The
following table describes:
l
ZVRobot metric. Click the associated hyperlink to display a description of the metric.
l
Name of the corresponding ZVRobot metric specified in ZVRobot.prop.
l
Name of the corresponding manifest file listed in AzulZVRobotIndex.html.
ZVRobot Manifest
See Section
Metric Page Name
File Created
ZVRobot
Snapshot Data
ZVProperties.prop
Metric
AzulZVRobotIndex.html
Manifest File Name
Configuration
ZVRobotPage.CONFIG_
PROCESS
Manifest.Frame.CONFIG_
PROCESS.html
Environment
Variables
ZVRobotPage.
ENVIRONMENT_VARIABLES
Manifest.Frame.ENVIRONMENT_
VARIABLES.html
HotSpot Flags
ZVRobotPage.HOTSPOT_
FLAGS
Manifest.Frame.HOTSPOT_
FLAGS.html
Azul Systems
Confidential and Proprietary
253
Zing Installation and Administration Guide
ZVRobot Manifest
See Section
Metric Page Name
File Created
Statistics
ZVRobotPage.RUNTIME_
STATISTICS
Manifest.Frame.RUNTIME_
STATISTICS.html
Settings
ZVRobotPage.SETTINGS_
GC_LOG
Manifest.Frame.SETTINGS_ GC_
LOG.html
Features
ZVRobotPage.FEATURES
Manifest.Frame.FEATURES.html
Threads List
ZVRobotPage.THREADS_
LIST
Manifest.Frame.THREADS_
LIST.html
Stack Trace
ZVRobotPage.THREADS_
STACK_TRACE
Manifest.Frame.THREADS_
STACK_TRACE.html
Deadlocks
ZVRobotPage.THREADS_
DEADLOCKS
Manifest.Frame.THREADS_
DEADLOCKS.html
Contention
ZVRobotPage.THREADS_
MONITOR_CONTENTION
Manifest.Frame.THREADS_
MONITOR_CONTENTION.html
Tick Profile
ZVRobotPage.TICKS_
PROFILE_TAG
Manifest.Frame.TICKS_
PROFILE_TAG.html
Tick Profile
ZVRobotPage.TICKS_
PROFILE_FLAG
Manifest.Frame.TICKS_
PROFILE_LOG.html
Memory
Summary
ZVRobotPage.MEMORY_
SUMMARY
Manifest.Frame.MEMORY_
SUMMARY.html
GC Summary
ZVRobotPage.MEMORY_
GC_ SUMMARY
Manifest.Frame.MEMORY_GC_
SUMMARY.html
GC History
ZVRobotPage.MEMORY_
GC_ HISTORY
Manifest.Frame.MEMORY_GC_
HISTORY.html
Compilers Tasks
ZVRobotPage.COMPILER_
TASKS
Manifest.Frame.COMPILER_
TASKS.html
Class List
ZVRobotPage.
APPLICATIONS_CLASS_
LIST
Manifest.Frame.APPLICATIONS_
CLASS_LIST.html
Azul Systems
Confidential and Proprietary
254
Zing Tools Management
Configuration Window
The Configuration window provides a detailed view of all the process settings for the ZVM.
Click a hyperlink in the left column to display the collected information for a snapshot. The number of
snapshots and the interval between snapshot collection is specified by the ZVRobotVars.COUNT and
ZVRobotVars.SECOND variables in the ZVRobot.prop file.
The following table describes the Configuration window elements.
Configuration Window Elements
Element
Description
Version
ZVM version.
Command line
List of classes and command line options used to launch the
Java application.
Prepended flags / filename
List of option flags from the identified file that were inserted
into the front of the Java command line.
Appended flags / filename
List of option flags from the identified file that were appended
to the end of the Java command line.
Azul Systems
Confidential and Proprietary
255
Zing Installation and Administration Guide
Configuration Window Elements
Element
Description
Current working directory
Current working directory.
Java Home
Java Home path.
Class path
Path to the class.
Boot class path
Path to the boot class.
Extensions directories
Directories for extensions.
Endorsed directories
Directories for classes implementing newer versions of
endorsed standards.
Library path
Native library path.
Boot library path
Boot library path.
ARTA Version
Version of the instrumentation in the ZVM that provides data
to the ZVRobot server.
Open descriptor count limit
Limit in bytes for open file descriptors for the ZVM front end.
Core file size limit
Maximum size in bytes for a core dump file.
Environment Variables Window
The Environment variables window provides a detailed view of all environmental variables set by the
shell that launched the ZVM.
Since the information in this window does not change over time, set the ZVRobotVars.COUNT
variable in the ZVRobot.prop file to 1. This tells ZVRobot to collect only one snapshot. By default,
this one snapshot is displayed in work area of the page.
Azul Systems
Confidential and Proprietary
256
Zing Tools Management
HotSpot Flags Window
The HotSpot flags window displays all -XX flags that the ZVM uses, their description, and current
values. This screen is for advanced diagnostics and is not intended for beginning users. Azul customer
support might instruct users to look up flag values in this window.
Since the information in this window does not change over time, set the ZVRobotVars.COUNT
variable in the ZVRobot.prop file to 1. This tells ZVRobot to collect only one snapshot. By default,
this one snapshot is displayed in work area of the page.
Azul Systems
Confidential and Proprietary
257
Zing Installation and Administration Guide
HotSpot Flags Window Elements
Element
Description
The type of flag. Possible values are:
Type
l
Boolean: the flag requires a boolean value of true (+) or false (-).
l
Integer or Unsigned Integer: the flag requires an integer value.
l
String: the flag requires a string.
Name
The name of the flag.
Value
The value configured for the flag.
Description
The description of the flag.
Statistics Window
The Statistics window displays information about running applications.
Click a hyperlink in the left column to display the collected information for a snapshot. The number of
snapshots and the interval between snapshot collection is specified by the ZVRobotVars.COUNT and
ZVRobotVars.SECOND variables in the ZVRobot.prop file. The following table describes the
Statistics window elements.
Azul Systems
Confidential and Proprietary
258
Zing Tools Management
Statistics Window Elements
Element
Description
SystemDictionary Stats
Displays the number of Java classes loaded in the
ZVM.
Thread Count Stats
Displays the number of running threads.
Settings Window
The Settings window provides controls for enabling GC logging, choosing the level of detail logged, and
whether the output is sent to standard out or saved to a new or existing file. ZVRobot captures the state
of this window's settings.
Since the information in this window does not change over time, set the ZVRobotVars.COUNT
variable in the ZVRobot.prop file to 1. This tells ZVRobot to collect only one snapshot. By default,
this one snapshot is displayed in work area of the page.
The following table describes the Settings window elements.
Settings Window Elements
Element
Description
Off
Disables GC logging for this ZVM instance. This is the default
setting.
Print GC
Enables GC logging with summarized output. Output is equivalent to
using the command line option -XX:+PrintGC.
Print GC details
Enables GC logging with detailed output. Output is equivalent to
Azul Systems
Confidential and Proprietary
259
Zing Installation and Administration Guide
Settings Window Elements
Element
Description
using the command line option -XX:+PrintGCDetails.
Options: File name
Type a file name to send GC log output instead of sending it to
standard out. Select the check box and type an existing file name to
append the output to the end of an existing file.
Features Window
The Features window provides controls for user access to certain ZVision and ZVRobot features in a
running ZVM. ZVRobot captures the state of his window's settings.
Since the information in this window does not change over time, set the ZVRobotVars.COUNT
variable in the ZVRobot.prop file to 1. This tells ZVRobot to collect only one snapshot. By default,
this one snapshot is displayed in work area of the page.
Features Window Elements
Element
Description
Azul Support
Enables user access to Azul Customer Support Options. Use of this
feature is at your risk because some of these diagnostic features can
interfere with the proper functioning of the ZVM, or cause the system to
crash.
Browse Instance
Enables users to browse a class instance, potentially exposing
sensitive data within the object.
Browse Class
Enables users to browse the static class data, potentially exposing
sensitive data within the class.
Browse Method
Enables users to browse a class method, potentially exposing
sensitive sensitive data within the method.
Environment
variables
Enables users to view the Environment Variables, potentially exposing
sensitive sensitive data.
Stack trace
Enables users to use the Stack Trace window, potentially exposing
sensitive sensitive data.
Azul Systems
Confidential and Proprietary
260
Zing Tools Management
Threads List Window
The Threads List window displays all threads, status of each thread, and links to detailed information
about a thread.
Click a hyperlink in the left column to display the collected information for a snapshot. The number of
snapshots and the interval between snapshot collection is specified by the ZVRobotVars.COUNT and
ZVRobotVars.SECOND variables in the ZVRobot.prop file.
The following table describes the Threads List window elements.
Threads List Window Elements
Element
Name
Description
Lists the name of the thread as assigned by the application.
The most common states for threads are:
l
running: the thread is executing on a processor core.
l
I/O wait: the thread is waiting for socket or file I/O.
l
acquiring monitor: the thread is waiting to acquire a synchronization monitor.
l
acquiring and releasing monitor: the thread a requires and releases a
synchronization monitor. This typically happens when the critical section of the
State
monitor is extremely small.
sleeping: the thread executed java.lang.Thread.sleep().
Other states, such as idle, semaphore wait, waiting on ZVM monitor Threads
lock, or Waiting on Monitor SLT lock indicate threads that are internal to the
ZVM. These states are not normally useful for analysis of application
performance. Also, during start up, compiler threads might be running. The time
(in seconds) that the thread has spent since entering the state is provided in
parenthesis after the waiting state indicators.
Details
Azul Systems
Provides links to a stack trace, tick profile, and object profile for the selected
thread. These functions are the same as the regular tick and profile features,
Confidential and Proprietary
261
Zing Installation and Administration Guide
Threads List Window Elements
Element
Description
except that only information on the selected thread displays. The following links
appear inline with threads listed in this window:
l
Stack: displays a thread execution details window.
l
Ticks: displays a per-thread profile.
l
Allocations: displays a thread-level Object profile. This window displays realtime counters for allocated bytes, allocated count, and average object size per
object type allocated by the thread since the most recent GC cycle start.
Stack Trace Window
The Stack Trace window lists all threads and displays whether a thread is executing or is blocked. This
can help to determine changes to make in the application code. CPU time displays for each thread.
Click a hyperlink in the left column to display the collected information for a snapshot. The number of
snapshots and the interval between snapshot collection is specified by the ZVRobotVars.COUNT and
ZVRobotVars.SECOND variables in the ZVRobot.prop file.
[Insert Graphic]
Deadlocks Window
The Deadlocks window lists detected deadlock cycles. Threads involved for each cycle of locks are
listed with the monitor type the thread is trying to acquire and the thread that is the current owner.
Click a hyperlink in the left column to display the collected information for a snapshot. The number of
snapshots and the interval between snapshot collection is specified by the ZVRobotVars.COUNT
and ZVRobotVars.SECOND variables in the ZVRobot.prop file.
Contention Window
The Contention window provides statistical information for each monitor and sorts monitors by total
acquire time. To sort by field, click on the column heading.
Azul Systems
Confidential and Proprietary
262
Zing Tools Management
Click a hyperlink in the left column to display the collected information for a snapshot. The number of
snapshots and the interval between snapshot collection is specified by the ZVRobotVars.COUNT and
ZVRobotVars.SECOND variables in the ZVRobot.prop file.
The following types of monitors appear in the Contention window:
l
Java-level monitors: identified by their class name.
l
Internal JVM monitors: identified with the extension _lock. If there are a high number of internal
JVM monitors, examine the verbose garbage collection (GC) logs. Observe if there are a large
number of GC pauses. Contact Azul customer support if this becomes a problem.
The total acquire time parameter is the time, in milliseconds, for acquires to occur for a given internal or
Java-level monitor.
The following table describes the Contention List window elements.
Contention Window Elements
Element
Name
Description
The name of the monitor.
l
Total: the time, in milliseconds, for acquires to occur for a given internal
or Java-level monitor. This is not tracked for lightly contended thin
Acquire Time (ms)
monitors.
Blocking acquires
Azul Systems
l
Max (ms): the maximum time, in milliseconds, to perform an acquire.
l
Count: the number of blocking acquires.
Confidential and Proprietary
263
Zing Installation and Administration Guide
Contention Window Elements
Element
Waits
Description
l
Count: the number of wait cycles for the thread.
l
Max : the maximum time, in milliseconds, of wait cycles that have
occurred.
l
Total: Cumulative wait time in milliseconds
Tick Profile Window
The Tick Profile window provides information to allow calculation of how processor resources are used
for a recorded period of time.
The time interval over which the shown ticks are collected depends on the number and activity of the
threads in the Java process. The busier the threads, the faster the data is collected into the fixed-size
structures that hold the data. See Statistical Tick Profiler in Using Zing Vision for more information the
statistical tick profiler.
Note
Method names in the tick profiler represent compiled methods, those no longer
being interpreted. All time spent by methods being interpreted is summed as one
entry named Interpreter.
Click a hyperlink in the left column to display the collected information for a snapshot. The number of
snapshots and the interval between snapshot collection is specified by the ZVRobotVars.COUNT and
ZVRobotVars.SECOND variables in the ZVRobot.prop file.
The following table describes the Tick Profile window elements.
Azul Systems
Confidential and Proprietary
264
Zing Tools Management
Tick Profile Window Elements
Element
Description
Percent
The percentage of CPU resources (% of the total number of ticks
collected) the method or function used.
Ticks
The number of (ticks) attributed to the function or method.
A description of the function being counted. The following are
definitions for specific functions:
l
VM_GC Task: the garbage collector for the ZVM.
l
ARTA_Performance_Monitor: the function that collects and
Source
displays ZVRobot data.
l
VM_C2 Compiler: the ZVM JIT compiler compiling methods
indicating that an application is still warming up.
l
generate_monitorenter: the amount of time spent busy-waiting to
acquire monitors.
Memory Summary Window
The Memory Summary window provides a summary of total memory usage and Contingency memory
usage. See Using the Zing Virtual Machine for detailed information about Contingency memory.
Click a hyperlink in the left column to display the collected information for a snapshot. The number of
snapshots and the interval between snapshot collection is specified by the ZVRobotVars.COUNT and
ZVRobotVars.SECOND variables in the ZVRobot.prop file.
The following table describes the Memory Summary window elements.
Azul Systems
Confidential and Proprietary
265
Zing Installation and Administration Guide
Memory Summary Window Elements – Java Heap Usage
Element
Description
Used
The amount of memory currently used by the ZVM.
Capacity
The amount of memory allocated for use by the ZVM.
Requested
capacity
The maximum amount of memory requested for use by the ZVM (Xmx).
GC cycle count
The number of garbage collection cycles.
Memory Summary Window Elements – Memory Accounts
Element
Description
VM internal
Shows allocated, balance, Reserved, and Contingency memory usage for
the ZVM.
Java heap
Shows allocated, balance, Reserved, and Contingency memory usage for
the Java heap.
Java Pause
Prevention
Shows allocated, balance, Reserved, and Contingency memory usage for
the Java Pause Prevention pool.
GC Summary Window
The GC Summary window provides a cumulative summary of the recent GC cycle history. See
Generational Pauseless Garbage Collection (GPGC) in Using the Zing Virtual Machine for more
information about the GPGC garbage collector.
Click a hyperlink in the left column to display the collected information for a snapshot. The number of
snapshots and the interval between snapshot collection is specified by the ZVRobotVars.COUNT
and ZVRobotVars.SECOND variables in the ZVRobot.prop file.
Azul Systems
Confidential and Proprietary
266
Zing Tools Management
The following table describes the Memory Summary window elements.
GC Summary Window Elements
Element
Description
New generation
cycles
Lists the summary values for memory usage during current cycle.
Old generation cycle
Lists the summary values for memory usage during generational
cycle.
GC History Window
The GC history window provides a detailed summary of the recent GC cycle history with a row per
cycle. Click column titles to sort by relevance.
Azul Systems
Confidential and Proprietary
267
Zing Installation and Administration Guide
Click a hyperlink in the left column to display the collected information for a snapshot. The number of
snapshots and the interval between snapshot collection is specified by the ZVRobotVars.COUNT and
ZVRobotVars.SECOND variables in the ZVRobot.prop file.
The following table describes the GC Memory History window elements.
GC Memory History Window Elements
Element
Description
Interval (sec): elapsed time between successive GC cycle starts.
Cycle
Pause ratio: ratio of time application spent in safe point pauses during the
cycle to cycle interval time.
Mode
l
p: indicates if allocation from pause pool is enabled
l
g: indicates if allocation from Contingency pool is enabled.
l
Live (MB): the amount of Java memory used by all application objects
currently not known to be dead.
Heap
l
Peak (MB): the peak size of the Java heap recorded during the GC cycle.
l
Peak (pgs): the peak size of the Java heap recorded during the GC cycle (in
MB by page increments).
l
Max (MB): maximum amount of heap memory guaranteed to the
application. This is generally set by –Xmx option.
Reserved
Pause
Generations
l
Peak (MB): the peak size of the memory Reserved during the GC cycle.
l
Used (MB): the amount of Reserved memory used during the GC cycle.
These values are used for internal Zing diagnostic purposes only.
l
Total fragmentation (MB): the total fragmentation in the heap.
l
New used (MB): the amount of memory used by all young objects not known
to be dead during the GC cycle.
l
Azul Systems
Old used (MB): the amount of memory used by all old objects not known to
Confidential and Proprietary
268
Zing Tools Management
GC Memory History Window Elements
Element
Description
be dead during the GC cycle.
l
Perm used (MB): the amount of memory used by all permanent objects not
known to be dead during the GC cycle.
Garbage
Pages
Pauses
l
Live (MB): the live memory in the generation(s) being collected.
l
Fragmentation (MB): the fragmentation in the generation(s) being collected.
l
Found (MB): the garbage found in the generation(s) being collected.
l
Freed (MB): the garbage freed in the generation(s) being collected.
l
Sideband limited (MB): used for internal Zing diagnostics purposes only.
These values are used for internal Zing diagnostic purposes only.
l
Pause 1 start (sec): the start time of the pause 1 safepoint.
l
Pause 1 duration (ms): the duration time of the pause 1 safepoint.
l
Pause 2 start (sec): the start time of the pause 2 safepoint.
l
Pause 2 duration (ms): the duration of the pause 2 safepoint
l
Pause 3 duration (ms): the total duration of all pause 3 safepoints for this GC
cycle.
Pauses
(GPGC only)
l
Pause 3 start (sec): the start time of the pause 3 safepoint.
l
Pause 3 count: the number of pause 3 safepoints.
l
Pause 3 max: the maximum duration of any one pause 3 safepoint.
l
Duration (sec.): the elapsed time from the end of the last GC cycle to the
start of this GC cycle.
Intercycle
l
Allocation rate (MB/s): the object allocation rate in the generation being
collected in the period between the last GC cycle to the start of this GC
cycle.
Intercycle
(GPGC only)
Intracycle
l
Perm allocation rate (MB/s): the object allocation rate in the permanent
generation during this GC cycle.
l
Duration (sec.): the GC cycle elapsed time; the duration of time from the
beginning to the end of this GC cycle.
Azul Systems
Confidential and Proprietary
269
Zing Installation and Administration Guide
GC Memory History Window Elements
Element
Description
l
Allocation rate (MB/s): the object allocation rate in the generation being
collected during this GC cycle.
Intracycle
(GPGC only)
l
Perm allocation rate (MB/s): the object allocation rate in the permanent
generation during this GC cycle.
l
Total threads: the total count of Java threads in the ZVM.
l
Threads delayed: the number of threads blocked trying to allocate memory
during the GC cycle. When the number of delayed threads is greater than 0,
application performance is affected due to lack of available memory.
App threads
l
Average thread delay (sec): the average delay for threads blocked trying to
allocate during the GC cycle.
l
Max thread delay (sec): the maximum delay for threads blocked trying to
allocate.
Compilers – Task Window
The Compilers Task window of ZVRobot displays the status of Server and Client compiler tasks.
Click a hyperlink in the left column to display the collected information for a snapshot. The number of
snapshots and the interval between snapshot collection is specified by the ZVRobotVars.COUNT and
ZVRobotVars.SECOND variables in the ZVRobot.prop file.
The following table describes the Compiler Tab window elements.
Azul Systems
Confidential and Proprietary
270
Zing Tools Management
Compiler Tab Window Elements
Element
Description
Compiling
Status of threads compiling.
Waiting
Status of threads waiting compiling.
Class List Window
The Class List window displays a list of application and Java class library classes that are loaded in the
ZVM.
Click a hyperlink in the left column to display the collected information for a snapshot. The number of
snapshots and the interval between snapshot collection is specified by the ZVRobotVars.COUNT
and ZVRobotVars.SECOND variables in the ZVRobot.prop file.
Using Zing System Tools
This section describes how to use standalone tool, zing-ps, that is packaged with Zing System
Tools.
For information on ZVM packaged tools, ZVision and ZVRobot, refer to Using Zing Vision and Using
Zing Vision Robot.
Using the zing-ps Tool
The memory management functions of the Zing System Tools are performed automatically when a
ZVM instance runs a Java application. The zing-ps command returns a snapshot of current Zing
memory usage. It displays information about active Zing processes, system behavior, and Zing
memory usage.
zing-ps is installed automatically when ZST is installed. It is run from the command line. Use this
command before you install, update, or restart the ZST or modify the ZST configuration.
This tool does not collect history. To collect zing-ps information over time, use a while loop to run the
zing-ps command at specified time increments, for example every 10 seconds, and log the data
generated. Then use the logged data in a numbers-to-graph tool, such as Excel to generate graphs.
For quick reference, a man page is provided for this command. At the command line, type:
$ man zing-ps
The zing-ps command usage is:
zing-ps [-p pid] [-a] [-V] [-n] [-c] [-j] [-s] [-l] [-d] [-f] [-b]
[-m] [-h] [--comma] [--acct] [--acct2] [-partition P]
Where –
Azul Systems
Confidential and Proprietary
271
Zing Installation and Administration Guide
Command
Description
-V
Prints the ZST version.
-p
[PID]
Prints information about specified process.
-a
Prints the arguments included in the process generating command.
-n
Prints only the name from the process generating command.
-c
Prints only the process generating command.
-j
Prints pre-defined portions of the process generating command.
-s
Prints the memory summary.
-l
Prints the legend.
-d
Suppresses printing of the details for each process.
-f
Suppresses printing of fund information.
-b
Prints values in raw byte counts.
-m
Rounds the byte counts to megabytes.
-h
Rounds the byte counts to, (human readable), multiples of powers of
1024 and prints appropriate suffix: K=Kilo, M=Mega, G=Giga, T=Tera
--comma
Comma separated tabular format.
--acct
Print basic memory accounting information.
--acct2
Print advanced memory accounting information.
partiti
on P
For Internal Use Only.
Some options are exclusive to each others. Many options can be used in combination. See Format
Command Option Samples for listing of mutually exclusive options.
Command Return Overview
Depending upon the zing-ps options specified, zing-ps responses provide the following:
Azul Systems
Confidential and Proprietary
272
Zing Tools Management
l
Legend for Memory Usage Labels
l
Summary of Total Memory Usage
l
Memory Usage for Each Running ZVM Process
l
Fund Breakdown
l
Memory Usage by Account
Legend for Memory Usage Labels
The memory usage report labels include the following:
Values as noted K=Kilo, M=Mega, G=Giga, T=Tera
LRSS=Linux Memory in use
ZRSS=Zing Memory in -use
ZCTG=Zing Contingency Memory in use
ZXmx=Reserved from Zing Memory
To display the legend, add the -1 option to the zing-ps command.
Additional Field Descriptions
Depending upon the options used, zing-ps returns four tables of information. The following are samples
of the table headings.
l
System Memory Summary
Memory Usage Summary
Linux Memory
l
Reservable
Contingency
Process usage
USER
l
Zing Memory
PID
TIME
LRSS
ZRSS
ZCTG
ZXmx
CMD
System fund usage (--acct and --acct2)
NAME
BALANCE
MAXIMUM
NAME
BALANCE(now/min/max KB)
ALLOCATED(now/min/max KB)
MAXIMUM(KB) ACS_TO_CREDIT
FND
ODFND
l
Account usage per process
NAME
BALANCE
ALLOCATED
MAXIMUM
FND
ODFND
The following are brief descriptions of the zing-ps information table headings. See Understanding
Zing Memory Use for additional descriptions.
l
USER – Username used to issue the Java command that started the process.
l
CMD – Java command used to start the process.
Azul Systems
Confidential and Proprietary
273
Zing Installation and Administration Guide
l
PID – Process identification number.
l
TIME – Process CPU time.
l
NAME – Type of process.
l
BALANCE – Available balance. This can be negative if contingency memory is being used.
l
ALLOCATED – Amount used.
l
now – Amount at this current moment in time. When the zing-ps command was issued.
l
min and max – The min and max values since the end of the last garbage collection.
Values are reset after the garbage collection completes and the previous garbage collection
that ended could, in this case, be of a type that starts and completes before the current type
has completed.
l
ACS_TO_CREDIT – The internally labeled name of the Java memory, the Java committable
memory. For example:
JAVA_COMMITTED_MEMORY_ACCOUNT = AZMM_JHEAP_ACCOUNT,
// 2)
0xfff… {CommitableMemoryAccount==2}{sequence number}
l
MAXIMUM – Amount reserved.
l
FND – Fund.
l
ODFND – Overdraft Fund.
Summary of Total Memory Usage
Memory Usage Summary Linux Memory Zing Memory Reservable Contingency
Used: 20433 6144 6144 0
Free: 11828 90636 80960 4838
Total: 32261 96780 87104 4838
Total Physical Memory: 129041
To view the summary, add the -s option to the zing-ps command.
Memory Usage for Each Running ZVM Process
Found 8 processes
USER
PID
TIME
LRSS
ZRSS
ZCTG
ZXmx
63
434
0
1024
CMD
xguan
25160
00:00:03
java -cp invokeinstruction MH
preames
12045
00:01:22
81
454
0
1024
java -cp hamcrest-core-1.3.jar:./ org.junit.runner.JUnitCore
MHTests
lakshv
21990
00:02:25
203
1152
java -jar gcLogAnalyser-99.99.99.99-295.jar
Azul Systems
Confidential and Proprietary
0
6144
274
Zing Tools Management
preames
30900
00:01:43
tnameserv -ORBInitialPort 18322
66
472
0
1024
preames
30849
08:26:34
rmid -port 18321 -log Rmid
181
452
0
1024
preames
30337
00:02:44
tnameserv -ORBInitialPort 16848
61
460
0
1024
preames
30043
00:02:30
rmid -port 16847 -log Rmid
44
446
0
1024
0
1024
ltene
4324
00:31:18
47
tnameserv tnameserv -ORBInitialPort 20735
570
The default for zing-ps return is the same as zing-ps -j. Including zing-ps command options
adds or removes return content relative to the default -j return. To not display the per process detail,
add the -d option.
Fund Breakdown
Fund breakdown (total 8)
NAME
BALANCE
MAXIMUM
fund[0]:
Committed
84892712960
91335163904
fund[1]:
Overdraft
5073010688
5073010688
fund[2]:
Unknown
0
0
fund[3]:
PausePrevention
5073010688
5073010688
fund[4]:
Unknown
0
0
fund[5]:
Unknown
0
0
fund[6]:
Unknown
0
0
fund[7]:
Unknown
0
0
To view the fund breakdown, add the --acct or --acct2 option to the zing-ps command. To not
display the fund breakdown, add the -f option.
Memory Usage by Account per Process
Found 6 processes
USER
PID PROCESS
preames
30900 /home/preames/work/java7/sandbox/azlinux/jdk6/x86_
64/
product/bin/tnameserv
NAME
BALANCE
ALLOCATED MAXIMUM
account[0]: default
62914560
2097152
65011712
account[1]: emergency_gc
0
0
0
account[2]: heap
Azul Systems
FND ODFND
1
1
-1
-1
517996544 490733568 6081740800 1
1
Confidential and Proprietary
275
Zing Installation and Administration Guide
account[3]: pause_prevention 0
0
5073010688 3
3
account[4]: unknown
0
0
0
-1
-1
account[5]: unknown
0
0
0
-1
-1
account[6]: unknown
0
0
0
-1
-1
account[7]: unknown
0
0
0
-1
-1
To view the fund breakdown, add the --acct or --acct2 option to the zing-ps command.
Format Command Sample Options
The five examples in the following subsections provide sample format command options to use with
Zing System Tools.
Example 1: No parameters used, returns the number of ZVM processes.
With no parameters, zing-ps returns the number of ZVM processes and memory usage for each
itemized process. The -j option conditions are used as the default option for formatting the CMD field.
$ zing-ps
Found 2 processes
USER
CMD
PID
TIME
LRSS
ZRSS
ZCTG
root 25310
00:00:08
117260288
568328192
0
/home/buildmaster/nightly/AVM/dev/in_progress/avm-dev115/sandbox/azlinux/jdk6/x86_64/product/bin/java
ZXmx
1073741824
root 25267
00:00:05
88256512
486539264
0
1073741824
home/buildmaster/nightly/AVM/dev/in_
progress/latest/sandbox/azlinux/jdk6/x86_64/product/bin/java
If there are no ZVMs running then it is safe to stop the ZST service.
$ zing-ps
Found 0 processes
Example 2: Comma separated.
The –comma option includes commas as a separator between table columns. This format can be
imported into CSV accepting tool, such as Excel. The option applies to all table formats. Sufficient
commas are included to align all columns.
$ zing-ps --acct --comma
,Fund breakdown (total 8),,
Azul Systems
FUND,
NAME,
BALANCE,
MAXIMUM
0 ,
Committed,
84892712960,
91335163904
1 ,
Overdraft,
5073010688,
5073010688
2 ,
Unknown,
0,
0
3 ,
PausePrevention,
5073010688,
5073010688
4 ,
Unknown,
0,
0
Confidential and Proprietary
276
Zing Tools Management
5 ,
Unknown,
0,
0
6 ,
Unknown,
0,
0
7 ,
Unknown,
0,
0
Example 3: CMD column options.
Select one command (CMD) column display option at a time: -a, -c, -j, or -n.
l
Using the -a option displays all the arguments included in the command used to initiate the
process.
$ zing-ps -a
Found 2 processes
USER
PID
TIME
LRSS
ZRSS
ZCTG
ZXmx CMD
preames
32691
00:00:05
94
626
0
1024
/home/preames/work/java7/sandbox/azlinux/jdk6/x86_
64/product/bin/java -XX:-ShowMessageBoxOnError XX:+UseGenPauselessGC -Xloggc:gc.log -Xmx1g Djava.awt.headless=true -Xverify Djava.security.policy=/home/qatest/sw/java/openjdk/tck/6b/lib/jck.
policy -Dsun.net.maxDatagramSockets=1024 -Djaxp.debug=true Djava.io.tmpdir=/home/preames/work/java7/sandbox/test_results_
tck/TCK-6b/5101-to-5200/tmp -cp
/home/buildmaster/nightly/ZVM/dev/in_progress/avm-dev143/sandbox/qa/AVM/lib/qajlib6.jar:.::/home/qatest/sw/java/openjdk
/tck/6b/classes com.azulsystems.qa.stress.runthese.Runthese -t 6b
/home/preames/work/java7/sandbox/test_results_tck/TCK-6b/5101-to5200/script
preames
32467
00:00:06
93
544
0
1024
/home/preames/work/java7/sandbox/azlinux/jdk6/x86_
64/product/bin/java -XX:-ShowMessageBoxOnError XX:+UseGenPauselessGC -Xloggc:gc.log -Xmx1g Djava.awt.headless=true -Xverify Djava.security.policy=/home/qatest/sw/java/openjdk/tck/6b/lib/jck.
policy -Dsun.net.maxDatagramSockets=1024 -Djaxp.debug=true Djava.io.tmpdir=/home/preames/work/java7/sandbox/test_results_
tck/TCK-6b/7301-to-7400/tmp -cp
/home/buildmaster/nightly/ZVM/dev/in_progress/avm-dev143/sandbox/qa/AVM/lib/qajlib6.jar:.::/home/qatest/sw/java/openjdk
/tck/6b/classes com.azulsystems.qa.stress.runthese.Runthese -t 6b
/home/preames/work/java7/sandbox/test_results_tck/TCK-6b/7301-to7400/script
l
Using the -c option displays the command used to initiate the process without the command
options.
$ zing-ps -c
Azul Systems
Confidential and Proprietary
277
Zing Installation and Administration Guide
Found 2 processes
USER
CMD
l
PID
TIME
LRSS
ZRSS
ZCTG
ZXmx
preames
32467
00:00:16
154
572
/home/preames/work/java7/sandbox/azlinux/jdk6/x86_
64/product/bin/java
0
1024
preames
30684
00:00:03
80
538
/home/preames/work/java7/sandbox/azlinux/jdk6/x86_
64/product/bin/java
0
1024
Using the -j option displays pre-defined options from the command used to initiate the
process. This is the default zing-ps return.
$ zing-ps -j
Found 8 processes
USER
PID
TIME
LRSS
ZRSS
ZCTG
ZXmx CMD
xguan
468
00:00:03
65
436
0
1024
java -cp classes com.azulsystems.qa.stress.runthese.Runthese -t 7
invokedynamic
preames
12045
00:03:44
64
454
0
1024
java -cp hamcrest-core-1.3.jar:./ org.junit.runner.JUnitCore
MHTests
lakshv
21990
00:13:13
236
960
java -jar gcLogAnalyser-99.99.99.99-295.jar
0
6144
preames
30900
00:06:47
tnameserv -ORBInitialPort 18322
43
478
0
1024
528
436
0
1024
preames
30337
00:07:43
tnameserv -ORBInitialPort 16848
43
462
0
1024
preames
30043
00:07:05
rmid -port 16847 -log Rmid
33
446
0
1024
ltene
4324
00:48:24
tnameserv tnameserv -
53
636
0
1024
preames
30849 1-16:43:45
rmid -port 18321 -log Rmid
l
Using the -n option displays only the name of the command used to initiate the process.
$ zing-ps -n
Found 2 processes
USER
CMD
Azul Systems
PID
TIME
LRSS
Confidential and Proprietary
ZRSS
ZCTG
ZXmx
278
Zing Tools Management
preames
tnameserv
30900
00:00:04
64
470
0
1024
preames
rmid
30849
00:00:03
63
480
0
1024
Example 4: Units displayed options.
Select one unit display option at a time: -b, -h, or -m.
l
Using the -b option displays the memory values in bytes. Fund and per-process account
information is in bytes by default, unless otherwise noted.
$ zing-ps -b
Found 9 processes
USER
PID
TIME
LRSS
ZRSS
ZCTG
ZXmx CMD
preames 14096 00:00:07 71397376 473956352 0
1073741824
java -cp hamcrest-core-1.3.jar:./ org.junit.runner.JUnitCore
MHTests
xguan
468 00:00:03 67801088 457179136 0
1073741824
java -cp classes com.azulsystems.qa.stress.runthese.Runthese -t 7
invokedynamic
preames 12045 00:03:44 67457024 476053504 0
1073741824
java -cp hamcrest-core-1.3.jar:./ org.junit.runner.JUnitCore
MHTests
l
lakshv
21990 00:13:12 247308288 1006632960 0
java -jar gcLogAnalyser-99.99.99.99-295.jar
6442450944
preames 30900 00:06:47 44728320
tnameserv -ORBInitialPort 18322
501219328
0
1073741824
preames 30849 1-16:43:33 553218048 457179136
rmid -port 18321 -log Rmid
0
1073741824
preames 30337 00:07:42
4716032
tnameserv -ORBInitialPort 16848
484442112
0
1073741824
preames 30043 00:07:05
34119680
rmid -port 16847 -log Rmid
467664896
0
1073741824
ltene
4324 00:48:24
55144448 666894336
tnameserv tnameserv -ORBInitialPort 20735
0
1073741824
Using the -h option displays the memory values in human readable values rounded to the units
as noted K=Kilo, M=Mega, G=Giga, T=Tera.
$ zing-ps -h
Found 8 processes
USER
CMD
Azul Systems
PID
TIME
LRSS
ZRSS
ZCTG
Confidential and Proprietary
ZXmx
279
Zing Installation and Administration Guide
xguan
468
00:00:03
65 M
436 M
0
1024 M
java -cp classes com.azulsystems.qa.stress.runthese.Runthese -t 7
invokedynamic
preames 12045
00:03:44
64 M
454 M
0
1024 M
java -cp hamcrest-core-1.3.jar:./ org.junit.runner.JUnitCore
MHTests
l
lakshv 21990
00:13:13
236 M
60 M
java -jar gcLogAnalyser-99.99.99.99-295.jar
0
6144 M
preames 30900
00:06:47
43 M
tnameserv -ORBInitialPort 18322
478 M
0
1024 M
preames 30849
1-16:43:44
rmid -port 18321 -log Rmid
528 M
436 M
0
1024 M
preames 30337
00:07:43
43 M
tnameserv -ORBInitialPort 16848
462 M
0
1024 M
preames 30043
00:07:05
rmid -port 16847 -log Rmid
446 M
0
1024 M
ltene
4324
00:48:24
53 M
636 M
tnameserv tnameserv -ORBInitialPort 20735
0
1024 M
33 M
Sample using -h with -s options.
$ zing-ps -s -h
Memory Usage Summary
Linux Memory
Zing Memory
Reservable
Contingency
Used:
11 G
14 G
14 G
0
Free:
21 G
81 G
71 G
4838 M
Total:
32 G
95 G
85 G
4838 M
Total Physical Memory:
126 G
Found 9 processes
USER
PID
TIME
LRSS
ZRSS
ZCTG
ZXmx CMD
preames
5862
00:00:06
78 M
452 M
0
1024 M
java -cp hamcrest-core-1.3.jar:./ org.junit.runner.JUnitCore
MHTests
xguan
468
00:00:03
65 M
436 M
0
1024 M
java -cp classes com.azulsystems.qa.stress.runthese.Runthese -t 7
invokedynamic
preames 12045
00:03:44
64 M
454 M
0
1024 M
java -cp hamcrest-core-1.3.jar:./ org.junit.runner.JUnitCore
MHTests
Azul Systems
Confidential and Proprietary
280
Zing Tools Management
l
lakshv
21990
00:13:14
236 M
960 M
java -jar gcLogAnalyser-99.99.99.99-295.jar
0
6144 M
preames 30900
00:06:48
43 M
tnameserv -ORBInitialPort 18322
478 M
0
1024 M
preames 30849 1-16:47:05
rmid -port 18321 -log Rmid
529 M
436 M
0
1024 M
preames 30337
00:07:44
43 M
tnameserv -ORBInitialPort 16848
462 M
0
1024 M
preames 30043
00:07:06
rmid -port 16847 -log Rmid
446 M
0
1024 M
ltene
4324
00:48:25
53 M
636 M
tnameserv tnameserv -ORBInitialPort 20735
0
1024 M
33 M
Using the -m option displays the memory values in megabytes. No suffix is included. This is the
default value for basic zing-ps (without --acct or --acct2 options). Used with the --
acct or --acct2 options, this changes the display to megabytes, rounded to the nearest
megabyte.
$ zing-ps --acct -m
Fund breakdown (total 8)
NAME
BALANCE
MAXIMUM
fund[0]:
Committed
73792
87104
fund[1]:
Overdraft
4838
4838
fund[2]:
Unknown
0
0
fund[3]:
PausePrevention
4838
4838
fund[4]:
Unknown
0
0
fund[5]:
Unknown
0
0
fund[6]:
Unknown
0
0
fund[7]:
Unknown
0
0
Found 8 processes
USER
PID PROCESS
xguan
468 /home/xguan/opt/zing7/bin/java
NAME
BALANCE
ALLOCATED
MAXIMUM
FND
account[0]:
256
default
60
2
62
1
-
account[1]:
255
emergency_gc
0
0
0
-1
-
ODFND
Azul Systems
Confidential and Proprietary
281
Zing Installation and Administration Guide
account[2]:
65021
heap
528
434
5800
1
-
account[3]: pause_prevention
65021
0
0
4838
3
-
account[4]:
252
unknown
0
0
0
-1
-
account[5]:
251
unknown
0
0
0
-1
-
account[6]:
250
unknown
0
0
0
-1
-
account[7]:
249
unknown
0
0
0
-1
-
preames
12045 ././sandbox/azlinux/cayenne/x86_64/debug/bin/java
NAME
BALANCE
ALLOCATED
MAXIMUM
FND
ODFND
account[0]:
-256
default
60
2
62
1
account[1]:
-255
emergency_gc
0
0
0
-1
account[2]:
65021
heap
510
452
5800
1
-
account[3]:
65021
pause_prevention
0
0
4838
3
-
account[4]:
-252
unknown
0
0
0
-1
account[5]:
-251
unknown
0
0
0
-1
account[6]:
-250
unknown
0
0
0
-1
account[7]:
-249
unknown
0
0
0
-1
lakshv
21990 /home/buildmaster/nightly/AVM/dev/in_
progress/avm-dev-152/sandbox/azlinux/jdk6/x86_64/product/bin/java
NAME BALANCE
ALLOCATED
MAXIMUM
FND
2
370
1
ODFND
account[0]:
256
Azul Systems
default
368
Confidential and Proprietary
-
282
Zing Tools Management
account[1]:
255
emergency_gc
0
0
0
-1
-
account[2]:
65021
heap
4816
958
10612
1
-
account[3]: pause_prevention
65021
0
0
4838
3
-
account[4]:
252
unknown
0
0
0
-1
-
account[5]:
251
unknown
0
0
0
-1
-
account[6]:
250
unknown
0
0
0
-1
-
account[7]:
249
unknown
0
0
0
-1
-
ALLOCATED
MAXIMUM
FND
preames
30900
/home/preames/work/java7/sandbox/azlinux/jdk6/x86_
64/product/bin/tnameserv
NAME BALANCE
ODFND
account[0]:
256
default
60
2
62
1
-
account[1]:
255
emergency_gc
0
0
0
-1
-
account[2]:
65021
heap
486
476
5800
1
-
account[3]: pause_prevention
65021
0
0
4838
3
-
account[4]:
252
unknown
0
0
0
-1
-
account[5]:
251
unknown
0
0
0
-1
-
account[6]:
250
unknown
0
0
0
-1
-
account[7]:
249
unknown
0
0
0
-1
-
preames
30849
/home/preames/work/java7/sandbox/azlinux/jdk6/x86_
64/product/bin/rmid
NAME
BALANCE
ALLOCATED MAXIMUM
FND
ODFND
Azul Systems
Confidential and Proprietary
283
Zing Installation and Administration Guide
account[0]:
256
default
60
2
62
1
-
account[1]:
255
emergency_gc
0
0
0
-1
-
account[2]:
65021
heap
528
434
5800
1
-
account[3]: pause_prevention
65021
0
0
4838
3
-
account[4]:
252
unknown
0
0
0
-1
-
account[5]:
251
unknown
0
0
0
-1
-
account[6]:
250
unknown
0
0
0
-1
-
account[7]:
249
unknown
0
0
0
-1
-
preames
30337
/home/preames/work/java7/sandbox/azlinux/jdk6/x86_
64/product/bin/tnameserv
NAME
BALANCE
ALLOCATED
MAXIMUM
FND
account[0]:
256
default
60
2
62
1
-
account[1]:
255
emergency_gc
0
0
0
-1
-
account[2]:
65021
heap
502
460
5800
1
-
account[3]: pause_prevention
65021
0
0
4838
3
-
account[4]:
252
unknown
0
0
0
-1
-
account[5]:
251
unknown
0
0
0
-1
-
account[6]:
250
unknown
0
0
0
-1
-
account[7]:
249
unknown
0
0
0
-1
-
ODFND
Azul Systems
Confidential and Proprietary
284
Zing Tools Management
preames
30043
/home/preames/work/java7/sandbox/azlinux/jdk6/x86_
64/product/bin/rmid
NAME
BALANCE
ALLOCATED
MAXIMUM
FND
ODFND
account[0]:
256
default
60
2
62
1
-
account[1]:
255
emergency_gc
0
0
0
-1
-
account[2]:
65021
heap
518
444
5800
1
-
account[3]: pause_prevention
65021
0
0
4838
3
-
account[4]:
252
unknown
0
0
0
-1
-
account[5]:
251
unknown
0
0
0
-1
-
account[6]:
250
unknown
0
0
0
-1
-
account[7]:
249
unknown
0
0
0
-1
-
ltene
4324 /home/buildmaster/nightly/AVM/dev/in_
progress/avm-dev-146/sandbox/azlinux/jdk6/x86_
64/product/bin/tnameserv
NAME
BALANCE
ALLOCATED
MAXIMUM
FND
account[0]:
256
default
60
2
62
1
-
account[1]:
255
emergency_gc
0
0
0
-1
-
account[2]:
65021
heap
328
634
5800
1
-
account[3]: pause_prevention
65021
0
0
4838
3
-
account[4]:
252
unknown
0
0
0
-1
-
account[5]:
251
unknown
0
0
0
-1
-
account[6]:
250
unknown
0
0
0
-1
-
ODFND
Azul Systems
Confidential and Proprietary
285
Zing Installation and Administration Guide
account[7]:
249
unknown
0
0
0
-1
-
Example 5: Details of specific process by process id (PID).
l
Using the -p [PID] option lists the memory information for the specified process only.
$ zing-ps -p 3135
Found 1 processes
USER
PID
TIME
LRSS
ZRSS
ZCTG
ZXmx CMD
jcoha
135
01:33:44
786
4430
3466
1024 compiler.compiler
Samples Memory Usage by Account
Examples 6 through 10 in the following subsections depict methods of using the zing-ps command to
list information about ZVM processes.
Example 6: To monitor a test run.
If you run a while loop, logging it into a file every 10 seconds or so, you can then make graphs from
the log data, using your favorite raw-number-to-nice-graphs conversion tool.
# while true; do zing-ps --acct >> /tmp/zing-ps_log ; sleep 10;
done
Example 7: For ZVM fault diagnosis.
In the rare occurrence of a ZVM crash, to diagnose the fault it can be useful to have the output of
zing-ps --acct from the system after the crash occurs, but before a system reboot or a ZST
restart. If ZVM crashed, please contact your Azul representative who might direct you to collect the
zing-ps --acct output and upload to the Azul ftp site along with other data, logs and cores.
Note:
Giving the ZST too much memory (and exhausting the Linux memory for C heap and
other Linux stuff as a result) can have "very negative" effects on the system.
Example 8: Memory statistics with zero ZVM processes running.
This example shows memory funding with no ZVM processes running.
# zing-ps --acct
Fund breakdown (total 8)
fund[0]:
Azul Systems
NAME
BALANCE
MAXIMUM
Committed
22745710592
22745710592
Confidential and Proprietary
286
Zing Tools Management
fund[1]:
Overdraft
1262485504
1262485504
fund[2]:
Unknown
0
0
fund[3]:
PausePrevention
1262485504
1262485504
fund[4]:
Unknown
0
0
fund[5]:
Unknown
0
0
fund[6]:
Unknown
0
0
fund[7]:
Unknown
0
0
Found 0 processes
Example 9: Memory account details with 1 ZVM process running.
$ zing-ps --acct2
Process level Maximum: 0
NAME
KB)
BALANCE(now/min/max KB) ALLOCATED(now/min/max
MAXIMUM(KB) ACS_TO_CREDIT
FND ODFND
AC[0]:Default
32768/ 30720/ 32768
32768
0xffffffffffffff00
0
1
AC[1]:Unknown
0
0/ 0/ 0
0xffffffffffffff01
AC[2]:JavaHeap
2201600
2048/ 0/ 4096
0/ 0/ 0
-1
-1
11360/ 911360/ 1873920 2201600/ 1239040/
8067072
0xffffffffffff0203
0
1
AC[3]:PausePrevention 0/ 0/ 0
4954112
0xffffffffffff0203
0/ 0/ 0
3
3
Field descriptions:
l
Values are KB bytes.
l
account [0]: The maximum is equal to the Partition. The allocated plus the balance are
equal to maximum.
l
account [2]: Java Heap. If the balance is negative, the application is using contingency
memory.
l
account [3]: Maximum value allocated for Pause Prevention.
The value for account[2] heap ALLOCATED (used) is higher than AC[2] javaHeap
ALLOCATED now (used) because the zing-ps --acct (in these examples) was followed
by the zing-ps --acct2 and the application continued to allocate objects into the Java
heap.
Example 10: Memory funding details with 1 ZVM process running.
This example shows memory funding breakdown for ZST and allocation to the ZVM process PID
15479.
$ zing-ps --acct
Azul Systems
Confidential and Proprietary
287
Zing Installation and Administration Guide
Fund breakdown (total 8)
NAME BALANCEMAXIMUM
fund[0]:Committed 819357286442553311232
fund[1]:Overdraft 23634903042363490304
fund[2]:Unknown 00
fund[3]:PausePrevention23634903042363490304
fund[4]:Unknown 00
fund[5]:Unknown 00
fund[6]:Unknown 00
fund[7]:Unknown 00
Found 1 processes.
USER PID PROCESS
root 15479 /opt/zing/zing-jdk1.6.0_23-4.2.0.0.beta-14-x86_
64/bin/java
NAMEBALANCEALLOCATEDMAXIMUMFNDODFND
account[0]:default341835776209715234393292801
account[1]:emergency_gc000-1-1
account[2]:heap3122659328027892121603637929574401
account[3]:pause_prevention0027487790694433
account[4]:unknown000-1-1
account[5]:unknown000-1-1
account[6]:unknown000-1-1
account[7]:unknown000-1-1
Explanation of zing-ps Fund Breakdown
The Fund breakdown in Example 10: Memory funding details with 1 ZVM process running, shows the
memory funding as set up for the ZST. The output from the command then also shows that the zingps acct command has found one Zing Java process PID 15749 and gives the memory breakdown
for that Java process.
l
The table below takes a look at the example Fund Breakdown in more detail. l
The figures have been converted from bytes to megabytes (because it is easier to think about
megabytes than large numbers of bytes).
NAME
fund[0]:
Azul Systems
Committed
BALANCE
7814
Confidential and Proprietary
MAXIMUM
40582 288
Zing Tools Management
NAME
BALANCE
MAXIMUM
The Committed Maximum is the maximum Reservable Heap
The Committed Balance is the maximum Available Heap size for the JVM Xmx command option; and if other JVMs have already been started this will be
less than the maximum Reservable value.
Java Process ID 15479 has been started and set a –Xmx of 32768MB reducing
the Available heap from 40582MB to 7814MB
fund[1]:
Overdraft
2254
2254
The Overdraft is the Contingency Memory (using the ZST defaults, this is 5%
of the Zing Memory)
fund[3]:
PausePrevention
2254
2254
The PausePrevention is the Pause prevention memory (using the ZST
defaults, this is 5% of the Zing Memory)
The sum of the Committed Maximum and the Overdraft and Pause prevention maximums is
equal to the size of the Zing Memory (using the ZST defaults this is 75% of the available
Operating System memory, leaving 25% for the Linux Memory). In this case 40852 + 2254 +
2254 = 45090MB from which we can infer that if the ZST was configured with a 75/25
allocation then the available Operating System memory at the time the ZST was configured
was 60120MB (58.7GB). This can be useful to know; for example, if we made another
assumption that this was a 64GB RAM machine, then 5416MB has been allocated away
somewhere else before the Operating System was loaded – perhaps some DMA buffers for
10GBe NIC interfaces or some such.
Example Memory Fund Allocation for a Process
Carrying on with Example 10: Memory funding details with 1 ZVM process running, using Process
15479 for illustration.
l
The table below takes a look at the example Fund allocation in more detail. l
The figures have been converted from bytes to megabytes (because it is easier to think about
megabytes than large numbers of bytes).
/opt/zing/zing-jdk1.6.0_23-4.2.0.0.beta-14-x86_64/bin/PID: 15479
java
NAME
Account 0
Azul Systems
default
BALANCE
326
ALLOCATED MAXIMUM FND ODFND
2
328
Confidential and Proprietary
0
1
289
Zing Installation and Administration Guide
/opt/zing/zing-jdk1.6.0_23-4.2.0.0.beta-14-x86_64/bin/PID: 15479
java
NAME
BALANCE
ALLOCATED MAXIMUM FND ODFND
The Account 0 value is in bytes, and it amounts to 328MB. Its max is equal to
its funding (that is, the ALLOCATED plus the BALANCE are equal to max). Account 0 is used for some ZST-based memory stuff that is not purely in the
heap (such as kernel structures for holding the kernel page tables and such). The JVM assigns 1% of -Xmx to Account 0, and the rest to Account 2 (both
draw from Committed memory).
Account 1
emergency_
gc
0
0
0
-1
-1
heap
29780
2660
34694
0
1
Account 2 is the Java Heap. If its BALANCE shows as negative, your
application is using contingency memory.
Account 2
The Account 2 funding (BALANCE + ALLOCATED) together with Account 0
funding adds up exactly to the -Xmx value.
In this case -Xmx => 29780 + 2660 + 328 = 32768MB
The option flag -XX:JavaMemMax can be used to set the maximum for
Account 2 (heap).
Note: When -XX:JavaMemMax=<val> equals -Xmx<val>, then the ZVM
instance is not allowed to use any Contingency memory.
pause_
prevention
Account 3
0
0
262144
3
3
The Account 3 MAXIMUM value for the pause_prevention pool computes to
256GB (exactly). Having a large MAXIMUM value doesn't mean you have
funding for that much. It just means you can draw up to that much total
allocation (including dipping into overdraft accounts) if the ZST has resources
to give you in the fund you are drawing from. In the case of pause prevention, a max of 256GB just means that this process
can draw down the entire pause prevention pool if it wants to (it has zero
funding for account 3, so all of that draw shows up in negative balances equal
to the allocation level).
The MAXIMUM values for each account are not funding numbers. They are just absolute
limits on how much can be allocated in that account. This matters a lot when overdraft
Azul Systems
Confidential and Proprietary
290
Zing Tools Management
accounts are involved, as the MAXIMUM caps the total amount of allocation, even after the
balance goes negative. The committed funding is always the sum of BALANCE + ALLOCATED, this remains true
including times when the ZVM dips into an overdraft account, such as contingency (the
BALANCE will go negative at that point, and even though ALLOCATED is greater than
committed funding, the sum is still equal to committed).
The ZVM uses Zing memory only for the -Xmx (Java heap). So you need to leave enough Linux
memory to accommodate not only the other Linux stuff that's there, but also the ZVM’s C-heap. Use zing-ps and awk Commands to Show Available Heap Sizes in MB
The output of the zing-ps -acct command is in bytes; you might prefer to think in Megabytes.
This technique is based on the zing-ps –acct command and is a one-line script that uses the Linux
grep command to search the output combined with the Linux awk command to calculate and format
the output.
The output from the zing-ps –acct command is searched by grep for the line containing the
Committed string. The awk command uses $3 to select the third field which is the available balance
in column three of the output with the heading BALANCE. l
The following one-line script prints the maximum Available Heap size for the JVM command
option –Xmx and if other JVMs have already been started this is less than the maximum
Reservable value in column four headed MAXIMUM.
$ echo "Zing Available Heapsize (MB) is "`zing-ps --acct|grep
Committed|awk '{print int($3/(1024*1024))}'
l
To read the total Reservable value in MB, in this one-line script the awk command is modified
to use $4 to select the fourth field which is the MAXIMUM balance
$ echo "Zing Total Heapsize (MB) is "`zing-ps --acct|grep
Committed|awk '{print int($4/(1024*1024))}
Example 11: No JVMs have been started.
[root@localhost bin]# zing-ps --acct
Fund breakdown (total 8)
Azul Systems
NAME
BALANCE
MAXIMUM
fund[0]:
Committed
4041211904
4041211904
fund[1]:
Overdraft
222298112
222298112
fund[2]:
Unknown
0
0
fund[3]:
PausePrevention
222298112
222298112
fund[4]:
Unknown
0
0
fund[5]:
Unknown
0
0
fund[6]:
Unknown
0
0
Confidential and Proprietary
291
Zing Installation and Administration Guide
fund[7]:
Unknown
0
0
Found 0 processes.
[root@localhost bin]# echo "Zing Available Heapsize (MB) is "`zingps --acct|grep Committed|awk '{print int($3/(1024*1024))}'`
Zing Available Heapsize (MB) is 3854
[root@localhost bin]# echo "Zing Total Heapsize (MB) is "`zing-ps -acct|grep Committed|awk '{print int($4/(1024*1024))}'`
Zing Total Heapsize (MB) is 3854
root
5785 java
NAME
BALANCE
ALLOCATED
MAXIMUM FND
account[0]:
1
default
10485760
2097152
12582912
0
account[1]:
-1
emergency_gc
0
0
0
-1
1283457024
0
3
ODFND
account[2]:
1
heap 555745280
505413632
account[3]: pause_prevention
3
0
0 274877906944
account[4]:
-1
unknown
0
0
0
-1
account[5]:
-1
unknown
0
0
0
-1
account[6]:
-1
unknown
0
0
0
-1
account[7]:
-1
unknown
0
0
0
-1
[root@localhost bin]# echo "Zing Available Heapsize (MB) is "`zingps --acct|grep Committed|awk '{print int($3/(1024*1024))}'`
Zing Available Heapsize (MB) is 2830
[root@localhost bin]# echo "Zing Total Heapsize (MB) is "`zing-ps -acct|grep Committed|awk '{print int($4/(1024*1024))}'`
Zing Total Heapsize (MB) is 3854
Example 12: An Azul Inspector 1GB JVM has been started.
See Azul Systems website to download free Open Source tool Azul Inspector, at
http://www.azulsystems.com/dev_resources/azul_inspector.
[root@localhost bin]# zing-ps --acct
Fund breakdown (total 8)
fund[0]:
Azul Systems
NAME
BALANCE
MAXIMUM
Committed
2967470080
4041211904
Confidential and Proprietary
292
Zing Tools Management
fund[1]:
Overdraft
222298112
222298112
fund[2]:
Unknown
0
0
fund[3]:
PausePrevention
222298112
222298112
fund[4]:
Unknown
0
0
fund[5]:
Unknown
0
0
fund[6]:
Unknown
0
0
fund[7]:
Unknown
0
0
Found 1 processes.
USER
root
PID PROCESS
5785 java
NAME
BALANCE
ALLOCATED
MAXIMUM FND
account[0]:
1
default
10485760
2097152
12582912
0
account[1]:
-1
emergency_gc
0
0
0
-1
1283457024
0
3
ODFND
account[2]:
1
heap 555745280
505413632
account[3]: pause_prevention
3
0
0 274877906944
account[4]:
-1
unknown
0
0
0
-1
account[5]:
-1
unknown
0
0
0
-1
account[6]:
-1
unknown
0
0
0
-1
account[7]:
-1
unknown
0
0
0
-1
[root@localhost bin]# echo "Zing Available Heapsize (MB) is "`zingps --acct|grep Committed|awk '{print int($3/(1024*1024))}'`
Zing Available Heapsize (MB) is 2830
[root@localhost bin]# echo "Zing Total Heapsize (MB) is "`zing-ps -acct|grep Committed|awk '{print int($4/(1024*1024))}'`
Zing Total Heapsize (MB) is 3854
Azul Systems
Confidential and Proprietary
293
Zing Installation and Administration Guide
Troubleshooting Zing
Troubleshooting Zing describes using core bundles and lists common errors with possible solutions
when using to run your Java applications.
Core Files and Core Bundling with Zing
If you contact your Azul technical support representative regarding a failure, you might be asked to
provide core files or a core bundle that includes the core files and crash associated files, (such as, log
files generated by the ZVM instance, system log files, and versions of shared libraries used by the ZVM
instance). Providing this information, in addition to the core files, helps Azul to properly diagnose and
quickly identify failures due to Zing.
If a failure occurs due to a ZVM instance or ZVM instance-related Java application failure a core file is
created, provided ulimit -c unlimited is set. The ulimit setting is required to generate a core
file.
To create a core bundle, Zing provides the following:
l
Automatic ZVM Core Bundling, using the zing-core-pattern tool. By default, this is not
enabled. l
Manual ZVM Core Bundling, using the zing-core-bundler script. There is an option, -XX:+DumpMiniCore, that creates significantly smaller Zing core files. To adjust
the size of the core file, use this option with the -XX:DumpMiniCoreLevel=<x> option. Where <x>
specifies the information that is included in the mini core file.
Linux systems provide core_pattern to name core dump files. Standard Red Hat Enterprise Linux
6.x includes a core bundling tool, Automatic Bug Reporting Tool (ABRT), that contains the abrt
daemon.
l
Note:
l
If you are using core_pipe_limit, it is set to 4 by default. Changing the limit
can affect system responsiveness. Though setting the limit to 0 (zero) allows
unlimited concurrent core dumps, the cores might be corrupted. Therefore, setting
the core_pipe_limit to 0 is not recommended.
Using the zing-core-pattern tool, replaces the RHEL, SLES, or Ubuntu
system core_pattern, but provides a means to automatically bundle Zing core
files with diagnostically required support files.
Automatic ZVM Core Bundling (zing-core-pattern) for
RHEL/CentOS/Oracle Linux 5.x and Amazon Linux is not supported. Use the
Manual method. See "Manual ZVM Core Bundling" on page 299.
Azul Systems
Confidential and Proprietary
294
Troubleshooting Zing
Mini-Core File Option
There is an option, -XX:+DumpMiniCore, that creates significantly smaller Zing core files. This
option is specified through the Java application launch command and applied to the runtime Java
application. It is not applied to the ZVM host machine. When you use this option:
l
Generated Zing core files do not include application objects stored in the Java heap.
l
Time required to produce a Zing core file or core file bundle is reduced.
To further adjust the amount of data that is included in a mini-core file, include the option, XX:DumpMiniCoreLevel=<x>, where <x> is in [0, 4].
l
Level 0 (the current default) includes page infos and audit records
l
Level 1 includes marking information and the card tables
l
Level 2 includes trampolines, forwarding hash tables, preallocated pages etc.
l
Level 3 includes the heap mirror pages
l
Level 4 includes the object heap itself
Mini-Core File Option Requirements
The mini-core file -XX:+DumpMiniCore option requires ZST 5.7.5 or later and ZVM 5.10.x or later.
The mini-core file level option, -XX:DumpMiniCoreLevel=<x>, requires ZST 5.7.11 or later and
ZVM 15.05.0.0 or later.
If the option is specified and either the ZST version or ZVM version is not compatible, a message is
issued.
l
If the ZST version supports the -XX:+DumpMiniCore option, but the ZVM version does
not, an Unrecognized VM option warning is printed to STDERR. For example:
/home/buildmaster/nightly/ZVM/5.x-zvm/released/zvm-5.9.0.07/sandbox/azlinux/jdk6/x86_64/product/bin/java -XX:+DumpMiniCore
-version
Zing 64-Bit Tiered VM error: Unrecognized VM option
'+DumpMiniCore'
Could not create the Java virtual machine.
l
If the ZVM version supports the -XX:+DumpMiniCore option, but the ZST version does
not, a warning message is issued. For example:
/bin/java -jar -XX:+DumpMiniCore -version
Zing VM Warning: -XX:+DumpMiniCore is not supported by this ZST
(found API version 0x5e, requires API version >= 0x5f)
java version "1.6.0-internal-zing_99.99.99.99.dev"
Zing Runtime Environment for Java Applications (build 1.6.0internal-zing_99.99.99.99.dev-b25)
Zing 64-Bit Tiered VM (build 1.6.0-zing_99.99.99.99.dev-b501product-azlinuxM-X86_64, mixed mode)
Azul Systems
Confidential and Proprietary
295
Zing Installation and Administration Guide
Setting the Option
Include the option -XX:+DumpMiniCore, and optionally, -XX:DumpMiniCoreLevel=<x>, with
your Java launch command.
Some failures cannot be effectively diagnosed with a mini-core file. In this case, Azul support may
request that you try to reproduce the failure and produce a full-sized core file, by removing XX:+DumpMiniCore from your Java launch command.
l
When -XX:+DumpMiniCore option is not specified, a full core file, including the
application objects stored in the Java heap, is created if Zing crashes.
l
When -XX:+DumpMiniCore option is specified, a core file, that does not include the
application objects stored in the Java heap, is created if Zing crashes.
l
When -XX:+DumpMiniCore and the -XX:DumpMiniCoreLevel=<x> options are
specified, a core file, that includes the specified information, is created if Zing crashes.
Automatic ZVM Core Bundling
When the automatic core bundling tool, zing-core-pattern, is explicitly enabled, Zing can
automatically bundle core files with the shared libraries that were in use by the ZVM instance running
the Java application when it failed.
Notes:
Using the zing-core-pattern tool, replaces the RHEL, SLES, or Ubuntu system core_
pattern, but provides a means to automatically bundle Zing core files with diagnostically
required support files.
Automatic ZVM Core Bundling (zing-core-pattern) for RHEL/CentOS/Oracle Linux 5.x
and Amazon Linux is not supported. Use the manual method. See "Manual ZVM Core
Bundling" on page 299.
A core bundle called <corepid>.tgz.gpg is typically generated in the current working directory of
the application, containing the core file and all extra information needed for forensic analysis of the
core. The core bundle created with zing-core-pattern is encrypted.
Enabling Automatic ZVM Core Bundling
Note:
Automatic ZVM Core Bundling (zing-core-pattern) for
RHEL/CentOS/Oracle Linux 5.x and Amazon Linux is not supported.Use the
manual method. See "Manual ZVM Core Bundling" on page 299.
To enable automatic core bundling in RHEL, SLES, or Ubuntu systems:
1. Scan the file, core_pattern, for any reference piped ( | ) core_pattern.
$ more /proc/sys/kernel/core_pattern
Azul Systems
Confidential and Proprietary
296
Troubleshooting Zing
2. Search for the string:
l
For RHEL and SLES
|/usr/libexec/abrt-hook-ccpp /var/spool/abrt %s %c %p %u %g
%t %h %e 636f726500
l
For Ubuntu
|/usr/share/apport/apport %p %s %c
3. If a piped ( | ) core_pattern is found:
Stop the process included in core_pattern. For example:
l
For RHEL and SLES
$ service abrtd stop
l
For RHEL 6.3:
$ service abrt-ccpp stop
l
For Ubuntu
$ service apport stop
4. Ensure /usr/sbin is in your Zing executable path. For example:
To your appropriate profile file, add:
export PATH=/usr/sbin:$PATH
5. Enable system-config-zing-core-pattern, run the command:
$ system-config-zing-core-pattern enable
l
Enabling zing-core-pattern stores a copy of existing core_
pattern to a /var/run/core_pattern.save and overwrites
any existing core pattern in /proc/sys/kernel/core_
Notes:
pattern.
l
Automatic ZVM Core Bundling for RHEL/CentOS 5.x and Amazon
Linux is not supported. Use the Manual method. See "Manual ZVM
Core Bundling" on page 299.
Example system response:
You are about to enable zing-core-pattern
This replaces core pattern in '/proc/sys/kernel/core_pattern'
This affects core dump process on this system
Current /proc/sys/kernel/core_pattern : |/usr/sbin/zing-corepattern-hook %p %u %g %s %t %h %e %c
New core pattern will be : |/usr/sbin/zing-core-pattern-hook %p %u
%g %s %t %h %e %c
Azul Systems
Confidential and Proprietary
297
Zing Installation and Administration Guide
6. Choose to replace core_pattern with zing-core-pattern. Type y.
Do you want to proceed? (y)es or (n)o [default 'y']
7. Choose to include application .jar files in the bundle.
Do you want to bundle application data, including 'jar' files? (y)es
or (n)o [default 'y']:
Yes – collects and adds .jar, application related files, and .so files to the core bundle.
No – does not add .jar iles to the core bundle. .so files are included in the core bundle.
Example system response:
zing-core-pattern: INFO: zing-core-pattern enabled
8. Choose to include the non-java executable in the bundle.
Do you want the non-java executable (if any) to be bundled ? : (y)es
or (n)o [default 'y']:
Example system response:
Current /proc/sys/kernel/core_pattern : core
Saving existing core pattern to /var/run/core-pattern.save
zing-core-pattern: INFO: zing-core-pattern enabled
9. Ensure the piped ( | ) core_pattern process is not restarted on boot.
l
For RHEL or SLES
$ chkconfig abrdt off
l
For RHEL 6.3
$ chkconfig abrt-ccpp off
l
For Ubuntu
$ chkconfig apport off
Disabling Automatic ZVM Core Bundling
To disable automatic core bundling in RHEL, SLES, or Ubuntu systems:
1. Disable system-config-zing-core-pattern, run the command and confirm at the prompt:
$ system-config-zing-core-pattern disable
Note:
Disable zing-core-pattern retrieves a copy of the previously stored core_
pattern from /var/run/core_pattern.save and overwrites the zingcore-pattern. If a saved core_pattern is not found, then the system's core_
pattern is set to core.
Example system response:
You are about to disable zing-core-pattern.
This replaces the core pattern in '/proc/sys/kernel/core_pattern'
Azul Systems
Confidential and Proprietary
298
Troubleshooting Zing
Current core pattern: |/usr/sbin/zing-core-pattern-hook %p %u %g %s
%t %h %e
New core pattern will be: /tmp/cores/core.%e.%p.%h.%t
2. Confirm to disable automatic Zing core file bundling through the system-config-zing-corepattern command.
Do you want to proceed? (y)es or (n)o [default 'y']:y
zing-core-pattern: INFO: zing-core-pattern found, replacing it with
core-pattern in /var/run/core-pattern.save
zing-core-pattern: INFO: Zing-core-pattern disabled
Locating Automatic ZVM Core Bundles
If a ZVM instance failure occurs when this is enabled, a bundle including the core file and shared
libraries is created.
1. Locate the bundled libs.<corepid>.tgz.gpg file.
l
For RHEL/CentOS 6.x and 7.0, SLES, and Ubuntu systems:
The libs.<corepid>.tgz.gpg file is located in the folder where the ZVM instance Java
launch command is executed.
l
For RHEL/CentOS 5.3 or higher 5.x systems:
Thelibs.<corepid>.tgz.gpg file is located in /var/crash.
2. Send the generated bundle to Azul for failure analysis.
Manual ZVM Core Bundling
If automatic core bundling is not enabled, you can manually bundle the core files with the shared
libraries that were in use by the ZVM instance running the Java application when it failed.
If you are running a system based on Red Hat Enterprise Linux 5.2 or CentOS Linux 5.2, the automatic
core bundler is not on, so core files need to be bundled manually. For this purpose the script zingcore-bundler is provided. This script, as part of the ZST installation, is located in the /usr/sbin
subdirectory of the ZST installation.
The zing-core-bundler bundles the required files needed to debug a Zing Virtual Machine core. It
assembles, collects, and bundles the core file along with the shared objects, data, and executable files
in-use to an encrypted gzipped tarball (.tgz.gpg) archive.
To manually bundle the core file with shared libraries:
1. On RHEL, SLES, and Ubuntu systems, ensure the development tools package group is
installed.
$ yum groupinstall 'Development Tools'
Azul Systems
Confidential and Proprietary
299
Zing Installation and Administration Guide
Selected development tools are required to bundle RHEL 5 core files. If they are not
Note: installed, the shared libraries, which were used at the time of the crash and are
essential to processing the core file, are not included.
2. Locate the core file.
Core files are bundled and stored in the current working directory of the process that crashed.
Note:
Typically, you create a core bundle on the system where the core file was created, for
example hostA. If you run the zing-core-bundler command with a copy of the
core file on a different system, for example hostB, then include the -bundle-oncurrent-host option.
3. On Ubuntu systems, ensure that the strings command is available. If it is not, install the
package, binutils.
# apt-get install binutils
4. If you use the jsvc tool to start your application, to generate Zing core files:
# kill -SIGABRT <PID>
5. Ensure /usr/sbin is in your Zing executable path. For example:
To your appropriate profile file, add:
export PATH=/usr/sbin:$PATH
6. Run the zing-core-bundler command. At the ZVM host command line,where ZST is
installed:
To create a bundle with only related shared libraries that does not include the .jar files.
/usr/sbin/zing-core-bundler -core core<pid>
For example:
$ /usr/sbin/zing-core-bundler -core core.1234 -getall -hsfile hs_
err_pid1234.log
A compressed and encrypted tar file ending in .tgz.gpg is returned.
In this example, the package contains the core file, the .jar files, and because the -getall
option is specified, it also contains the related shared libraries which were used in the ZVM
instance running the Java application.
l
The -getall option ensures the bundle contains all application related data including
the .jar files.
l
The -hsfile option creates an additional file that provides additional information
useful for support to identify issues.
Notes:
l
Encryption error
If there is a file with the same name as the core bundle, the core bundle will not be
encrypted and the following message is displayed in the core bundling response.
Azul Systems
Confidential and Proprietary
300
Troubleshooting Zing
INFO: Encrypting core bundle libs.1234.tgz
...
gpg: libs.1234.tgz: encryption failed: File exists
ERROR: Encryption failed.
WARNING: Core bundle is not encrypted
INFO: Core bundle at libs.1234.tgz
l
Bundler host error
Typically, you create the bundle on the same system where the core file was created.
If you are not creating the bundle on the originating core file system, and you did not
specify the required option, the following message is displayed in the core bundling
response.
ERROR: The bundler must run on host <hostname>
ERROR: Use the argument '-bundle-on-current-host' to bundle
on this host
To create the bundle on a different system:
1. Copy the core file from the originating core file system (hostA) to hostB.
2. On hostB, include the -bundle-on-current-host option when you run the zingcore-bundler command.
The zing-core-bundler syntax and options are listed below.
zing-core-bundler [-core COREFILE][-getexe] [-getall] [-exepath
executable file path] [-hsfile hs_err_pid<pid>.log file] [-decrypt file]
[-passwordfile password file path] [-bundle-on-current-host
Option
Description
-core
Specify the core file to be bundled.
-bundle-on-cur-
Create bundle on current host. This allows bundling a core file that was gen-
rent-host
erated on another host.
-exepath
Specify the path of the binary that was used to invoke Zing Virtual Machine.
-decrypt
Specify file to decrypt
Include all possible files associated with the crash. This includes shared
-getall
objects, data, and executable files.
Include non-java executable with the bundle. This is bundled if it is available
-getexe
on the host at the time of bundling.
Azul Systems
Confidential and Proprietary
301
Zing Installation and Administration Guide
Option
Description
-hsfile
Explicitly specify the path of the hs_err file.
-passwordfile
Specify decryption password file.
-help
Print options help.
7. Locate the bundled libs.<corepid>.tgz.gpg file.
The libs.<corepid>.tgz.gpg file is created in the current directory.
For example, If you execute the command from /home/test. The bundled
libs.<corepid>.tgz.gpg file is created in /home/test.
Ensure you have write permissions in the executing directory. In this example, /home/test.
8. Send the generated bundle to Azul for failure analysis.
Decrypting Core Files
Core bundles created through zing-core-pattern, whether they are automatically or manually
created and bundled, are encrypted. Files contained in the core bundle are not encrypted individually.
The core bundle filename format is: libs.<corepid>.tgz.gpg, where <corepid> is a unique ID
specific to the process that encountered an error and crashed.
Note:
Core files are not encrypted if zing-core-pattern is disabled.
To decrypt the Zing core bundle:
1. Locate the core bundle file.
2. Run the command:
$ /usr/sbin/zing-core-bundler –decrypt libs.<corepid>.tgz.gpg
For example
$ /usr/sbin/zing-core-bundler -decrypt libs.12830.tgz.gpg
INFO: Decrypting file libs.12830.tgz.gpg
gpg: CAST5 encrypted session key
gpg: encrypted with 1 passphrase
gpg: encrypted with RSA key, ID 2934892D
SUCCESS: Decrypted file libs.12830.tgz
Azul Systems
Confidential and Proprietary
302
Troubleshooting Zing
Zing License Core Files
If your Zing licensing feature fails, the default behavior is to follow the core settings. You have the
option to explicitly specify generating a core file when the Zing license fails.
To automatically enable core dump for zing-licensed process:
1. Run the zing-licensed --enable_core option.
# zing-licensed --enable_core
Enabled core dump for zing-licensed
WARNING: Restart zing-licensed service after enabling core dump
2. Restart zing-licensed service:
#service zing-licensed restart.
Zing licensed generated core files are stored at:
l
For RHEL or SLES, /var/crash/zing-licensed and /var/crash/zing-licensem.
l
For Ubuntu, /var/crash/zing-licensed.
To disable core dump for zing-licensed process:
1. Run the zing-licensed --disable_core option.
# zing-licensed --disable_core
Disabled core dump for zing-licensed
WARNING: Restart zing-licensed service after disabling core dump
2. Restart zing-licensed service:
#service zing-licensed restart.
This reverts zing-licensed core behavior to your system core settings.
To check if core dump is enable or disabled for zing-licensed:
1. Run the zing-licensed --display_core_info option.
# zing-licensed –display_core_info
To locate the Zing licensed core files:
l
For RHEL 6.X and SLES
See /var/crash/zing-licensed or /var/crash/zing-license
l
For Ubuntu
See /var/crash/zing-licensed
Azul Systems
Confidential and Proprietary
303
Zing Installation and Administration Guide
Zing Installation Errors
If you received an error or warning message during installation or configuration, locate the message
below, then complete the steps listed for your operating system.
/user/bin Access
Message:
/usr/bin/ld: Permission denied
Action:
ZST installation requires access to /user/bin/ld. Ensure acceptable permissions are set. For
example:
# chmod 755 /usr/bin/ld
Known Intel chip errata can cause spurious system-wide delays
Description:
Recent Intel chips in combination with some aggressive power-saving driver techniques are known to
cause sporadic system lockups, up to a minute long. This is an issue which can occur with or without
Zing present but can be more noticeably manifest in the high-performance, low-latency applications
commonly deployed in Zing environments.
This issue appears with RedHat Enterprise Linux 6.1 and its integrated Intel Idle Driver. The Intel Idle
Driver aims to improve power efficiency but can lead to system lockups when run on some recent Intel
CPUs (some Nehalems and Westmeres have been reported to have problems). Note that this issue
does not occur on all Nehalem and Westmere processors.
Solution:
To solve the issue, you can simply boot your system with the Intel idle driver disabled. To do so, modify
your kernel boot parameters and add the argument "intel_idle.max_cstate=0" to the boot line. See
“Disabling Intel Idle Driver on Systems Using Power Efficiency Chips,” in Installing and Configuring
Zing System Tools for directions on RHEL, SUSE, and Ubuntu systems.
DKMS Installation Troubleshooting
If you received an error or warning message during installation or configuration, locate the message
below, then complete the steps listed for your operating system.
l
Kernel Mismatch on a DKMS RHEL System
l
Kernel Mismatch on a DKMS SLES System
Azul Systems
Confidential and Proprietary
304
Troubleshooting Zing
l
Kernel Mismatch on a DKMS Ubuntu System
See also the following appendices:
l
Zing Appendix A -- Installation Prerequisite Details
l
Zing Appendix C -- Zing System Tools Installed Files
Kernel Mismatch on a DKMS RHEL System
Message:
Error! Your kernel headers for kernel 2.6.32-71.el6.x86_64 cannot
be found at
/lib/modules/2.6.32-71.el6.x86_64/build
or
/lib/modules/2.6.32-71.el6.x86_64/source
Run the following commands to fix the error:
- yum install kernel-devel-2.6.32-71.el6.x86_64
- dkms build -m zing_mm -v 99.99.99.99-712
- dkms install -m zing_mm -v 99.99.99.99-712
DKMS Error. Resolve the errors listed above and run 'system-configzing-memory'
Action:
Complete the commands listed in the error message.
Kernel Mismatch on a DKMS SLES System
Message:
Error! Your kernel headers for kernel 3.0.101-0.35-default cannot
be found at
/lib/modules/3.0.101-0.35-default/build
or
/lib/modules/3.0.101-0.35-default/source
You attempted to install ZST but the correct kernel headers are not
installed.
Complete the following commands to fix the errors. For example:
# zypper install kernel-devel-3.0.101-0.35-default
# dkms build -m zing_mm -v 5.7.7.0-10
# dkms install -m zing_mm -v 5.7.7.0-10
DKMS Error. Resolve the errors listed above and run 'system-configzing-memory'
Action:
Complete the commands listed in the error message.
Azul Systems
Confidential and Proprietary
305
Zing Installation and Administration Guide
Kernel Mismatch on a DKMS Ubuntu or Debian System
Message:
Error! Your kernel headers for kernel 3.0.101-0.35-default cannot
be found at
/lib/modules/3.0.101-0.35-default/build
or
/lib/modules/3.0.101-0.35-default/source
You attempted to install ZST when DKMS is installed, but the
correct kernel headers are not installed.
Complete the following command to fix the errors.
# apt-get install kernel-devel-$(uname -r)
DKMS Error. Resolve the errors listed above and run 'system-configzing-memory'
Action:
Complete the command listed in the error message.
Native Launch with ZVM
A native launch application is any application that is written in a language other than Java and calls
JNI_CreateJavaVM to create the VM and makes calls to Java code.This is usually written in C/C++
and contains the “main” function to start execution.
Invoking Native Launch with Zing
If Native (C/C++) applications invoke jZVM, consider these issues before you execute your code.
l
Native launch application -- Identify how the native app creates the VM to find possible
collisions with areas used by or allocated by the ZVM.
l
Consider known limitations when running native apps on Zing. Identify the issues by the failure
messages.
l
Invoking the ZVM -- check for differences between a stock build and a Zing build.
l
Native apps should also know how to run Zing specific features such as the Tick Profiler
mechanism and C-Heap Leak detection.
Consult the cases listed below to determine your best course of action.
Invoking the ZVM in Native Launch Apps
There are two primary ways of invoking the ZVM in native apps. Key sections of code are highlighted.
Azul Systems
Confidential and Proprietary
306
Troubleshooting Zing
Case 1: C-main dlopens the libjvm.so and tries to find "JNI_CreateJavaVM” using
dlsym.
This is the most common case of native apps. These applications are pre-built apps. For given JAVA_
HOME, these applications should run.
Sample source:
……
JavaVMInitArgs vm_args;
JavaVMOption options[1];
options.optionString = "-Djava.class.path=./";
vm_args.version = JNI_VERSION_1_6;
vm_args.options = options;
vm_args.nOptions = 1;
/* Create the Java VM */
void* libjvm_handle;
jint (*createVM_handle) (JavaVM **pvm, void **penv, void *args);
libjvm_handle = dlopen("libjvm.so", RTLD_LOCAL | RTLD_LAZY);
if (!libjvm_handle) {
fprintf(stderr, "Can't load libjvm.so with error %s\n",dlerror());
exit(1);
}
createVM_handle = dlsym(libjvm_handle, "JNI_CreateJavaVM");
res = createVM_handle(&jvm, (void**)&env, &vm_args);
if (res < 0) {
fprintf(stderr, "Can't create Java VM\n");
exit(1);
}
……
Sample run:
LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/amd64/server:$LD_LIBRARY_PATH
./hello_world
Case 2: C-main app directly calls JNI_CreateJavaVM by statically linking libjvm.
This is *not* most common case of the native applications. For given JAVA_HOME, these applications
could be built/compiled and run.
Sample source:
……
Azul Systems
Confidential and Proprietary
307
Zing Installation and Administration Guide
JavaVMInitArgs args;
JavaVMOption options;
args.version = JNI_VERSION_1_6;
args.nOptions = 1;
options.optionString = "-Djava.class.path=./";
args.options = &options;
int rv;
rv = JNI_CreateJavaVM(jvm, (void**)&env, &args);
if (rv < 0 || !env)
printf("Unable to Launch JVM %d\n",rv);
else
printf("Launched JVM! \n");
…….
Sample build/compile line:
gcc -fPIC -I${JAVA_HOME}/include -I${JAVA_HOME}/include/linux L${JAVA_HOME}/jre/lib/amd64/server –ljvm –o hello_world hello_
world.c
Sample run:
LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/amd64/server:$LD_LIBRARY_PATH
./hello_world
Known Issues Running Native Launch Apps on Zing
Because threads, signals, and memory are managed differently under Zing, users should be aware of
the following issues.
Case 1: Virtual address space collisions when native launch applications allocate
memory where ZVM allocates certain internal structures like the CodeCache and the
OopTable.
l
Failure message: Error: temporary az_mreserve(CodeCache) failed: Bad
address
l
at (0xNNNNNN for 0xNNNNNN bytes)
Failure message: Error: temporary az_mreserve(OopTable) failed: Bad
address
at (0xNNNNNN for 0xNNNNNN bytes)
If the above messages are encountered, native launch applications should try to LD_
PRELOAD libc_hooks2.so or call JNI_CreateJavaVM early on in the main function
before calling routines that may allocate memory.
Azul Systems
Confidential and Proprietary
308
Troubleshooting Zing
Case 2: Virtual address space collisions when native launch applications allocate
memory where libjvm.so needs to be loaded
l
Failure message: fatal: libjvm.so must load under 1Gb.
(JNI_
CreateJavaVM is located at 0xNNNNNNNNN)
If this message is encountered, native launch applications should try and load libjvm.so
early before calling routines that may allocate memory.
Case 3: Virtual address space collisions when native launch applications allocate
memory where ZVM allocates other VM internal data structures
This should usually manifest itself as an az_mreserve or mmap failure in the ZVM.
Case 4: Native launch application uses signal handlers for signals used by the ZVM
Please use libjsig.so to chain user signal handlers correctly behind the ZVM’s signal
handlers.libjsig.so is available under $JAVA_HOME/jre/lib/amd64/server/libjsig.so.
Loading /usr/lib/zing/libc_hooks2.so
Native launch applications will have to link against libc_hooks2.so or LD_PRELOAD libc_
hooks2.so in the following scenariosl
When using -XX:+UseTickProfiler
If profiling threads created in native code is desired, libc_hooks2.so should be loaded.
If libc_hooks2.so is not loaded, the tick profiler will profile threads when those threads
attach themselves to the ZVM through JNI_AttachCurrentThread.
l
When using -XX:+CHeapLeakDetection
The ZVM relies on the libc intercepts in libc_hooks2.so to track memory
allocations.libc_hooks2.so should be loaded in order to use CHeapLeakDetection.
l
When there is a collision in the low 2 GB of virtual address space
Applications can either modify the code to load libjvm.so early or load libc_
hooks2.so, which will reserve upfront the space needed for the VM in the low 2 GB of virtual
address space.
Native launch app colliding with areas used or assumed by the ZVM.
Areas that are used by or assumed to be allocated for a certain purpose by the ZVM should be avoided
by a native launch app. It is possible that such areas have been impacted by a native launch app such
as the case described below where the native launch has allocated a code cache or Oop table.
Azul Systems
Confidential and Proprietary
309
Zing Installation and Administration Guide
Case 1: Native launch app allocates/mmaps specfic areas like CodeCache/OopTable
In this case a message indicates that the temporary az_mreserve failed:
"Error: temporary az_mreserve(CodeCache) failed: Bad
address
at (oxNNNNNN for 0xNNNNNN bytes)"
If this message is encountered, please contact Azul Technical Support for assistance.
Tick Profiler and C-Heap Leak Detection
For a native launch app that needs to get under the tick profiler and leak detection the scanner should
have LD_PRELOAD libc_hooks.so with options set at (-XX:+UseTickProfiler and XX:+CHeapLeakDetection) thru JAVA_TOOL_OPTIONS or JAVA_OPTS.
Threads can come under the tick profiler scanner and allocations can come under the leak detection
scanner only after VM initialization.
Native launch apps can choose to not LD_PRELOAD libc_hooks.so, if you don’t want to get
threads under the tick profiler, but they automatically come under the scanner after they attach to VM.
Zing Errors
This section lists common errors and possible solutions when using Zing to run your Java applications.
When abrtd running zing core_pattern cannot overwrite /proc/sys/kernel/core_pattern
Description:
When abrtd running zing core_pattern cannot overwrite /proc/sys/kernel/core_pattern.
Solution:
1. Scan the file, core_pattern, for any reference piped ( | ) core_pattern. For example:
$ more /proc/sys/kernel/core_pattern
|/usr/libexec/abrt-hook-ccpp /var/spool/abrt %s %c %p %u %g %t %h %e
636f726500
2. If a piped ( | ) core_pattern is found:
a. Stop the process included in core_pattern. For example:
$ service abrtd stop
For RHEL 6.3:
# service abrt-ccpp stop
Azul Systems
Confidential and Proprietary
310
Troubleshooting Zing
b. Clear or change core-pattern to not include any piped (|) core pattern.
$ echo core > /proc/sys/kernel/core_pattern
c. Restart zing-core-pattern.
$ /etc/init.d/zing-core-pattern start
d. Ensure the piped ( | ) core-pattern process is not restarted on boot.
$ chkconfig abrdt off
For RHEL 6.3:
$ chkconfig abrt-ccpp off
ZVM WARNING: The Event Tick Buffer Profiling system status is:
"Shutdown."
Reason:
Max Buffer Exceeded.
Description:
The tick profiler shutdowns automatically when an attempt is made to allocate more than the configured
maximum number of tick profiling buffers. The reason is printed.
Background:
When using tick profiling, each thread in the process uses a buffer to store the events that contain the
tick profiling data. A pool of these tick profiling buffers is allocated when the ZVM starts. The number of
buffers in the pool is increased if the ZVM’s tick profiling system determines that more buffers are
required. The pool expansion occurs, by default, in “chunks” of 2048 tick profiling buffers. The
expansion usually occurs when more threads are created in the application than the number of tick
profiling buffers available. To limit the overhead of tick profiling on application performance, a default
upper bound on the number of tick profiling buffers is enforced. This is currently set to a maximum of
4096 tick profiling buffers.
Two ZVM command line arguments control override the default sizes and their associated behaviors:
l
EventTickBuffersMaxAllocatedBeforeShutoff
This sets the number of tick profiling buffers initially allocated and the chunk size used for expansion
when the pool is resized. The default value is 2048 and the maximum value is 65536.
l
EventTickBuffersAllocated
This sets the maximum number of tick profiling buffers that can be allocated before tick profiling
shuts down completely. The default value for this upper bound on the number of tick profiling buffers
is 4096 and the maximum value is 4294967295. If, using the default value of 4096, the ZVM process
tries to create a new thread that requires a tick profiling buffer and 4096 tick profiling buffers are
already in use, then the tick profiler is turned off.
Azul Systems
Confidential and Proprietary
311
Zing Installation and Administration Guide
Solution:
Increase the maximum number of permissible tick profiling buffers by increasing the value of
EventTickBuffersMaxAllocatedBeforeShutoff in the command line arguments. To avoid data structure
resizes, use EventTickBuffersAllocated to simultaneously increase the chunk size used at initialization
and for expansion. For example:
# java -XX:ARTAPort=9990 –XX:+UseTickProfiling XX:EventTickBuffersAllocated=4096 XX:EventTickBuffersMaxAllocatedBeforeShutoff=12288 <java-class>
For best performance, keep the number of threads in the application to a reasonable number based on
the CPU resources available.
Crash When Using Tight Polling Loops
Description:
Application hangs, and eventually crashes with a checkpoint timeout, when real time threads run tight
loops.
Background:
In these conditions, lower priority threads are not getting sufficient CPU time to reach checkpoint.
Solution:
When the parameter CheckpointBoostPriority is configured, Zing temporarily boosts the priority
of real time threads to the configured value.
For example, one way to use this safely includes:
1. Configure -XX:CheckpointBoostPriority=<Y>
2. Start the ZVM on node 0 with SCJED_RR and a priority, <X>, ensuring that the value for <X> is
greater than <Y>.
3. Using JNI, move the threads with relative real time priority requirements to node 1.
4. Using JNI, assign relative priorities to the threads you moved to node 1, so that the highest among
them is <Y>.
Out of Space for OOP (Ordinary Object Pointer) Table - KlassTable
Description:
The KlassTable contains a unique ID for each loaded class. Each entry is 8 bytes long. The entries will
be freed when the corresponding classes are unloaded.
Azul Systems
Confidential and Proprietary
312
Troubleshooting Zing
Solution:
The KlassTableSize option needs to be set appropriately for the number of live classes in the
application, which can be monitored using PrintGCDetails.
The default value of KlassTableSize is based on the Java heap size, For example:
l
Java heaps less than 2 GB - default 2 MB
l
Java heaps 2 GB or greater - default 8 MB
pthread_exit called without calling JNI_DetachCurrentThread first
Description:
The flag, -XX:+JNIDetachThreadsIfNotDetached, is needed by any application that attaches a
JNI thread to the VM but does not detach the thread before calling pthread_exit. For example with IBM
WebSphere MQ Low Latency Messaging (LLM) applications.
If this error occurs, an error report file with more information is saved in, for example,
/home/<user>/tmp5/hs_err_pid16620.log.
Solution:
If you receive this error, include the following flag in your ZVM command:
-XX:+JNIDetachThreadsIfNotDetached
%preun(zing-zst-<version>) scriptlet failed, exit status1
Description:
You must stop all Zing Java processes before uninstalling or upgrading.
Solution:
Run the zing-ps tool to list any running Zing processes. Stop the processes. Re-run zing-ps to
verify all Zing processes are stopped. Reissue the uninstallation or upgrade commands.
The Zing System Tools is Not Loaded Properly
Description:
Could not create the Java virtual machine.
Azul Systems
Confidential and Proprietary
313
Zing Installation and Administration Guide
Solution:
Please confirm the zing-zst package is properly installed and configured.
The Zing System Tools Has Not Been Configured Yet
Description:
Could not create the Java virtual machine.
Solution:
Please run system-config-zing-memory as root. See man zing-installation for more
information.
Not Enough Free Memory During Zing Memory Configuration
Description 1:
Running Linux processes are taking up a large amount of memory while attempting to configure Zing
memory.
Solution 1:
Stop any processes that use significant resources while configuring Zing. You need to reevaluate your
resource budget if you plan to run these concurrently with Zing.
Description 2:
You have selected values that are too aggressive using the wizard or directly in the pmem.conf
configuration file.
Solution 2:
Try again with less aggressive memory settings.
Error example:
[root@localhost ~]# system-config-zing-memory
Welcome to the Zing memory configuration wizard by Azul Systems.
This wizard sets up the memory driver configuration file
/etc/zing/pmem.conf and initializes the memory driver. (Read the
Azul Systems
Confidential and Proprietary
314
Troubleshooting Zing
man pages for zing, zing-zst, and zing-pmem-conf for more
information about this configuration file.)
By default, Azul recommends setting up your system with 25% Linux
memory and 75% Zing memory. Choose yes to accept this default, or
no to enter the wizard expert flow.
** accept default configuration
** (y)es or (n)o [default y]:
zing-memory: INFO: Restarting...
zing-memory: INFO: Stopping...
zing-memory: INFO: stop successful
zing-memory: INFO: Starting...
Fatal error: Not enough free memory (memoryForAzulBytes
(12545163264) > MemFree (3725369344)).
zing-memory: ERROR: Configure pmem failed
Fatal error: os::safeSystem: command /sbin/service zing-memory
restart failed.
ZST Fails to Allocate Requested Memory
Description:
Long after system startup (and after much memory has been consumed back and forth), ZST might fail
to allocate the requested amount of memory, as it might (a) already be occupied or (b) exist in
fragmented state that cannot be defragmented into 2MB pages. Transparent huge pages on RHEL 6.x
can help recover 2MB pages, but might not be absolutely reliable, and on other systems (RHEL 5.x,
SuSE 11, and other non-RHEL-6 kernel prior to 2.6.38) it is unlikely that you can find 75% of system
memory in 2MB page form after the system has run loads for a while and cached files.
Solution:
Start ZST immediately after a fresh system reboot.
Cannot find kernel config /boot/config-<kernel version>
Description:
The kernel you are running has no associated configuration file in the system /boot directory. A normal
kernel installation and build will always include such a file. This is a serious error indicating that you are
running a kernel in a non-standard way.
Solution:
You are likely using a non-supported system configuration. The solution is to use a standard
configuration.
Azul Systems
Confidential and Proprietary
315
Zing Installation and Administration Guide
Qualification Failed Errors
The Zing configuration process checks the system for known issues that could prevent Zing from
running. When such issues are detected you might see the message Qualification Failed along
with descriptive text about the specific issue. You generally see this message on your screen but if the
error is detected at boot time it is logged by the kernel, usually in /var/log/messages. The following
sections provide information for handling some qualification issues.
Processor support insufficient for Zing (constant_tsc support
required)
Description:
Zing requires a specific feature of modern X86 chips in order to function properly. This feature,
constant_tsc, provides a reliable, fast time-source even in the presence of power-saving operations
performed by modern processors. This reliable time source is required by ZVM.
Solution:
Run Zing on a processor supporting the constant_tsc feature. Nearly all server-class X86
processors made in the last five years have this feature. See the Zing System Requirements for a list
of supported processors.
UseLargePages Not Supported with Zing System Tools
If you are running non-Zing Java and using XX:+UseLargePages, you must choose one of the
following options to run Zing:
l
Disable the hugepages facility on your Linux system when you are running Zing.
l
Ensure there is sufficient memory allocated to each: Zing, hugepages, and Linux memory.
In Linux 2.6 kernel large page support was added. In Java, large memory pages are enabled on Linux
platforms by using these option flags:
l
HotSpot: -XX:+UseLargePages
l
Jrockit: -XXlargePages
l
IBM: -Xlp
l
Zing: Zing pages are always large pages. They come from the ZST.
Zing does not support the -XX:+UseLargePages option. Zing pages are always large pages and they come from the ZST.
ZST and the HugePages pool conflict, as they both reserve memory for themselves. The ZST memory
service reserves and manages its own memory space, so Large Memory Pages configured by the
operating system make it difficult or impossible for the ZST memory service to reserve enough pages
for Zing. Therefore set the output from the three Huge variables to zero.
Azul Systems
Confidential and Proprietary
316
Troubleshooting Zing
To set the huge variables to zero:
1. Check if your system can support large page memory:
# cat /proc/meminfo | grep Huge
l
If Large Pages are available but not configured or reserved, the response is similar to:
# cat /proc/meminfo | grep Huge
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB
l
If Large Pages are available, configured, and reserved, the response is similar to the following.
# cat /proc/meminfo | grep Huge
HugePages_Total: 30000
HugePages_Free: 264
HugePages_Rsvd: 88
Hugepagesize: 2048 kB
2. Set the operating system hugepages memory settings to zero.
a. Log in as root.
b. Reset the hugepages value, type:
# echo 0 > /proc/sys/vm/nr_hugepages
c. Optionally, because the /proc values resets after reboot, set the value in an init script,
such as rc.local or sysctl.conf.
Azul Systems
Confidential and Proprietary
317
Zing Installation and Administration Guide
Appendix A – Zing Installation Prerequisites
Details
While Zing runs fine with certain default system settings, there are default OS settings that can cause
response time sensitive applications to experience multi-hundred-of-millisecond stalls regardless of the
JVM. Since Zing tends to eliminate or dramatically reduce JVM-related stalls, these remaining OS
related stalls will tend to dominate the worst-case latency behavior of applications. It is therefore highly
recommended that the following settings be applied.
To ensure that OS settings are appropriate for performance sensitive applications, apply the following
settings.
l
SWAP – Turn this off, either by removing swap or by setting swappiness to 0. Best practice
removes swap.
l
vm.min_free_kbytes – Set this to 1 GB or more. Best practice assigns more than 1 GB.
l
Transparent Huge Pages (THP) – Turn this off.
If you are using a Nehalem power efficiency chip set, disable the Intel idle driver. If you do not do this,
then the ZST installation aborts.
SWAP Settings
Ensure your SWAP is either turned off or set to 0. Add the required parameter to the sysctl.conf file.
A common method used to change the parameter value is to edit /etc/sysctl.conf then use the sysctl
command to write the changes into the kernel. This ensures the swap setting is persistent across
reboots.
The following steps apply to RHEL, SUSE, and Ubuntu systems.
Persistent change:
This is the recommended method for changing your SWAP settings. It takes effect immediately and is
persistent across reboots.
1. Verify the limit of the swappiness parameter and the effects of changing it on your system.
2. Edit the /etc/sysctl.conf file. Add the following entry:
vm.swappiness=0
Ensure that you do not have duplicate entries for vm.swappiness in the file.
3. Load the sysctl setting from the default file /etc/sysctl.conf. Run sysctl as follows:
# sysctl -p
Azul Systems
Confidential and Proprietary
318
Appendix A – Zing Installation Prerequisites Details
Non-persistent change:
This method changes your SWAP settings temporarily. It is not persistent across reboots.
1. Verify the limit of the swappiness parameter and the effects of changing it on your system.
2. Run the sysctl command with the -w option from the command line:
# sysctl -w vm.swappiness=0
vm.min_free Settings
Set your vm.min_free_kbytes memory value to 1 GB or more. Best practice sets this value greater
than 1 GB.
A common method used to change the parameter value is to edit /etc/sysctl.conf then use the
sysctl command to write the changes into the kernel. This ensures the swap setting is persistent
across reboots.
The following steps apply to RHEL, SUSE, and Ubuntu systems.
Persistent change:
This is the recommended method for changing your vm.min_free_kbytes settings. It takes effect
immediately and is persistent across reboots.
1. Verify the limit of the vm.min_free_kbytes parameter and the effects of changing it on your
system.
2. Edit the /etc/sysctl.conf file. Add the following entry:
vm.min_free_kbytes=1048576
Ensure that you do not have duplicate entries for vm.min_free_kbytes in the file.
3. Load the sysctl setting from the default file /etc/sysctl.conf. Run sysctl as follows:
# sysctl -p
Non-persistent change:
This method changes your vm.min_free_kbytes settings temporarily. It is not persistent across
reboots.
1. Verify the limit of the vm.min_free_kbytes parameter and the effects of changing it on your
system.
2. Run the sysctl command with the -w option from the command line:
# sysctl -w vm.min_free_kbytes=1048576
Transparent HugePages Settings
If your operating system supports Transparent HugePages (THP), the THP feature must be turned off.
For RHEL systems, see https://access.redhat.com/site/solutions/422283.
Azul Systems
Confidential and Proprietary
319
Zing Installation and Administration Guide
To disable Transparent HugePages on RHEL, SUSE, and Ubuntu systems:
1. Check the current HugePage setting.
# cat /sys/kernel/mm/transparent_hugepage/enabled
Sample response.
[always] madvise never
2. Open the grub configuration file for editing.
l
For RHEL: /boot/grub/grub.conf
l
For SLES: /boot/grub/menu.1st
l
For Ubuntu: /etc/default/grub
3. Add transparent_hugepage=never to the kernel boot line, for example:
title Red Hat Enterprise Linux Server (2.6.32-431.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_
qaesp3-lv_root rd_LVM_LV=vg_qaesp3
/lv_root rd_LVM_LV=vg_qaesp3/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM
LANG=en_US.UTF-8 SYSFONT=latarcyrhebsun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet
intel_idle.max_cstate=0 transparent_hugepage=never
initrd /initramfs-2.6.32-431.el6.x86_64.img
Note
This example includes the line for disabling the Intel idle driver. See Intel Idle Driver.
4. Reboot the server. This is required to have the change take effect.
5. Check the change is applied after reboot.
# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
Disabling the Intel Idle Driver
Message:
zing-system: ERROR: Qualification Failed. Known Intel chip errata
can cause spurious system-wide delays; to correct, see Zing
documentation (man zing-installation).
Background:
Intel Nehalem chips can be subject to sporadic pauses in certain circumstances unrelated to Zing.
The issue seems to be an erratum in Nehalem and some Westmere chips related to aggressive powersaving optimizations introduced in these chips. The issue is that your system might be seen to freeze
for periods as small at 1/10 of a second and as much as 30 seconds. This pause is provoked by the use
of a new driver which Intel introduced and which is enabled by default in Red Hat Enterprise Linux 6.1.
Azul Systems
Confidential and Proprietary
320
Appendix A – Zing Installation Prerequisites Details
It is not clear which Nehalem/Westmere chips are subject to this issue. Therefore, in this release of
Zing, we check for the presence of the driver that provokes the chip failure, and abort Zing initialization
if the driver is enabled. When the issue is fixed by Intel and/or Red Hat, we will remove this check.
Reference:
l
Red Hat bug related to Nehalem and Westmere, see https://bugzilla.redhat.com/show_
bug.cgi?id=710265
l
Citrix discussion of the issue, including pointers to Intel chip errata documentation, see
http://support.citrix.com/article/CTX127395
l
Dell discussion, see http://en.community.dell.com/dell-blogs/enterprise/b/tech-
center/archive/2011/07/15/low-latency-and-c-states-in-linux.aspx
zing-memory: ERROR: Bailing due to previous errors (error 1)
Action:
If you are using a Nehalem power efficiency chip set, disable the Intel idle driver. If you do not do this,
then the ZST installation aborts.
To disable the Intel Idle Driver, boot your system with the additional kernel flag, intel_idle.max_
cstate=0.
l
Notes:
l
Disabling the Intel Idle driver, in many cases, resolves the issue of unexpected
pauses. In some cases you might also need to disable C-State power saving
transitions in the BIOS of your machine. Instructions for disabling C-States vary
for each BIOS vendor.
In some situations, the kernel checks the max_cstate and if it finds max_
cstate=0, it sets it to max_cstate=1. To prevent this check of the max_
cstate setting:
In /etc/zing/directory, create a file, disable_cstate_check.
To automatically disable the Intel Idle Driver:
On RHEL/CentOS 7.x systems:
1. Edit /etc/default/grub to add the argument, intel_idle.max_cstate=0, to GRUB_
CMD_LINE_LINUX.
For example: If your /etc/default/grub file contains the following:
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
Azul Systems
Confidential and Proprietary
321
Zing Installation and Administration Guide
GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel_qaesp3/root rd.lvm.lv=rhel_
qaesp3/swap crashkernel=auto vconsole.font=latarcyrheb-sun16
vconsole.keymap=us rhgb quiet "
GRUB_DISABLE_RECOVERY="true"
Append: intel_idle.max_cstate=0 The resulting line should look like this:
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel_qaesp3/root rd.lvm.lv=rhel_
qaesp3/swap crashkernel=auto vconsole.font=latarcyrheb-sun16
vconsole.keymap=us rhgb quiet intel_idle.max_cstate=0"
GRUB_DISABLE_RECOVERY="true"
2. Update the grub entries, run the command.
$ sudo grub2-mkconfig --output=/boot/grub2/grub.cfg
3. Reboot the system.
4. Verify that your running kernel has the Intel Idle driver disabled.
View /proc/cmdline and search for the string intel_idle.max_cstate=0. If the
string is present then the currently-running kernel has the Intel Idle driver disabled.
On RHEL/CentOS 5.x or 6.x systems:
1. Edit /boot/grub/grub.conf to add the argument, intel_idle.max_cstate=0, to the
kernel line for the kernel you are booting.
For example: If your /boot/grub/grub.conf file contains the following:
kernel /vmlinuz-2.6.32-122.el6.x86_64 ro \
root=/dev/mapper/vg_bogrhel61-lv_root rd_LVM_LV=vg_bogrhel61/lv_root
\
rd_LVM_LV=vg_bogrhel61/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM \
LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc
KEYTABLE=us \
crashkernel=auto rhgb quiet
Append: intel_idle.max_cstate=0. The resulting line should look like this:
kernel /vmlinuz-2.6.32-122.el6.x86_64 ro \
root=/dev/mapper/vg_bogrhel61-lv_root rd_LVM_LV=vg_bogrhel61/lv_root
\
rd_LVM_LV=vg_bogrhel61/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM \
LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc
KEYTABLE=us \
crashkernel=auto rhgb quiet intel_idle.max_cstate=0
Azul Systems
Confidential and Proprietary
322
Appendix A – Zing Installation Prerequisites Details
2. Reboot the system.
3. Verify that your running kernel has the Intel Idle driver disabled.
View /proc/cmdline and search for the string intel_idle.max_cstate=0. If the string is
present then the currently-running kernel has the Intel Idle driver disabled.
On Ubuntu systems:
1. Edit /etc/default/grub to add the argument, intel_idle.max_cstate=0, to GRUB_
CMDLINE_LINUX.
For example: If your /etc/default/grub file contains the following:
GRUB_CMDLINE_LINUX=" "
Append: intel_idle.max_cstate=0. The resulting line should look like this:
GRUB_CMDLINE_LINUX="intel_idle.max_cstate=0"
2. Update the grub entries, run the command.
$ sudo update-grub
3. Reboot the system.
4. Verify that your running kernel has the Intel Idle driver disabled.
View /proc/cmdline and search for the string intel_idle.max_cstate=0. If the string
is present then the currently-running kernel has the Intel Idle driver disabled.
On SUSE systems:
1. Edit /boot/grub/menu.lst to add the argument, intel_idle.max_cstate=0, to the
kernel line for the kernel you are booting.
For example: If your /boot/grub/menu.lst file contains the following:
title SUSE Linux Enterprise Server 11 SP2 - 3.0.13-0.27
root (hd0,1)
kernel /boot/vmlinuz-3.0.13-0.27-default root=/dev/sda2
resume=/dev/sda1 splash=silent crashkernel=128M-:64M showopts
vga=0x314
initrd /boot/initrd-3.0.13-0.27-default
Append: intel_idle.max_cstate=0. The resulting line should look like this:
title SUSE Linux Enterprise Server 11 SP2 - 3.0.13-0.27
root (hd0,1)
kernel /boot/vmlinuz-3.0.13-0.27-default root=/dev/sda2
resume=/dev/sda1 splash=silent crashkernel=128M-:64M showopts
vga=0x314 \
intel_idle.max_cstate=0
initrd /boot/initrd-3.0.13-0.27-default
Azul Systems
Confidential and Proprietary
323
Zing Installation and Administration Guide
2. Reboot the system.
3. Verify that your running kernel has the Intel Idle driver disabled.
View /proc/cmdline and search for the string intel_idle.max_cstate=0. If the string
is present then the currently-running kernel has the Intel Idle driver disabled.
Azul Systems
Confidential and Proprietary
324
Appendix C – Zing System Tools Installed Files
Appendix C – Zing System Tools Installed
Files
ZST software consists of a kernel loadable module, and licensing daemon. and supporting scripts and
tools. This section contains a list of all the files included with the driver software installation.
The following files are installed as part of the ZST software package.
User Tools
/usr/bin/zing-ps
List Zing system and process memory usage.
Administrator Tools
/usr/sbin/zing-llpmem
Low-level Zing memory and account manipulation. Not meant to be invoked directly by the user.
/usr/sbin/zing-configure-pmem
High-level Zing memory manipulation. Only expert users should consider running this directly.
Normally, this tool is called automatically by higher-level tools. Reads pmem.conf configuration file.
Invokes zing-llpmem to do low-level work. Called by zing-memory init script.
/usr/sbin/system-config-zing-memory
Zing memory configuration wizard. Users who want to quickly get started with Zing should run this tool.
Creates pmem.conf file. See Configure Zing Memory
/usr/sbin/system-config-zing-core-pattern
Zing core bundling configuration wizard. For automatically generating Zing core bundles. See
Troubleshooting Zing.
Azul Systems
Confidential and Proprietary
325
Zing Installation and Administration Guide
Configuration Files
/etc/zing/pmem.conf.default
Default Zing physical memory configuration file. Software relies on this file to be in a pristine state. Do
not modify this file.
/etc/zing/pmem.conf
Active Zing physical memory configuration file. Created by system-config-zing-memory, or
manually by the user. See the zing-pmem-conf man page for more information about this file.
/etc/zing/pmem.conf.*example*
Various example configuration files.
Man Pages
/usr/share/man/man7/zing-*.7.gz
/usr/share/man/man1/zing-*.1.gz
/usr/share/man/man5/zing-*.5.gz
Title
Content
zing (7)
Top-level description of Zing components and high level
installation instructions.
zing-installation
(7)
Basic description of how to install Zing components including
ZST and ZVM.
zing-zst (7)
Description of ZST components.
zing-pmem-conf (5)
Describes how to configure, replace, and load Zing physical
memory configuration file.
zing-ps (1)
Report a snapshot of current Zing usage.
ZST Configuration File Relationships
The following flow diagram shows how the configuration files and tools relate to one another:
Azul Systems
Confidential and Proprietary
326
Appendix C – Zing System Tools Installed Files
/usr/sbin/system-config-zing-memory
--> /etc/init.d/zing-memory start
--> /sbin/insmod .../mri_mm.ko
--> /usr/sbin/zing-configure-pmem
--> /usr/sbin/zing-llpmem
ZST Boot Sequence File Relationships
After installation and configuration, on bootup, the zing-memory init script loads the kernel module and
partition memory.
The following flow diagram shows how the boot sequence files and tools relate to one another:
/etc/init.d/zing-memory start
--> /sbin/insmod .../mri_mm.ko
--> /usr/sbin/zing-configure-pmem
--> /usr/sbin/zing-llpmem
ZST Update Software File Relationships
The following flow diagram shows how the update files and tools relate to one another:
/usr/sbin/system-config-zing-memory
--> /etc/init.d/zing-memory start
--> /sbin/insmod .../mri_mm.ko
--> /usr/sbin/zing-configure-pmem
--> /usr/sbin/zing-llpmem
.
Azul Systems
Confidential and Proprietary
327
Zing Installation and Administration Guide
Legal Notice
Published, May 15, 2015.
© 2005–2015, Azul Systems, Incorporated 1173 Borregas Avenue, Sunnyvale, CA 94089-1306. All
rights reserved.
Products and specifications discussed in this document may reflect future versions and are subject to
change without notice. Azul Systems assumes no responsibility or liability for any errors or
inaccuracies that may appear in the informational content contained in this guide.
No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or
by any means, electronic, mechanical, recording, or otherwise, without the prior written permission of
Azul Systems. Please note that the content in this document is protected under copyright law even if it
is not distributed with software that includes an end user license agreement.
Azul Systems, Azul Zing, Zing, and the Azul logo are trademarks or registered trademarks of Azul
Systems, Inc. Linux is a registered trademark of Linus Torvalds. Red Hat is the property of Red Hat,
Inc. Java is a registered trademark of Oracle Corporation. Microsoft and Windows are registered
trademarks of Microsoft Corporation. Other marks are the property of their respective owners and are
used here only for identification purposes.
Azul Systems
Confidential and Proprietary
328