Download Out of Memory Errors - Java

Out of Memory Errors
a list of all of them
Kees Jan Koster
[email protected]
Wednesday, March 13, 13
Wednesday, March 13, 13
•
http://java-monitor.com/livedemo.html
•
•
•
•
free, on-line service
up and running in 5 minutes
access server stats from your mobile
get help in the integrated forum
Wednesday, March 13, 13
java-monitor
probe
java-monitor
probe
java-monitor
probe
java-monitor
server
java-monitor
plugin
java-monitor
probe
java-monitor
probe
Wednesday, March 13, 13
Wednesday, March 13, 13
Wednesday, March 13, 13
Wednesday, March 13, 13
Wednesday, March 13, 13
Out of Memory Errors
a list of all of them
Kees Jan Koster
[email protected]
Wednesday, March 13, 13
Wednesday, March 13, 13
JVM
cannot continue
operation
Wednesday, March 13, 13
java.lang.OutOfMemoryError: Java heap space
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.start(JIoEndpoint.java:478)
at org.apache.tomcat.util.net.JIoEndpoint.newWorkerThread(JIoEndpoint.java:697)
at org.apache.tomcat.util.net.JIoEndpoint.createWorkerThread(JIoEndpoint.java:676)
at org.apache.tomcat.util.net.JIoEndpoint.getWorkerThread(JIoEndpoint.java:708)
at org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:743)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:320)
at java.lang.Thread.run(Thread.java:619)
at ...
Wednesday, March 13, 13
java.lang.OutOfMemoryError:
at
at
at
at
at
at
at
at
at
at
Java heap space
java.lang.Thread.start0(Native Method)
java.lang.Thread.start(Thread.java:597)
org.apache.tomcat.util.net.JIoEndpoint$Worker.start(JIoEndpoint.java:478)
org.apache.tomcat.util.net.JIoEndpoint.newWorkerThread(JIoEndpoint.java:697)
org.apache.tomcat.util.net.JIoEndpoint.createWorkerThread(JIoEndpoint.java:676)
org.apache.tomcat.util.net.JIoEndpoint.getWorkerThread(JIoEndpoint.java:708)
org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:743)
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:320)
java.lang.Thread.run(Thread.java:619)
...
Wednesday, March 13, 13
•
non-fatal OOME’s
• Java heap space
• PermGen space
• GC overhead limit exceeded
• Requested array size exceeds VM limit
• unable to create new native thread
Wednesday, March 13, 13
•
fatal OOME’s
• thr_setspecific: out of swap space
• Out of swap space to map in thread stack
• CodeCache: no room for %s
• CodeCache: no room for vtable chunks
• offset table expansion
• G1 heap expansion
• Cannot create GC thread. Out of system resources
• object start array expansion
• card table last card
• unable to allocate initial hashtable for jvmti object
•
•
Wednesday, March 13, 13
tags
Cannot create GangWorker array
...
•
non-fatal OOME’s
• Java heap space
• PermGen space
• GC overhead limit exceeded
• Requested array size exceeds VM limit
• unable to create new native thread
Wednesday, March 13, 13
heap
Wednesday, March 13, 13
nonheap
Wednesday, March 13, 13
Wednesday, March 13, 13
•
memory profiling
• what does not belong?
• what’s big when it should be small?
• what’s many that should be few?
Wednesday, March 13, 13
•
JavaRanch.com
•
•
•
•
appalling performance
JVM pauses
inexplicably large memory consumption
memory leak?
Wednesday, March 13, 13
Wednesday, March 13, 13
•
analysed heap dump and found:
• org.apache.lucene.search.FieldCache$StringIndex
• 3 copies, each about 115MB
• similar, but not identical
• containing strings like “20081010125634”
•
developer changed type from String to long
• timestamp index
Wednesday, March 13, 13
before
after
Wednesday, March 13, 13
seconds
milliseconds
Wednesday, March 13, 13
•
Java heap space
• check for memory leaks
• get heap dumps
•
•
• -XX:+HeapDumpOnOutOfMemoryError
• use a memory profiler
bigger heap using -Xmx=…
PermGen space
• check class loading
• clean up redeployment leaks
• check for string interning
• bigger non-heap using -XX:MaxPermSize=…
Wednesday, March 13, 13
•
non-fatal OOME’s
• Java heap space
• PermGen space
• GC overhead limit exceeded
• Requested array size exceeds VM limit
• unable to create new native thread
Wednesday, March 13, 13
Wednesday, March 13, 13
young
generation
old
generation
heap
scavenger
collector
Wednesday, March 13, 13
full garbage
collector
nonheap
•
GC
Java heap space
• check for memory leaks
• get heap dumps
•
•
d
e
d
e
e
c
x
e
t
i
m
i
l
d
a
e
h
r
ove
• -XX:+HeapDumpOnOutOfMemoryError
• use a memory profiler
bigger heap using -Xmx=…
PermGen space
• check class loading
• clean up redeployment leaks
• check for string interning
• bigger non-heap using -XX:MaxPermSize=…
Wednesday, March 13, 13
42536.572: [GC 42536.572: [ParNew: 353920K->2080K(353920K),
>1137843K(2520704K), 0.1249020 secs]
42546.733: [GC 42546.733: [ParNew: 316704K->3604K(353920K),
>1139368K(2520704K), 0.0602080 secs]
42555.527: [GC 42555.527: [ParNew: 318228K->5034K(353920K),
>1140797K(2520704K), 0.0611270 secs]
42564.154: [GC 42564.154: [ParNew: 319658K->5214K(353920K),
>1142411K(2520704K), 0.0782850 secs]
42571.444: [GC 42571.444: [ParNew: 319838K->5514K(353920K),
>1144161K(2520704K), 0.0651220 secs]
42576.714: [GC 42576.714: [ParNew
0.1246980 secs] 1450967K0.0599980 secs] 1452467K0.0609210 secs] 1453992K0.0780650 secs] 1455421K0.0649090 secs] 1457035K-
(promotion failed): 320138K-
>320138K(353920K), 0.2365970 secs]42576.951: [CMS: 1139969K->1120688K(
2166784K), 9.2214860 secs] 1458785K->1120688K(2520704K), 9.4584090 secs]
42587.017: [GC 42587.017: [ParNew: 314624K->34454K(353920K), 0.1340880 secs] 1435312K>1205264K(2520704K), 0.1344800 secs]
42588.700: [GC 42588.701: [ParNew: 349078K->23091K(353920K),
Wednesday, March 13, 13
Wednesday, March 13, 13
CMS is non-compacting
Wednesday, March 13, 13
can’t meet “young generation guarantee”
promotion failed
Wednesday, March 13, 13
•
promotion failed
• -Xloggc:/var/log/gc.log -verbose:gc
•
•
•
Wednesday, March 13, 13
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps
bigger heap
smaller eden
...
•
non-fatal OOME’s
• Java heap space
• PermGen space
• GC overhead limit exceeded
• Requested array size exceeds VM limit
• unable to create new native thread
Wednesday, March 13, 13
heap
Wednesday, March 13, 13
nonheap
ObjectInputStream is = ...
int size = is.readInt();
String[] someArray = new String[size];
Wednesday, March 13, 13
ObjectInputStream is = ...
int size = is.readInt();
if (size > 100) {
throw
new IllegalArgumentException(“insane “ + size);
}
String[] someArray = new String[size];
Wednesday, March 13, 13
Wednesday, March 13, 13
•
Requested array size exceeds VM limit
• use the stack trace (this once!)
• allocate smaller arrays
• validate external array sizes
Wednesday, March 13, 13
•
non-fatal OOME’s
• Java heap space
• PermGen space
• GC overhead limit exceeded
• Requested array size exceeds VM limit
• unable to create new native thread
Wednesday, March 13, 13
heap
nonheap
native
code
thread stacks are
allocated here
Wednesday, March 13, 13
OS
1-2 GB
heap
non- native
heap code
4GB
hard 32-bit process limit
Wednesday, March 13, 13
•
unable to create new native thread
• -Xss to limit thread stacks
• *reduce* the heap and/or non-heap pools
• switch to 64-bit
• use sane heap/non-heap/thread stack sizes
Wednesday, March 13, 13
•
non-fatal OOME’s
• Java heap space
• PermGen space
• GC overhead limit exceeded
• Requested array size exceeds VM limit
• unable to create new native thread
Wednesday, March 13, 13
•
fatal OOME’s
• thr_setspecific: out of swap space
• Out of swap space to map in thread stack
• CodeCache: no room for %s
• CodeCache: no room for vtable chunks
• offset table expansion
• G1 heap expansion
• Cannot create GC thread. Out of system resources
• object start array expansion
• card table last card
• unable to allocate initial hashtable for jvmti object
•
•
Wednesday, March 13, 13
tags
Cannot create GangWorker array
...
physical memory
RAM
SWAP
fast
Wednesday, March 13, 13
disk-based memory
very, very slow
RAM
Wednesday, March 13, 13
SWAP
RAM
Wednesday, March 13, 13
SWAP
RAM
Wednesday, March 13, 13
SWAP
•
fatal OOME’s
• thr_setspecific: out of swap space
• Out of swap space to map in thread stack
• CodeCache: no room for %s
• CodeCache: no room for vtable chunks
• offset table expansion
• G1 heap expansion
• Cannot create GC thread. Out of system resources
• object start array expansion
• card table last card
• unable to allocate initial hashtable for jvmti object
n
o
i
t
a
c
o
l
l
a
M
A
R
k
c
e
h
s
c
t
i
•
m
i
l
m
e
t
s
y
s
k
c
e
h
c
•
•
•
Wednesday, March 13, 13
tags
Cannot create GangWorker array
...