| Exit Print View | |
Troubleshooting Guide for HotSpot VM |
 |
 |
  |
1. Diagnostic Tools and Options
3. Troubleshooting Memory Leaks
3.1 Meaning of OutOfMemoryError
3.1.1 Detail Message: Java heap space
3.1.2 Detail Message: PermGen space
3.1.3 Detail Message: Requested array size exceeds VM limit
3.1.4 Detail Message: request <size> bytes for <reason>. Out of swap space?
3.1.5 Detail Message: <reason> <stack trace> (Native method)
3.2 Crash Instead of OutOfMemoryError
3.3 Diagnosing Leaks in Java Language Code
3.3.3.4 -XX:+HeapDumpOnOutOfMemoryError Command-line Option
3.3.4 Obtaining a Heap Histogram on a Running Process
3.3.5 Obtaining a Heap Histogram at OutOfMemoryError
3.3.6 Monitoring the Number of Objects Pending Finalization
3.3.7 Third Party Memory Debuggers
3.4 Diagnosing Leaks in Native Code
3.4.1 Tracking All Memory Allocation and Free Calls
3.4.2 Tracking Memory Allocation in a JNI Library
4. Troubleshooting System Crashes
5. Troubleshooting Hanging or Looping Processes
6. Integrating Signal and Exception Handling
If your application's execution time becomes longer and longer, or if the operating system seems to be performing slower and slower, this could be an indication of a memory leak. In other words, virtual memory is being allocated but is not being returned when it is no longer needed. Eventually the application or the system runs out of memory, and the application terminates abnormally.
This chapter provides some suggestions on diagnosing problems involving possible memory leaks.
One common indication of a memory leak is the java.lang.OutOfMemoryError error. This error is thrown when there is insufficient space to allocate an object in the Java heap or in a particular area of the heap. The garbage collector cannot make any further space available to accommodate a new object, and the heap cannot be expanded further.
When the java.lang.OutOfMemoryError error is thrown, a stack trace is printed also.
A java.lang.OutOfMemoryError can also be thrown by native library code when a native allocation cannot be satisfied, for example, if swap space is low.
An early step to diagnose an OutOfMemoryError is to determine what the error means. Does it mean that the Java heap is full, or does it mean that the native heap is full? To help you answer this question, the following subsections explain some of the possible error messages, with reference to the detail part of the message:
Exception in thread "main": java.lang.OutOfMemoryError: Java heap space
Exception in thread "main": java.lang.OutOfMemoryError: PermGen space
Exception in thread "main": java.lang.OutOfMemoryError: Requested array size exceeds VM limit
See 
3.1.3 Detail Message: Requested array size exceeds VM
limit.
Exception in thread "main": java.lang.OutOfMemoryError: request <size> bytes for <reason>. Out of swap space?
See 3.1.4 Detail Message: request <size> bytes for
<reason>. Out of swap space?.
Exception in thread "main": java.lang.OutOfMemoryError: <reason> <stack trace> (Native method)
See 3.1.5 Detail Message: <reason> <stack
trace> (Native method).
The detail message Java heap space indicates that an object could not be allocated in the Java heap. This error does not necessarily imply a memory leak. The problem can be as simple as a configuration issue, where the specified heap size (or the default size, if not specified) is insufficient for the application.
In other cases, and in particular for a long-lived application, the message might be an indication that the application is unintentionally holding references to objects, and this prevents the objects from being garbage collected. This is the Java language equivalent of a memory leak. Note that APIs that are called by an application could also be unintentionally holding object references.
One other potential source of OutOfMemoryError arises
with applications that make excessive use of finalizers. If a class
has a finalize method, then objects of that type do not have their
space reclaimed at garbage collection time. Instead, after garbage
collection the objects are queued for finalization, which occurs at
a later time. In the Sun implementation, finalizers are executed by
a daemon thread that services the finalization queue. If the
finalizer thread cannot keep up with the finalization queue, then
the Java heap could fill up and OutOfMemoryError would be
thrown. One scenario that can cause this situation is when an
application creates high-priority threads that cause the
finalization queue to increase at a rate that is faster than the
rate at which the finalizer thread is servicing that queue. Section
3.3.6 Monitoring the Number of Objects Pending
Finalization discusses how to monitor objects for which
finalization is pending.
The detail message PermGen space indicates that the permanent generation is full. The permanent generation is the area of the heap where class and method objects are stored. If an application loads a very large number of classes, then the size of the permanent generation might need to be increased using the -XX:MaxPermSize option.
Interned java.lang.String objects are no longer stored in the permanent generation. The java.lang.String class maintains a pool of strings. When the intern method is invoked, the method checks the pool to see if an equal string is already in the pool. If there is, then the intern method returns it; otherwise it adds the string to the pool. In more precise terms, the java.lang.String.intern method is used to obtain the canonical representation of the string; the result is a reference to the same class instance that would be returned if that string appeared as a literal.
When this kind of error occurs, the text ClassLoader.defineClass might appear near the top of the stack trace that is printed.
The jmap -permgen command prints statistics for the
objects in the permanent generation and also information about
internalized String instances. See 2.7.4
Getting Information on the Permanent Generation.
The detail message Requested array size exceeds VM limit indicates that the application (or APIs used by that application) attempted to allocate an array that is larger than the heap size. For example, if an application attempts to allocate an array of 512MB but the maximum heap size is 256MB then OutOfMemoryError will be thrown with the reason Requested array size exceeds VM limit. In most cases the problem is either a configuration issue (heap size too small), or a bug that results in an application attempting to create a huge array, for example, when the number of elements in the array are computed using an algorithm that computes an incorrect size.
The detail message request <size> bytes for <reason>. Out of swap space? appears to be an OutOfMemoryError. However, the HotSpot VM code reports this apparent exception when an allocation from the native heap failed and the native heap might be close to exhaustion. The message indicates the size (in bytes) of the request that failed and the reason for the memory request. In most cases the <reason> part of the message is the name of a source module reporting the allocation failure, although in some cases it indicates a reason.
When this error message is thrown, the VM invokes the fatal
error handling mechanism, that is, it generates a fatal error log
file, which contains useful information about the thread, process,
and system at the time of the crash. In the case of native heap
exhaustion, the heap memory and memory map information in the log
can be useful. See Appendix C,
Fatal Error Log for detailed information about this file.
If this type of OutOfMemoryError is thrown, you might
need to use troubleshooting utilities on the operating system to
diagnose the issue further. See 2.16
Operating-System-Specific Tools.
The problem might not be related to the application, for example:
The operating system is configured with insufficient swap space.
Another process on the system is consuming all memory resources.
If neither of the above issues is the cause, then it is possible that the application failed due to a native leak, for example, if application or library code is continuously allocating memory but is not releasing it to the operating system.
If the detail part of the error message is <reason> <stack trace> (Native method) and a stack trace is printed in which the top frame is a native method, then this is an indication that a native method has encountered an allocation failure. The difference between this and the previous message is that the allocation failure was detected in a JNI or native method rather than in Java VM code.
If this type of OutOfMemoryError is thrown, you might
need to use utilities on the operating system to further diagnose
the issue. See 2.16
Operating-System-Specific Tools.
Sometimes an application crashes soon after an allocation from the native heap fails. This occurs with native code that does not check for errors returned by memory allocation functions.
For example, the malloc system call returns NULL if there is no memory available. If the return from malloc is not checked, then the application might crash when it attempts to access an invalid memory location. Depending on the circumstances, this type of issue can be difficult to locate.
However, in some cases the information from the fatal error log
or the crash dump might be sufficient to diagnose this issue. The
fatal error log is covered in detail in Appendix C,
Fatal Error Log. If the cause of a crash is determined to be
the failure to check an allocation failure, then the reason for the
allocation failure must be examined. As with any other native heap
issue, the system might be configured with insufficient swap space,
another process on the system might be consuming all memory
resources, or there might be a leak in the application (or in the
APIs that it calls) that causes the system to run out of
memory.
Diagnosing leaks in Java language code can be a difficult task. In most cases it requires very detailed knowledge of the application. In addition the process is often iterative and lengthy. This section provides the following subsections:
The NetBeans Profiler (previously known as JFluid) is an excellent profiler, which can locate memory leaks very quickly. Most commercial memory leak debugging tools can often take a long time to locate a leak in a large application. The NetBeans Profiler, however, uses the pattern of memory allocations and reclamations that such objects typically demonstrate. This process includes also the lack of memory reclamations. The profiler can check where these objects were allocated, which in many cases is sufficient to identify the root cause of the leak.
More details can be found at http://profiler.netbeans.org.
The jhat utility (see 2.5
jhat Utility) is useful when debugging unintentional
object retention (or memory leaks). It provides a way to browse an
object dump, view all reachable objects in the heap, and understand
which references are keeping an object alive.
To use jhat you must obtain one or more heap dumps of
the running application, and the dumps must be in binary format.
Once the dump file is created, it can be used as input to
jhat, as described in 2.5
jhat Utility.
A heap dump provides detailed information on the allocation of heap memory. The following sections describe several ways to produce a heap dump:
The HPROF profiler agent can create a heap dump while the application is executing. The following is an example of the command line:
$ java -agentlib:hprof=file=snapshot.hprof,format=b application
If the VM is embedded or is not started using a command line
launcher that allows additional options to be provided, then it
might be possible to use the JAVA_TOOLS_OPTIONS
environment variable so that the -agentlib option is
automatically added to the command line. See A.2
JAVA_TOOL_OPTIONS Environment Variable for further
information on this environment variable.
Once the application is running with HPROF, a heap dump is created by pressing Ctrl-\ or Ctrl-Break (depending on the platform) on the application console. An alternative approach on Solaris OS and Linux is to send a QUIT signal with the kill -QUIT pid command. When the signal is received, a heap dump is created; in the above example the file snapshot.hprof is created.
The heap dump file contains all the primitive data and stack traces.
A dump file can contain multiple heap dumps. If Ctrl-\ or Ctrl-Break is pressed a number of times then the subsequent dumps are appended to the file. The jhat utility uses the #n syntax to distinguish the dumps, where n is the dump number.
A heap dump can also be obtained using the jmap utility
(see 2.7 jmap Utility). The
following is an example of the command line:
$ jmap -dump:format=b,file=snapshot.jmap process-pid
Regardless of how the Java VM was started, the jmap tool will produce a head dump snapshot, in the above example in a file called snapshot.jmap. The jmap output files should contain all the primitive data, but will not include any stack traces showing where the objects have been created.
Another way to obtain a heap dump is with the JConsole utility. In the MBeans tab, select the HotSpotDiagnostic MBean, then the Operations display, and choose the dumpHeap operation.
If you specify the -XX:+HeapDumpOnOutOfMemoryError command-line option, and if an OutOfMemoryError is thrown, the VM generates a heap dump.
You can try to quickly narrow down a memory leak by examining a heap histogram. This information can be obtained in several ways:
A heap histogram can be obtained from a running process using the command jmap -histo pid. The output shows the total size and instance count for each class type in the heap. If a sequence of histograms is obtained (for example, every 2 minutes), then you might be able to observe a trend that can lead to further analysis.
On Solaris OS and Linux, the jmap utility can also provide a histogram from a core file.
If the Java process is started with the -XX:+PrintClassHistogram command-line option, then the Ctrl-Break handler will produce a heap histogram.
If you specify the -XX:+HeapDumpOnOutOfMemoryError command-line option, and if an OutOfMemoryError is thrown, the VM generates a heap dump. You can then use the jmap utility to obtain a histogram from the heap dump.
If a core file is produced when the OutOfMemoryError is thrown, you can execute jmap on the core file to get a histogram, as in the following example.
$ jmap -histo \ /java/re/javase/6/latest/binaries/solaris-sparc/bin/java core.27421 Attaching to core core.27421 from executable /java/re/javase/6/latest/binaries/solaris-sparc/bin/java, please wait... Debugger attached successfully. Server compiler detected. JVM version is 1.6.0-beta-b63 Iterating over heap. This may take a while... Heap traversal took 8.902 seconds. Object Histogram: Size Count Class description ------------------------------------------------------- 86683872 3611828 java.lang.String 20979136 204 java.lang.Object[] 403728 4225 * ConstMethodKlass 306608 4225 * MethodKlass 220032 6094 * SymbolKlass 152960 294 * ConstantPoolKlass 108512 277 * ConstantPoolCacheKlass 104928 294 * InstanceKlassKlass 68024 362 byte[] 65600 559 char[] 31592 359 java.lang.Class 27176 462 java.lang.Object[] 25384 423 short[] 17192 307 int[] :
The example shows that the OutOfMemoryError is caused by the number of java.lang.String objects (3611828 instances in the heap). Without further analysis it is not clear where the strings are allocated. However, the information is still useful and the investigation can continue with tools such as HPROF or jhat to find out where the strings are allocated, as well as what references are keeping them alive and preventing them from being garbage collected.
As noted in 3.1.1 Detail Message:
Java heap space, excessive use of finalizers can be
the cause of OutOfMemoryError. You have several options
for monitoring the number of objects that are pending
finalization.
The JConsole management tool (see 2.3
JConsole Utility) can be used to monitor the number of objects
that are pending finalization. This tool reports the pending
finalization count in the memory statistics on the
“Summary” tab pane. The count is approximate but it can
be used to characterize an application and understand if it relies
a lot on finalization.
On Solaris OS and Linux, the jmap -finalizerinfo option prints information on objects awaiting finalization.
An application can report the approximate number of objects
pending finalization using the
getObjectPendingFinalizationCount method in the
java.lang.management.MemoryMXBean class. Links to the API
documentation and example code can be found in 2.17 Developing Diagnostic Tools. The
example code can easily be extended to include the reporting of the
pending finalization count.
In addition to the tools mentioned in the previous chapters, there are a large number of third-party memory debuggers available. JProbe from Quest Software, and OptimizeIt from Borland are two examples of commercial tools with memory debugging capability. There are many others and no specific product is recommended.
Several techniques can be used to find and isolate native code memory leaks. In general there is no single ideal solution for all platforms.
A very common practice is to track all allocation and free calls of the native allocations. This can be a fairly simple process or a very sophisticated one. Many products over the years have been built up around the tracking of native heap allocations and the use of that memory.
Tools like Purify and Sun's dbx Run Time Checking (see
3.4.4 Using dbx to Find Leaks) functionality
can be used to find these leaks in normal native code situations
and also find any access to native heap memory that represents
assignments to uninitialized memory or accesses to freed
memory.
Not all these types of tools will work with Java applications that use native code, and usually these tools are platform-specific. Since the virtual machine dynamically creates code at runtime, these tools can wrongly interpret the code and fail to run at all, or give false information. Check with your tool vendor to make sure the version of the tool works with the version of the virtual machine you are using.
Many simple and portable native memory leak detecting examples
can be found at http://sourceforge.net/. Most of
these libraries and tools assume that you can recompile or edit the
source of the application and place wrapper functions over the
allocation functions. The more powerful of these tools allow you to
run your application unchanged by interposing over these allocation
functions dynamically. This is the case with the library
libumem.so, starting with Solaris 9 OS update 3; see
3.4.5 Using libumem to Find Leaks.
If you write a JNI library, it would probably be wise to create some kind of localized way to make sure your library does not leak memory, using a simple wrapper approach.
The following procedure is an easy localized allocation tracking approach for a JNI library. First, define the following lines in all source files:
#include <stdlib.h> #define malloc(n) debug_malloc(n, __FILE__, __LINE__) #define free(p) debug_free(p, __FILE__, __LINE__)
Then you can use the following functions to watch for leaks.
/* Total bytes allocated */
static int total_allocated;
/* Memory alignment is important */
typedef union { double d; struct {size_t n; char *file; int line;} s; } Site;
void *
debug_malloc(size_t n, char *file, int line)
{
char *rp;
rp = (char*)malloc(sizeof(Site)+n);
total_allocated += n;
((Site*)rp)->s.n = n;
((Site*)rp)->s.file = file;
((Site*)rp)->s.line = line;
return (void*)(rp + sizeof(Site));
}
void
debug_free(void *p, char *file, int line)
{
char *rp;
rp = ((char*)p) - sizeof(Site);
total_allocated -= ((Site*)rp)->s.n;
free(rp);
}
The JNI library would then need to periodically (or at shutdown) check the value of the total_allocated variable to make sure that it made sense. The above code could also be expanded to save in a linked list the allocations that remained and report where the leaked memory was allocated. This is a localized and portable way to track memory allocations in a single set of sources. You would need to make sure that debug_free() was called only with a pointer that came from debug_malloc(), and you would also need to create similar functions for realloc(), calloc(), strdup(), and so forth, if they were used.
A more global way to look for native heap memory leaks would involve interposition of the library calls for the entire process.
Most operating systems include some form of global allocation tracking support.
On Windows, go to http://msdn.microsoft.com/library/default.asp and search for debug support. The Microsoft C++ compiler has the /Md and /Mdd compiler options that will automatically include extra support for tracking memory allocations.
Linux systems have tools such as mtrace and libnjamd to help in dealing with allocation tracking.
Solaris Operating Systems provide the watchmalloc tool.
Solaris 9 OS update 3 started providing the libumem tool
(see 3.4.5 Using libumem to Find Leaks).
The Sun debugger dbx includes the Run Time Checking (RTC) functionality, which can find leaks. The dbx debugger is also available on Linux.
Below is a sample dbx session.
$ dbx ${java_home}/bin/java
Reading java
Reading ld.so.1
Reading libthread.so.1
Reading libdl.so.1
Reading libc.so.1
(dbx) dbxenv rtc_inherit on
(dbx) check -leaks
leaks checking - ON
(dbx) run HelloWorld
Running: java HelloWorld
(process id 15426)
Reading rtcapihook.so
Reading rtcaudit.so
Reading libmapmalloc.so.1
Reading libgen.so.1
Reading libm.so.2
Reading rtcboot.so
Reading librtc.so
RTC: Enabling Error Checking...
RTC: Running program...
dbx: process 15426 about to exec("/net/bonsai.sfbay/export/home2/user/ws/j2se/build/solaris-i586/bin/java")
dbx: program "/net/bonsai.sfbay/export/home2/user/ws/j2se/build/solaris-i586/bin/java"
just exec'ed
dbx: to go back to the original program use "debug $oprog"
RTC: Enabling Error Checking...
RTC: Running program...
t@1 (l@1) stopped in main at 0x0805136d
0x0805136d: main : pushl %ebp
(dbx) when dlopen libjvm { suppress all in libjvm.so; }
(2) when dlopen libjvm { suppress all in libjvm.so; }
(dbx) when dlopen libjava { suppress all in libjava.so; }
(3) when dlopen libjava { suppress all in libjava.so; }
(dbx) cont
Reading libjvm.so
Reading libsocket.so.1
Reading libsched.so.1
Reading libCrun.so.1
Reading libm.so.1
Reading libnsl.so.1
Reading libmd5.so.1
Reading libmp.so.2
Reading libhpi.so
Reading libverify.so
Reading libjava.so
Reading libzip.so
Reading en_US.ISO8859-1.so.3
hello world
hello world
Checking for memory leaks...
Actual leaks report (actual leaks: 27 total size: 46851 bytes)
Total Num of Leaked Allocation call stack
Size Blocks Block
Address
========== ====== =========== =======================================
44376 4 - calloc < zcalloc
1072 1 0x8151c70 _nss_XbyY_buf_alloc < get_pwbuf < _getpwuid <
GetJavaProperties < Java_java_lang_System_initProperties <
0xa740a89a< 0xa7402a14< 0xa74001fc
814 1 0x8072518 MemAlloc < CreateExecutionEnvironment < main
280 10 - operator new < Thread::Thread
102 1 0x8072498 _strdup < CreateExecutionEnvironment < main
56 1 0x81697f0 calloc < Java_java_util_zip_Inflater_init < 0xa740a89a<
0xa7402a6a< 0xa7402aeb< 0xa7402a14< 0xa7402a14< 0xa7402a14
41 1 0x8072bd8 main
30 1 0x8072c58 SetJavaCommandLineProp < main
16 1 0x806f180 _setlocale < GetJavaProperties <
Java_java_lang_System_initProperties < 0xa740a89a< 0xa7402a14<
0xa74001fc< JavaCalls::call_helper < os::os_exception_wrapper
12 1 0x806f2e8 operator new < instanceKlass::add_dependent_nmethod <
nmethod::new_nmethod < ciEnv::register_method <
Compile::Compile #Nvariant 1 < C2Compiler::compile_method <
CompileBroker::invoke_compiler_on_method <
CompileBroker::compiler_thread_loop
12 1 0x806ee60 CheckJvmType < CreateExecutionEnvironment < main
12 1 0x806ede8 MemAlloc < CreateExecutionEnvironment < main
12 1 0x806edc0 main
8 1 0x8071cb8 _strdup < ReadKnownVMs < CreateExecutionEnvironment < main
8 1 0x8071cf8 _strdup < ReadKnownVMs < CreateExecutionEnvironment < main
The output shows that the dbx debugger reports memory leaks if memory is not freed at the time the process is about to exit. However, memory that is allocated at initialization time and needed for the life of the process is often never freed in native code. Therefore, in such cases the dbx debugger can report memory leaks that are not leaks in reality.
Note that the example used two suppress commands to suppress the leaks reported in the virtual machine (libjvm.so) and the Java support library (libjava.so).
Starting with Solaris 9 OS update 3, the libumem.so library and the modular debugger (mdb) can be used to debug memory leaks. Before using libumem, you must preload the libumem library and set an environment variable as follows:
$ LD_PRELOAD=libumem.so $ export LD_PRELOAD $ UMEM_DEBUG=default $ export UMEM_DEBUG
Now, run the Java application but stop it before it exits. The following example uses truss to stop the process when it calls the _exit system call:
$ truss -f -T _exit java MainClass arguments
At this point you can attach the mdb debugger, as follows:
$ mdb -p pid >::findleaks
The ::findleaks command is the mdb command to find memory leaks. If a leak is found, the findleaks command prints the address of the allocation call, buffer address, and nearest symbol.
It is also possible to get the stack trace for the allocation which resulted in the memory leak by dumping the bufctl structure. The address of this structure can be obtained from the output of the ::findleaks command. The description of the commands to perform these functions, as well as more information on using libumem to identify memory managements bugs, is located at the following address: http://download.oracle.com/docs/cd/E19424-01/820-4814/geogv/index.html.
| |