| Exit Print View | |
Troubleshooting Guide for HotSpot VM |
 |
 |
  |
1. Diagnostic Tools and Options
3. Troubleshooting Memory Leaks
4. Troubleshooting System Crashes
5. Troubleshooting Hanging or Looping Processes
6. Integrating Signal and Exception Handling
This section describes some command line options that can be useful in diagnosing problems.
Command-line options that are prefixed with -XX are specific to the Java HotSpot Virtual Machine. Many of these options are important for performance tuning and diagnostic purposes, and are therefore described in this appendix. All the -XX options are described at http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html.
With the jinfo -flag command (
2.6
jinfo Utility) and with the JConsole utility
(2.3 JConsole Utility), you can
dynamically set, unset, or change the value of certain Java VM
flags for a specified Java process.
For the complete list of these flags, use the MBeans tab of the JConsole utility. See the list of values for the DiagnosticOptions attribute of the HotSpotDiagnostic MBean, which is in the com.sun.management domain. These flags are the following:
HeapDumpOnOutOfMemoryError
HeapDumpPath
PrintGC
PrintGCDetails
PrintGCTimeStamps
PrintClassHistogram
PrintConcurrentLocks
The -XX:+HeapDumpOnOutOfMemoryError command-line option tells the HotSpot VM to generate a heap dump when an allocation from the Java heap or the permanent generation cannot be satisfied. There is no overhead in running with this option, and so it can be useful for production systems where OutOfMemoryError takes a long time to surface.
You can also specify this option at runtime with the MBeans tab in the jconsole utility.
The heap dump is in HPROF binary format, and so it can be
analyzed using any tools that can import this format. For example,
the jhat tool can be used to do rudimentary analysis of
the dump. See 2.5
jhat Utility.
The following example shows the result of running out of memory with this flag set.
$ java -XX:+HeapDumpOnOutOfMemoryError -mn256m -mx512m ConsumeHeap
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid2262.hprof ...
Heap dump file created [531535128 bytes in 14.691 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at ConsumeHeap$BigObject.(ConsumeHeap.java:22)
at ConsumeHeap.main(ConsumeHeap.java:32)
The ConsumeHeap fills up the Java heap and runs out of memory. When java.lang.OutOfMemoryError is thrown, a heap dump file is created. In this case the file is 507MB and is created as java_pid2262.hprof in the current directory.
By default the heap dump is created in a file called java_pidpid.hprof in the working directory of the VM, as in the example above. You can specify an alternative file name or directory with the -XX:HeapDumpPath= option. For example -XX:HeapDumpPath=/disk2/dumps will cause the heap dump to be generated in the /disk2/dumps directory.
When a fatal error occurs, the HotSpot Virtual Machine can optionally execute a user-supplied script or command. The script or command is specified using the -XX:OnError=string command line option, where string is a single command, or a list of commands separated by a semicolon. Within string, all occurrences of %p are replaced with the current process ID (pid), and all occurrences of %% are replaced by a single %. The following examples demonstrate how this option can be used.
On Solaris OS the pmap command displays information about the address space of a process. In the following example, if a fatal error occurs, the pmap command is executed to display the address space of the process.
java -XX:OnError="pmap %p" MyApplication
The following example shows how the fatal error report can be mailed to a support alias when a fatal error is encountered
java -XX:OnError="cat hs_err_pid%p.log|mail support@acme.com" \ MyApplication
On Solaris OS the gcore command creates a core image of the specified process, and the dbx command launches the debugger. In the following example, the gcore command is executed to create the core image, and the debugger is started to attach to the process.
java -XX:OnError="gcore %p;dbx - %p" MyApplication
In the following Linux example, the gdb debugger is launched when an unexpected error is encountered. Once launched, gdb will attach to the VM process
java -XX:OnError="gdb - %p" MyApplication
On Windows the Dr. Watson debugger can be configured as the
post-mortem debugger so that a crash dump is created when an
unexpected error is encountered. See 7.4.4
Collecting Crash Dumps on Windows for other details.
An alternate approach to obtaining a crash dump on Windows it to use the -XX:OnError option to execute the userdump.exe utility, as follows.
java -XX:OnError="userdump.exe %p" MyApplication
The userdump utility is part of the Microsoft OEM
Support Tools package, which can be downloaded from the Microsoft
site. See 7.4.4
Collecting Crash Dumps on Windows for further details.
The above example assumes that the path to the userdump.exe utility is defined in the PATH variable.
When the option -XX:+ShowMessageBoxOnError is set and a fatal error is encountered, the HotSpot VM will display information about the fatal error and prompt the user to specify whether the native debugger is to be launched. In the case of Solaris OS and Linux, the output and prompt are sent to the application console (standard input and standard output). In the case of Windows, a Windows message box pops up.
Below is an example from a fatal error encountered on a Linux system.
============================================================================== Unexpected Error ------------------------------------------------------------------------------ SIGSEGV (0xb) at pc=0x2000000001164db1, pid=10791, tid=1026 Do you want to debug the problem? To debug, run 'gdb /proc/10791/exe 10791'; then switch to thread 1026 Enter 'yes' to launch gdb automatically (PATH must include gdb) Otherwise, press RETURN to abort... ==============================================================================
In this case a SIGSEGV error has occurred and the user is prompted to specify whether the gdb debugger is to be launched to attach to the process. If the user enters y or yes, gdb will be launched (assuming it is on the PATH variable).
On Solaris OS the message is similar to the above except that
the user is prompted to start the dbx debugger. On Windows
a message box is displayed. If the user presses the YES button, the
VM will attempt to start the default debugger. This debugger is
configured by a registry setting; see 7.4.4
Collecting Crash Dumps on Windows for further details. If
Microsoft Visual Studio is installed, the default debugger is
typically configured to be msdev.exe.
In the above example the output includes the process ID (10791 in this case) and also the thread ID (1026 in this case). If the debugger is launched, one of the initial steps in the debugger might be to select the thread and obtain its stack trace.
As the process is waiting for a response it is possible to use other tools to obtain a crash dump or query the state of the process. On Solaris OS, for example, a core dump can be obtained using the gcore utility.
On Windows a Dr. Watson crash dump can be obtained using the
userdump or windbg programs. The windbg
program is included in Microsoft's Debugging Tools for Windows. See
7.4.4 Collecting Crash Dumps on Windows
for further information on windbg and the link to the
download location. In windbg select the Attach to a
Process menu option, which displays the list of processes and
prompts for the process ID. The HotSpot VM displays a message box,
which includes the process ID. Once selected the .dump /f
command can be used to force a crash dump. In the following example
a crash dump is created in file crash.dump.
In general the -XX:+ShowMessageBoxOnError option is more useful in a development environment where debugger tools are available. The -XX:OnError option is more suitable for production environments where a fixed sequence of commands or scripts are executed when a fatal error is encountered.
Several other -XX command-line options can be useful in troubleshooting.
-XX:OnOutOfMemoryError=string is used to specify a command or script to execute when an OutOfMemoryError is first thrown.
-XX:ErrorFile=filename is used to specify a
location for the fatal error log file. See C.1
Location of Fatal Error Log.
-XX:HeapDumpPath=path is used to specify a
location for a heap dump. See B.1.2
-XX:+HeapDumpOnOutOfMemoryError Option.
-XX:MaxPermSize=size is used to specify the
size of the permanent generation memory. See 3.1.2
Detail Message: PermGen space.
-XX:+PrintCommandLineFlags is used to print all the VM
command-line flags. See 7.3
Collecting Data for a Bug Report.
-XX:+PrintConcurrentLocks will cause the Ctrl-Break handler to print a list of concurrent locks owned by each thread.
-XX:+PrintClassHistogram will cause the Ctrl-Break handler to print a heap histogram.
-XX:+PrintGCDetails and -XX:+PrintGCTimeStamps
are used to print detailed information about garbage collection.
See B.2.3 -verbose:gc Option.
-XX:+UseAltSigs is used (on Solaris 8 and 9 OS) to
instruct the HotSpot VM to use alternate signals to
SIGUSR1 and SIGUSR2. See 6.1
Signal Handling on Solaris OS and Linux.
-XX:+UseConcMarkSweepGC, -XX:+UseSerialGC, and
-XX:+UseParallelGC specify the garbage collection policy
to be used. See 4.2.2 Crash
During Garbage Collection.
In addition to the -XX options, many other command-line options can provide troubleshooting information. This section describes a few of these options.
The -Xcheck:jni option is useful in diagnosing problems with applications that use the Java Native Interface (JNI). Sometimes bugs in the native code can cause the HotSpot VM to crash or behave incorrectly.
The -Xcheck:jni option is added to the command line that starts the application, as in the following example.
java -Xcheck:jni MyApplication
The -Xcheck:jni option causes the VM to do additional validation on the arguments passed to JNI functions. Note that the option is not guaranteed to find all invalid arguments or diagnose logic bugs in the application code, but it can help diagnose a large number of such problems.
When an invalid argument is detected, the VM prints a message to the application console or to standard output, prints the stack trace of the offending thread, and aborts the VM.
In the following example, a NULL value was incorrectly passed to a JNI function that does not allow a NULL value.
FATAL ERROR in native method: Null object passed to JNI
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:343)
- locked <0x450b9f70> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:439)
at java.net.ServerSocket.accept(ServerSocket.java:410)
at org.apache.tomcat.service.PoolTcpEndpoint.acceptSocket
(PoolTcpEndpoint.java:286)
at org.apache.tomcat.service.TcpWorkerThread.runIt
(PoolTcpEndpoint.java:402)
at org.apache.tomcat.util.ThreadPool$ControlRunnable.run
(ThreadPool.java:498)
at java.lang.Thread.run(Thread.java:536)
In the following example, an incorrect argument was provided to a JNI function that expects a jfieldID argument.
FATAL ERROR in native method: Instance field not found in JNI get/set
field operations
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359)
- locked <0xf082f290> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.bind(ServerSocket.java:318)
at java.net.ServerSocket.<init>(ServerSocket.java:185)
at jvm003a.<init>(jvm003.java:190)
at jvm003a.<init>(jvm003.java:151)
at jvm003.run(jvm003.java:51)
at jvm003.main(jvm003.java:30)
The following list presents examples of other problems that the -Xcheck:jni option can help diagnose.
Cases where the JNI environment for the wrong thread is used
Cases where an invalid JNI reference is used
Cases where a reference to a non-array type is provided to a function that requires an array type
Cases where a non-static field ID is provided to a function that expects a static field ID
Cases where a JNI call is made with an exception pending
In general, all errors detected by the -Xcheck:jni option are fatal errors, that is, the error is printed and the VM is aborted. There is one exception to this behavior. When a JNI call is made within a JNI critical region, the following non-fatal warning message is printed.
Warning: Calling other JNI functions in the scope of Get/ReleasePrimitiveArrayCritical or Get/ReleaseStringCritical
A JNI critical region is created when native code uses the JNI functions GetPrimitiveArrayCritical or GetStringCritical to obtain a reference to an array or string in the Java heap. The reference is held until the native code calls the corresponding release function. The code between the get and release is called a JNI critical section and during that time the HotSpot VM cannot bring the VM to a state that allows garbage collection to occur. The general recommendation is not to use other JNI functions within a JNI critical section, and in particular any JNI function that could potentially cause a deadlock. The warning printed above by the -Xcheck:jni option is thus an indication of a potential issue; it does not always indicate an application bug.
For more information on JNI, refer to the Java Native Interface documentation web site.
The -verbose:class option enables logging of class loading and unloading.
The -verbose:gc option enables logging of garbage collection (GC) information. It can be combined with other HotSpot VM specific options such as -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps to get further information about the GC. The information output includes the size of the generations before and after each GC, total size of the heap, the size of objects promoted, and the time taken.
These options, together with detailed information about GC analysis and tuning, are described at the GC Portal site.
The -verbose:gc option can be dynamically enabled at
runtime using the management API or JVM TI. See section 2.17 Developing Diagnostic Tools for
further information on these APIs.
The jconsole monitoring and management tool can also
enable or disable the option when the tool is attached to a
management VM. See 2.3
JConsole Utility.
The -verbose:jni option enables logging of JNI. When a JNI or native method is resolved, the HotSpot VM prints a trace message to the application console (standard output). It also prints a trace message when a native method is registered using the JNI RegisterNative function. The -verbose:jni option can be useful in diagnosing issues with applications that use native libraries.
| |