When a fatal error occurs, an error log is created with information and the state obtained at the time of the fatal error.
Note that the format of this file can change slightly in update releases.
This appendix contains the following sections.
The product flag -XX:ErrorFile=file can be used to specify where the file will be created, where file represents the full path for the file location. The substring %% in the file variable is converted to %, and the substring %p is converted to the process ID of the process.
In the following example, the error log file will be written to the directory /var/log/java and will be named java_errorpid.log.
java -XX:ErrorFile=/var/log/java/java_error%p.log
If the -XX:ErrorFile=file flag is not specified, by default the file name is hs_err_pidpid.log, where pid is the process ID of the process.
In addition, if the -XX:ErrorFile=file flag is not specified, the system attempts to create the file in the working directory of the process. In the event that the file cannot be created in the working directory (insufficient space, permission problem, or other issue), the file is created in the temporary directory for the operating system. On Solaris OS and Linux the temporary directory is /tmp. On Windows the temporary directory is specified by the value of the TMP environment variable; if that environment variable is not defined, the value of the TEMP environment variable is used.
The error log contains information obtained at the time of the fatal error, including the following information, where possible.
The operating exception or signal that provoked the fatal error
Version and configuration information
Details on the thread that provoked the fatal error and thread's stack trace
The list of running threads and their state
Summary information about the heap
The list of native libraries loaded
Command line arguments
Environment variables
Details about the operating system and CPU
Note - In some cases only a subset of this information is output to the error log. This can happen when a fatal error is of such severity that the error handler is unable to recover and report all details.
The error log is a text file consisting of the following sections:
A header that provides a brief description of the crash. See B.3 Header Format.
A section with thread information. See B.4 Thread Section Format.
A section with process information. See B.5 Process Section Format.
A section with system information. See B.6 System Section Format.
Note - Note that the format of the fatal error log described here is based on Java SE 6. The format might be different with other releases.
The header section at the beginning of every fatal error log file contains a brief description of the problem. The header is also printed to standard output and may show up in the application's output log.
The header includes a link to the HotSpot Virtual Machine Error Reporting Page, where the user can submit a bug report.
The following is a sample header from a crash.
# # An unexpected error has been detected by Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x417789d7, pid=21139, tid=1024 # # Java VM: Java HotSpot(TM) Client VM (1.6.0-rc-b63 mixed mode, sharing) # Problematic frame: # C [libNativeSEGV.so+0x9d7] # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp #
This example shows that the VM crashed on an unexpected signal. The next line describes the signal type, program counter (pc) that caused the signal, process ID and thread ID, as follows.
# SIGSEGV (0xb) at pc=0x417789d7, pid=21139, tid=1024 | | | | +--- thread id | | | +------------- process id | | +--------------------------- program counter | | (instruction pointer) | +--------------------------------------- signal number +---------------------------------------------- signal name
The next line contains the VM version (Client VM or Server VM), an indication whether the application was run in mixed or interpreted mode, and an indication whether class file sharing was enabled.
# Java VM: Java HotSpot(TM) Client VM (1.6.0-rc-b63 mixed mode, sharing)
The next information is the function frame that caused the crash, as follows.
# Problematic frame: # C [libNativeSEGV.so+0x9d7] | +-- Same as pc, but represented as library name and offset. | For position-independent libraries (JVM and most shared | libraries), it is possible to inspect the instructions | that caused the crash without a debugger or core file | by using a disassembler to dump instructions near the | offset. +----------------- Frame type
In this example, the “C” frame type indicates a native C frame. The following table shows the possible frame types.
|
Internal errors will cause the VM error handler to generate a similar error dump. However, the header format is different. Examples of internal errors are guarantee() failure, assertion failure, ShouldNotReachHere(), and so forth. Here is an example of how the header looks for an internal error.
# # An unexpected error has been detected by HotSpot Virtual Machine: # # Internal Error (4F533F4C494E55583F491418160E43505000F5), pid=10226, tid=16384 # # Java VM: Java HotSpot(TM) Client VM (1.6.0-rc-b63 mixed mode)
In the above header, there is no signal name or signal number. Instead the second line now contains the text “Internal Error” and a long hexadecimal string. This hexadecimal string encodes the source module and line number where the error was detected. In general this “error string” is useful only to engineers working on the HotSpot Virtual Machine.
The error string encodes a line number and therefore it changes with each code change and release. A crash with a given error string in one release (for example 1.6.0) might not correspond to the same crash in an update release (for example 1.6.0_01), even if the strings match.
Note - Do not assume that a workaround or solution that worked in one situation associated with a given error string will work in another situation associated with that same error string. Note the following facts:
Errors with the same root cause might have different error strings.
Errors with the same error string might have completely different root causes.
Therefore, the error string should not be used as the sole criterion when troubleshooting bugs.
This section contains information about the thread that just crashed. If multiple threads crash at the same time, only one thread is printed.
The first part of the thread section shows the thread that provoked the fatal error, as follows.
Current thread (0x0805ac88): JavaThread "main" [_thread_in_native, id=21139] | | | | +-- ID | | | +------------- state | | +-------------------------- name | +------------------------------------ type +-------------------------------------------------- pointer
The thread pointer is the pointer to the Java VM internal thread structure. It is generally of no interest unless you are debugging a live Java VM or core file.
The following list shows possible thread types.
JavaThread
VMThread
CompilerThread
GCTaskThread
WatcherThread
ConcurrentMarkSweepThread
The following table shows the important thread states.
|
The thread ID in the output is the native thread identifier.
If a Java thread is a daemon thread, then the string daemon is printed before the thread state.
The next information in the error log describes the unexpected signal that caused the VM to terminate. On a Windows system the output appears as follows.
siginfo: ExceptionCode=0xc0000005, reading address 0xd8ffecf1
In the above example, the exception code is 0xc0000005 (ACCESS_VIOLATION), and the exception occurred when the thread attempted to read address 0xd8ffecf1.
On Solaris OS and Linux systems the signal number (si_signo) and signal code (si_code) are used to identify the exception, as follows.
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00004321
The next information in the error log shows the register context at the time of the fatal error. The exact format of this output is processor-dependent. The following example shows output for the Intel (IA32) processor.
Registers: EAX=0x00004321, EBX=0x41779dc0, ECX=0x080b8d28, EDX=0x00000000 ESP=0xbfffc1e0, EBP=0xbfffc1f8, ESI=0x4a6b9278, EDI=0x0805ac88 EIP=0x417789d7, CR2=0x00004321, EFLAGS=0x00010216
The register values might be useful when combined with instructions, as described below.
After the register values, the error log contains the top of stack followed by 32 bytes of instructions (opcodes) near the program counter (PC) when the system crashed. These opcodes can be decoded with a disassembler to produce the instructions around the location of the crash. Note that IA32 and AMD64 instructions are variable in length, and so it is not always possible to reliably decode instructions before the crash PC.
Top of Stack: (sp=0xbfffc1e0) 0xbfffc1e0: 00000000 00000000 0818d068 00000000 0xbfffc1f0: 00000044 4a6b9278 bfffd208 41778a10 0xbfffc200: 00004321 00000000 00000cd8 0818d328 0xbfffc210: 00000000 00000000 00000004 00000003 0xbfffc220: 00000000 4000c78c 00000004 00000000 0xbfffc230: 00000000 00000000 00180003 00000000 0xbfffc240: 42010322 417786ec 00000000 00000000 0xbfffc250: 4177864c 40045250 400131e8 00000000 Instructions: (pc=0x417789d7) 0x417789c7: ec 14 e8 72 ff ff ff 81 c3 f2 13 00 00 8b 45 08 0x417789d7: 0f b6 00 88 45 fb 8d 83 6f ee ff ff 89 04 24 e8
Where possible, the next output in the error log is the thread stack. This includes the addresses of the base and the top of the stack, the current stack pointer, and the amount of unused stack available to the thread. This is followed, where possible, by the stack frames, and up to 100 frames are printed. For C/C++ frames the library name may also be printed. It is important to note that in some fatal error conditions the stack may be corrupt, and in this case this detail may not be available.
Stack: [0x00040000,0x00080000), sp=0x0007f9f8, free space=254k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0x83d77] C [App.dll+0x1047] j Test.foo()V+0 j Test.main([Ljava/lang/String;)V+0 v ~StubRoutines::call_stub V [jvm.dll+0x80f13] V [jvm.dll+0xd3842] V [jvm.dll+0x80de4] C [java.exe+0x14c0] C [java.exe+0x64cd] C [kernel32.dll+0x214c7] Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j Test.foo()V+0 j Test.main([Ljava/lang/String;)V+0 v ~StubRoutines::call_stub
The log contains two thread stacks.
The first thread stack is Native frames, which prints the native thread showing all function calls. However this thread stack does not take into account the Java methods that are inlined by the runtime compiler; if methods are inlined they appear to be part of the parent's stack frame.
The information in the thread stack for native frames provides important information about the cause of the crash. By analyzing the libraries in the list from the top down, you can generally determine which library might have caused the problem and report it to the appropriate organization responsible for that library.
The second thread stack is Java frames, which prints the Java frames including the inlined methods, skipping the native frames. Depending on the crash it might not be possible to print the native thread stack but it might be possible to print the Java frames.
If the error occurred in the VM thread or in a compiler thread, then further details may be printed. For example, in the case of the VM thread, the VM operation is printed if the VM thread is executing a VM operation at the time of the fatal error. In the following output example, the compiler thread provoked the fatal error. The task is a compiler task and the HotSpot Client VM is compiling method hs101t004Thread.ackermann.
Current CompileTask: HotSpot Client Compiler:754 b nsk.jvmti.scenarios.hotswap.HS101.hs101t004Thread.ackermann(IJ)J (42 bytes)
For the HotSpot Server VM the output for the compiler task is slightly different but will also include the full class name and method.
The process section is printed after the thread section. It contains information about the whole process, including thread list and memory usage of the process.
The thread list includes the threads that the VM is aware of. This includes all Java threads and some VM internal threads, but does not include any native threads created by the user application that have not attached to the VM. The output format follows.
=>0x0805ac88 JavaThread "main" [_thread_in_native, id=21139] | | | | | +----- ID | | | | +------------------- state | | | | (JavaThread only) | | | +--------------------------------- name | | +------------------------------------------ type | +---------------------------------------------------- pointer +------------------------------------------------------ "=>" current thread
An example of this output follows.
Java Threads: ( => current thread ) 0x080c8da0 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=21147] 0x080c7988 JavaThread "CompilerThread0" daemon [_thread_blocked, id=21146] 0x080c6a48 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=21145] 0x080bb5f8 JavaThread "Finalizer" daemon [_thread_blocked, id=21144] 0x080ba940 JavaThread "Reference Handler" daemon [_thread_blocked, id=21143] =>0x0805ac88 JavaThread "main" [_thread_in_native, id=21139] Other Threads: 0x080b6070 VMThread [id=21142] 0x080ca088 WatcherThread [id=21148]
The thread type and thread state are described in B.4 Thread Section Format.
The next information is the VM state, which indicates the overall state of the virtual machine. The following table describes the general states.
|
The VM state output is a single line in the error log, as follows.
VM state:not at safepoint (normal execution)
The next information in the error log is a list of mutexes and monitors that are currently owned by a thread. These mutexes are VM internal locks rather than monitors associated with Java objects. Below is an example to show how the output might look when a crash happens when VM locks are held. For each lock the log contains the name of the lock, its owner, and the addresses of a VM internal mutex structure and its OS lock. In general this information is useful only to those who are very familiar with the HotSpot VM. The owner thread can be cross-referenced to the thread list.
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])[0x007357b0/0x0000031c] Threads_lock - owner thread: 0x00996318 [0x00735978/0x000002e0] Heap_lock - owner thread: 0x00736218
The next information is a summary of the heap. The output depends on the garbage collection (GC) configuration. In this example the serial collector is used, class data sharing is disabled, and the tenured generation is empty. This probably indicates that the fatal error occurred early or during start-up and a GC has not yet promoted any objects into the tenured generation. An example of this output follows.
Heap def new generation total 576K, used 161K [0x46570000, 0x46610000, 0x46a50000) eden space 512K, 31% used [0x46570000, 0x46598768, 0x465f0000) from space 64K, 0% used [0x465f0000, 0x465f0000, 0x46600000) to space 64K, 0% used [0x46600000, 0x46600000, 0x46610000) tenured generation total 1408K, used 0K [0x46a50000, 0x46bb0000, 0x4a570000) the space 1408K, 0% used [0x46a50000, 0x46a50000, 0x46a50200, 0x46bb0000) compacting perm gen total 8192K, used 1319K [0x4a570000, 0x4ad70000, 0x4e570000) the space 8192K, 16% used [0x4a570000, 0x4a6b9d48, 0x4a6b9e00, 0x4ad70000) No shared spaces configured.
The next information in the log is a list of virtual memory regions at the time of the crash. This list can be long in the case of large applications. The memory map can be very useful when debugging some crashes, as it can tell you what libraries are actually being used, their location in memory, as well as the location of heap, stack, and guard pages.
The format of the memory map is operating-system-specific. On the Solaris Operating System, the base address and library name are printed. On the Linux system the process memory map (/proc/pid/maps) is printed. On the Windows system, the base and end addresses of each library are printed. The following example output was generated on Linux/x86. Note that most of the lines have been omitted from the example for the sake of brevity.
Dynamic libraries: 08048000-08056000 r-xp 00000000 03:05 259171 /h/jdk6/bin/java 08056000-08058000 rw-p 0000d000 03:05 259171 /h/jdk6/bin/java 08058000-0818e000 rwxp 00000000 00:00 0 40000000-40013000 r-xp 00000000 03:0a 400046 /lib/ld-2.2.5.so 40013000-40014000 rw-p 00013000 03:0a 400046 /lib/ld-2.2.5.so 40014000-40015000 r--p 00000000 00:00 0 Lines omitted. 4123d000-4125a000 rwxp 00001000 00:00 0 4125a000-4125f000 rwxp 00000000 00:00 0 4125f000-4127b000 rwxp 00023000 00:00 0 4127b000-4127e000 ---p 00003000 00:00 0 4127e000-412fb000 rwxp 00006000 00:00 0 412fb000-412fe000 ---p 00083000 00:00 0 412fe000-4137b000 rwxp 00086000 00:00 0 Lines omitted. 44600000-46570000 rwxp 00090000 00:00 0 46570000-46610000 rwxp 00000000 00:00 0 46610000-46a50000 rwxp 020a0000 00:00 0 46a50000-46bb0000 rwxp 00000000 00:00 0 46bb0000-4a570000 rwxp 02640000 00:00 0 Lines omitted.
The format of a line in the above memory map is as follows.
40049000-4035c000 r-xp 00000000 03:05 824473 /jdk1.5/jre/lib/i386/client/libjvm.so |<------------->| ^ ^ ^ ^ |<----------------------------------->| Memory region | | | | | | | | | | Permission --- + | | | | r: read | | | | w: write | | | | x: execute | | | | p: private | | | | s: share | | | | | | | | File offset ----------+ | | | | | | Major ID and minor ID of -------+ | | the device where the file | | is located (i.e. /dev/hda5) | | | | inode number ------------------------+ | | File name -------------------------------------------------------+
In the memory map output, each library has two virtual memory regions: one for code and one for data. The permission for the code segment is marked with r-xp (readable, executable, private), and the permission for the data segment is rw-p (readable, writable, private).
The Java heap is already included in the heap summary earlier in the output, but it can be useful to verify that the actual memory regions reserved for heap match the values in the heap summary and that the attributes are set to rwxp.
Thread stacks usually show up in the memory map as two back-to-back regions, one with permission ---p (guard page) and one with permission rwxp (actual stack space). In addition, it is useful to know the guard page size or stack size. For example, in this memory map, the stack is located from 4127b000 to 412fb000.
On a Windows system, the memory map output is the load and end address of each loaded module, as in the example below.
Dynamic libraries: 0x00400000 - 0x0040c000 c:\jdk6\bin\java.exe 0x77f50000 - 0x77ff7000 C:\WINDOWS\System32\ntdll.dll 0x77e60000 - 0x77f46000 C:\WINDOWS\system32\kernel32.dll 0x77dd0000 - 0x77e5d000 C:\WINDOWS\system32\ADVAPI32.dll 0x78000000 - 0x78087000 C:\WINDOWS\system32\RPCRT4.dll 0x77c10000 - 0x77c63000 C:\WINDOWS\system32\MSVCRT.dll 0x08000000 - 0x08183000 c:\jdk6\jre\bin\client\jvm.dll 0x77d40000 - 0x77dcc000 C:\WINDOWS\system32\USER32.dll 0x7e090000 - 0x7e0d1000 C:\WINDOWS\system32\GDI32.dll 0x76b40000 - 0x76b6c000 C:\WINDOWS\System32\WINMM.dll 0x6d2f0000 - 0x6d2f8000 c:\jdk6\jre\bin\hpi.dll 0x76bf0000 - 0x76bfb000 C:\WINDOWS\System32\PSAPI.DLL 0x6d680000 - 0x6d68c000 c:\jdk6\jre\bin\verify.dll 0x6d370000 - 0x6d38d000 c:\jdk6\jre\bin\java.dll 0x6d6a0000 - 0x6d6af000 c:\jdk6\jre\bin\zip.dll 0x10000000 - 0x10032000 C:\bugs\crash2\App.dll
The next information in the error log is a list of VM arguments, followed by a list of environment variables. An example follows.
VM Arguments: java_command: NativeSEGV 2 Environment Variables: JAVA_HOME=/h/jdk PATH=/h/jdk/bin:.:/h/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin: /usr/dist/local/exe:/usr/dist/exe:/bin:/usr/sbin:/usr/ccs/bin: /usr/ucb:/usr/bsd:/usr/etc:/etc:/usr/dt/bin:/usr/openwin/bin: /usr/sbin:/sbin:/h:/net/prt-web/prt/bin USERNAME=user LD_LIBRARY_PATH=/h/jdk6/jre/lib/i386/client:/h/jdk6/jre/lib/i386: /h/jdk6/jre/../lib/i386:/h/bugs/NativeSEGV SHELL=/bin/tcsh DISPLAY=:0.0 HOSTTYPE=i386-linux OSTYPE=linux ARCH=Linux MACHTYPE=i386
Note that the list of environment variables is not the full list but rather a subset of the environment variables that are applicable to the Java VM.
On Solaris OS and Linux, the next information in the error log is the list of signal handlers.
Signal Handlers: SIGSEGV: [libjvm.so+0x3aea90], sa_mask[0]=0xfffbfeff, sa_flags=0x10000004 SIGBUS: [libjvm.so+0x3aea90], sa_mask[0]=0xfffbfeff, sa_flags=0x10000004 SIGFPE: [libjvm.so+0x304e70], sa_mask[0]=0xfffbfeff, sa_flags=0x10000004 SIGPIPE: [libjvm.so+0x304e70], sa_mask[0]=0xfffbfeff, sa_flags=0x10000004 SIGILL: [libjvm.so+0x304e70], sa_mask[0]=0xfffbfeff, sa_flags=0x10000004 SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGUSR2: [libjvm.so+0x306e80], sa_mask[0]=0x80000000, sa_flags=0x10000004 SIGHUP: [libjvm.so+0x3068a0], sa_mask[0]=0xfffbfeff, sa_flags=0x10000004 SIGINT: [libjvm.so+0x3068a0], sa_mask[0]=0xfffbfeff, sa_flags=0x10000004 SIGQUIT: [libjvm.so+0x3068a0], sa_mask[0]=0xfffbfeff, sa_flags=0x10000004 SIGTERM: [libjvm.so+0x3068a0], sa_mask[0]=0xfffbfeff, sa_flags=0x10000004 SIGUSR2: [libjvm.so+0x306e80], sa_mask[0]=0x80000000, sa_flags=0x10000004
The final section in the error log is the system information. The output is operating-system-specific but in general includes the operating system version, CPU information, and summary information about the memory configuration.
The following example shows output on a Solaris 9 OS system.
--------------- S Y S T E M --------------- OS: Solaris 9 12/05 s9s_u5wos_08b SPARC Copyright 2005 Sun Microsystems, Inc. All Rights Reserved. Use is subject to license terms. Assembled 21 November 2005 uname:SunOS 5.9 Generic_112233-10 sun4u (T2 libthread) rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity load average:0.41 0.14 0.09 CPU:total 2 has_v8, has_v9, has_vis1, has_vis2, is_ultra3 Memory: 8k page, physical 2097152k(1394472k free) vm_info: Java HotSpot(TM) Client VM (1.5-internal) for solaris-sparc, built on Aug 12 2005 10:22:32 by unknown with unknown Workshop:0x550
On Solaris OS and Linux, the operating system information is contained in the file /etc/*release. This file describes the kind of system the application is running on, and in some cases the information string might include the patch level. Some system upgrades are not reflected in the /etc/*release file. This is especially true on the Linux system, where the user can rebuild any part of the system.
On Solaris OS the uname system call is used to get the name for the kernel. The thread library (T1 or T2) is also printed.
On the Linux system the uname system call is also used to get the kernel name. The libc version and the thread library type are also printed. An example follows.
uname:Linux 2.4.18-3smp #1 SMP Thu Apr 18 07:27:31 EDT 2002 i686 libc:glibc 2.2.5 stable linuxthreads (floating stack) |<- glibc version ->|<-- pthread type -->|
On Linux there are three possible thread types, namely linuxthreads (fixed stack), linuxthreads (floating stack), and NPTL. They are normally installed in /lib, /lib/i686, and /lib/tls.
It is useful to know the thread type. For example, if the crash appears to be related to pthread, then you might be able to work around an issue by selecting a different pthread library. A different pthread library (and libc) can be selected by setting LD_LIBRARY_PATH or LD_ASSUME_KERNEL.
The glibc version usually does not include the patch level. The command rpm -q glibc might provide more detailed version information.
On Solaris OS and Linux, the next information is the rlimit information. Note that the default stack size of the VM is usually smaller than the system limit. An example follows.
rlimit: STACK 8192k, CORE 0k, NPROC 4092, NOFILE 1024, AS infinity | | | | virtual memory (-v) | | | +--- max open files (ulimit -n) | | +----------- max user processes (ulimit -u) | +------------------------- core dump size (ulimit -c) +---------------------------------------- stack size (ulimit -s) load average:0.04 0.05 0.02
The next information specifies the CPU architecture and capabilities identified by the VM at start-up, as in the following example.
CPU:total 2 family 6, cmov, cx8, fxsr, mmx, sse | | |<----- CPU features ---->| | | | +--- processor family (IA32 only): | 3 - i386 | 4 - i486 | 5 - Pentium | 6 - PentiumPro, PII, PIII | 15 - Pentium 4 +------------ Total number of CPUsThe following table shows the possible CPU features on a SPARC system.
|
The following table shows the possible CPU features on an Intel/IA32 system.
|
The following table shows the possible CPU features on an AMD64/EM64T system.
|
The next information in the error log is memory information, as follows.
unused swap space total amount of swap space | unused physical memory | | total amount of physical memory | | | page size | | | | v v v v v Memory: 4k page, physical 513604k(11228k free), swap 530104k(497504k free)
Some systems require swap space to be at lease twice the size of real physical memory, whereas other systems do not have any such requirements. As a general rule, if both physical memory and swap space are almost full, there is good reason to suspect that the crash was due to insufficient memory.
On Linux systems the kernel may convert most of unused physical memory to file cache. When there is a need for more memory, the Linux kernel will give the cache memory back to the application. This is handled transparently by the kernel, but it does mean the amount of unused physical memory reported by fatal error handler could be close to zero when there is still sufficient physical memory available.
The final information in the SYSTEM section of the error log is vm_info, which is a version string embedded in libjvm.so/jvm.dll. Every Java VM has its own unique vm_info string. If you are in doubt about whether the fatal error log was generated by a particular Java VM, check the version string.