Where Are My Dump Files?

In this post I want to write a little bit about what happens when JRockit crashes. Specifically, I want to talk about some of the situations where you may have trouble finding the dump files (text and binary dumps). While this is usually not an issue, it can be very frustrating to experience a JVM crash and not be able to find any of the important data needed to troubleshoot the issue further. Hopefully this post will shed some light on some of the scenarios where these files are not created or not where you expect them to be.

The dump files are described in detail here. For this discussion, you only need to know the difference between the text dump file (jrockit.<pid>.dump) and binary dump file ("core.<pid>" on Linux, "core" on Solaris, and "jrockit.<pid>.mdmp" on Windows).

The default behavior is for these two files to be written to the current working directory (cwd) of the JVM process. This behavior is consistent with other JVMs (including HotSpot) and most other software. But there are situations where this is not ideal. For example, if the JVM's cwd is on a file system with limited space, you may want to designate another location for crash files to be output. It is often a good idea to isolate crash file output from the rest of the system so that a series of crashes do not exhaust the free space on a filesystem that is needed to keep your application up and running. This reasoning is similar to using a dedicated filesystem for log file output.

On Linux and Solaris, one option is to use the tools built into the OS to specify an alternate output location for all binary dump files. On Linux, you can edit the /proc/sys/kernel/core_pattern file to specify a different path and/or naming convention for binary dump files. Likewise, on Solaris you can use coreadm to specify a different path for output. Note that with Linux, your only option for configuration at the OS level is to change the global behavior for all processes, not just JRockit (Solaris thankfully gives you per-process control, if needed). Also note that both of these options only impact where the binary dump file is output; the text dump file will still be output to the cwd of the JVM, regardless of the OS-level settings. This is often not a concern as the size of most text dump files is usually negligible compared to the space requirements for binary dump files.

When JR crashes, another consideration is that the dump file paths printed as part of the error message assume the default OS behavior. If you use either of the above OS-level methods to specify a non-default output location for binary dump files, JRockit has no way of knowing this and will output incorrect paths. It is best to think of the path output for the binary dump as simply an "educated guess".

Another option is to set the JRockit specific environmental variable, JROCKIT_DUMP_PATH. This will work even on Windows. It also differs from the OS-level settings above in that both binary and text dump files will be written to the specified directory. Note that JROCKIT_DUMP_PATH depends on default settings for OS-level configuration. If you use either of the above methods to override the location for binary dump file output, JROCKIT_DUMP_PATH will only impact the location of the text dump file.

There are also two well-known exceptions to my description above. In both cases, our signal handler code (or exception handler code on Windows) is not able to run successfully (or at all), therefor causing issues. The result is that JROCKIT_DUMP_PATH may be ignored, or the dump files could be truncated or even completely missing. Depending on the platform, output of both dump files can be dependent on the correct operation of the signal handler.

The first case is when a binary dump is produced via an external signal. The most common scenario is when you use the kill command to send a signal like SIGABRT to intentionally abort the process and generate a binary dump file. On both Linux and Solaris, this will result in the signal being "delivered" to the process's primordial thread. Starting from Java SE 6, the java launcher forks off from the primordial thread as soon as possible and the primordial thread does nothing but wait for the JVM to shutdown. As the signal handlers are never registered for the primordial thread (What would be the point? It doesn't do anything but wait.), a signal delivered to the primordial thread will result in the OS-defined default action, and JRockit will never have a chance to influence the output any binary dump created. This is also why you will not get a text dump in this situation. The recommended way to "artificially" trigger a crash is to use the force_crash diagnostic command and avoid this issue.

The other scenario where you may not find the expected dump files is when the JVM's state is so corrupted that our signal handler can not run without itself crashing. By far, the most common cause of this is a stack overflow. Especially on Windows, it is very common to end up with a 0-byte mdmp (mini-dump) file when you blow the stack. If you ever find missing or truncated dump files, a stack overflow should be the first culprit you suspect.

By default dump files will be output into the JVM's current working directory, but you can override that behavior with OS-level settings (on Linux/Solaris) or the JROCKIT_DUMP_PATH environmental variable. Also remember that JROCKIT_DUMP_PATH may be totally ignored if an external signal is received, and the dump files may never even get created correctly if you suffer a stack overflow.