How to generate and download Java Application heap dump from a Tanzu Application Service (TAS) for VMs container

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

There are restrictions on how you can capture heap dumps for your application because your application runs in a container on a remote host. This article explains how to generate and download Java Application heap dump from a Tanzu Application Service (TAS) for VMs container.

Resolution

Method #1 - Spring Boot Actuators

The easiest option is to use Spring Boot Actuators. For Spring Boot apps with actuators enabled, a heap dump is available at the /actuator/heapdump endpoint in Spring Boot 2.x. Please refer details at Spring Boot v2.1.x.RELEASE - Part V. Spring Boot Actuator: Production-ready features.

Method #2 - APM Tools

Another easy option is to use an APM Tool (NewRelic, AppDynamics, Dynatrace, etc.). These tools are actively monitoring your app and can report heap memory usage. The Java buildpack supports these and other APM Tools.

Method #3 - Java Memory Assistant

The Java Memory Assistant is a Java Agent that can be installed by the Java buildpack - it can be configured to automatically take heap dumps on configured intervals. This works great in conjunction with a bound volume service, such that heap dumps are taken periodically and saved to persistent storage.

The Java Memory Assistant does not currently support a trigger to take heap dumps at a specific moment, it only works on scheduled intervals. It also only supports Oracle & SAP JVMs, so it does not work out-of-the-box with the JVM provided by Pivotal's Java buildpack.

Method #4 - JMX

Another option is to trigger a heap dump is to do it over JMX. Start by using the instructions here to enable JMX with your app (please note this requires a restage of your application). With JMX enabled, create an SSH tunnel and connect with Jconsole.

In JConsole, you can trigger a heap dump by going to the MBeans tab, expanding "com.sun.management" and then HotSpotDiagnostics from the tree on the left. Click "Operations". You should now see a button that says "dumpHeap". This requires two arguments. The first is the path to which you want to send the heap dump. The second argument indicates if you want only live objects (true) or if you want all objects (false). For a path, enter /home/vcap/tmp/dump.hprof or whatever you want to name the file.

To retrieve the file, you can use the Security Copy Protocol (SCP). Please note this requires SSH access to your app container, or you can bind a volume service and place the heap dumps there.

To retrieve via SCP, do the following:

1. Run cf ssh-code to get a one-time access code for SSH into your container.
2. Get the SSH endpoint to use. Run cf curl /v2/info | grep app_ssh_endpoint
3. Obtain the App GUID for your app. Run cf app APP_NAME --guid
4. Download the heap dump with scp -P 2222 -o User=cf:<your-app-guid>/<app-index> <SSH-endpoint>:/home/vcap/tmp/dump.hprof ./dump.hprof. Make sure to substitute your app guid, the app index and the SSH endpoint that you retrieved from steps 2 and 3.

Method #5 - Java Kill Agent

The Java Kill Agent is another agent that is bundled with the Java buildpack. It is installed and configured to run when there is a resource exhaustion event. It will then print a histogram of the 100 largest types by a total number of bytes and a summary of usage for all the JVM's memory spaces.

It can also be configured to write a heap dump. If a Volume Service with the string heap-dump in its name or tag is bound to the application, terminal heap dumps will be written to it with the pattern:

<CONTAINER_DIR>/<SPACE_NAME>-<SPACE_ID[0,8]>/<APPLICATION_NAME>-<APPLICATION_ID[0,8]>/<INSTANCE_INDEX>-<TIMESTAMP>-<INSTANCE_ID[0,8]>.hprof.

Method #6 - with jattach

Most Java developers are probably familiar with jmap and jcmd for taking heap dumps, however since TAS for VMs does not have a full JDK, these commands are not available. The jattach is a 3rd party command that is like jmap + jstack + jcmd + jinfo all rolled into one and utilizes the JVM's Dynamic Attach mechanism. Best yet, it works perfectly with only a JRE.

Here are the instructions:

1. Download jattach from the Releases page on Github. If you have access to the Internet from inside your container, you can do this with curl -L -O <release-url>. You can then skip steps #2, #3 and #4.

2. Get a one-time code for SCP. You will need to do this for each SCP command you issue.

$ cf ssh-code
xxxxxxxx

3. Get SSH endpoint for SCP. Replace "ssh.SYSTEM_DOMAIN" with the domain returned from this command.

cf curl /v2/info | grep app_ssh_endpoint

4. Upload the required files from the JDK download using the following commands:

scp -P 2222 -o User=cf:$(cf app APP_NAME --guid)/0 path/to/jattach ssh.SYSTEM_DOMAIN:/home/vcap/app/.java-buildpack/

4. Get into the app container:

cf ssh APP_NAME -i INDEX

5. List the process in the container, find the Java app process ID:

$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   1084     4 ?        S<s  Oct05   0:00 /tmp/garden-init
vcap          16  0.2  2.0 2202944 343068 ?      S<sl Oct05  11:41 /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java -agentpath:/home/vcap/app/.java-buildpack/open_jdk_jre/bin/jvmkill-1.12.0_RELEASE=printH
...

5. Generate heap dump and optionally gzip it.

$ /home/vcap/app/.java-buildpack/jattach 16 dumpheap dump.hprof
Connected to remote JVM
Response code = 0
Heap dump file created
$ gzip /home/vcap/app/dump.hprof

6. Get one-time code for SCP: 

$ cf ssh-code
xxxxxxxx

7. Download dump.hprof file from the container. Replace the app name, index, and SSH system domain again. If you gzip your heap dump, append `.gz` to the end of the file names.

$ scp -P 2222 -o User=cf:$(cf app APP_NAME --guid)/INDEX ssh.SYSTEM_DOMAIN:/home/vcap/app/dump.hprof dump.hprof
...
dump.hprof                                                                                                                                                                100%   44MB 754.9KB/s   00:59

Note: Default SSH settings may suppress the password prompt. You may need to supply the additional -o switch, to re-enable password authentication and password prompt to show. 

scp -o StrictHostKeyChecking=no -o PasswordAuthentication=yes -P 2222 -oUser=cf:$(cf app APP_NAME --guid)/INDEX ssh.SYSTEM_DOMAIN:/home/vcap/app/dump.hprof dump.hprof

 

 

You should now have your heap dump on the local machine.

This process will download the heap dump to the local file system inside your application container. Heap dumps can be large, so be careful when doing this and make sure that there is sufficient disk space inside the application container. If you fill-up the disk space in the container, this could impact the operation of the application instance.

If the heap dump is very large and will not fit in the available local disk space then you will need to attach a volume service instance to your application so that the heap dump can be written there. Due to limitations in the JVM (Oracle/OpenJDK), it is not currently possible to stream out a heap dump to another process or the network.

Please see the Java buildpack instructions for more details on how to bind a volume service to store heap dumps.

 

Additional Information

This process will download the heap dump to the local file system inside your application container. Heap dumps can be large, so be careful when doing this and make sure that there is sufficient disk space inside the application container. If you fill up the disk space in the container, this could impact the operation of the application instance.

If the heap dump is very large and will not fit in the available local disk space then you will need to attach a volume service instance to your application so that the heap dump can be written there. Due to limitations in the JVM (Oracle/OpenJDK), it is not currently possible to stream out a heap dump to another process or the network.

Please see the Java buildpack instructions for more details on how to bind a volume service to store heap dumps.