How to take thread dumps for VMware Tanzu Gemfire VMs
Article ID: 294040
Updated On:
Products
VMware Tanzu Gemfire
Issue/Introduction
This article describes the process of taking Java thread dumps for VMware Tanzu GemFire service instance VMs.
The Support team may request you to take thread dumps for VMware Tanzu GemFire VMs to troubleshoot issues. Usually thread dumps are needed when we see some member or members are not responsive.
The Support team may request you to take thread dumps for VMware Tanzu GemFire VMs to troubleshoot issues. Usually thread dumps are needed when we see some member or members are not responsive.
Resolution
VMware Tanzu GemFire VMs run GemFire server and locator processes under the hood. Java thread dumps for GemFire server and locator are useful when troubleshooting GemFire issues.
For any Java process, you run `jcmd Thread.print` to take thread dump. However, for VMware Tanzu Gemfire, the BOSH process manager runs the GemFire servers and locators within its own container. Therefore, taking thread dumps and exporting them out of the VM is not as straightforward. To do this, refer to the steps below:
1. bosh ssh into the VMware Tanzu Gemfire VM from where you want to take thread dump bosh -e <ENV> -d <DEPLOYMENT> ssh server/0. VMware Tanzu Gemfire VMs can be referenced using 0-based index like server/0, locator/2 etc.
2. Get into the BOSH Process Manager (bpm) shell by running sudo /var/vcap/packages/bpm/bin/bpm shell <job-name>, where job-name is gemfire-server or gemfire-locator depending on which VMware Tanzu Gemfire VM you are on.
3. Find the PID which is running the GemFire Java process by running ps -aux | grep java. Typically PID is 1.
4. Take at-least 3 thread dumps every couple of seconds apart using /var/vcap/packages/jdk8/bin/jcmd 1 Thread.print > /tmp/threaddump1.txt. Make sure to redirect the output to different files each time. Files in the /tmp on the container will be visible under /var/vcap/data/gemfire-server/tmp directory on the VM.
5. Move the files to the /tmp directory on the VM by running mv var/vcap/data/gemfire-server/tmp/thread.dump /tmp/.
6. Files can be SSH'ed to your local machine using bosh scp command. From your local machine run:
For any Java process, you run `jcmd Thread.print` to take thread dump. However, for VMware Tanzu Gemfire, the BOSH process manager runs the GemFire servers and locators within its own container. Therefore, taking thread dumps and exporting them out of the VM is not as straightforward. To do this, refer to the steps below:
1. bosh ssh into the VMware Tanzu Gemfire VM from where you want to take thread dump bosh -e <ENV> -d <DEPLOYMENT> ssh server/0. VMware Tanzu Gemfire VMs can be referenced using 0-based index like server/0, locator/2 etc.
2. Get into the BOSH Process Manager (bpm) shell by running sudo /var/vcap/packages/bpm/bin/bpm shell <job-name>, where job-name is gemfire-server or gemfire-locator depending on which VMware Tanzu Gemfire VM you are on.
3. Find the PID which is running the GemFire Java process by running ps -aux | grep java. Typically PID is 1.
4. Take at-least 3 thread dumps every couple of seconds apart using /var/vcap/packages/jdk8/bin/jcmd 1 Thread.print > /tmp/threaddump1.txt. Make sure to redirect the output to different files each time. Files in the /tmp on the container will be visible under /var/vcap/data/gemfire-server/tmp directory on the VM.
5. Move the files to the /tmp directory on the VM by running mv var/vcap/data/gemfire-server/tmp/thread.dump /tmp/.
6. Files can be SSH'ed to your local machine using bosh scp command. From your local machine run:
`bosh -d scp :/tmp/threaddumppppp.txt /tmp Ex: bosh -d service-instance_1fd2850e-b754-4c5e-aa5c-ddb54ee301e6 scp server/b76e9665-2299-4f40-88e0-eaeb7c7dfb10:/tmp/threaddump1.txt /tmp
Feedback
Yes
No
Powered by
