How to generate heapdump for VMware Aria/VCF Operations services

Products

VCF Operations/Automation (formerly VMware Aria Suite) VCF Operations

Issue/Introduction

While working with VMware by Broadcom Support, you may be requested to provide a heapdump from your VMware Aria/VCF Operations deployment for further troubleshooting. This article provide the steps to generate a heapdump in VMware Aria/VCF Operations

Environment

VMware Aria Operations 8.18.x

VMware Cloud Foundation Operations 9.x

Resolution

The heapdump generation is on a per node basis. If multiple heapdumps per process or node are requested, repeat the below steps as needed.

Log into the desired VMware Aria/VCF Operations node as root via SSH or Console.
Ensure the service for which the heapdump should be generated is in an active (running) state by running the following command:
```
service vmware-vcops status service_name
```
Note: Replace service_name with the service name of the service you want to generate the heapdump for. Use the table outlined in step 5 for a list of service names.
Example: service vmware-vcops status collector
Run the following command to see the available disk space of the /storage/db drive:
```
df -H /storage/db
```
Note: The size of heapdump depends on multiple factors, such as node size, or the service the heapdump is generated on. For example, an Analytics heapdump generated on an extra large sized node could be 110GB or more. Ensure there is enough available space left on the /storage/db drive. See Troubleshooting Storage Issues in Aria Operations (318408) if you need to free up space.
Run the following command to change to the /usr/lib/vmware-vcops/common/lib directory:
```
cd /usr/lib/vmware-vcops/common/lib
```
Run the following command to generate the heapdump:

VMware Aria Operations 9.1.x and later
```
java -Djavax.net.ssl.trustStore=NONE -cp "$VCOPS_BASE/common/lib/*" com.vmware.vcops.common.JMXCommandExecutor service-port-number dumpHeap /storage/db/vcops/heapdump/
```
Note: Replace service-port-number with the port number of the VMware VCF Operations service you need to generate a heapdump for.
Service Service Name Port Number
Analytics analytics 9004
Collector collector 9005
UI (tcServer) ui 9006
API (REST) api 9007

VMware Aria/VCF Operations 8.18.x and 9.0.x (FIPS OFF)
```
java -Djavax.net.ssl.trustStore=NONE -cp "alive_platform.jar:log4j-core-2.17.2.jar:vrops-logging-1.0-SNAPSHOT.jar:log4j-api-2.17.2.jar:vrops-jmx-metrics-1.0.jar:vrops-trustmanager-3.0-SNAPSHOT.jar:vcops-security-1.0-SNAPSHOT.jar:bc-fips-1.0.2.3.jar:bctls-fips-1.0.12.3.jar" com.vmware.vcops.common.JMXCommandExecutor service-port-number dumpHeap /storage/db/vcops/heapdump
```
Note: Replace service-port-number with the port number of the VMware Aria/VCF Operations service you need to generate a heapdump for. Please check above mentioned table.
Example: java -Djavax.net.ssl.trustStore=NONE -cp "alive_platform.jar:log4j-core-2.17.2.jar:vrops-logging-1.0-SNAPSHOT.jar:log4j-api-2.17.2.jar:vrops-jmx-metrics-1.0.jar:vrops-trustmanager-3.0-SNAPSHOT.jar:vcops-security-1.0-SNAPSHOT.jar:bc-fips-1.0.2.3.jar:bctls-fips-1.0.12.3.jar" com.vmware.vcops.common.JMXCommandExecutor 9004 dumpHeap /storage/db/vcops/heapdump

VMware Aria/VCF Operations 8.18.x and 9.0.x (FIPS ON)
```
java -Djavax.net.ssl.trustStore=NONE -cp "alive_platform.jar:log4j-core-2.17.2.jar:vrops-logging-1.0-SNAPSHOT.jar:log4j-api-2.17.2.jar:vrops-jmx-metrics-1.0.jar:vrops-trustmanager-3.0-SNAPSHOT.jar:vcops-security-1.0-SNAPSHOT.jar:bc-fips-1.0.2.3.jar:bctls-fips-1.0.12.3.jar:bcutil-fips-current.jar:bctls-fips-current.jar:bcpkix-fips-current.jar:bc-fips-current.jar" com.vmware.vcops.common.JMXCommandExecutor service-port-number dumpHeap /storage/db/vcops/heapdump
```
Note: Replace service-port-number with the port number of the VMware Aria/VCF Operations service you need to generate a heapdump for. Please check above mentioned table.
Once the generation is complete, the resulting file will be located in the /storage/db/vcops/heapdump directory. An SCP Utility, such as WinSCP, can be used to copy the heapdump so it can be uploaded to VMware by Broadcom Support.