Method to collect heap dump on provisioning issue investigation

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Provisioning may failed due to index query exceeded issue.

Which object(network, storage and etc) cause the overload should be confirmed in advanced .

Such information can be fetched through JAVA heap dump that fetched at the same time when issue occurred  

Environment

VMware Aria Automation 8.x

Resolution

When issue observed, execute below command immediately in Aria Automation command line: 

for pp in `kubectl -n prelude get pods | grep provisioning-service-app | awk '{print $1}'`; do NOW=$(date +"%Y-%d-%m-%H-%M-%S") ; kubectl -n prelude exec -it $pp -- jcmd ProvisioningSpringApplication GC.heap_dump /var/log/$pp-$NOW.hprof && sleep 1; done 

The command will produce a heap dump per provisioning pod, the file can be located in the log mapped directory:

/services-logs/prelude/provisioning-service-app/file-logs 

Look for file name with patterns:

provisioning-service-app-<pod-id>-<timestamp>.hprof

or

provisioning-service-app-<pod-id>-<timestamp>.hprof.xz

The latter has to be unzipped by xz -d utility program. If both present they should be the same so one of them is sufficient for the analysis.