VMware Cloud Director UI is not reachable when generating multicell log bundle.

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• VMware Cloud director UI is not reachable while collecting a multicell log bundle using the following command

/opt/vmware/vcloud-director/bin/vmware-vcd-support --all -m

• From the commandline of the cell,we see the following

2022-07-19 12:57:15 | INFO | Output directory created successfully.

2022-07-19 12:57:15 | INFO | Collecting multi cell logs...

Collecting logs... [Time to timeout: 30 seconds]

No log bundles have been received from any of the cells or the tar did not successfully complete. Exiting.

2022-07-19 13:04:15 | ERROR | Failed to collect logs from multiple cells

2022-07-19 13:04:15 | INFO | Exporting DB...

2022-07-19 13:04:15 | INFO | Loading DB config properties.

INFO: Starting database export ...

Environment

VMware Cloud Director 10.x

Cause

This issue occurs due to Java heap dumps files (.hprof) on any/all of the cells. These files are created when the java process runs out of memory on the cells.

Resolution

To resolve the issue, the stale hprof can be deleted using the steps below:

1. SSH into each cell.
2. Check the “/opt/vmware/vcloud-director/logs” directory for java heap dump files (.hprof) on each cell.

cd /opt/vmware/vcloud-director/logs

      3. Remove the files having the “.hprof” extension.
 

[ /opt/vmware/vcloud-director/logs ]# rm java_xxxxx.hprof
 

      4. Perform a service restart on all the cells serially.

service vmware-vcd restart

      5. Check the Cloud director UI and verify it is working.

Workaround:
To workaround this issue, run the single-cell log collection command on each cell.

/opt/vmware/vcloud-director/bin/vmware-vcd-support

Additional Information

Impact/Risks:
Warning: Downtime is required as the cell services need to be restarted.