Operations Manager becomes unresponsive with OutOfMemoryError in SDDC Manager

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• The SDDC Manager UI becomes extremely slow or unresponsive after running for an extended period.
• Users may encounter "Gateway Time-out" or "Failed to fetch inventory details" errors when navigating the UI.
• Secondary workflows, such as backup failures, may occur.
• Restarting the operationsmanager service temporarily resolves the issue.

Environment

VMware SDDC Manager 5.x
VMware SDDC Manager 9.x

Cause

This issue occurs due to a memory leak in the Operations Manager service. A background metrics-refresh job, which runs every 5 minutes per vCenter, continuously creates new HTTP client objects (Reactor Netty) to query the NSX Trust Management endpoint. These objects are not properly released and accumulate in the JVM heap, eventually exhausting the default 2 GB memory limit and causing an OutOfMemoryError.

Resolution

To mitigate this issue and delay the OOM crash, increase the heap size of the Operations Manager service.

Note: Before applying this workaround, ensure that the SDDC Manager VM has sufficient RAM allocated to accommodate the increased heap size.

1. SSH into the SDDC Manager appliance using the vcf user.

2. Switch to the root user:

   su -
   

3. Stop the operationsmanager service by running the following command:

   systemctl stop operationsmanager

4. Create a backup of the Operations Manager configuration file:

   cp /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf.bak
   

5. Open the configuration file in a text editor (e.g., vim):

   vim /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf
   

6. Locate the line containing the maximum heap size setting (it will include -Xmx2G) and change it to -Xmx3G or -Xmx4G.

   • Original: -Xmx2G

   • New: -Xmx4G

7. Save the file and exit the editor  (if using vi, press Esc, type :wq!, and press Enter).

8. Restart the Operations Manager service to apply the new settings and clear any leaked memory:

   systemctl start operationsmanager
   

As a temporary workaround, periodically restarting the operationsmanager service (Step 8) will also reclaim all leaked memory until a permanent patch is applied.