SDDC Manager GUI stuck on loading screen due to "java.lang.OutOfMemoryError: Java heap space" events in Operations Manager service

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• SDDC Manager UI is slow or stuck on the loading screen.
• SDDC Manager appliance shows 100% CPU usage on a Java process.
• SOS Health Check Command does not complete
• Log files show java.lang.OutOfMemoryError: Java heap space events for the operationsmanager service.
  
  /var/log/vmware/vcf/operationsmanager/operationsmanager.log
  

  YYYY-MM-DDThh:mm:ss ERROR [vcf_om,#####] [o.s.a.i.SimpleAsyncUncaughtExceptionHandler,om-exec-28] Unexpected exception occurred invoking async method: public void com.vmware.vcf.telemetry.vac.service.impl.VcfMetricsCollectorServiceImpl.triggerMetricsCollection(java.util.List)
  java.lang.OutOfMemoryError: Java heap space
  ERROR [vcf_om,#####] [c.v.v.p.v.u.CredentialsValidationTaskExecutor,om-exec-2] Error occurred in validation tasks: java.lang.OutOfMemoryError: Java heap space
  java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
          at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
          at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)

Environment

• VCF 5.x
• VCF 9.x

Cause

• The root cause of the service unavailability is resource exhaustion, specifically related to the Java Virtual Machine (JVM) memory allocation for the Operations Manager service.
• The default maximum heap size of 2GB is insufficient to handle the memory demands generated by a high volume of tasks, complex inventory data, or concurrent operations especially in larger or highly active VMware Cloud Foundation (VCF) environments. This insufficient allocation results in the service exceeding its memory limit, ultimately triggering a crash.

Resolution

A fix has been implemented in SDDC Manager 9.1 to alleviate this issue by greatly reducing the Operations Manager service memory consumption. However, in very large environments with high task volumes, complex inventory data, or frequent concurrent operations, the issue may persist.

For version prior to 5.2.1, use the following workaround to increase the heap size:

1. Take a snapshot of the SDDC Manager VM.

2. SSH to the SDDC Manager appliance using the vcf user and elevate to root using su.

3. Stop the operationsmanager service by running the following command:

   systemctl stop operationsmanager
   

4. Edit the service configuration file located at /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf. Use a text editor like vi:

   vi /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf
   

5. Locate the line containing the maximum heap size setting (it will include -Xmx2G) and change it to -Xmx4G

   • Original: -Xmx2G

   • New: -Xmx4G

6. Save the file and exit the text editor (if using vi, press Esc, type :wq!, and press Enter).

7. Start the operationsmanager service:

   systemctl start operationsmanager
   

The operationsmanager service will now run with a maximum Java Heap Memory of 4GB, resolving the OutOfMemoryError and restoring service stability.

Additional Information

If you encounter java.lang.OutOfMemoryError: Java heap space in the Domain Manager service, refer to: SDDC Manager Unresponsive: vcf-domain-manager Crashes with Java Heap Space OutOfMemoryError