SDDC Manager GUI stuck on loading screen due to "java.lang.OutOfMemoryError: Java heap space" events in Operations Manager service

Products

• VMware SDDC Manager

Issue/Introduction

• After logging into the SDDC Manager UI, all or some items on the SDDC Manager dashboard remain on the loading screen and are not displayed
• SOS Health Check Command does not complete
• SOS Log acquisition takes a long time
• "java.lang.OutOfMemoryError: Java heap space" is output in /var/log/vmware/vcf/operationsmanager/operationsmanager.log
  

  yyyy-mm-ddThh:mm:ss.xxx+0000 ERROR [vcf_om,0000000000000000,0000] [o.s.s.s.TaskUtils$LoggingErrorHandler,om-scheduler-2] Unexpected error occurred in scheduled task
  java.lang.OutOfMemoryError: Java heap space
  yyyy-mm-ddThh:mm:ss.xxx+0000 ERROR [vcf_om,xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,1d24] [o.s.a.i.SimpleAsyncUncaughtExceptionHandler,om-exec-28] Unexpected exception occurred invoking async method: public void com.vmware.vcf.telemetry.vac.service.impl.VcfMetricsCollectorServiceImpl.triggerMetricsCollection(java.util.List)
  java.lang.OutOfMemoryError: Java heap space
  yyyy-mm-ddThh:mm:ss.xxx+0000 ERROR [vcf_om,xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,c805] [c.v.v.p.v.u.CredentialsValidationTaskExecutor,om-exec-2] Error occurred in validation tasks: java.lang.OutOfMemoryError: Java heap space
  java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
          at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
          at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)

 

Environment

• VCF 5.x
• VCF 9.x

Cause

• The root cause of the service unavailability is resource exhaustion, specifically related to the Java Virtual Machine (JVM) memory allocation for the Operations Manager service (vcf-operations-manager).
• The default maximum heap size of 2GB is insufficient to handle the memory demands generated by a high volume of tasks, complex inventory data, or concurrent operations especially in larger or highly active VMware Cloud Foundation (VCF) environments. This insufficient allocation results in the service exceeding its memory limit, ultimately triggering a crash.
• This is explicitly confirmed by the log entries, which directly indicate that the service ran out of allocated memory: 

  java.lang.OutOfMemoryError: Java heap space

Resolution

A fix has been implemented in SDDC Manager 9.1 to alleviate this issue by greatly reducing the Operations Manager service memory consumption. However, in very large environments with high task volumes, complex inventory data, or frequent concurrent operations, the issue may persist.

To resolve this, increase the Java Heap Memory allocation for the Operations Manager service. Raising the limit from the default 2GB to 4GB quadruples the available resources, ensuring the service can handle high workloads without crashing.

Follow these steps to apply the necessary memory increase:

1. Take a snapshot of the SDDC Manager VM.

2. SSH to the SDDC Manager appliance using the vcf user and elevate to root using su.

3. Stop the operationsmanager service by running the following command:

   systemctl stop operationsmanager
   

4. Edit the service configuration file located at /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf. Use a text editor like vi:

   vi /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf
   

5. Locate the line containing the maximum heap size setting (it will include -Xmx512m) and change it to -Xmx2G:

   • Original: -Xmx2G

   • New: -Xmx4G

6. Save the file and exit the text editor (if using vi, press Esc, type :wq!, and press Enter).

7. Start the operationsmanager service:

   systemctl start operationsmanager
   

The operationsmanager service will now run with a maximum Java Heap Memory of 4GB, resolving the OutOfMemoryError and restoring service stability.

Additional Information

If you encounter java.lang.OutOfMemoryError: Java heap space in the Domain Manager service, refer to: SDDC Manager Unresponsive: vcf-domain-manager Crashes with Java Heap Space OutOfMemoryError