SDDC Manager Unresponsive: vcf-domain-manager Crashes with Java Heap Space OutOfMemoryError

Products

• VMware Cloud Foundation
• VMware SDDC Manager

Issue/Introduction

• This article addresses a issue where the VMware Cloud Foundation (VCF) SDDC Manager appliance becomes unresponsive or unavailable.
• The primary symptom is the domainmanager service crashing, particularly during periods of high task volume or inventory operations. 
• The underlying error identified in the /var/log/vmware/vcf/domainmanager/domainmanager.log is a Java Heap Space OutOfMemoryError.

  ERROR [vcf_dm,0000000000000000,0000] [o.a.c.c.C.[.[.[.[dispatcherServlet],http-nio-127.0.0.1-7200-exec-7]  Servlet.service() for servlet [dispatcherServlet] in context with path [/domainmanager] threw exception [Handler dispatch failed: java.lang.OutOfMemoryError: Java heap space] with root cause
  java.lang.OutOfMemoryError: Java heap space
  
  ERROR [vcf_dm,69#######################fa,6767] [o.s.a.i.SimpleAsyncUncaughtExceptionHandler,dm-exec-16]  Unexpected exception occurred invoking async method: void com.vmware.vcf.configreconciler.service.ResourceCacheService.refreshLocalInventoryWithConfigApplicability()
  java.lang.OutOfMemoryError: Java heap space

Environment

VCF 5.x

Cause

• The root cause of the service unavailability is resource exhaustion specifically related to the Java Virtual Machine (JVM) memory allocation for the vcf-domain-manager service.
• The default maximum heap size of 512MB is insufficient to handle the memory demands generated by a high volume of tasks, complex inventory data, or concurrent operations, especially in larger or busier VCF environments.
• This insufficient allocation results in the service exceeding its allocated memory limit, triggering the crash.
• The cause is explicitly confirmed by the following log entries, which directly show the service ran out of allocated memory.

  java.lang.OutOfMemoryError: Java heap space

Resolution

A fix has been implemented in SDDC Manager 5.2.x and later to alleviate this issue by greatly reducing the Domain Manager service memory consumption.

However, in very large environments with high task volumes, complex inventory data, or frequent concurrent operations, the issue may persist.

To resolve this, increase the Java Heap Memory allocation for the Domain Manager service.
Raising the limit from the default 512MB to 2GB quadruples the available resources, ensuring the service can handle high workloads without crashing.

Follow these steps to apply the necessary memory increase:

1. Take snapshot of SDDC Manager VM
2. SSH to the SDDC Manager appliance using the vcf user and elevate to root using su
3. Stop the domainmanager service

   systemctl stop domainmanager

4. Edit the service configuration file located at /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf. Use a text editor like vi

   vi /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf

5. Locate the line containing the maximum heap size setting (it will include -Xmx512m) and change it to -Xmx2G:

   Original: -Xmx512m
   
   New: -Xmx2G

6. Save the file and exit the text editor.

   Esc > :wq!

7. Start the domainmanager service

   systemctl start domainmanager

The domainmanager service will now run with a maximum Java Heap Memory of 2GB, resolving the OutOfMemoryError and restoring service stability.