VCF 9.0.x vCenter import fails on precheck validation with: Something went wrong. Please retry or contact the service provider and provide the reference token.

Products

• VMware vCenter Server

Issue/Introduction

When running the add Workload Domain and selecting vCenter import from VCF Operations It failed with error:  Something went wrong. Please retry or contact the service provider and provide the reference token.

Error seen in /var/log/vmware/vcf/domainmanager/domainmanager.log 

ERROR [vcf_dm,0000000000000000,0000] [o.a.c.c.C.[.[.[.[dispatcherServlet],http-nio-127.0.0.1-7200-exec-9] Servlet.service( ) for servlet [dispatcherServlet] in context with path [/domainmanager] threw exception [Handler dispatch failed: java. lang. OutOfMemoryError: Java heap space] with root cause java. lang. OutOfMemory Error: Java heap space

or: 

ERROR [vcf_dm,0000000000000000,0000] [c.v.e.s.e.h.VcfFallbackErrorController,http-nio-127.0.0.1-7200-exec-7] [SOPI80] - Error attributes:
{timestamp=Date Month 00 00:00:00 UTC YEAR, exception=java. lang. OutOfMemory Error, trace=java. lang. OutOfMemory Error: GC overhead limit exceeded

 

Error seen in /var/log/vmware/vcf/operationsmanager/operationsmanager.log

ERROR [vcf_om,0000000000000000,0000] [o.a.c.c.C.[.[.[.[dispatcherServlet],http-nio-127.0.0.1-7200-exec-9] Servlet.service( )
[Handler dispatch failed: java. lang. OutOfMemory Error: Java heap space] with root cause
java. lang. OutOfMemoryError: Java heap space

 

Environment

VCF 9.0.x

Cause

The logs explicitly show `java.lang.OutOfMemoryError` occurring immediately after `Fetching NSX Clusters data from inventory`. This indicates that the object heap reached its maximum capacity while attempting to deserialize or process inventory objects. The progression from `Java heap space` to `GC overhead limit exceeded` confirms the JVM was spending more than 98% of its time performing garbage collection to recover negligible amounts of memory.

 In very large environments with high task volumes, complex inventory data, or frequent concurrent operations, the issue may persist.

Resolution

To resolve this, increase the Java Heap Memory allocation for the Domain Manager service.
Raising the limit from the default  to 8GB quadruples the available resources, ensuring the service can handle high workloads without crashing.

Follow these steps to apply the necessary memory increase:

1. Take snapshot of SDDC Manager VM
2. SSH to the SDDC Manager appliance using the vcf user and elevate to root using su
3. Stop the domainmanager service

   systemctl stop domainmanager

4. Edit the service configuration file located at /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf. Use a text editor like vi

   vi /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf

5. Locate the line containing the maximum heap size setting (it will include -Xmx512m) and change it to -Xmx8G:

   Original: -Xmx512m
   
   New: -Xmx8G

6. Save the file and exit the text editor.

   Esc > :wq!

7. Start the domainmanager service

   systemctl start domainmanager

 

For the Operations Manager: 

1. Stop the operationsmanager service

   systemctl stop operationsmanager

2. Edit the service configuration file located at /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf. Use a text editor like vi

   vi /etc/vmware/vcf/operationsmanager/vcf-operations-manager.conf

3. Locate the line containing the maximum heap size setting (it will include -Xmx2G) and change it to -Xmx8G:

   Original: -Xmx2G
   
   New: -Xmx8G

4. Save the file and exit the text editor.

   Esc > :wq!

5. Start the operationsmanager service

   systemctl start domainmanager

 

Retry the precheck validation. 

NOTE: After the Import please revert back the heap size setting for both services back to the default values.

Additional Information

Increasing the heap size to 8GB provides sufficient headroom for the JVM to store the inventory metadata and perform complex validation logic without triggering aggressive garbage collection or exhausting the address space. This allows the `InventoryServiceAdapterImpl` to complete the data fetch and proceed with the workload domain import validation.