Unable to add Multiple hosts in the SDDC Manager

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

Unable to add large number of hosts into the SDDC Manager

Logging visible in domainmanager.log:

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-07-27T23:23:58.181+0000 ERROR [vcf_dm,0000000000000000,0000] [o.a.c.c.C.[.[.[.[dispatcherServlet],http-nio-127.0.0.1-7200-exec-3]  Servlet.service() for servlet [dispatcherServlet] in context with path [/domainmanager] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded] with root cause
java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-07-27T23:23:58.184+0000 DEBUG [vcf_dm,b9ba6f65a9ce7bc3,7cd7] [c.v.v.s.t.DynamicTrustManager,dm-exec-5]  Checking validity of certificate chain C=US, CN=xxxxxxx.xxxx.xxx
2021-07-27T23:23:58.185+0000 DEBUG [vcf_dm,b9ba6f65a9ce7bc3,7cd7] [c.v.v.s.t.DynamicTrustManager,dm-exec-5]  Certificate chain C=US, CN=xxxxxxx.xxxx.xxx is valid
2021-07-27T23:23:58.189+0000 ERROR [vcf_dm,8a09e48e1debc5b0,b6dc] [c.v.e.s.e.h.VcfFallbackErrorController,http-nio-127.0.0.1-7200-exec-3]  [NC64DU] - Error attributes: {timestamp=Tue Jul 27 23:23:58 UTC 2021, status=500, error=Internal Server Error, exception=java.lang.OutOfMemoryError, trace=java.lang.OutOfMemoryError: GC overhead limit exceeded
, message=GC overhead limit exceeded, path=/domainmanager/workflows/e6ef7d34-xxxx-xxxx-xxxx-xxxxxxxxxxxx}
2021-07-27T23:23:58.192+0000 WARN  [vcf_dm,b9ba6f65a9ce7bc3,7cd7] [c.v.v.v.c.h.i.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase,dm-exec-5]  Shutting down the connection monitor.
2021-07-27T23:23:58.192+0000 WARN  [vcf_dm,0000000000000000,0000]

Cause

Large number of objects being added to the SDDC Manager causes the Out of Memory state of domainmanager service.

Resolution

Workaround:

1. Take a snapshot of SDDC Manager VM from vCenter UI

2. SSH to the SDDC Manager VM (as root)

3. Take a backup of the vcf-domain-manager.conf file

cp -rf /etc/vmware/vcf/domainmanager/conf/vcf-domain-manager.conf /root/vcf-domain-manager.conf.BAK

4. Check the contents of the vcf-domain-manager.conf file for the heap size set:

cat /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf
The current heap size set in the file should be 512 Mb, i.e. it should have the below line set:
JAVA_OPTS=-Xmx512m

We need to change this and set the heap size as 2 GB (below steps explain the same)

5. vi /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf

6. Edit the file to set JAVA_OPTS flag as below:
JAVA_OPTS=-Xmx2G
Save and quit (wq!)

7. Restart domainmanager
systemctl restart domainmanager

8. Retry the failed workflow