SDDC manager upgrade from 5.2.2 to 5.2.4 fails at Validating Services.

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

SDDC manager upgrade from 5.2.2 to 5.2.4 fails at Validating SDDC Services.

Environment

SDDC Manager 5.2.x

Cause

The Domain Manager service fails and enters a crash loop due to a Java OutOfMemory (OOM) exception triggered within the ConfigreconcilerServiceStartupCleanUp process.

/var/log/vmware/vcf/domainmanager/ domainmanager.log file reveals that the application was running out of heap memory:

06:53:40 Starting DomainManagerApplication ... PID 1431
06:55:06 Started DomainManagerApplication in 87.898 seconds
06:55:06 ConfigreconcilerServiceStartupCleanUp: Cleaning up in-progress config reconciliation workflows...
06:55:14 ERROR BlobFsmContextStore: Exception reading the orchestrator's contract parameters ... (repeats per stale execution)
06:58:38 ERROR SpringApplication: Application run failed
          Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
              at com.google.gson.stream.JsonReader...
              at OrchestratorAssemblers.toProcessingTask
              at ConfigreconcilerServiceStartupCleanUp.cleanUpInProgressDriftReconciliationWorkflows:71
06:58:53 [new process instance] Initialize the resource cache done.

Resolution

Increase the heap size on /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf file for the Domain Manager service from 512 MB to 2048 MB.

Note: Take a snapshot of SDDC manager. SSH to SDDC Manager using vcf and switch to root.

1. First, create a backup of the current configuration file as a best practice:
cp /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf.bak

2. Run below command to perform the specific replacement:
sed -i 's/-Xmx512m/-Xmx2048m/g' /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf

This will update the 'Maximum heap size' from 512 MB to 2048 MB.

3. Restart the Domain Manager service: systemctl restart domainmanager

4. Try upgrading SDDC Manager from 5.2.2 to 5.2.4.