VCF 5.2.x upgrade failures at the SDDC Manager Deployment Drift stage caused by Java heap memory reset and Service Validation Failed errors.
search cancel

VCF 5.2.x upgrade failures at the SDDC Manager Deployment Drift stage caused by Java heap memory reset and Service Validation Failed errors.

book

Article ID: 448596

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

A VMware Cloud Foundation (VCF) upgrade fails specifically at the SDDC Manager Deployment Drift stage. This failure is characterized by the error "Service Validation Failed" and commonly occurs during the transition from VCF 5.2.2 to 5.2.4.

  • The VCF upgrade process halts at the SDDC Manager Deployment Drift stage.
  • The error message Service Validation Failed is displayed in the SDDC Manager UI.
  • The domainmanager service enters an ERROR state and fails to start correctly after the appliance reboot.
  • domainmanager.log contains the following error: java.lang.OutOfMemoryError: GC overhead limit exceeded
  • LCM logs (lcm.log and lcm-debug.log) report errorCode:1 during the config drift reconciliation phase.

Environment

VCF 5.2.x

SDDC Manager 5.2.x

Cause

During the upgrade to VCF 5.2.x, the RPM package for the domainmanager service does not contain the %config(noreplace) directive. Consequently, the RPM upgrade process overwrites custom configurations in /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf, resetting the Java heap memory to the default 512MB.

When the appliance reboots and initiates the "Deployment Drift" reconciliation, the default memory is insufficient for the cleanup tasks, leading to an OutOfMemory (OOM) crash-loop that prevents service validation from completing.

Resolution

A permanent fix involving a configuration save/restore mechanism is implemented in VCF 9.1 and higher. For VCF 5.2.x, follow the workaround below.

Important: You must apply this workaround immediately after the SDDC Manager appliance reboots during the patching process, but before the Drift configuration update phase begins.

  1. Initiate the SDDC Manager upgrade.
  2. Once the SDDC Manager appliance has completed its reboot, SSH to the appliance as root.
  3. Check the current heap size configuration: grep Xmx /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf
  4. If the value has been reset to -Xmx512m, edit the file to increase it to 2GB: sed -i 's/-Xmx512m/-Xmx2G/g' /etc/vmware/vcf/domainmanager/vcf-domain-manager.conf
  5. Restart the domainmanager service to apply the change: systemctl restart domainmanager
  6. Verify the service status: systemctl status domainmanager
  7. Monitor the SDDC Manager UI; the "Deployment Drift" validation should now proceed and complete successfully.

 

Powered by Wolken Software