vCenter Upgrade through SDDC Manager fails due to stale ESXi Host Entries in VMware Cloud Foundation DB

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

The vCenter upgrade process in VMware Cloud Foundation fails during the pre-check or execution phase. This issue is often caused by stale or orphaned ESXi host entries in the SDDC Manager database, which cause lifecycle management (LCM) operations to timeout. This article provides steps to identify and remove these stale records to allow the upgrade to proceed.

Symptoms: 

• The host is showing -- status on the SDDC Manager UI
  
• Upgrade starts and immediately fails with following error - System initiated upgrade cancel operation 

 

Environment

VMware Cloud Foundation 5.x, 

VMware Cloud Foundation 9.x

SDDC Manager

VMware ESX

Cause

Orphaned host records in the SDDC Manager platform database tables correspond to hosts that are unreachable, decommissioned, or in an ERROR state. When the LCM services attempt to orchestrate a vCenter upgrade, the presence of these stale records causes connection timeouts during certificate retrieval and status validation, effectively halting the upgrade workflow.

• This issue is typically caused by a mismatch between the physical state of the host and the records maintained in the SDDC Manager PostgreSQL database. This often occurs under the following conditions:
• The ESXi host has experienced an unrecoverable hardware failure.
• The host is in a persistent error state that prevents the SDDC Manager from communicating with the host agents.
• Orphaned references exist in the database that block the standard decommission workflow.
• Manual removal of host from vCenter prior to SDDC Manager decommissioning
• Hardware systemboard replacement
• Any process that might remove an ESXi host from the SDDC Manager environment, but where the SDDC Manager fails to fully remove the ESXi host object, or the SDDC Manager is unaware of the removal.

Resolution

Verification of Host Status:

1. Establish an SSH session to the SDDC Manager as vcf, then elevate to root. 
2. Check the status of the hosts in the database: 
   

   psql --host=localhost -U postgres -d platform -c "\x" -c "select id, hostname, status from host where status='ERROR' OR status='DEACTIVATING';"

3. Identify the specific ID for the host to be removed in the database:
   

   psql --host=localhost -U postgres -d platform -c "\x" -c "select id, hostname, status from host where hostname='<ESXi Host Name>';"

Log Collection Procedure:

1. Access the SDDC Manager via SSH using the vcf user.
2. Switch to the root user: su -
3. Change to the support utility directory: cd /opt/vmware/sddc-support
4. Execute the SOS utility to collect SDDC Manager logs: ./sos --sddc-manager-logs --zip
5. Collect the resulting .zip file located in the /var/log/vmware/vcf/sddc-support/ directory
6. Upload logs to the Support Request. See Uploading files to cases on the Broadcom Support Portal

Next Steps:

1. Engage Broadcom technical support. See Contact Support.
2. Provide the SOS log bundle generated in the previous steps.
3. Reference this article regarding the requirement for database cleanup.

Additional Information

ESXi host removal from VCF SDDC Manager fails due to database mismatch