Unable to remove ESXi host from VCF SDDC Manager due to database mismatch | Steps to remove ESXi Hosts from the SDDC Manager DB

Products

• VMware Cloud Foundation
• VMware SDDC Manager / VCF Installer

Issue/Introduction

In VMware Cloud Foundation (VCF) environments, an ESXi host may enter a state where it cannot be removed from the SDDC Manager inventory using the GUI. This condition prevents the decommissioning of hosts and may result in persistent error messages within the SDDC Manager dashboard.

Errors observed:

• IPs can't be resolved to hostnames This occurs when IPs are reused for new hosts while the old host is still in the "Uncommissioned" inventory.
• Failed to validate host spec - Insufficient free IPs and Failed to allocate IPs from networks with error.
• Unable to find task id when trying to remove node
• After a failed "Add Host" workflow, unable to Remove Host from SDDC UI.

Environment

• VMware Cloud Foundation 5.x
• VMware Cloud Foundation 9.x

Cause

This issue is typically caused by a mismatch between the physical state of the host and the records maintained in the SDDC Manager PostgreSQL database. This often occurs under the following conditions:

• The ESXi host has experienced an unrecoverable hardware failure.

• The host is in a persistent error state that prevents the SDDC Manager from communicating with the host agents.

• Orphaned references exist in the database that block the standard decommission workflow.

Resolution

Verification of Host Status:

1. Establish an SSH session to the SDDC Manager as vcf, then elevate to root. 
2. Connect to the database:
   

   psql --host=localhost -U postgres -d platform

3. Check the status of the hosts: 
   

   select id, hostname, status from host where status='ERROR' OR status='DEACTIVATING';

4. Identify the specific ID for the host to be removed:
   

   select id, hostname, status from host where hostname='<ESXi Host Name>';

Log Collection Procedure:

1. Access the SDDC Manager via SSH using the vcf user.
2. Switch to the root user: su -
3. Change to the support utility directory: cd /opt/vmware/sddc-support
4. Execute the SOS utility to collect SDDC Manager logs: ./sos --sddc-manager-logs --zip
5. The resulting .zip file located in the /var/log/vmware/vcf/sddc-support/ directory must be collected.
6. Upload logs to the Support Request. See Uploading files to cases on the Broadcom Support Portal

Next Steps:

1. Engage Broadcom technical support. See Contact Support.
2. Provide the SOS log bundle generated in the previous steps.
3. Reference this article regarding the requirement for database cleanup.