ESXi host removal from VCF SDDC Manager fails due to database mismatch

Products

• VMware Cloud Foundation
• VMware SDDC Manager / VCF Installer

Issue/Introduction

In VMware Cloud Foundation (VCF) environments, an ESXi host may enter a state where it cannot be removed from the SDDC Manager inventory using the GUI. This condition prevents the decommissioning of hosts and may result in persistent error messages within the SDDC Manager dashboard.

Errors observed:

• IPs can't be resolved to hostnames This occurs when IPs are reused for new hosts while the old host is still in the "Uncommissioned" inventory.
• Failed to validate host spec - Insufficient free IPs and Failed to allocate IPs from networks with error.
• Unable to find task id when trying to remove node
• After a failed "Add Host" workflow, unable to Remove Host from SDDC UI.
• Fetch Remove ESX Host(s) Data from the SDDC Manager Inventory with an Internal Server error and Cause being: Cannot determine IP address for FQDN and Name or service not known when ESXi host was incorrectly removed directly from vCenter. 

Environment

• VMware Cloud Foundation 5.x
• VMware Cloud Foundation 9.x
• VMware Cloud Foundation running in a VxRail environment

Cause

This issue is typically caused by a mismatch between the physical state of the host and the records maintained in the SDDC Manager PostgreSQL database. This often occurs under the following conditions:

• The ESXi host has experienced an unrecoverable hardware failure.

• The host is in a persistent error state that prevents the SDDC Manager from communicating with the host agents.

• Orphaned references exist in the database that block the standard decommission workflow.

• Manual removal of host from vCenter prior to SDDC decommissioning
  
  
• Hardware systemboard replacement
  
  
• Any process that might remove an ESXi host from the SDDC manager environment, but where the SDDC Manager fails to fully remove the ESXi host object, or the SDDC Manager is unaware of the removal.

Resolution

Verification of Host Status:

1. Establish an SSH session to the SDDC Manager as vcf, then elevate to root. 
2. Connect to the database:
   

   psql --host=localhost -U postgres -d platform

3. Check the status of the hosts: 
   

   select id, hostname, status from host where status='ERROR' OR status='DEACTIVATING';

4. Identify the specific ID for the host to be removed:
   

   select id, hostname, status from host where hostname='<ESXi Host Name>';

Log Collection Procedure:

1. Access the SDDC Manager via SSH using the vcf user.
2. Switch to the root user: su -
3. Change to the support utility directory: cd /opt/vmware/sddc-support
4. Execute the SOS utility to collect SDDC Manager logs: ./sos --sddc-manager-logs --zip
5. The resulting .zip file located in the /var/log/vmware/vcf/sddc-support/ directory must be collected.
6. Upload logs to the Support Request. See Uploading files to cases on the Broadcom Support Portal

Next Steps:

1. Engage Broadcom technical support. See Contact Support.
2. Provide the SOS log bundle generated in the previous steps.
3. Reference this article regarding the requirement for database cleanup.