ESXi host removal from VCF SDDC Manager fails due to database mismatch

Products

• VMware Cloud Foundation
• VMware SDDC Manager / VCF Installer

Issue/Introduction

In VMware Cloud Foundation (VCF) environments, an ESXi host may enter a state where it cannot be removed from the SDDC Manager inventory using the GUI. This condition prevents the decommissioning of hosts and may result in persistent error messages within the SDDC Manager dashboard.

Errors observed:

• IPs cannot be resolved to hostnames - This occurs when IPs are reused for new hosts, while the old host is still in the "Uncommissioned" inventory.
• Failed to validate host spec - Insufficient free IPs and failed to allocate IPs from networks with error.
• Unable to find task id when trying to remove node
• Unable to Remove Host from the SDDC Manager UI following a failed 'Add Host' workflow.
• The host is stuck in an "Activating" status on the SDDC Manager UI
• Unable to decommission an unassigned host in SDDC Manager.
• Failed to decommission the host. The attempted functionality HOST_DECOMMISSION is not supported in the current system state. Function requires a context id, but was unable to calculate one.
• Internal server error: Fetch Remove ESX Host(s) Data from the SDDC Manager Inventory cause being: Cannot determine IP address for FQDN

Environment

• VMware Cloud Foundation 5.x
• VMware Cloud Foundation 9.x

Cause

This issue is typically caused by a mismatch between the physical state of the host and the records maintained in the SDDC Manager PostgreSQL database. This often occurs under the following conditions:

• The ESXi host has experienced an unrecoverable hardware failure.

• The host is in a persistent error state that prevents the SDDC Manager from communicating with the host agents.

• Orphaned references exist in the database that block the standard decommission workflow.

• Manual removal of host from vCenter prior to SDDC Manager decommissioning
  
  
• Hardware system-board replacement
  
  
• Any process that might remove an ESXi host from the SDDC Manager environment, but where the SDDC Manager fails to fully remove the ESXi host object, or the SDDC Manager is unaware of the removal.

Resolution

Make sure the ESXi host(s) cannot be removed through normal processes: Remove a Host from a vSphere Cluster in a Workload Domain

Verification of Host Status:

1. Establish an SSH session to the SDDC Manager as vcf then elevate to root. 
   
   
2. Check the status of the hosts in the database: 
   
   psql --host=localhost -U postgres -d platform -c "\x" -c "select id, hostname, status from host where status='ERROR' OR status='DEACTIVATING';"
   
   
3. Identify the specific ID for the host to be removed in the database:
   
   psql --host=localhost -U postgres -d platform -c "\x" -c "select id, hostname, status from host where hostname='<ESXi Host Name>';"

Log Collection Procedure:

1. Access the SDDC Manager via SSH using the vcf user.
2. Switch to the root user: su -
3. Change to the support utility directory: cd /opt/vmware/sddc-support
4. Execute the SOS utility to collect SDDC Manager logs: ./sos --sddc-manager-logs --zip
5. Collect the resulting .zip file located in the /var/log/vmware/vcf/sddc-support/ directory
6. Upload logs to the Support Request. See Uploading files to cases on the Broadcom Support Portal

Next Steps:

1. Engage Broadcom technical support. See Contact Support.
2. Provide the SOS log bundle generated in the previous steps.
3. Reference this article regarding the requirement for database cleanup.