Configuration Status of Decommissioned ESXi Hosts Stuck in “Deactivating” in SDDC Manager

Products

• VMware SDDC Manager

Issue/Introduction

• After removing and permanently decommissioning multiple ESXi hosts from the environment, the SDDC Manager UI continues to display these hosts with their configuration status stuck in “Deactivating.”
  
• The spinning “Deactivating” state does not change even after rebooting the SDDC manager.
• The SDDC Manager UI does not provide an option to manually remove the hosts that remain in the “Deactivating” state.
• As per the /var/log/vmware/vcf/domainmanager/domainmanager.log:
  Caused by: com. vmware. vim. vmomi. client. exception. ConnectionException: https://<ESXi-FQDN>443/sdk invocation failed with "org. apache.http.conn.ConnectTimeoutException: Connect to <ESXi-FQDN>:443 [ESXi-FQDN/##.##.##.##] failed: Connect timed out"

Environment

• VMware Cloud Foundation 4.x
• VMware Cloud Foundation 5.x

Cause

This issue occurs when the ESXi hosts are not in a healthy state from the perspective of the VMware Cloud Foundation (VCF) SDDC Manager. It is typically caused by:

• A mismatch or inconsistency between the SDDC Manager database and the actual ESXi host inventory.
• An ESXi host in an error or unrecoverable state, such as a hardware failure, which prevents proper communication with the SDDC Manager.
• Hosts that have been physically removed or decommissioned but are still referenced in the SDDC Manager database.

Resolution

Resolving this issue requires assistance from Broadcom Technical Support. We recommend contacting Broadcom Support for further investigation. Collecting and providing a log bundle from the SDDC Manager will help expedite the troubleshooting process.

Steps to Generate the SDDC Manager Log Bundle:

• Log in to the SDDC Manager via SSH using the vcf user account.
• Switch to the root user and run the following command to generate the SDDC Manager logs:
  • /opt/vmware/sddc-support/sos --sddc-manager-logs --zip
    NOTE: Unless specified the default log location is /var/log/vmware/vcf/sddc-support/
• Transfer the generated log bundle from /var/log/vmware/vcf/sddc-support/ to your desktop using your preferred file transfer tool (for example, WinSCP).
• Upload the log bundle to the Broadcom Support Portal.