Host Deletion stuck at 20%

Products

• VMware NSX

Issue/Introduction

• Unable to decommission NSX Transport Node hosts. 
• Hosts were already removed from vCenter. 
• The recommended way of decommissioning NSX Hosts was not followed, Best Practice for Decommissioning an NSX-Prepared ESXi Host. 
• In NSX > System > Fabric > Hosts : Hosts appear under the stand-alone tab.
• Trying to "Delete NSX" in the Web UI with the "Force" box checked hangs the de-commission process at 20%
• Trying deleting via REST API : DELETE https://<mgr-ip>/api/v1/transport-nodes/<Transport-Node-UUID>?force=true&unprepare_host=false
  returns a 200 OK, and the process also hangs at 20%
• The Manager log /var/log/proton/localhost_access_log.txt shows the GET API fail with a 404:
  

  ####.####.#### - "GET /nsxapi/api/v1/transport-nodes/e3###################/status HTTP/1.1" 404 181 52 51 +

• The Manager log /var/log/nsxapi.log shows the DELETE operation is hung at 20 percent 
  

  ####.####.####Z INFO L2HostConfigTaskExecutor5 DeploymentProgressServiceImpl FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the DeploymentProgress from: DeploymentProgress [ deploymentType=HOST_TN, operationType=DELETE, progress=8, stateDescription=deployment.progress.tn.delete.waiting_for_host_config_reply, removeNsxFlag=false] to DeploymentProgress [ id=e36##########, deploymentType=HOST_TN, operationType=DELETE, progress=20, stateDescription=deployment.progress.tn.delete.waiting_for_host_config_reply, removeNsxFlag=false]

Environment

• VMware NSX 9.0.2.0

Cause

• When an ESXi host is removed directly from vCenter without first removing NSX, it can result in entries for that host to remain in the NSX database.
• The correct procedure to remove NSX from a single host: Uninstall NSX Data Center from a Managed Host in a vSphere Cluster

Note: This is also referred to as a "Stale Host" and will prevent adding back a refreshed or new host using the same name.

Resolution

• Broadcom has prepared two scripts which will detect and cleanup stale TransportNode entries, which is especially helpful for large data centers having 40 or more hosts per cluster.
• If you believe these scripts are required for cleaning up stale TransportNode entries , please open a support case with Broadcom Support and refer to this KB article.

Note: For more information, see Creating and managing Broadcom support cases.

Additional Information

.