SDDC Manager API for cluster deletion may leave stale transport nodes on NSX

Products

• VMware NSX

Issue/Introduction

• Host Cluster is being decommissioned.
• SDDC API is being used to delete the Cluster as per below API documentation
  
  Delete a Cluster
  
  
• No manual removal of NSX was done on NSX before running the above API.
• After cluster deletion, upon inspection on NSX UI, it may be seen some stale transport nodes are created. 
• These TNs may appear under the stand-alone tab at NSX > System > Fabric > Hosts.

Environment

• VMware NSX-T 3.x
• VMware NSX 4.x
• VMware SDDC Manager 4.4.x
• VMware SDDC Manager 5.x

Cause

By design, SDDC Manager cluster Delete API does not ensure complete cleanup of NSX entries. If the TNs have some references on NSX, it can result in stale TNs. The cluster deletion workflow is a best-effort model. It will attempt the cleanup but if NSX dependencies are encountered, the decommission moves forward.

To summarize, NSX cleanup is best effort in all versions of SDDC, and in some sites, it may leave stale Transport Nodes.

Resolution

It is a valid behavior. 

Additional Information

Broadcom prepared 2 scripts which can help with cleaning up large number of stale hosts on NSX. 

If you see a large number of stale TNs please open a new support case with Broadcom Support and refer to this KB article : KB 406768 : Script for removing large number of stale transport nodes from NSX

Note: These scripts will only work with NSX 3.2.x and higher versions.