Manual removal of hosts from VC without removing NSX from them at first have resulted in stale host entries on the NSX appliance

Products

• VMware NSX

Issue/Introduction

• Hosts and vCenter are being decommissioned.
• ESXi hosts were already manually removed on vCenter.
• vCenter also removed and uninstalled. 
• Hosts may or may not have been removed from SDDC Manager.
• The best practice of removal of NSX from hosts or transport nodes before decommissioning hosts as per KB 403287 was not followed.
• On NSX UI at System > Fabric > Hosts, these hosts show as 'Host Disconnected' or as 'Unknown' status.
• A clean up process is required for these stale hosts/clusters in the NSX appliance.

Environment

• VMware NSX-T 3.x
• VMware NSX 4.x

 

Cause

If ESXi hosts are being decommissioned and these ESXis use NSX (i.e., they are configured as Transport Nodes on NSX), then the best practice or the recommended way is to remove NSX from these ESXis at first using the NSX UI, before removing the ESXis from vCenter. Please check Best Practice for Decommissioning an NSX-Prepared ESXi Host to Avoid Stale Entries

If this best practice is not followed and ESXis are removed from vCenter directly before removing NSX, then this will lead to stale and orphaned hosts at NSX UI. NSX Managers will not be aware that these hosts have been decommissioned. NSX Mangers will conclude that they have lost communication with the ESXi hosts and will show 'Host Disconnected' or "Unknown" at the NSX configuration/ Status tab.

 

Resolution

If hosts were already removed from vCenter and that resulted in stale hosts in 'Host Disconnected' state on NSX , then proceed with below steps to clean up these stale and orphaned hosts:

1) If a Transport Node Profile is in use, detach the Transport Node Profile.

2) Select 'Remove NSX' at the host cluster level or at individual host level. Use the “Force Delete” checkbox as shown below. Please proceed with this step only if you are confirmed that the hosts have been decommissioned.

 

Picture caption: Using the 'Force Delete' option.

3) The above steps will change the status of the hosts from 'Host Disconnected' status to 'Not configured' status.

4) Remove the vCenter as Compute Manager on NSX following below step. Please follow the below step only if you are confirmed there are no hosts managed by this vCenter using NSX, and this vCenter has been already decommissioned and uninstalled.

Go to System > Fabric > Compute Managers 

Select the compute manager and click on DELETE.

5) Refresh the NSX UI and wait for a couple of minutes.

After the above steps, you will no longer see the stale and decommissioned hosts on NSX UI at System > Fabric > Hosts . 

Additional Information

Please check Best Practice for Decommissioning an NSX-Prepared ESXi Host to Avoid Stale Entries to prevent this problem from happening.