VMware NSX Manager node stuck at "1% Deleting..." in the NSX GUI.

Products

• VMware NSX

Issue/Introduction

• A VMWare NSX Manager node delete task was initiated in the NSX GUI.
• After some time, the Manager node remains present in the NSX GUI and is stuck at "1% Deleting...". 
• Rebooting the remaining NSX Managers does not clear this state. 
• The NSX Manager VM is not present in the relevant Compute Manager.
  

Environment

VMware NSX

Cause

Under certain conditions, an NSX Manager node deletion task can lead to a stale entry in the VMware NSX Manager Corfu database.

Resolution

Workaround

• Make sure you have an up to date backup in place before proceeding.
• Please run the following API calls to remove the records of the NSX manager node that is no longer in use:
  • To identify the NSX-T manager which needs to be cleaned up, run the following API Call:

GET https://<NSX-IP>/api/v1/cluster/nodes/deployments
...
"vm_id": "########-d106-48f5-ad0a-##########",
"roles": [
"MANAGER",
"CONTROLLER"
...
"hostname": "NSX-MANAGER-##",
...

• Take note of the UUID in the "vm_id" value for the VMware NSX Manager which no longer exists, in above example it is ######-d106-48f5-ad0a-##########. Then replace <deleted-vm-id> in below POST API call with the UUID.
• First we run a POST to Delete the node:

POST https://<manager_ip>/api/v1/cluster/nodes/deployments/<deleted-vm-id>?action=delete

• If the above POST API fails, then run the below POST API which will force the Delete operation:

POST https://<manager_ip>/api/v1/cluster/nodes/deployments/<deleted-vm-id>?action=delete&force_delete=true

• If the issue still exists after executing the above API calls, please open a support request with Broadcom Support and refer to this KB article.

For more information, see How to Submit a Support Request.

NOTE: This API call will show all managers that were auto-deployed using the VMware NSX GUI.  Manually deployed managers (via vCenter using OVA) will not appear in this output.  For instance, the original manager used to build a VMWare NSX environment will not appear, revealing only 2 managers in the output of this API.  If the original manager is ever detached from the cluster and redeployed via the VMware NSX GUI, this API call will show all three Managers.