When reconfiguring NSX Manager node in NSX-T Data Center, Deployment fails with the "Error code: 21016"
Article ID: 317159
Updated On:
Products
Issue/Introduction
- When attempting to deploy a new NSX Manager node after deleting an old NSX manager node, the process fails with the error "
The specified hostname is already in use by an existing auto-deployed cluster node VM. Duplicates of hostnames are not allowed. (Error code: 21016)". - In the log file var/log/proton/nsxapi.log we see the following entries:
2023-04-14T14:16:34.447Z INFO http-nio-127.0.0.1-7440-exec-437 NsxBaseRestController 4264 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/cluster/nodes/deployments caused by exception com.vmware.nsx.management.common.exceptions.InvalidArgumentException: {"moduleName":"CLUSTER_NODE_VM_DEPLOYMENT","errorCode":21016,"errorMessage":"The specified hostname '<nsx-manager-fqdn>' is already in use by an existing auto-deployed cluster node VM. Duplicates of hostnames are not allowed."}
- You've run detach for the old NSX Manager node from the NSX Manager cluster through CLI and it has been run successfully. For example:
nsx-manager> detach node <node-UUID>Node has been detached. Detached node must be deleted permanently.
- Running the below GET API call to display information relating to cluster node VM deployments, shows the deleted NSX manager node UUID in the response.
GET https://<NSX-IP>/api/v1/cluster/nodes/deploymentsEnvironment
VMware NSX 4.x
VMware NSX-T Data Center 3.x
Cause
"detach node <node-UUID>", under certain conditions, can lead to a stale entry in the NSX Manager database.Resolution
This issue is resolved in VMware NSX-T Data Center 3.2.2.1 and VMware NSX 4.0.1. Available by following instructions found at Downloading Broadcom products and software.
Workaround
Make sure you have an up to date backup in place before proceeding.
Please run the following API calls to remove the records of the NSX manager node that are no longer in use:
- To identify the NSX-T manager which needs to be cleaned up, run the following API Call:
GET https://<NSX-IP>/api/v1/cluster/nodes/deployments..."vm_id": "xxxxxxx-d106-48f5-ad0a-xxxxxxxxxxx","roles": ["MANAGER","CONTROLLER"..."hostname": "NSX-MANAGER-02",
... - Take note of the UUID for the NSX-T manager which no longer exists, in above example it is xxxxxxx-d106-48f5-ad0a-xxxxxxxxxxx. Then replace <deleted-vm-id> in below POST API call with the UUID.
- First we run a POST to Delete the node:
POST https://<manager_ip>/api/v1/cluster/nodes/deployments/<deleted-vm-id>?action=delete - If the above POST API fails, then run the below POST API which will force the Delete operation:
POST https://<manager_ip>/api/v1/cluster/nodes/deployments/<deleted-vm-id>?action=delete&force_delete=true - If the issue still exists after executing the above API calls, please open a support request with Broadcom Support and refer to this KB article.
For more information, see How to Submit a Support Request.
NOTE: This API call will show all managers that were autodeployed using the NSX-T GUI. Manually deployed managers (via vCenter using OVA) will not appear in this output. For instance, the original manager used to build an NSX-T environment will not appear, revealing only 2 managers in the output of this API. If the original manager is ever detached from the cluster and redeployed via the NSX-T GUI, this API call will show all three managers.
Feedback
