Unable to add ESXI host as a host transport node.

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

To provide manual steps to remove the partial records for an ESXI host transport node with API call first and if API fails, then from the embedded Corfu database within NSX-T.

When attempting to prepare a ESXI host as a host transport node, the process fails with an indication that the NSX Manager cannot find the record for the host, referencing old metadata elements for that host, i.e. old IP different external uuid, "Node <uuid> with same ip <ip-address> already exists", etc.

Or

you receive the following error while trying to configure NSX on ESXi host
"Error: Operation cannot be performed as the object(s) has been updated by another user while you were trying. Refresh and try again. (Error code: 500071)"

 

Environment

VMware NSX-T Data Center
VMware NSX

Cause

This behavior is caused by a partial record for a host transport node that includes metadata that references the host that is trying to prepare as a host transport node. The TN creation workflow can halt prematurely while retrieving the thumbprint for the host certificate. If the certificate is not in the correct format ( a common causes extra spaces or formatting characters in the certificate body) this will cause a parsing failure and leave a partial record for that host.

Resolution

 

Before going to the Corfu database it may be possible to force a cleanup by REST API:

DELETE  /api/v1/transport-nodes/<transport-node-id>?force=true&unprepare_host=false

You can validate the deletion by running the below API:

 GET api/v1/transport-nodes/<TN UUID>/state

1. Before running The above API call, run through the below steps.
2. "Remove NSX" from host in question in the NSX-T UI (or remove host from cluster if Transport Node Profile is present on cluster).
3. Ensure NSX-T VIBs have been removed from the host.
   1. esxcli software vib list | grep nsx
   2. If VIBs are still present, but NSX-T UI shows host is no longer configured:
      1. nsxcli
      2. del nsx
4. Search NSX-T UI for the host name of the ESXi host in question and if a UUID shows up, copy it out for the above API call.
5. Execute above API call with UUID obtained from step 4.
   1. If you don't have access to Postman, run the following cURL command on any one of the 3 NSX-T Managers from root mode.
   2. curl -l -k -u 'admin' -H 'Content-Type:application/json' -X DELETE  "https://localhost/api/v1/transport-nodes/<transport-node-UUID-goes-here>?force=true&unprepare_host=false"
6. From admin cli mode on ALL 3 NSX-T Managers, run "start search resync all"
7. Log out of NSX-T UI and log back in (reindexing may take a few minutes)
8. Check NSX-T search for host name in question to ensure no UUID shows up.
9. Re-install NSX on host in question.

 

Workaround:

This issue can be worked around by recreating the ESXI host with a full set of new metadata, assuming that the underlying certificate issue is no longer present on the "new" host.