NSX-T UI shows some transport nodes tunnels down after V2T migration using the migrator coordinator

Products

• VMware NSX

Issue/Introduction

Symptoms:

• You are running a version of NSX-T Data Center pre 3.2.1 and 3.1.3.8.
• You have recently completed a V2T migration using the built in migration tool.
• Navigating on the NSX-T UI to System > Fabric > Nodes, some host transport nodes (TN) are in "degraded" status and have some tunnels in "down" status.

Environment

VMware NSX-T Data Center

Cause

During a V2T migration, some hosts may fail to be migrated, this may lead to stale VTEPs for the transport nodes in NSX-T. The tunnels to these stale VTEPs will be down for existing hosts.

Resolution

This issue is fixed starting NSX-t Data Center 3.2.1

Workaround:

• SSH as root into the NSX-T Manager appliance where the migration service was run.
• Run the below command and record the output:

grep vtep-table /var/log/migration-coordinator/v2t/cm.log | cut -d ' ' -f 10 | sort | uniq

• The output of the previous command should have each line similar to the below one, which has the TN ID between "transport-nodes/" and "/vtep-table":

...

url=http://localhost:7440/nsxapi/api/v1/transport-nodes/<tn-id>/vtep-table,

...

• For each TN ID in the output of the above command, search for it in NSX-T UI using elastic search, if no result is found, record the TN ID and use it in the next command to remove the TN.
• For each TN ID which does not exist in the previous step, run the below API calls in the indicated order against the NSX-T manager where the migration service ran:

POST https://<nsx-mngr-ip>/api/v1/transport-nodes/<tn-id>/vtep-table

{"node_id": "<tn_id>"}

 

DELETE https://<nsx-mngr-ip>/api/v1/transport-nodes/vtep-labels/<tn_id>

Note: Please make sure a current up to data backup is in place before proceeding with workaround steps.