N-VDS to VDS migration fails with "APPLY_TOPOLOGY_FAILED" for a compute cluster that has one or more member nodes with overridden config that doesn't match the TNP applied to the cluster.

Products

• VMware NSX

Issue/Introduction

N-VDS to VDS migration fails with "APPLY_TOPOLOGY_FAILED".

The compute cluster, the migration failed for has one or more Transport Nodes that have a different config from the cluster's Transport Node Profile.

The APPLY_TOPOLOGY portion of the migration fails due to a conflict transaction exception.

2024-01-23T14:43:27.610Z ERROR NvdsUpgradeTaskExecutor1 NvdsUpgradeReadinessCheckServiceImpl 72329 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP250020" level="ERROR" subcomp="manager"] Failed to apply ExtraConfigHswProfile on TransportNode <UUID>, exception Error in Updated TransportNode. Exception: TX ABORT | Snapshot Time = Token(epoch=244, sequence=4573781384) | Failed Transaction ID = <UUID> | Offending Address = 4573781488 | Conflict Key = 5BE633B2773FD5B7 | Conflict Stream = 35dc0219-####-####-####-########022 | Cause = CONFLICT | Time = 1886 ms : com.vmware.nsx.management.switching.common.exceptions.SwitchingException: Error in Updated TransportNode. Exception: TX ABORT | Snapshot Time = Token(epoch=244, sequence=4573781384) | Failed Transaction ID = <UUID> | Offending Address = 4573781488 | Conflict Key = 5BE633B2773FD5B7 | Conflict Stream = 35dc0219-####-####-####-########022 | Cause = CONFLICT | Time = 1886 ms

Environment

VMware NSX-T Data Center
VMware NSX

In the environment where this issue was encountered, we could see that the configuration for a Transport Node is overridden and doesn't match with the TNP applied to the cluster. For this TN, uplink-2 is mapped to vmnicX but in TNP it is mapped to vmnicY. This resulted in a migration failure during the first attempt. Re-attempting migration should be successful as this is only a transaction exception.

One TN failed to update ExtraConfigHswProfile when applying VDS topology:

2024-01-23T14:43:26.762Z ERROR providerTaskExecutor-10 TransportNodeServiceImpl 72329 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP9508" level="ERROR" subcomp="manager"] Physical NIC vmnic3 not found on host <UUID>.
2024-01-23T14:43:26.762Z WARN providerTaskExecutor-10 PolicyResourceChangeNotificationManager 72329 POLICY [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Failure received invoking listener TransportNodeListener for change UPDATING on resource /infra/sites/default/enforcement-points/default/host-transport-nodes/<UUID>
2024-01-23T14:43:26.762Z WARN providerTaskExecutor-10 TransportNodeCollectionServiceImpl 72329 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Error occurred during HostTn update for discovered node <UUID>

Cause

In this issue, we can see that the APPLY_TOPOLOGY failed due to a conflict transaction exception. This happened due to an overridden host in the cluster. 

Resolution

1. Create a new VC cluster.
2. Create a new TNP with overridden TN's configuration and apply the TNP to a new cluster.
3. Move the overridden host to the new cluster.
4. Execute NVDS2CVDS.
5. Move the host back to the initial cluster.
6. Delete the new VC cluster.