SDDC workflow for commissioning Hosts, fails on NSX TEP IP assignment, because of an NSX PROFILE_MISMATCH.

Products

• VMware SDDC Manager

Issue/Introduction

Commissioning ESXi Hosts in SDDC 5.X fails with the following error 

Description    Validate IP Address Availability for Edge Overlay (TEP) IP Assignment
Progress Messages    Failed to validate the provided spec with error [Failed to validate IP assignment scheme of the cluster <Cluster_Name> using existing host in the cluster <ESXi_Hostname>]
Error
Message: Failed to validate the provided spec with error [Failed to validate IP assignment scheme of the cluster <Cluster_Name> using existing host in the cluster <ESXi_Hostname>]
Remediation Message:
Reference Token: <Token_ID>
Cause:

Environment

SDDC 5.X

Cause

This is usually a configuration issue.
Per the NSX UI, the view of the hosts (System > Fabric > Hosts) is showing the TransportNodes [TN/ TNs] in Mismatch, to their Cluster's assigned TransportNodeProfile [TNP]

# less -i domainmanager.log 

Date && Time 202*-**-**THH:MM:SS:MS DEBUG [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.h.NsxtCommonOperations,dm-exec-18]  Finished waiting for TN status to change to IN-PROGRESS or SUCCESS, currentState is [HostNodeStatus (com.vmware.nsx.model.host_node_status) => {
Date && Time 202*-**-**THH:MM:SS:MS INFO  [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.p.a.TransportNodeCollectionResolver,dm-exec-18]  Waiting for TNC status to change from in-progress state
Date && Time 202*-**-**THH:MM:SS:MS DEBUG [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.h.NsxtCommonOperations,dm-exec-18]  Waiting 600000 ms for TNC status change from IN-PROGRESS
[...]
Date && Time 202*-**-**THH:MM:SS:MS DEBUG [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.h.NsxtCommonOperations,dm-exec-18]  Waiting 600000 ms for TNC status change from IN-PROGRESS
Date && Time 202*-**-**THH:MM:SS:MS DEBUG [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.p.a.TransportNodeCollectionResolver,dm-exec-18]  Current state of cluster preparation IN_PROGRESS
[...]
Date && Time 202*-**-**THH:MM:SS:MS DEBUG [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.p.a.TransportNodeCollectionResolver,dm-exec-18]  Current state of cluster preparation PROFILE_MISMATCH
Date && Time 202*-**-**THH:MM:SS:MS INFO  [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.p.a.TransportNodeCollectionResolver,dm-exec-18]  Waiting for TNC status to switch to success or failure
[...]
Date && Time 202*-**-**THH:MM:SS:MS DEBUG [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.p.a.TransportNodeCollectionResolver,dm-exec-18]  Current state of cluster preparation PROFILE_MISMATCH
Date && Time 202*-**-**THH:MM:SS:MS INFO  [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.p.a.TransportNodeCollectionResolver,dm-exec-18]  Waiting for TNC status to switch to success or failure
[...]
Date && Time 202*-**-**THH:MM:SS:MS DEBUG [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.p.a.TransportNodeCollectionResolver,dm-exec-18]  Current state of cluster preparation PROFILE_MISMATCH
Date && Time 202*-**-**THH:MM:SS:MS DEBUG [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.h.NsxtCommonOperations,dm-exec-18]  Waiting 600000 ms for TNC status to change to SUCCESS or FAILURE
Date && Time 202*-**-**THH:MM:SS:MS DEBUG [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.p.a.TransportNodeCollectionResolver,dm-exec-18]  Current state of cluster preparation PROFILE_MISMATCH
Date && Time 202*-**-**THH:MM:SS:MS DEBUG [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.h.NsxtCommonOperations,dm-exec-18]  Timeout waiting for TNC status to change to SUCCESS or FAILURE
Date && Time 202*-**-**THH:MM:SS:MS DEBUG [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.p.a.TransportNodeCollectionResolver,dm-exec-18]  Exiting due to final result of cluster progress state PROFILE_MISMATCH
Date && Time 202*-**-**THH:MM:SS:MS INFO  [vcf_dm,6883b4c65c2c2dace9fcc4d93fd62cb0,3e43] [c.v.v.c.f.p.n.p.a.TransportNodeCollectionResolver,dm-exec-18]  Tracking cluster level failures and trigger resolve if cluster TNC status is failed

CreateTransportNodeCollectionAction fails because TransportNodeCollection State is PROFILE_MISMATCH state. 
TransportNodeCollection PROFILE_MISMATCH status, means that the cluster has one/more Host(s) with overriding configuration. 
That means one/more Host(s)'s TN configuration, differs from the TN config defined in TransportNodeProfile.
This will results in an "Add Host(s)" workflow failure, as this is expected.

FYI the process can take some time, as the workflow will have multiple reattempts.
Please wait out the process to end.

Resolution

Match the Clusters' Transport Node profiles' to the actual hosts switch configuration for NSX, so that they are all compliant.
PROFILE_MISMATCH state is an expected answer, if there is a misconfiguration, or a non compliant state.