'Create NSX transport node collection' task fails on the SDDC Manager with error 'Failed to realize transport node collection'

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• The task 'Install and Configure Overlay over Management Network Backed NSX for Domain <domain_name>' fails at subtask 'Create NSX transport node collection' on the SDDC Manager with the following error:
  
  Message: Error in configuring VLAN backed transport node collections
Remediation Message:
Reference Token: ######
Cause: com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException:
Error in configuring VLAN backed transport node collections for clusters <Cluster_name> Error in configuring VLAN backed transport node collections for clusters <Cluster_name> Failed to realize transport node collection


• On the vCenter Server that manages the cluster, the 'Apply NSX Solution' task for the cluster fails with the following error:
  
  'A general system error occurred: Solution(s) apply failed because the vSphere Lifecycle image contains either a new ESXi version or a new addons version or new components. Please proceed to the vSphere Client Lifecycle Manager to update ESXi or addons or components along with the solution(s) 'com.vmware.nsxt''


• On the vCenter Server, under Cluster -> Updates -> Images, the image configured for the cluster is 'out of compliance' with the ESXi hosts on the Cluster.
  
  
• The below alert is also observed under Cluster -> Updates -> Images:
  
  'The cluster needs remediation to finish enabling these Solutions: VMware NSX-T 9.0.2.0'

Environment

VMware vCenter 9.0.x

VMware Cloud Foundation 9.0.x

Cause

The failure occurs during the NSX Transport Node creation due to a compliance drift between the ESXi hosts and the vSphere Lifecycle Manager (vLCM) cluster image.

Resolution

Remediate the ESXi hosts against the Cluster Image by following the steps below:

1. On the vCenter Server, navigate to the target cluster -> Updates 
2. Verify the cluster compliance status. Refer Check the Compliance of Multiple Clusters and Standalone Hosts Against an Image
3. Remediate the hosts in the cluster to make them compliant against the vLCM image. Refer Remediate a Group of Clusters or Standalone Hosts
4. Re-initiate the 'Install and Configure Overlay over Management Network Backed NSX' task from SDDC Manager.