NSX cluster NSX configuration Failed with errors about Management tag and Failed to find all vmks

Products

• VMware NSX
• VMware Cloud Foundation

Issue/Introduction

• Hosts may have recently been removed from the vCenter and rebuilt/reimaged.
• NSX UI>System>Fabric>Hosts>Clusters shows hosts in Validation Error or Failed state in the NSX Configuration column.
• Expanding those errors shows the following for Validation Error:
  
  8828: Unable to fetch vmkernel nic enabled with Management tag for the Transport node TransportNode/<UUID> Overlay networking using Vmkernel Adaptor will not be configured.
  
  
• Expanding the Failed errors show an error like:
  
  Host configuration: invalid fetchDvpgTeaming reply: [## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##| Error = {'fault: "Failed to find all vmks [''] in dvs ## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##. Please check the vmks and try again."}|teaming=|active=|standby=|]; VTEP apply operation failed: [host_ip ###.###.###.### not found for dvs ## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##];
  
  
• A VDS was configured after the VCF installation was complete and the vmkernels were migrated to the new VDS.
• The Transport Node Profile shows the initial VDS under the VDS Switch configuration item (Go to NSX UI>System>Fabric>Hosts>Transport Node Profile> expand Host Switch and then expand the Transport Node Profile configuration in the Host Switch configuration box that shows up)

Environment

VCF 9.x
NSX 9.x

Cause

The Transport Node Profile (TNP) assigned to the cluster includes is configured to "Use VMKernel Adapter" which uses the Management VMK for the TEPs.  The VDS assigned to the TNP is not the VDS where the Management VMKernel port is assigned.

Resolution

In vCenter, move the Management VMK to the VDS in use by the TNP.  See Migrate ESXi vmkernel/VM networking from one DVS to another DVS, or a VSS, to another VDS for more information. 

Additional Information

For more information on planning and designing a VCF 9.1 deployment please see:

• VCF Network Detailed Design
• Deployment, Convergence, and Upgrade