Error: Unable to get uplinks for VDS when adding ESXi host to cluster in SDDC Manager

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Adding an ESXi host to the cluster in SDDC Manager fails with error :  "Unable to get uplinks for VDS" 
  
  
• Due to a configuration mismatch, there are scenarios where re-imaged ESXi hosts do not appear under "Add hosts," even when the hosts are in a commissioned and unassigned state.
  
  
• Add host to cluster keeps loading as shown below :

• The newly added host is not exactly identical to the existing host configuration (Number of physical NICs)
  
  
• On the SDDC manager logs: /var/log/vmware/vcf/domainmanager/domainmanager.log

<timestamp> ERROR [vcf_dm,68e3c9e68924cdc91145700bf2824e4d,dd81] [c.v.e.s.c.s.ClusterNetworkConfigFetcher,dm-exec-19]  Unable to get uplinks for VDS <VDS NAME>
<timestamp>  ERROR [vcf_dm,68e3c9e68924cdc91145700bf2824e4d,dd81] [c.v.e.s.c.s.ClusterNetworkConfigFetcher,dm-exec-19]  Exception occurred when fetching details from vCenter
java.lang.NullPointerException: Cannot read the array length because "<local6>" is null
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.getUplinksForDvs(VcManagerBase.java:12403)
        at com.vmware.evo.sddc.common.services.ClusterNetworkConfigFetcher.generateVdsConfiguration(ClusterNetworkConfigFetcher.java:307)
        at com.vmware.evo.sddc.common.services.ClusterNetworkConfigFetcher.lambda$generateVdsConfigurations$4(ClusterNetworkConfigFetcher.java:238)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at com.vmware.evo.sddc.common.services.ClusterNetworkConfigFetcher.generateVdsConfigurations(ClusterNetworkConfigFetcher.java:233)
        at com.vmware.evo.sddc.common.services.ClusterNetworkConfigFetcher.fetchNetworkDetailsFromVCenter(ClusterNetworkConfigFetcher.java:419)
        at com.vmware.evo.sddc.common.services.ClusterNetworkConfigQueryHelper.lambda$getVCenterNsxNetworkConfig$1(ClusterNetworkConfigQueryHelper.java:282)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
<timestamp>  DEBUG [vcf_dm,68e3c9e68924cdc91145700bf2824e4d,dd81] [c.v.e.s.c.s.ClusterNetworkConfigFetcher,dm-exec-19]  Returning vds configuration: {"name":"VDSNAME","portGroups":[{"name":"<PortGroupNAME>","transportType":"MANAGEMENT","activeUplinks":["uplink1","uplink2"],"standByUplinks":[],"policy":"loadbalance_loadbased","vlanId":##},{"name":"PortGroupNAME","transportType":"VMOTION","activeUplinks":["uplink1","uplink2"],"standByUplinks":[],"policy":"loadbalance_loadbased","vlanId":##},{"name":"PortGroupNAME","transportType":"VSAN","activeUplinks":["uplink1","uplink2"],"standByUplinks":[],"policy":"loadbalance_loadbased","vlanId":##}],"inventoryMismatchInfo":[],"uplinks":[]}
<timestamp>  DEBUG [vcf_dm,68e3c9e68924cdc91145700bf2824e4d,dd81] [c.v.e.s.c.s.ClusterNetworkConfigFetcher,dm-exec-19]  Preparing host configuration list
<timestamp>  DEBUG [vcf_dm,68e3c9e68924cdc91145700bf2824e4d,dd81] [c.v.e.s.c.s.ClusterNetworkConfigFetcher,dm-exec-19]  Getting vmnicToUplinkMap for host HOSTNAME
<timestamp>  DEBUG [vcf_dm,68e3c9e68924cdc91145700bf2824e4d,dd81] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-19]  Finding host with fqdn HOSTNAME
<timestamp>  DEBUG [vcf_dm,9023feec338a4854,5a69] [c.v.e.s.c.s.a.i.InventoryServiceAdapterImpl,http-nio-127.0.0.1-7200-exec-8]  Fetching Cluster with ID b5a#####-####-####-####-#####0d8 from inventory
<timestamp>  DEBUG [vcf_dm,68e3c9e68924cdc91145700bf2824e4d,dd81] [c.v.e.s.c.util.RetriableCallable,dm-exec-19]  Starting retriable operation 'Waiting for fetch DVS ports' with 10 retries.
<timestamp>  ERROR [vcf_dm,68e3c9e68924cdc91145700bf2824e4d,dd81] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-19]  Failed to get Distributed Virtual Ports for Distributed Virtual Switch 

Environment

SDDC Manager

VMware Cloud Foundation

Cause

Due to a data mismatch between SDDC Manager and the vCenter database, the ESXi host cannot be added.The issue is specifically triggers by out-of-band (OOB) changes to VDS configurations common across the cluster when done on the vCenter server.

Resolution

 Contact Broadcom Support  if you are encountering the same issue to validate the configuration.