VCF 4.x/5.x VxRail Bring up fails at task "Populate inventory of SDDC Manager"

Products

• VMware Cloud Foundation

Issue/Introduction

To troubleshoot VCF on VxRail bring up errors similar to the one described in sample log snippets

Symptoms:
VCF 4.4 and 5.x Cloud builder bring up fails with following errors in vcf-bringup-debug.log
 

2022-04-01T15:59:26.542+0000 [bringup,XXXXXXXXXXXXXXXX,XXXX] ERROR [c.v.e.s.o.model.error.ErrorFactory,pool-4-thread-5] [IN89PL] NSX_SDK_FAILED_TO_FIND_VM Virtual Machine : [VM Name] was not found in NSX-T response. Please deploy the VM first.

com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Virtual Machine : [VM Name] was not found in NSX-T response. Please deploy the VM first.

    at com.vmware.vcf.common.fsm.plugins.nsxt.action.UpdateFirewallExclusionListAction.preValidate(UpdateFirewallExclusionListAction.java:71)

    at com.vmware.vcf.common.fsm.plugins.nsxt.action.UpdateFirewallExclusionListAction.preValidate(UpdateFirewallExclusionListAction.java:34)

at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$0(FsmActionState.java:18)

at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)

at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:159)

at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:144)

at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:400)

at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:545)

at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:124)

at sun.reflect.GeneratedMethodAccessor381.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

    at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)

    at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)

    at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:67)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

    at java.lang.Thread.run(Thread.java:748)

 

Environment

VMware Cloud Foundation 4.x, 5.x

Cause

The VxRail bring up configuration excel sheet contained 4 ESXi hosts however there is a 5th ESXi host present in the vCenter server.
The NSX-T node in question in the log snip was migrated by DRS to the 5th ESXi host and SDDC does not have that ESXi host in its inventory.

Resolution

Ensure that the vCenter server contains only ESXi hosts that are part of the bring up configuration excel sheet.
If any other hosts are added to the vCenter - migrate any management VMs running in it and remove the host from vCenter inventory before running the bring up script. 

After the bring up is completed - any extra ESXi hosts can be added using SDDC manager standard process.

Workaround:
Migrate any running VMs from the extra ESXi host and place the host in maintenance mode, then retry the workflow.

Additional Information

Impact/Risks:
VCF on VxRail bring up will fail in this situation