Error: "Bootstrap failed during stage 'INFRA0002 - Waiting for VCF services runtime cluster nodes to become ready" during the VCF Management Services deployment phase of VCF 9.1

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• During the VCF Management Services deployment phase of VCF 9.1, the following error may appear in the VCF Installer UI: 

  Error: VCF Management Services cluster bootstrap failed with the following errors: "Bootstrap failed during stage 'INFRA0002 - Waiting for VCF services runtime cluster nodes to become ready': Command or operation failed with exit code 1. Please retry the operation or contact support if the issue persists. [VCFMS-BOOTSTRAP-999]".

  
  
• The following error is observed in /var/log/vmware/vcf/domainmanager/domainmanager.log: 

  ERROR [vcf_dm,######,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-###]  [####] PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED Failed to bootstrap VCF services runtime
  com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to bootstrap VCF services runtime
          at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:##)
          at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:##)
          at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:##)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:####)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:###)
          at java.base/java.lang.Thread.run(Thread.java:####)
  Caused by: com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to bootstrap VCF services runtime

  
  

Environment

• VMware Cloud Foundation (VCF) 9.1

Cause

• The bootstrap logic provisions the cluster and waits for status.phase to be "Provisioned". However, the "Provisioned" phase is not guaranteed to be persisted in the KubernetesCluster CR.
• When the kubernetescluster controller sets Cluster.Status.Phase to "Provisioned", this phase is set in memory.
• During that very same reconciliation pass, if ControlPlaneInitialized and InfrastructureProvisioned are already true, the field is immediately overwritten to "Ready" and persisted in the CR.
• As a result, the "Provisioned" phase may never be persisted, or may be so transient that the bootstrap code can completely miss.

Resolution

• This is a known issue, and VCF Engineering is working on a fix.
• To work around this, contact Broadcom Support for assistance.