The VCF Management Services cluster bootstrap failed with the following error: "Bootstrap failed during stage."

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• During the VCF Management Services deployment phase of VCF 9.1, the following error may appear in the VCF Installer UI:
  
  Error: VCF Management Services cluster bootstrap failed with the following errors: "Bootstrap failed during stage 'INFRA0002 - Waiting for VCF services runtime cluster nodes to become ready': Command or operation failed with exit code 1. Please retry the operation or contact support if the issue persists. [VCFMS-BOOTSTRAP-999]".
  
  
• The following error is observed in /var/log/vmware/vcf/domainmanager/domainmanager.log:
  
  ERROR [vcf_dm,######,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-###]  [####] PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED Failed to bootstrap VCF services runtime
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to bootstrap VCF services runtime
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:##)
        at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:##)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:##)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:####)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:###)
        at java.base/java.lang.Thread.run(Thread.java:####)
Caused by: com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to bootstrap VCF services runtime
  

Environment

VMware Cloud Foundation (VCF) 9.1

Cause

• The bootstrap logic provisions the cluster and waits for status.phase to be "Provisioned". However, the "Provisioned" phase is not guaranteed to be persisted in the KubernetesCluster CR.
• When the kubernetescluster controller sets Cluster.Status.Phase to "Provisioned", this phase is set in memory. During that very same reconciliation pass, if ControlPlaneInitialized and InfrastructureProvisioned are already true, the field is immediately overwritten to "Ready" and persisted in the CR. As a result, the "Provisioned" phase may never be persisted, or may be so transient that the bootstrap code can completely miss.

Resolution

This is a known issue, and VCF Engineering is working on a fix.

To work around this, contact Broadcom Support for assistance.