Failed to bootstrap VCF services runtime Reference Token: ####

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When deploying a new instance of VMware Cloud Foundation (VCF) 9.1, the process fails during the Deploy and configure VCF Management Platform phase (typically Step 4 of 14).
  
  
  
  
• The following error is observed in /var/log/vmware/vcf/domainmanager/domainmanager.log

YYYY-MM-DDTHH:MM:SS  DEBUG [vcf_dm,6a##13a67##526##93518a###733,9a76] [c.v.e.s.s.helper.SddcManagerTaskUtil,dm-exec-1070]  Response for Get task with id: 23####fd4-e##5-4##c-a##2-fe####fb##28 is {"id":"23####fd4-e##5-4##c-a##2-fe####fb##28","name":"Creating VSP Cluster <Example.com> in domain <domain-name>","type":"VSP_BOOTSTRAP","status":"In Progress","creationTimestamp":"YYYY-MM-DDTHH:MM:SS.Z","completionTimestamp":"YYYY-MM-DDTHH:MM:SS.Z","subTasks":[{"name":"Validate VSP cluster creation specification","description":"Validate VSP cluster creation specification","status":"SUCCESSFUL","creationTimestamp":"YYYY-MM-DDTHH:MM:SS.Z","completionTimestamp":"YYYY-MM-DDTHH:MM:SS.Z","errors":[]},{"name":"Bootstrap VCF Services Platform","description":"Bootstrap VCF Services Platform","status":"IN_PROGRESS","creationTimestamp":"YYYY-MM-DDTHH:MM:SS.Z","completionTimestamp":"YYYY-MM-DDTHH:MM:SS.Z","errors":[{"errorCode":"PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED_DETAILED","errorType":"ERROR","arguments":[""],"message":"VCF Management Services cluster bootstrap failed with the following errors: .","causes":[],"referenceToken":"####"}]},{"name":"Configure VCF Services Platform","description":"Configure VCF Services Platform","status":"PENDING","creationTimestamp":"YYYY-MM-DDTHH:MM:SS.Z","completionTimestamp":"YYYY-MM-DDTHH:MM:SS.Z","errors":[]},{"name":"Add VM Name Prefix to NSX firewall exclusion list","description":"Add VM Name Prefix to NSX firewall exclusion list","status":"PENDING","creationTimestamp":"YYYY-MM-DDTHH:MM:SS.Z","completionTimestamp":"YYYY-MM-DDTHH:MM:SS.Z","errors":[]}],"errors":[],"resolutionStatus":"UNRESOLVED","isCancellable":false,"isRetryable":true}
YYYY-MM-DDTHH:MM:SS  ERROR [vcf_dm,6a##13a67##526##93518a###733,9a76] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-1070]  [G####AL] PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED Failed to bootstrap VCF services runtime
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to bootstrap VCF services runtime
        at com.vmware.evo.sddc.sddcmanager.MonitorVspClusterBootstrapOnSddcManagerAction.execute(MonitorVspClusterBootstrapOnSddcManagerAction.java:79)
Caused by: java.util.concurrent.TimeoutException: Task did not complete within 600 seconds.
        at com.vmware.evo.sddc.sddcmanager.helper.SddcManagerTaskUtil.monitorTask(SddcManagerTaskUtil.java:97)

Environment

VCF 9.1

Cause

The failure is caused by the presence of uppercase or mixed upper with lower case letters in the Management Domain Name or Fully Qualified Domain Name (FQDN) provided for VCF components (such as Instance Services, Fleet, or SDDC Manager) during deployment.

The VCF Services Platform (VSP) utilizes internal Kubernetes-based orchestration. Per RFC 1123 standards, Kubernetes resource names and metadata must be strictly lowercase. If an FQDN containing uppercase characters (e.g - Example.com) is used, the underlying services fail to initialize, resulting in a bootstrap timeout.

 

Note: VCF 9.1 may now permit all upper case for FQDN and other names for VCF management objects. However, it currently does not support mixed case (upper with lower case) for the object names.

Resolution

To resolve this issue, ensure all FQDNs used in the VCF deployment are configured in strictly lowercase.

1. Validate DNS Records: Ensure all Forward (A) and Reverse (PTR) DNS records for VCF components are configured in lowercase on your DNS server
2. Correct Deployment Parameters: If using JSON, ensure every hostname and FQDN is entered in lowercase (e.g., change InstanceName to instancename).
3. Restart the Deployment:
   
   • Perform a cleanup of the failed deployment following the steps for this scenario outlined in KB Roll back and and clean up after failed convergence of vSphere 8.x to VCF 9.1.
   • Restart the VCF 9.1 installation from scratch using the corrected, strictly lowercase FQDNs.

Additional Information

VCF Components FQDNs and IP addresses


Use a JSON Specification File to Deploy VMware Cloud Foundation or vSphere Foundation