VCF 9.1 instance failing at 97% with the message "Failed to bootstrap VCF services runtime"

Products

VMware SDDC Manager / VCF Installer VMware Cloud Foundation

Issue/Introduction

Issue specific to VCF 9.x deployment which is failing at 97% (ideally at 90% and above) with the following message -

Bootstrap VCF Services Platform Failed
Description Bootstrap VCF Services Platform
Progress Messages Failed to bootstrap VCF services runtime

Error
Message: Failed to bootstrap VCF services runtime
Remediation Message:
Reference Token: ######
Cause: java.lang.StringIndexOutOfBoundsException: Range [-1, O) out of
bounds for length O Range [-1, O) out of bounds for length O

domainmanager.log reports the following events -
YYYY-MM-DDTHH:MM:SS.493+0000 DEBUG [vcf_dm,################################,babf] [c.v.e.s.c.util.LocalProcessService,dm-exec-300] Error Output: YYYY/MM/DD HH:MM:SS IP address not specified. Trying to look up the bootstrap VM in vCenter
YYYY/MM/DD HH:MM:SS Using appliance name from props: bootstrap-vm-######
Error: failed to find bootstrap VM 'bootstrap-vm-######': failed to find VM /####-mgmt-####/vm/vcf-management-services/bootstrap-vm-######: vm '/####-mgmt-####/vm/vcf-management-services/bootstrap-vm-######' not found
YYYY/MM/DD HH:MM:SS error: failed to find bootstrap VM 'bootstrap-vm-######': failed to find VM /####-mgmt-####/vm/vcf-management-services/bootstrap-vm-######: vm '/####-mgmt-####/vm/vcf-management-services/bootstrap-vm-######' not found
YYYY-MM-DDTHH:MM:SS.715+0000 ERROR [vcf_dm,################################,babf] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-300] [Reference Token] PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED Failed to bootstrap VCF services runtime
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to bootstrap VCF services runtime
at com.vmware.vcf.vimanager.fsm.actions.BootstrapVspClusterAction.execute(BootstrapVspClusterAction.java:218)
at com.vmware.vcf.vimanager.fsm.actions.BootstrapVspClusterAction.execute(BootstrapVspClusterAction.java:89)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:66)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:161)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:147)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:401)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:515)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:128)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:63)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.RuntimeException: java.lang.StringIndexOutOfBoundsException: Range [-1, 0) out of bounds for length 0
at com.vmware.vcf.vimanager.services.VspServiceImpl.bootstrapVspCluster(VspServiceImpl.java:237)
at com.vmware.vcf.vimanager.fsm.actions.BootstrapVspClusterAction.execute(BootstrapVspClusterAction.java:185)
... 15 common frames omitted
Caused by: java.lang.StringIndexOutOfBoundsException: Range [-1, 0) out of bounds for length 0
at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:55)
at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:52)
at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)
at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)
at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)
at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromToIndex(Preconditions.java:112)
at java.base/jdk.internal.util.Preconditions.checkFromToIndex(Preconditions.java:349)
at java.base/java.lang.String.checkBoundsBeginEnd(String.java:4865)
at java.base/java.lang.String.substring(String.java:2834)
at java.base/java.lang.String.substring(String.java:2807)
at com.vmware.vcf.vimanager.services.VspServiceImpl.getBootstrapTask(VspServiceImpl.java:247)
at com.vmware.vcf.vimanager.services.VspServiceImpl.bootstrapVspCluster(VspServiceImpl.java:218)
... 16 common frames omitted
YYYY-MM-DDTHH:MM:SS.545+0000 DEBUG [vcf_dm,################################,ba68] [c.v.v.d.notif.impl.pc.PcPollingTask,vcf-ds-27] [Sub Id: '########-####-####-####-########:', Data source id: '########-####-####-####-########:'] Timeout while waiting for updates. Retrying with version '1'
YYYY-MM-DDTHH:MM:SS.210+0000 DEBUG [vcf_dm,################################,babf] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-300] Collected the following errors for task with name BootstrapVspClusterAction and ID ########-####-####-####-########:: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.evo.sddc.common.services.sddcmanagermessages)]]
YYYY-MM-DDTHH:MM:SS.332+0000 DEBUG [vcf_dm,################################,c8d3] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-307] Invoking task BootstrapVspClusterAction.UNDO, Plugin: ViDomainServicePlugin, ParamBuilder null, Input map: {vspClusterCreationSpec=vspClusterCreationSpec}, Id: ########-####-####-####-######## ...
YYYY-MM-DDTHH:MM:SS.345+0000 DEBUG [vcf_dm,################################,c8d3] [c.v.e.s.o.c.c.ContractParamBuilder,dm-exec-307] Contract task Bootstrap VSP cluster input: {"vspClusterCreationSpec":{"domainId":"########-####-####-####-########","platformFqdn":"####srvsruntime###.ram.ram","systemUserPassword":"*****","type":"MANAGEMENT","networkMoId":"dvportgroup-22","gatewayCidrIpv4":"<IP>/21","ipv4Pool":{"ipRange":{"startIpAddress":"<IP>","endIpAddress":"<IP>"}},"size":"small","internalClusterCidrIpv4":"<ip>/15","name":"####-mgmt-vmsp-01","instanceFqdn":"####msddcinst###.ram.ram","fleetFqdn":"<fleet-mgr>.ram.ram","version":"9.1.0.0.25370367"}}
YYYY-MM-DDTHH:MM:SS.346+0000 DEBUG [vcf_dm,################################,c8d3] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-307] Collected the following errors for task with name BootstrapVspClusterAction and ID ########-####-####-####-########: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.evo.sddc.common.services.sddcmanagermessages)]]

Environment

VMware Cloud Foundation 9.1

Cause

This issue can be caused due to multiple reasons - network or storage configuration mismatch.
In this article, this issue reported and caused due to changes done at the storage layer related to vlan and speed/mtu mismatch
Supporting log snippet from associated vSphere/ESXi-
Error caused by file /vmfs/volumes/<datastore name/id>/vcf-services-runtime-template-9.1.0.0.25370367_3/vcf-services-runtime-template-9.1.0.0.25370367.vmdk
That error pinpoints it: the VMDK file on datastore b3c12a21-bbf4f49d is blocking the OVF deploy. Error caused by file /vmfs/volumes/<datastore name/id>/vcf-services-runtime-template-9.1.0.0.25370367_3/vcf-services-runtime-template-9.1.0.0.25370367.vmdk What this means SDDC Manager extracted the OVF template to that datastore, then tried to clone/create VMs from it. The _3 suffix means this is the 3rd VCFMS node. The VMDK is either:Corrupt - bad download/extraction Locked - stale VM still attached to it Datastore full/inaccessible - no space or APD Permissions - SDDC Manager service account can't access it Fix it now - 5 steps1. Check datastore health first SSH to any mgmt ESXi host:bash df -h /vmfs/volumes/<datastore name/id>

Resolution

This issue can be resolved after fixing the problems at the inclined storage (NFS/SAN/vSAN) from the vSphere layer related to vlan and speed/MTU mismatch, as reported in the above log snippet (mentioned in Cause)

Additional Information

The error PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED during VMware Cloud Foundation (VCF) deployment typically indicates a failure in the VCF Services Platform (VMSP) bootstrap process. Common causes include network connectivity issues, naming constraint violations, or duplicate infrastructure names.

Common Causes and Troubleshooting

1. Network Connectivity and Firewall (Port 5480)

The SDDC Manager must reach the Health API of the temporary bootstrap VM via port 5480.

Identify VM IP: Check /var/log/vmware/vcf/domainmanager/domainmanager.log to find the IP assigned to the bootstrap VM.
Test Connectivity: From SDDC Manager, run:
- nc -zv <Bootstrap_VM_IP> 5480
- curl -v -k https://<Bootstrap_VM_IP>:5480/health
Resolution: Ensure port 5480 is allowed bi-directionally between SDDC Manager and the Management network.

2. Duplicate Datastore Names

If a datastore name exists in multiple datacenters, the process may fail with a NullPointerException.

Resolution: Temporarily rename the datastore to a unique name across all datacenters and retry.

3. Naming Constraint Violations (Spaces or Uppercase)

VMSP uses Kubernetes orchestration, which follows RFC 1123 naming standards (lowercase, no spaces/underscores).

Object Names: Ensure Datacenter and Cluster names do not contain spaces or unsupported characters.
FQDNs: All hostnames and FQDNs must be strictly lowercase.
Resolution: Rename affected vSphere objects (e.g., change Prod Cluster to prod-cluster) and ensure DNS records are lowercase.

4. IP Address Conflict

The bootstrap fails if any IP in the assigned pool is already pingable.

Check Logs: Look for IP <IP_ADDRESS> is already in use (pingable) in domainmanager.log.
Resolution: Free up the occupied IP or update the deployment JSON spec with a clean IP range.