VCF 9.1 Deployment Fails at Bootstrap VCF Services Runtime with VMSP Agent Not Healthy

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

VMware Cloud Foundation (VCF) deployments may become blocked at the "Bootstrap VCF services runtime" stage.

The /var/log/vmware/vcf/domainmanager/domainmanager.log records a java.lang.StringIndexOutOfBoundsException within VspServiceImpl.getBootstrapTask, which acts as the root blocker for the initialization process. Following the exception, the system reports the following error:

VMSP agent is not healthy: failed to connect to health endpoint: Get "https://<Bootstrap IP>:5480/health": dial tcp <Bootstrap IP>:5480: connect: no route to host.

During this failure state, service binding and connectivity checks reveal:

• The VMSP bootstrap VM is powered on, and initial VMSP runtime VMs are created, but the bootstrap process fails while waiting for the VMSP cluster to become ready.

• Connections to the control plane nodes of the VMSP cluster from other infrastructure VMs (such as SDDC Manager or vCenter) fail with a "no route to host" error.

• Port 5480 is not in a LISTEN state on the SDDC/Installer host , and local connection attempts return "Connection refused."

• Mitigation attempts such as manually adding iptables rules on the SDDC/Installer host do not resolve the issue.

 

Environment

VMware Cloud Foundation 9.1

 

Cause

Certain ESXi hosts in the environment have underlying network isolation or routing issues. When the VMSP cluster or bootstrap VMs are scheduled on these problematic ESXi hosts by DRS, they become unroutable. This prevents the bootstrap VM from connecting to the API server for the VMSP cluster, causing the deployment to time out and fail.

 

Resolution

1. Identify the specific ESXi hosts where the unroutable VMSP bootstrap or runtime VMs currently reside.

2. Migrate (vMotion) the affected VMs to a known-good ESXi host (e.g., an ESXi host where the vCenter Server is successfully operating and is fully routable).

3. Place the problematic ESXi host(s) into Maintenance Mode to prevent the bootstrap process from scheduling new VMSP cluster VMs onto them.

4. Attempt to ping the newly migrated VMs from the SDDC Manager or vCenter to validate that they are now routable.

5. If the bootstrap process creates additional VMs for the VMSP cluster, monitor their placement. If any are unroutable, migrate them to a known-good host and place the offending host in Maintenance Mode.

6. Once connectivity to all VMSP VMs is verified, retry the deployment workflow from the SDDC Manager interface.

7. Investigate and resolve the underlying network routing or uplink issues on the isolated ESXi hosts before removing them from Maintenance Mode.