VCF Management Services bootstrap fails with PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED due to port blockage on firewall

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When deploying or upgrading to VMware Cloud Foundation (VCF) 9.1, the process fails during the Deploy and configure VCF Management Platform phase with a generic bootstrap error similar to below: 
  
  VCF Management Services cluster bootstrap failed with the following errors.
  
  
• The /var/log/vmware/vcf/domainmanager/domainmanager.log on VCF Installer/SDDC-Manager contains errors indicating a timeout or failure during the VSP cluster bootstrap:
  
  YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,<task_id>,<subtask_id>] [c.v.v.v.services.VspServiceImpl,pool-3-thread-####]  YYYY-MM-DDThh:mm:ss install API Task with Id ######## has status Failed
YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,<task_id>,<subtask_id>] [c.v.v.v.services.VspServiceImpl,pool-3-thread-####]  Error: install failed: Failed
YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,<task_id>,<subtask_id>] [c.v.v.v.services.VspServiceImpl,pool-3-thread-####]  YYYY-MM-DD hh:mm:ss error: install failed: Failed
YYYY-MM-DDThh:mm:ss DEBUG [vcf_dm,<task_id>,<subtask_id>] [c.v.e.s.c.util.LocalProcessService,dm-exec-####]  Command exit code: 1
YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,<task_id>,<subtask_id>] [c.v.e.s.c.util.LocalProcessService,dm-exec-####]  Local command  /opt/vmware/vcf/domainmanager/tmp/vmsp######## bootstrap cluster deploy --wait /nfs/vmware/vcf/nfs-mount/bundle/####-####-####/####-####-####/vcf-services-platform-template-9.1.0.####.ova --props-file=/opt/vmware/vcf/domainmanager/tmp/vmsp.props.########.tpl failed with exit code 1
Out Stream: LocalProcess INFO: YYYY-MM-DD hh:mm:ss - Bootstraping cluster with ID ####-####-####-####
...
LocalProcess ERROR:  YYYY-MM-DD hh:mm:ss - YYYY-MM-DD hh:mm:ss install API Task with Id ######## has status Running
LocalProcess ERROR:  YYYY-MM-DD hh:mm:ss - YYYY-MM-DD hh:mm:ss install API Task with Id ######## has status Failed
LocalProcess ERROR:  YYYY-MM-DD hh:mm:ss - Error: install failed: Failed
LocalProcess ERROR:  YYYY-MM-DD hh:mm:ss - YYYY-MM-DD hh:mm:ss error: install failed: Failed
...
YYYY-MM-DDThh:mm:ss DEBUG [vcf_dm,<task_id>,<subtask_id>] [c.v.e.s.c.util.LocalProcessService,dm-exec-####]  Output: VM bootstrap-vm-<vm_name>has IP: <bootstrap_vm_ip>
{
  "id": "########",
  "type": "com.vmware.vcfms.task.InstallVcfmsComponent",
  "createTime": "YYYY-MM-DDThh:mm:ss",
  "startTime": "YYYY-MM-DDThh:mm:ss",
  "endTime": "YYYY-MM-DDThh:mm:ss",
  "cancellable": false,
  "retryable": true,
  "phase": "Failed",
  "status": "Failed",
  "createdBy": "vmware-system-user"
}
YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,<task_id>,<subtask_id>] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-####]  [REFERENCE_TOKEN] PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED_DETAILED VCF Management Services cluster bootstrap failed with the following errors: .
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: VCF Management Services cluster bootstrap failed with the following errors: 
  
  
  
• Attempting to fetch vcenter certificates using openssl command from bootstrap vm fails with "Connection refused" or "Connection timed out" similar to below:
  
  # openssl s_client -showcerts -connect <vc_fqdn>:443
#######:error:########:system library:connect:Connection timed out

Cause

This issue occurs when a network firewall blocks HTTPS (port 443) connectivity between the temporary VCF Services Platform (VMSP) Bootstrap VM and the vCenter Server. The bootstrap process requires the Bootstrap VM to communicate with vCenter to initialize the Management Services cluster. If this traffic is intercepted or dropped, the bootstrap task times out.

Resolution

To resolve this issue, identify the IP address of the temporary Bootstrap VM and ensure port 443 is open to the vCenter Server.

1. Make sure DRS on the cluster is set to fully automated 
   
   
2. From  the Bootstrap VM, or by console access, verify if port 443 on the vCenter Server is reachable using openssl command:
   
    openssl s_client -connect <vcenter_fqdn>:443 -showcerts
   
   
3. Review the physical or virtual firewall rules between the Management Services network (where the Bootstrap VM resides) and the vCenter Server.
   
   
4. Modify or disable any firewall rule blocking TCP port 443 (HTTPS) traffic originating from the VMSP Bootstrap VM and targeted at the vCenter Server.
   
   
5. Once connectivity is confirmed, restart the deployment or upgrade task from the SDDC Manager UI.

Additional Information

To enable verbose logging and gather more detailed logs for bootstrap tasks, refer to VMSP deployment fails at "Bootstrap VCF Services Platform" step during VCF upgrade