VMSP deployment is hung for prolonged period - no errors in VCF Operations UI

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• The vmsp cluster deployment keeps spinning in VCF Operations with no error status
• The bootstrap VM appears to have been deployed successfully
• Rebooting the VCF Operations cluster does not clear the spinning deployment task.
• VMSP nodes appear to be up and running.
• Checking the 'packagedeployments' in the vmsp-platform namespace all show a status of "package deployment failed"

root [ /home/vmware-system-user ]# kubectl get packagedeployments -n vmop-platform
NAME AGE PHASE STATUS
vmop-platform-app-vmop-config 9d Failed package deployment failed
vmop-platform-group-bootstrap 9d Failed package deployment failed
vmop-platform-group-core 9d Failed package deployment failed
vmop-platform-groupingress-init 9d Failed package deployment failed
root [ /home/vmware-system-user ]

• Checking the status of the VMSP nodes show that all have a READY status of 'true':

root@/home/vmware-system-user: # kubectl get vspherenamespaces -A
NAMESPACE NAME CLUSTER READY PROVIDERID AGE
vmsp-platform dstmp-default-xxxx-xxxx-xxxx dstmp true vsphere://21179668-f79c-1385-c012-4563c3a76c41 9d
vmsp-platform dstmp-default-xxxx-xxxx-xxxx dstmp true vsphere://ea1736c1-155e-7905-0cde-169b0790c5c1 9d
vmsp-platform dstmp-default-xxxx-xxxx-xxxx dstmp true vsphere://4317969e-4586-480c-7054-1fce7b7166e4 9d
vmsp-platform dstmp-xxxx-xxxx dstmp true vsphere://9217c710-5a9b-6b59-f519-5d6dc6b04623 9d

• The /var/log/vmware/vcf/domainmanger/domainmanager.log show entries similar to:

xxxx-xx-xxT14:33:33.566+0000 ERROR [vcf_dm,6a04514252a0b7e8027faca80eaf2f8c,8ff4] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-301] [U9FE3L] PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED_DETAILED VCF Management Services cluster bootstrap failed with the following errors: "Bootstrap failed during stage 'DEPLOY0001 - Preparing VCF services runtime cluster': Command or operation failed with exit code 1. Please retry the operation or contact support if the issue persists. [VCFMS-BOOTSTRAP-999]".
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: VCF Management Services cluster bootstrap failed with the following errors: "Bootstrap failed during stage 'DEPLOY0001 - Preparing VCF services runtime cluster': Command or operation failed with exit code 1. Please retry the operation or contact support if the issue persists. [VCFMS-BOOTSTRAP-999]".
at com.vmware.vcf.vimanager.services.VspServiceImpl.bootstrapVspCluster(VspServiceImpl.java:226)
at com.vmware.vcf.vimanager.fsm.actions.BootstrapVspClusterAction.execute(BootstrapVspClusterAction.java:185)
at com.vmware.vcf.vimanager.fsm.actions.BootstrapVspClusterAction.execute(BootstrapVspClusterAction.java:89)

• You have confirmed that the environment does not have a CIDR range that overlaps with the internal CIDR range of 198.18.0.0/15 used by VCF services runtime (see KB 440541)

 

Environment

VCF 9.1.x

Cause

This issue can be caused when the VMSP cluster deployment exceeds the 240 minute limit set in the product for the bootstrap VM to complete it's task.

Resolution

• Please open a Service Request with Broadcom Support citing this Knowledge Article