Deployment of VMSP cluster is failing with "Configuration error occurred. Invalid VCF services runtime cluster name"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• During upgrade of VCF to 9.1, VMSP cluster deployment is failing with "Deployment of VMSP cluster is failing with "Configuration error occurred. Invalid VCF services runtime cluster name \u0027<cluster name>\u0027"
• The deployment was initiated with the VCF Service Runtime FQDN in uppercase
• /var/log/vmware/vcf/domainmanager/domainmanager.log: (on SDDC Manager)

YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,6a469a993f496aede124c8d331237ab8,626c] [c.v.e.s.s.helper.SddcManagerTaskUtil,dm-exec-30] Response for Get task with id: ########-####-####-####-######fe06ac is {"id":"########-####-####-####-######fe06ac","name":"Creating VSP Cluster <VMSP cluster FQDN> in domain <VCF domain name>","type":"VSP_BOOTSTRAP","status":"Failed","creationTimestamp":
"YYYY-MM-DDTHH:MM:SS","completionTimestamp":"YYYY-MM-DDTHH:MM:SS","subTasks":[{"name":"Validate VSP cluster creation specification","description":"Validate VSP cluster creation specification","status":"SUCCESSFUL","creationTimestamp":"YYYY-MM-DDTHH:MM:SS","completionTimestamp":"YYYY-MM-DDTHH:MM:SS","errors":[]}
,{"name":"Bootstrap VCF Services Platform","description":"Bootstrap VCF Services Platform","status":"FAILED" ,"creationTimestamp":"YYYY-MM-DDTHH:MM:SS","completionTimestamp":"YYYY-MM-DDTHH:MM:SS","errors": [{"errorCode":"PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED_DETAILED","errorType":"ERROR","arguments":["\"Configuration error occurred. Invalid VCF services runtime cluster name \u0027<cluster name>\u0027. Please configure and retry the operation or contact support if the issue persists. [VCFMS-BOOTSTRAP-002]\""],"message":"VCF Management Services cluster bootstrap failed with the following errors: \"Configuration error occurred. Invalid VCF services runtime cluster name \u0027<cluster name>\u0027. Please configure and retry the operation or contact support if the issue persists. 

YYYY-MM-DDTHH:MM:SS INFO [vcf_dm,6a469a993f496aede124c8d331237ab8,626c] [c.v.e.s.s.helper.SddcManagerTaskUtil,dm-exec-30] Status for task with id: ########-####-####-####-######fe06ac is: Failed
YYYY-MM-DDTHH:MM:SS INFO [vcf_dm,6a469a993f496aede124c8d331237ab8,626c] [c.v.e.s.s.MonitorVspClusterBootstrapOnSddcManagerAction,dm-exec-30] Vsp bootstrap task /v1/tasks/########-####-####-####-######fe06ac failed or cancelled. Status 'Failed'.

Environment

VCF 9.1

Cause

The deployment failure is caused by entering hostnames or FQDNs containing uppercase letters in the VCF Installer UI. VCF Management Services (VSP Cluster) strictly enforces Kubernetes-compliant naming standards, which require all-lowercase alphanumeric characters and hyphens

Resolution

Manually update the deployment workflow JSON to convert all hostnames to lowercase.

1. SSH to the SDDC Manager/Installer VM
2. Execute the below command to to enable internal vault access
   
   echo vcf.vault.http-access=true >> /etc/vmware/vcf/domainmanager/application.properties && systemctl restart domainmanager
   
   
3. Identify the task ID (from UI/domainmanager.log) and extract the json
   
   curl -X GET http://localhost/domainmanager/internal/vault/<task id> | json_pp > /tmp/workflow.json
   
   Sample:
   
   curl -X GET http://localhost/domainmanager/internal/vault/########-####-####-####-######fe06ac | json_pp > /tmp/workflow.json

Task ID is captured from the domainmanager.log
   
   
4. Use vi editor to manually edit /tmp/workflow.json to change all instances of hostnames (e.g., sddc address, fleetfqdn, instance fqdn) to lowercase.
5. Import the updated JSON
   
   curl -H 'Content-Type:text/plain' -X PUT http://localhost/domainmanager/internal/vault/<task id> -d @/tmp/workflow.json
   
   
6. Restart the domainmanager service on SDDC
   
   systemctl restart domainmanager
   
   
7. Cleanup the partially deployed bootstrap VM
   
   
   1. Log in to vCenter server via vSphere Client
   2. Identify the bootstrap VM and delete the machine
      
      
      1. Navigate to the VMs and Templates tab under inventory
      2. Expand the folder vcf-management-services
      3. right click and power off the bootstrap VM and delete
         
         
8. Log in to VCF Operations UI/VCF Installer UI and restart the task