Converting vSphere environment failed on "Bootstrap VCF Services Platform"

Products

• VMware Cloud Foundation

Issue/Introduction

• The task of converging vSphere environment to VVF fails at the deploy and configure VCF management platform stage.
• Follwing error in domainmanager.log

Caused by: com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: VCF Management Services cluster bootstrap failed with the following errors: "Bootstrap failed during stage 'INIT0002 - Initializing bootstrap machine': Command or operation failed with exit code 124. Please retry the operation or contact support if the issue persists. [VCFMS-BOOTSTRAP-999]".
        at com.vmware.vcf.vimanager.services.VspServiceImpl.bootstrapVspCluster(VspServiceImpl.java:226)
        at com.vmware.vcf.vimanager.fsm.actions.BootstrapVspClusterVvfAction.execute(BootstrapVspClusterVvfAction.java:115)
        ... 15 common frames omitted


• The vsp-install-#####-bootstrap-execute-script pod status shows error, and 'kubectl logs vsp-install-#####-bootstrap-execute-script' command returns error.
  
  No resource found in vmsp-platform namespace

Environment

VMware Cloud Foundation 9.x
VMware vSphere Foundation 9.x

Cause

• API Parsing Error: The SDDC Manager's /v1/appliance-info API incorrectly handles DNS names containing more than two dots. It incorrectly parses a single FQDN like domain.example.com into two separate search domains: domain.example.com and example.com.
• DNS Property Inheritance: These malformed search domains are passed to the bootstrap VM via the dnsSearchDomains OVA property. Consequently, they are written to the host's /etc/resolv.conf and inherited by all Kubernetes pods and containers.
• DNS Suffix Collision: When a containerized process attempts to reach the internal registry (registry.vmsp-platform.svc.cluster.local), the DNS resolver incorrectly appends the public example.com suffix. This results in an attempt to resolve registry.vmsp-platform.svc.cluster.local.example.com.
• External Resolution Conflict: Since example.com is a valid public top-level domain, the query is directed to a public IP address instead of the internal service. Because there is no listener on port 5000 at that public address, the connection times out, leading to the bootstrap failure.

Resolution

To update the vCenter DNS domain using the API Explorer, follow these steps:

1. Log in to API Explorer: Access the vCenter API Explorer by navigating to https://<vcenter-fqdn>/ui/api-explorer. Log in using [e-mail] or an equivalent account.
2. Locate the DNS API: In the Select API dropdown menu, select appliance. Scroll down to find the network > dns > domain section.
3. Execute GET: Expand the GET /appliance/network/dns/domain operation and click Execute to see the current configured domain and verify the current state.
   
   {
   "domain.example.com",
   "example.com"
}
   
   
4. Execute SET/UPDATE: Expand the PUT /appliance/network/dns/domain operation.
   Click the Try it out button.
   In the Request body, provide the correct FQDN (e.g., domain.example.com).
   Example Body:
   
   {
  "domian.example.com"
}

   Click Execute.
5. Verify the Change: Ensure the response code is 200 OK. 
6. Restart Deployment: Return to the SDDC Manager UI and retry the failed VVF installation task. The bootstrap VM should now inherit the correct, single DNS search domain.