VCF 9.1 Convergence Fails With "Failed to bootstrap VCF services runtime" Due to vCenter Certificate Trust Issues

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• During a vSphere to VMware Cloud Foundation (VCF) convergence deployment, the process fails at the VCF Management Platform stage with the following error: "Failed to bootstrap VCF services runtime."
• Review of the /var/log/vmware/vcf/domainmanager/domainmanager.log reveals certificate chain validation errors, "Unable to construct a valid chain" and subsequent SSL handshake failures.
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm, 0000000000000000, 0000] [c. v. v. s. t. DynamicTrustManager, VLSI-I/0 reactor-0] Error checking certificate chain O-<vCenter_HOSTNAME>, L=Palo Alto, ST=California, C=US, CN=<vCenter_HOSTNAME>, SerialNumber=17808127622352709007 for validity. java. security. cert. CertificateException: Unable to construct a valid chain
YYYY-MM-DDTHH:MM:SS WARN [vcf_dm, 0000000000000000, 0000] [c. v.v.v.c.http. impl. VlsiTlsStrategy, VLSI-I/0 reactor-0] SSL handshake from /<SDDC_IP> to <REDACTED_HOSTNAME>: 443 failed

Environment

VMware Cloud Foundation (VCF) 9.1

Cause

• This issue is caused by the vCenter Server hostname containing a mix of uppercase and lowercase characters, which is an unsupported configuration in VCF 9.1. Consequently, the issuing certificate within the vCenter trusted roots store populates the issuer field exclusively in uppercase characters.
• If a manual hostname and PNID change was performed to fix it lowercase in the Virtual Appliance Management Interface (VAMI) without following the correct procedure mentioned in Addressing vCenter FQDN and Hostname Case Mismatches , followed by a manual certificate regeneration, the old uppercase issuer may persist in the certificate store, creating an invalid certificate chain. When the VCF Installer or SDDC Manager attempts to validate this certificate against its internal trusted roots and castore, the case discrepancy results in a validation failure.

Resolution

1. Take a snapshot of the VCF Installer/SDDC Manager appliance before performing any modifications. This ensures a recovery point is available if the procedure needs to be reversed.
2. Ensure the vCenter's hostname and PNID are configured correctly in lowercase via the VAMI. Follow the standard procedure for hostname updates as documented in: Addressing vCenter FQDN and Hostname Case Mismatches
3. Construct the valid trusted root chain by combining the vCenter Machine SSL certificate and the root certificate in a text editor on SDDC manager: vi certificate_chain.cer
4. Manually import the constructed certificate chain into the SDDC Manager keystores using the keytool utility.
   1. Get the Trust store key: cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key
   2. Import the certificate into the trusted_certificates.store: keytool -importcert -alias <aliasname> -file <certificate file from step 3> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store --storepass <trust store key from step 4.1>
   3. Import into the Java cacerts store: keytool -importcert -alias <aliasname> -file <certificate file from step 3> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit
      
      Note regarding Alias Names: While the alias name can be arbitrary, it is recommended to use the alias name present in the vCenter's trusted root store for consistency.
      
      
5. Modify the domain JSON specification file to update the vCenter's FQDN and hostname entries to lowercase and re-try the VCF Installer workflow by following the guidance in: Retry VCF 9.0 Installer workflow by modifying the deployment workflow JSON SPEC file

Additional Information

• Manually reviewing certificates in VMware Endpoint Certificate Store for vSphere 6.x and 7.x
• Keytool Commands for interacting with certificates in SDDC Manager Java Keystores
• To create and manage your support cases, see Creating and Managing Broadcom Support Cases.