Replace Expired vCenter Certificates in a VMware Cloud Foundation Environment

Products

• VMware Cloud Foundation

Issue/Introduction

• vCenter certificates in a VMware Cloud Foundation (VCF) environment are expired.
• SDDC Manager UI is inaccessible or returns "Identity Internal Server Error."
• Automated certificate rotation in SDDC Manager fails with PKIX path validation errors.

Environment

• VMware Cloud Foundation 5.x
• VMware Cloud Foundation 4.x

Cause

• vCenter Server Machine SSL certificates reached hard expiration. This breaks the cryptographic trust between SDDC Manager and vCenter, preventing SDDC Manager from executing automated lifecycle management (LCM) certificate renewal workflows.

Resolution

Manual certificate replacement on the vCenter appliance is required to restore trust.

1. Take Snapshots: 
   • Power off all vCenter Server nodes in the SSO domain simultaneously and take offline snapshots.
     
     
2. Renew certificate of the vCenter manually.
   • STS certificates are expired, 
     • STS certificates are not managed by SDDC-Manager, and should be replaced using the appropriate KBs:
       • Please refer the article to verify and renew the STS certificate: "Signing certificate is not valid" or "No healthy upstream" error in vCenter Server Appliance
   • Using Self signed certificates. 
     • Please refer the article: Regenerate vSphere 6.x, 7.x, and 8.0 certificates using self-signed VMCA
   • Using External CA signed certificates.
     • Generate CSR: 
       • Connect to the workload vCenter via SSH and generate a Certificate Signing Request (CSR):

         • openssl req -new -newkey rsa:2048 -nodes -keyout server.key -out server.csr

         • Note: only if you want to have 2048 as RSA key length
     • Sign and Replace: 
       • Submit the CSR to the External Certificate Authority. Use the vCenter Certificate Manager utility to manually replace the Machine SSL certificate on the appliance.
       • Please refer the article: Machine SSL certificate renewal using Custom Certificate Authority (CA) in vCenter Server
3. Reestablish SDDC Manager Trust:
   
   
   1. Copy the root CA certificate to /tmp on the SDDC Manager VM.
   2. Run the below command to get commonsvcs truststore password

      • cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key

   3. Import the root CA to the commonsvcs truststore: 

      • keytool -importcert -alias <alias> -file <path_to_root_CA> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store

      • Note:
        • <aliasname> : This can be any user defined name.
        • <certificate file> : Provide the path of root CA Certificate
      • When prompted, provide the password to add certificate to truststore obtained in step 2.
   4. Type yes when prompted to trust the certificate.
   5. Verify if new root CA is added to common services truststore, enter the password for truststore key obtained in step 2.

      • keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store

   6. Import the root CA to the SDDC Manager truststore: 

      • keytool -importcert -alias <alias> -file <path_to_root_CA> -keystore /etc/alternatives/jre/lib/security/cacerts

      • Note:
        • <aliasname> : This can be any user defined name.
        • <certificate file> : Provide the path of root CA Certificate
      • When prompted, provide the password to add certificate to truststore. The default password is "changeit"
   7. Type yes when prompted to trust the certificate.
   8. Verify if new root CA is added to SDDC-Manager truststore.

      • keytool -list -v -keystore /etc/alternatives/jre/lib/security/cacerts

   9. Restart SDDC Manager services: 

      • /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

         

4. Removing Expired certificates
   • Leaving old expired certificates might cause future issue. 
   • Please refer to KB Remove Old or Unused Certificates to remove these.