How to add/delete Custom CA Certificates to SDDC Manager and Common Services trust stores

Products

• VMware Cloud Foundation
• VMware SDDC Manager

Issue/Introduction

• If the custom CA root certificate is not published to the SDDC Manager truststores, then the subsequent errors might be evident:

  PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

• In SOS logs:

  ERROR restutil.py::get::128::_collect_https_esx_logsThread0] GET call failed for https://esxi-1.vcf.example.com/cgi-bin/vm-support.cgi due to HTTPSConnectionPool(host='esxi-1.vcf.example.com', port=443): Max retries exceeded with url: /cgi-bin/vm-support.cgi (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

  
  
  
  
  

Environment

VMware Cloud Foundation 4.5.x

VMware Cloud Foundation 5.x

Resolution

The custom CA root certificate should be added to SDDC-Manager trust stores.

To add custom CA root certificate to SDDC Manager (versions prion to 4.1) trust store:

1. Take a snapshot of the SDDC Manager VM prior to starting this process
2. Use a file transfer utility to copy the trusted certificate file to the /tmp directory on the SDDC Manager VM.
3. SSH to the SDDC Manager VM as the VCF user and then issue the su - command to switch to the root user.
4. Obtain the trusted certificates key by issuing the following command:

   cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key

   Sample Output:

   q_0EZjUI7Z^B7V@2A+ ----> Key Store Password

5. Issue a command similar to the following to import the certificate into the SDDC Manager trust store

   keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store --storepass <Key from Step # 4>
   
   <Note:  if the above fails run the command without the storepass and paste the key from step #4 once prompted:>
   
   keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store
   

    

6. Issue a command similar to the following to import the certificate into the java trust store:

   keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

   Notes:
   • Type yes when prompted to trust the certificate.
   • Enter alias name, for the <aliasname> value(Can be any name given to identify which component the certificate belongs to).
   • Replace <certificate file> with the full/absolute path to the certificate file that was uploaded in Step #2.

• • Issue a command similar to the following to verify that the new trusted certificate has been added to the SDDC Manager trust store:

    keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass <Key from Step # 4>

  • Issue the following command to restart the SDDC Manager services:

    /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

To delete a stale certificate in trust store : (Please reach out to Broadcom support if you are not sure)

1. Take Snapshot of SDDC manager VM
2. Find the alias of the duplicate cert by running this command

   keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass <Key from Step # 4>

3. Delete the duplicate alias by running

   keytool -delete -alias <alias from Step #1> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store

To add custom CA root certificate to SDDC Manager (versions 4.1 and later) trust store:

Public API can also be used from VCF 4.1 to add/delete trusted certificates to the SDDC manager trust store.

1. Import the trusted certificate using Public API - Refer VMware Cloud Foundation API Reference Guide
2. SSH to SDDC Manager and after importing CA certificate to the trust store.
3. Restart all the SDDC Manager Services using the below command

   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

To add custom CA root certificate to SDDC Manager (versions 4.5.1 and later) trust store:

• The trusted certificate can also be added to the SDDC Manager trust store using the SDDC Manager UI.
  • Follow the steps mentioned in  Add a Trusted Certificate to the SDDC Manager Trust Store

 

Workaround:
The attached python script can also be run from SDDC Manager to add the trusted certificates to the SDDC Manager.

 

Additional Information

This issue is being checked by Diagnostics for VMware Cloud Foundation.

The check is as follows:

• Product: SDDC
• Log File: sos.log
• Log Expression Check "Caused by SSLError" AND "certificate verify failed"