How to add/delete Custom CA Certificates to SDDC Manager and Common Services trust stores

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

If the custom CA root certificate is not published to the SDDC Manager truststores, then the subsequent errors might be evident:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

In SOS logs:
2020-03-26T11:04:28.661Z [ERROR restutil.py::get::128::_collect_https_esx_logsThread0] GET call failed for https://esxi-1.vcf.example.com/cgi-bin/vm-support.cgi due to HTTPSConnectionPool(host='esxi-1.vcf.example.com', port=443): Max retries exceeded with url: /cgi-bin/vm-support.cgi (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Environment

Vmware Cloud Foundation 4.5.1

Resolution

The custom CA root certificate should be added to SDDC-Manager trust stores.

To add custom CA root certificate to SDDC Manager (versions prion to 4.1) trust store:
 
Note: Take a snapshot of the SDDC Manager VM prior to starting this process

1. Use a file transfer utility to copy the trusted certificate file to the /tmp directory on the SDDC Manager VM.
2. SSH to the SDDC Manager VM as the VCF user and then issue the su - command to switch to the root user.
3. Obtain the trusted certificates key by issuing the following command:
   • cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key
   • Note: You will see output similar to the following:
     • q_0EZjUI7Z^B7V@2A+ ----> Key Store Password
4. Issue a command similar to the following to import the certificate into the SDDC Manager trust store
   • keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store 

• When prompted, enter the password.

Notes:

1. Type yes when prompted to trust the certificate.

2. Enter alias name, for the <aliasname> value.

3. Replace <certificate file> with the full path to the certificate file that was uploaded in Step 1.

4. Replace <trust store key> with the key value returned in Step 3.

• Issue a command similar to the following to import the certificate into the java trust store:

keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

 

Notes:

1. Type yes when prompted to trust the certificate.

2. Replace <certificate file> with the full path to the certificate file that was uploaded in Step 1

3. Issue a command similar to the following to verify that the new trusted certificate has been added to the SDDC Manager trust store:
   
   keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass <trust store key>

Note: Replace <trust store key> with the trusted certificates key value returned in Step 3
Issue the following command to restart the SDDC Manager services:

• /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

To delete a stale certificate in trust store : (Please reach out to GSS if you are not sure)

a. Take Snapshot of SDDC manager VM 
b. Find the alias of the duplicate cert by running this command 
keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass <trust store key>
c.Delete the duplicate alias by running 

• keytool -delete -alias alias_fetched_from_above -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store

To add custom CA root certificate to SDDC Manager (versions 4.1 and later) trust store:

Public API can also be used from VCF 4.1 to add/delete trusted certificates to the SDDC manager trust store.

1. Import the trusted certificate using Public API.

VMware Cloud Foundation API Reference Guide

2. SSH to SDDC Manager and after importing CA certificate to the trust store.
3. Restart all the SDDC Manager Services using the below command
   • /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

 

To add custom CA root certificate to SDDC Manager (versions 4.5.1 and later) trust store:

The trusted certificate can also be added to the SDDC Manager trust store using the SDDC Manager UI.
Follow the steps mentioned in  Add a Trusted Certificate to the SDDC Manager Trust Store  

Workaround:
The attached python script can also be run from SDDC Manager to add the trusted certificates to the SDDC Manager.

Additional Information

This issue is being checked by Diagnostics for VMware Cloud Foundation.

The check is as follows:

• Product: SDDC
• Log File: sos.log
• Log Expression Check "Caused by SSLError" AND "certificate verify failed"