How to import the vCenter root certificate into the SDDC manager TrustStore

Products

• VMware Cloud Foundation

Issue/Introduction

If the vCenter root certificate is not published to the SDDC Manager truststores then the subsequent errors could occur.

• Unable to see utilization details in SDDC UI.
• Unable to commission hosts.
• Unable to add hosts into clusters.
• Unable to login to the SDDC UI.

Similar exception traces can be seen in commonvsvc log.

/var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log

YYYY-MM-DDTHH:MM:SS ERROR [common,52343a2f26ac349f,6b2b] [c.v.v.i.sync.utils.VcSyncManagerUtil,cs-exec-3] Error connecting to vCenter vcenter-1, with exception {} com.vmware.vim.vmomi.client.exception.SslException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:250) at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

The SDDC Manager GUI may display a blank screen with an error message similar to the following:

{"message":"500 - \"{\\\"errorCode\\\":\\\"IDENTITY_INTERNAL_SERVER_ERROR\\\",\\\"arguments\\\":[],\\\"message\\\":\\\"Identity Internal Server Error\\\",\\\"referenceToken\\\":\\\"ABC123\\\"}\""}

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

Cause

The vCenter root certificate has changed and the SDDC Manager is still referencing the old root certificate. Occurs when option 8 is run in the certificate manager utility on vCenter.

Resolution

Scripted process:

Upload the attached script VcRootCaSync.py to the SDDC Manager, and run the script as root user:

python VcRootCaSync.py

Expected Output:

/home/vcf ]# python VcRootCaSync.py
Please provide SSO administrator user[[email protected]]:
Provide password for [email protected]:
Available vCenter Servers:
[1] ACTIVE | example.vcsa1.com
[2] ACTIVE | example.vcsa2.com

        Select a vCenter server by entering the corresponding number: 1
        Selected vCenter: example.vcsa1.com

         Session token created successfully
         Root certificate saved to /tmp/root.cer

vCenter example.vcsa1.com Root Certificate Found:
-----BEGIN CERTIFICATE-----
MIIDETCCAfmgAwIBAgI***************************ANBgkqhkiG9w0BAQsFADAb

-----END CERTIFICATE-----

         Using randomly generated Alias: abc-def-vcsa1_RootCrt_EA0G

         vCenter Root certificate added to SDDC Manager trust stores.
         Refreshing certificate store
         Deleting root certificate from temp

 

Manual Process:

• SSH into the vCenter server and get the root certificate

• If customer is using default certificate use the below command:

/usr/lib/vmware-vmca/bin/certool --getrootca --cert=/tmp/root.cer

• If customer is using a custom root certificate.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store trusted_roots

• Copy the root certificate to the temp directory on the SDDC Manager.

vi /tmp/root.cer to create an empty file with vi.

Press i to enter insert mode.

Paste the body of the  certificate into the root.cer file.

Press esc to exit insert mode.

Type wq! and hit enter to write your changes to the file and exit the vi editor.

• Obtain the trusted certificates key by issuing the following command

pass=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)

• Import the certificate into the trusted_certificates_store with the keytool.

keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass <trust store key>

example:

keytool -importcert -alias new_mgmt_root -file /tmp/root.cer -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass

• Import the certificate into the cacerts store.

keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

example:

keytool -importcert -alias new_mgmt_root -file /tmp/root.cer -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

• Verify the new certificate has been successfully added.

keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass  | less

• Refresh the trusted certificates using the API from the SDDC-Manager

curl -X POST localhost/appliancemanager/trustedCertificates/refresh

 

Workaround:

If the SDDC UI is accessible you can also import the root certificate from API Explorer.

• Convert the root certificate on the vCenter into single line PEM format and copy the output.

awk 'NF {sub(//, ""); printf "%s\\n",$0;}' /tmp/vmca.crt

For example:

 root@vcenter-1 [ ~ ]# awk 'NF {sub(//, ""); printf "%s\\n",$0;}' /tmp/vmca.crt
-----BEGIN CERTIFICATE-----
<certicate..............>
-----END CERTIFICATE-----\n

 

• Navigate to the API Explorer in the SDDC UI and input the certificate and certificateUsageType.

API Explorer > APIs for managing Trusted Certificates > POST

• Input the certificate in one line PEM format in the certificate field.
• Input "TRUSTED_FOR_OUTBOUND" into the certificateUsageType field.

• Click on Execute. 

Additional Information

Manually reviewing certificates in VMware Endpoint Certificate Store for vSphere 6.x and 7.x