SDDC Manager fails to trust vCenter certificate with error "Failed to trust Certificate(s). Error while validating certificate"

Products

• VMware SDDC Manager

Issue/Introduction

• vCenter certificate status has an alert with message: "Connection cannot be established because certificate is not trusted".
  
  
• Upon selecting "review" option to attempt to trust the vCenter certificate, the certificate validation fails with error: "Failed to trust Certificate(s). Error while validating certificate"
• vCenter root and machine certificates had been recently renewed directly from the vCenter.
• /var/log/vmware/vcf/operationsmanager/operationsmanager.log
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om, 16#####################, ar##] [c.v.v.s. t. DynamicTrustManager, om-exec-7] Error checking certificate chain
O=####, L=#####, ST=#####, C=##, CN=<vCenter-FQDN>. for validity.
sun. security. validator. ValidatorException: PKIX path building failed: sun. security.provider. certpath. SunCertPathBuilderException: unable to find valid certification path to requested target

Environment

• VMware Cloud Foundation 4.x
• VMware Cloud Foundation 5.x

Cause

• This issue occurs when the vCenter Server root certificate is updated or regenerated (for example, by running option 8 in the vCenter Certificate Manager utility), but the SDDC Manager continues to reference the previous certificate chain in its database. This typically happens when the vCenter certificate renewal or import process is performed outside of SDDC Manager, preventing the updated certificate information from being synchronized.

Resolution

The following scripted procedure can be used to update the SDDC Manager trust store with the new vCenter certificate.

1. Take Snapshot of SDDC Manager Appliance from the vCenter
2. Upload the VcRootCaSync.py script (as described in “How to import the vCenter root certificate into the SDDC Manager TrustStore”) to the /home/vcf directory on the SDDC Manager, and execute the script as the root user.
   python VcRootCaSync.py
3. Select the vCenter server which is in error state.
   
   Example:
   

   /home/vcf ]# python VcRootCaSync.py
Please provide SSO administrator user[[email protected]]:
Provide password for [email protected]:
Available vCenter Servers:
[1] ERROR | example.vCenter1.com
[2] ACTIVE | example.vCenter2.com

           Select a vCenter server by entering the corresponding number: 1
        Selected vCenter: example.vCenter1.com

            Session token created successfully
         Root certificate saved to /tmp/root.cer

   vCenter example.vCenter1.com Root Certificate Found:
-----BEGIN CERTIFICATE-----
M#############################################b

   -----END CERTIFICATE-----

            vCenter Root certificate added to SDDC Manager trust stores.
         Refreshing certificate store
         Deleting root certificate from temp

4. Refresh the SDDC manager GUI and the status for vCenter certificate should be active.

Additional Information

Alternatively, The following methods can be referred to update the SDDC manager trust with the new vCenter certificate.

Manual method:

1. SSH into the vCenter server and get the root certificate.
2. If customer is using default certificate use the below command:
   /usr/lib/vmware-vmca/bin/certool --getrootca --cert=/tmp/root.cer
3. If customer is using a custom root certificate.
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store trusted_roots
4. Copy the root certificate to the temp directory on the SDDC Manager.
   vi /tmp/root.cer
5. Obtain the trusted certificates key by issuing the following command
   pass=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)
6. Import the certificate into the trusted_certificates_store with the keytool.
   keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass <trust store key>
   
   example:
   keytool -importcert -alias new_mgmt_root -file /tmp/root.cer -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass
7. Import the certificate into the cacerts store.
   keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit
   
   example:
   keytool -importcert -alias new_mgmt_root -file /tmp/root.cer -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit
8. Verify the new certificate has been successfully added.
   keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass  | less
9. Refresh the trusted certificates using the API from the SDDC-Manager
   curl -X POST localhost/appliancemanager/trustedCertificates/refresh

API method:

1. Convert the root certificate on the vCenter into single line PEM format and copy the output.
   awk 'NF {sub(//, ""); printf "%s\\n",$0;}' /tmp/vmca.crt

   For example:
   # awk 'NF {sub(//, ""); printf "%s\\n",$0;}' /tmp/vmca.crt
-----BEGIN CERTIFICATE-----
<certicate..............>
-----END CERTIFICATE-----\n
2. Navigate to the API Explorer in the SDDC UI and input the certificate and certificateUsageType.
   API Explorer > APIs for managing Trusted Certificates > POST
3. Input the certificate in one line PEM format in the certificate field.
4. Input "TRUSTED_FOR_OUTBOUND" into the certificateUsageType field.
   
5. Click on Execute.