SDDC Manager fails to trust integrated components certificate with error "Failed to trust Certificate(s). Error while validating certificate"

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

In a VMware Cloud Foundation (VCF) environment, integrated component communication (vCenter, NSX, VxRail) fails after a certificate renewal or replacement. Symptoms include:

Accessing the SDDC Manager UI fails with the following error: {"message":"Identity Internal Server Error","code":"IDENTITY_INTERNAL_SERVER_ERROR","status":500}
vCenter certificate status has an alert with message: "Connection cannot be established because certificate is not trusted".
Upon selecting "review" option to attempt to trust the vCenter certificate, the certificate validation fails with error: "Failed to trust Certificate(s). Error while validating certificate"
vCenter root and machine certificates had been recently renewed directly from the vCenter.
From SDDC manager /var/log/vmware/vcf/operationsmanager/operationsmanager.log
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om, 16#####################, ar##] [c.v.v.s. t. DynamicTrustManager, om-exec-7] Error checking certificate chain
O=####, L=#####, ST=#####, C=##, CN=<vCenter-FQDN>. for validity.
sun. security. validator. ValidatorException: PKIX path building failed: sun. security.provider. certpath. SunCertPathBuilderException: unable to find valid certification path to requested target.

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x

Cause

This issue occurs when integrated component certificates are updated or regenerated outside of SDDC Manager (out-of-band), causing a mismatch between the certificate presented by the component and the trust store held in the SDDC Manager database.

Resolution

The below methods can be used to update the SDDC Manager trust store with the new vCenter certificate.

Script method
Manual method:
API method:

Script method

Take Snapshot of SDDC Manager Appliance from the vCenter
Upload the VcRootCaSync.py script attached to the KB to the /home/vcf directory on the SDDC Manager, and execute the script as the root user.
python VcRootCaSync.py
Select the vCenter server which is in error state, or the management vCenter Server. (Refer Broadcom KB: Import the vCenter root certificate into the SDDC manager TrustStore for details)

Example:
# python VcRootCaSync.py
Please provide SSO administrator user[[email protected]]:
Provide password for [email protected]:
Available vCenter Servers:
[1] ERROR | vcsa1.example.com
[2] ACTIVE | vcsa2.example.com
Select a vCenter server by entering the corresponding number: 1
Selected vCenter: vcsa1.example.com
Session token created successfully
Root certificate saved to /tmp/root.cer
vCenter vcsa1.example.com Root Certificate Found:
-----BEGIN CERTIFICATE-----
M#############################################b
-----END CERTIFICATE-----
vCenter Root certificate added to SDDC Manager trust stores.
Refreshing certificate store
Deleting root certificate from temp
Refresh the SDDC manager GUI and the status for vCenter certificate should be active.
If the previous step does not resolve the issue, restart the SDDC Manager as root user by running the command: /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh, or restart the SDDC Manager VM.

Manual method:

Extract the certificate from the integrated components.
Copy the certificate to the temp directory on the SDDC Manager.
vi /tmp/certificate.cer
Obtain the trusted certificates key by issuing the following command
pass=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)
Import the certificate into the trusted_certificates_store with the keytool.
keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass <trust store key>

example:
keytool -importcert -alias new_mgmt_root -file /tmp/certificate.cer -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass
Import the certificate into the cacerts store.
keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

example:
keytool -importcert -alias new_mgmt_certificate -file /tmp/certificate.cer -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit
Verify the new certificate has been successfully added.
keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass | less
Refresh the trusted certificates using the API from the SDDC-Manager
curl -X POST localhost/appliancemanager/trustedCertificates/refresh

API method:

Convert the certificate on the integrated components into single line PEM format and copy the output.
awk 'NF {sub(//, ""); printf "%s\\n",$0;}' /tmp/certificate.crt
For example:
# awk 'NF {sub(//, ""); printf "%s\\n",$0;}' /tmp/certificate.crt
-----BEGIN CERTIFICATE-----
<certicate..............>
-----END CERTIFICATE-----\n
Navigate to the API Explorer in the SDDC UI and input the certificate and certificateUsageType.
API Explorer > APIs for managing Trusted Certificates > POST
Input the certificate in one line PEM format in the certificate field.
Input "TRUSTED_FOR_OUTBOUND" into the certificateUsageType field.
Click on Execute.

Additional Information

Steps to extract certificate for vCenter Server :

If using default certificate use the below command:
/usr/lib/vmware-vmca/bin/certool --getrootca --cert=/tmp/root.cer
If using a custom root certificate.
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store trusted_roots
Copy the root certificate to the temp directory on the SDDC Manager.
vi /tmp/root.cer

Steps to extract certificate for NSX Manager :

Use a file transfer utility to copy the following certificates to the /tmp directory on the SDDC Manager Appliance.
1. Extract NSX Manager's full certificate chain (Leaf - Intermediate - Root) for each of the NSX Manager nodes and NSX Manager VIP.
2. Another certificate file for only the Root certificate.

Note: If the same CA is signing all three NSX manager nodes and VIP, only one Root certificate file is needed.

Steps to extract certificate for VxRAIL Manager :

Use a file transfer utility to copy the following certificates to the /tmp directory on the SDDC Manager Appliance.
1. VXRail Manager's full certificate chain (leaf - intermediate - root)
2. Another certificate file for only the Root certificate.

Attachments

vcrootcasync.py get_app