SDDC Manager fails to trust certificate of integrated components (e.g., NSX Manager, VxRail Manager, vRSLCM) and reports error "Failed to trust Certificate(s). Error while validating certificate"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Component certificate status has an alert with message: "Connection cannot be established because certificate is not trusted".
  Below example, reports the error for the NSX Manager certificates. 
  
  
  
  
• Upon selecting "review" option to attempt to trust the certificate, the certificate validation fails with error: "Failed to trust Certificate(s). Error while validating certificate"
• Component certificate had been renewed directly from the appliance by importing the Custom CA signed certificate.
• Operations Manager log (/var/log/vmware/vcf/operationsmanager/operationsmanager.log) 

YYYY-MM-DDTHH:MM DEBUG [vcf_om,68ae############################,cf##] [c.v.v.s.t.DynamicTrustManager,##-###-##] Error checking certificate chain CN=<FQDN>, OU=<Org_Unit>, O=<Organization>, L=<Locality>, ST=<State>, C=<Country>,CN=<Custom_CA_Name>, DC=admin, DC=ad, DC=<Domain_name>, DC=<com>,CN=<Domain> Root Certificate Authority, O=<Organization>, L=<Locality>, ST=<State>, DC=<Domain>, DC=<com>, C=<Country>.
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:242)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:113)

Environment

• VCF 4.x
• VCF 5.x

Cause

The SDDC Manager database continues to reference an outdated certificate chain for the component. This discrepancy occurs because the new certificate was renewed or imported directly on the resource (e.g., NSX Manager, VxRail Manager, vRLSCM) rather than through the SDDC Manager. 

Resolution

The following steps can be used to update the SDDC Manager trust store with the new certificate.  

1. 1. Take Snapshot of SDDC Manager Appliance without virtual machine memory.
   2. Use a file transfer utility to copy the following certificates to the /tmp directory on the SDDC Manager Appliance.
      1. full certificate chain (Leaf - Intermediate - Root) 
      2. Another certificate file for only the Root certificate. 
   3. SSH to the SDDC Manager Appliance as the vcf user and then issue the su - command to switch to the root user.
   4. Obtain the trusted certificates key by issuing the following command

      KEY=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)

   5. Confirm the output by using the below command:

      echo $KEY

   6. Run the below command to import certificate for each of the NSX Managers into the SDDC Manager trust store:

      keytool -importcert -alias <aliasname> -file <certificate_file_path> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store --storepass $KEY

      • Note:
        Type yes when prompted to trust the certificate.
        The <aliasname> can be any name.
        Replace <certificat_file_path> with the full path to the certificate file that was uploaded in the step 2a.
   7. Run the below command to import the root certificate into the java trust store:

      keytool -importcert -alias <aliasname> -file <root_certificate_file_path> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

      • Note:
        Type yes when prompted to trust the certificate
        The <aliasname> can be any name.
        Replace <root_certificate_file_path> with the full path to the certificate file that was uploaded in the step 2b.
        
        
   8. Restart SDDC services by using following sddcmanager_restart_services script:

      /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

   9. Verify that the new NSX Manager certificate has been added to the SDDC Manager trust store:

      keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $KEY

Additional Information

For NSX Manager Certificate - 

• Step 6 and 7 need to be run for each of the NSX Manager nodes and NSX Manager VIP. 
• NSX Manager's full certificate chain for each of the NSX Manager nodes and NSX Manager VIP.
• Another certificate file for only the Root certificate. Note: If the same CA is signing all three NSX manager nodes and VIP, only one Root certificate file is needed.
• Scripted process to replace expired or self-signed VMware NSX Manager Certificates with VMCA-Signed Certificates can also be performed