SDDC Manager fails to trust NSX Manager certificate with error "Failed to trust Certificate(s). Error while validating certificate"

Products

• VMware SDDC Manager

Issue/Introduction

• NSX Manager certificate status has an alert with message: "Connection cannot be established because certificate is not trusted".
  
  
  
  
• Upon selecting "review" option to attempt to trust the NSX Manager certificate, the certificate validation fails with error: "Failed to trust Certificate(s). Error while validating certificate"
• NSX Manager certificate had been renewed directly from the NSX Manager by importing the Custom CA signed certificate.
• Operations Manager log (/var/log/vmware/vcf/operationsmanager/operationsmanager.log) 

YYYY-MM-DDTHH:MM DEBUG [vcf_om,68ae############################,cf##] [c.v.v.s.t.DynamicTrustManager,##-###-##] Error checking certificate chain CN=<NSX_Manager1a_FQDN>, OU=<Org_Unit>, O=<Organization>, L=<Locality>, ST=<State>, C=<Country>,CN=<Custom_CA_Name>, DC=admin, DC=ad, DC=<Domain_name>, DC=<com>,CN=<Domain> Root Certificate Authority, O=<Organization>, L=<Locality>, ST=<State>, DC=<Domain>, DC=<com>, C=<Country>.
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:242)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:113)

Environment

• VCF 4.x
• VCF 5.x

Cause

• SDDC Manager is still referencing the old NSX certificate chain in its database as the NSX certificate was not renewed/imported from the SDDC Manager. 

Resolution

• The following steps can be used to update the SDDC Manager trust store with the new certificate in the NSX managers.
• Step 6 and 7 need to be run for each of the NSX Manager nodes and NSX Manager VIP. 

1. 1. Take Snapshot of SDDC Manager Appliance without virtual machine memory.
   2. Use a file transfer utility to copy the following certificates to the /tmp directory on the SDDC Manager Appliance.
      1. NSX Manager's full certificate chain for each of the NSX Manager nodes and NSX Manager VIP.
      2. Another certificate file for only the Root certificate. Note: If the same CA is signing all three NSX manager nodes and VIP, only one Root certificate file is needed.
   3. SSH to the SDDC Manager Appliance as the vcf user and then issue the su - command to switch to the root user.
   4. Obtain the trusted certificates key by issuing the following command

      KEY=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)

   5. Confirm the output by using the below command:

      echo $KEY

   6. Run the below command to import certificate for each of the NSX Managers into the SDDC Manager trust store:

      keytool -importcert -alias <aliasname> -file <certificate_file_path> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store --storepass $KEY

      • Note:
        Type yes when prompted to trust the certificate.
        The <aliasname> can be any name.
        Replace <certificat_file_path> with the full path to the certificate file that was uploaded in the step 2a.
   7. Run the below command to import the root certificate into the java trust store:

      keytool -importcert -alias <aliasname> -file <root_certificate_file_path> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

      • Note:
        Type yes when prompted to trust the certificate
        The <aliasname> can be any name.
        Replace <root_certificate_file_path> with the full path to the certificate file that was uploaded in the step 2b.
        
        
   8. Restart SDDC services by using following sddcmanager_restart_services script:

      /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

   9. Verify that the new NSX Manager certificate has been added to the SDDC Manager trust store:

      keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $KEY