SDDC Manager Workload domain deployment fails with an error 'Failed to import certificate in vCenter vcenter.example.com trusted root certificates'

Products

• VMware SDDC Manager

Issue/Introduction

• Deployment of a workload domain fails with an error 'Failed to import certificate in vCenter vcenter.example.com trusted root certificates'

  FAILED_TO_IMPORT_VC_TRUSTED_ROOT_CERTIFICATE

• Error in /var/log/vmware/vcf/domainmanager/domainmanager.log

  [YYYY-MM-DDTHH:MM:SS] ERROR [vcf_dm,66c9704f6658f49e2c3222be114ab2c0,ad06] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-10]  [9MAVGA] FAILED_TO_IMPORT_VC_TRUSTED_ROOT_CERTIFICATE Failed to import certificate in vCenter vcenter.example.com trusted root certificates
  com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to import certificate in vcenter.example.com trusted root certificates
          at com.vmware.evo.sddc.vsphere.contract.ImportTrustedRootCertificatesAction.postValidate(ImportTrustedRootCertificatesAction.java:164)
          at com.vmware.evo.sddc.vsphere.contract.ImportTrustedRootCertificatesAction.postValidate(ImportTrustedRootCertificatesAction.java:31)
          at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$1(FsmActionState.java:23)
          at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
  ..
  ..
  Caused by: java.lang.RuntimeException: java.security.cert.CertificateException: java.security.cert.CertificateException: Can't get single X509 certificate from PEM. More then one certificate in PEM.
          at com.vmware.evo.sddc.vsphere.contract.ImportTrustedRootCertificatesAction.lambda$postValidate$2(ImportTrustedRootCertificatesAction.java:119)

Environment

VMware Cloud Foundation

Cause

One of the CA certificate is added as Chain of intermediate and root in vCenter's trusted roots store.

Resolution

1. Identify the certificate from TRUSTED_ROOTS store of mentioned vCenter and take the backup of the certificate using vecs-cli command
   • SSH to VCenter VM and run the below command

     /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias  <alias name> --output /certificate/<certificate usage name>.crt

2. Take the Power off snapshot of all vCenter Servers in ELM (Enhanced Linked Mode)
   
   
3. Delete / remove this certificate from TRUSTED_ROOTS using below command

   /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /certificate/<certificate usage name>.crt

4. Create separate certificates using above chain of certificate.
   However it may fail for same reason with an error as

   dir-cli failed error 13

5. an alternative to UNPUBLISH the chain from above  [dir-cli failed error 13]
   - export the certificate chain which was erroring in the log file to a single cert file.
   # /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <alias_name> --output /certificate/<cert_chain>.crt 
   
   - as /usr/lib/vmware-vmafd/bin/dir-cli might have an issue unpublishing chains, break the chain into two individual certificates files (the root + intermediate)
   use vi/ nano/ sublime etc
   
   - used /usr/lib/vmware-vmafd/bin/dir-cli to unpublish the two separate certificates
   # /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /certificate/sub_caroot.crt
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /certificate/root.crt 
   
   - then use /usr/lib/vmware-vmafd/bin/vecs-cli entry delete to delete the chain from TRUSTED_ROOTS
   # /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias <alias name>
   
   - lastly /usr/lib/vmware-vmafd/bin/dir-cli to publish the two individual certificates followed by /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
   # /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /certificate/sub_caroot.crt
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /certificate/caroot.crt

# /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
   
   
6. If it BOTH options from above fail, then remove the certificate chain using jxplorer with below steps.
   1. Install Jxplorer and connect to vCenter using the steps provided in KB How to export VMDir information from vCenter Server using the JXplorer utility
      
      NOTE: vsphere.local is a generic sso domain name that is commonly used in most environments.. If in case your sso domain name is customized - you would need to make appropriate changes.
      
      Use below command to confirm your SSO domain name.

      /usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost

       
   2. Once logged in and able to get the LDAP view of vC SSO. Follow below click path to access trusted store of vCenter,
      • vsphere >> Configuration >> Certificate-Authorities
        
   3. Identify the specific certificate chain and right click >> Delete.
7. After deleting, we need to import the individual certificates created from the chain using below commands in vCenter SSH

   /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /certificate/sub-caroot.crt
   /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /certificate/caroot.crt

8. Re run the task of deploying WLD.