SDDC Manager Workload domain deployment fails with an error 'Failed to import certificate in vCenter vcenter.example.com trusted root certificates'
search cancel

SDDC Manager Workload domain deployment fails with an error 'Failed to import certificate in vCenter vcenter.example.com trusted root certificates'

book

Article ID: 376040

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

  • Deployment of a workload domain fails with an error 'Failed to import certificate in vCenter vcenter.example.com trusted root certificates'

    FAILED_TO_IMPORT_VC_TRUSTED_ROOT_CERTIFICATE

  • Error in /var/log/vmware/vcf/domainmanager/domainmanager.log

    [YYYY-MM-DDTHH:MM:SS] ERROR [vcf_dm,66c9704f6658f49e2c3222be114ab2c0,ad06] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-10]  [9MAVGA] FAILED_TO_IMPORT_VC_TRUSTED_ROOT_CERTIFICATE Failed to import certificate in vCenter vcenter.example.com trusted root certificates
    com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to import certificate in vcenter.example.com trusted root certificates
            at com.vmware.evo.sddc.vsphere.contract.ImportTrustedRootCertificatesAction.postValidate(ImportTrustedRootCertificatesAction.java:164)
            at com.vmware.evo.sddc.vsphere.contract.ImportTrustedRootCertificatesAction.postValidate(ImportTrustedRootCertificatesAction.java:31)
            at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$1(FsmActionState.java:23)
            at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
    ..
    ..
    Caused by: java.lang.RuntimeException: java.security.cert.CertificateException: java.security.cert.CertificateException: Can't get single X509 certificate from PEM. More then one certificate in PEM.
            at com.vmware.evo.sddc.vsphere.contract.ImportTrustedRootCertificatesAction.lambda$postValidate$2(ImportTrustedRootCertificatesAction.java:119)

Environment

VMware Cloud Foundation 5.x

 

Cause

One of the CA certificate is added as chain of intermediate and root in vCenter's trusted roots store.

Resolution

NOTE: Take the offline snapshot of all the vCenter Servers in ELM (Enhanced Linked Mode). Refer: VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

  1. Identify the certificate from TRUSTED_ROOTS store of mentioned vCenter.
    • SSH to the vCenter and run the below command:

      /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep 'Alias|Entry type|Signature|Issuer|Subject' > certs.txt
    • The above command provides the output where one of the CA certificate is added as chain of intermediate and root in vCenter's trusted roots store. Example output:

      Alias : #######################
      Entry Type:        Trusted Cert
              Signatureignature Algorithm: sha256WithRSAEncryption
              Issuer: CN=Org
              Subject: DC=org, DC=organization, DC=domain, CN=Org
              Subject Public Key Info:
                  X509v3 Subject Key Identifier:
                      Digital Signature, Certificate Sign, CRL Sign
                      CA Issuers - URI:http://<certificateauthorityfqdn>/crl/Org\20Application\20Policy$20Authority-G1.crt
                      CA Issuers - URI: ldap:///CN=Org, CN=###, CN=Public\20Key&20Services, CN=Services, CN=Configuration, DC=domain, DC=organization, DC=org?cACertificate?base?objectClass=certificationAuthority
          Signature Algorithm: sha256WithRSAEncryption
          Signature Value:
              Signature Algorithm: sha256WithRSAEncryption
              Issuer: DC=Org, DC=organization, DC=domain, CN=ROOTCA
              Subject: CN=Org
              Subject Public Key Info:
                  X509v3 Subject Key Identifier: 
                      CA Issuers - URI:http://certificateauthorityfqdn/crl/ROOTCA.crt
                      CA Issuers - URI: ldap:///CN=ROOTCA, CN=AIA, CN=Public\20Key\20Services, CN=Services, CN=Configuration, DC=domain, DC=organization, DC=Org?cACertificate?base?objectClass=certificationAuthority
                      Digital Signature, Certificate Sign, CRL Sign
          Signature Algorithm: sha256WithRSAEncryption
          Signature Value:
              Signature Algorithm: sha256WithRSAEncryption
              Issuer: DC=Org, DC=organization, DC=domain, CN=ROOTCA
              Subject : DC=Org, DC=organization, DC=domain, CN=ROOTCA
              Subject Public Key Info:
                  Digital Signature, Certificate Sign, CRL Sign
              X509v3 Subject Key Identifier:
          Signature Algorithm: sha256WithRSAEncryption
          Signature Value:

    • Take the backup of the certificate using vecs-cli command and save it: 

      /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias  <alias name> --output /certificate/<certificate_name>.crt

    • Unpublish the certificate from TRUSTED_ROOTS as mentioned in the step 4 of the KB: Removing CA Certificates from the TRUSTED_ROOTS store 

    • Create separate certificates (sub-caroot.crt and caroot.crt) using chain of certificate.

    • After deleting the certificates, import the individual certificates created from the chain using below commands:

      /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /certificate/sub-caroot.crt

      /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /certificate/caroot.crt

    • Re run the task of deploying WLD.

 

If the unpublish command fails with an error: dir-cli failed error 13, follow the alternative steps to UNPUBLISH the chain:

  • Export the certificate chain which was erroring in the log file to a single cert file.

    /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <alias_name> --output /certificate/<certificate_name>.crt

  • dir-cli might fail unpublishing chains, break the chain into two individual certificates files (the root + intermediate) using vi.

  • Use below commands to unpublish the two separate certificates:

    /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /certificate/sub_caroot.crt

    /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /certificate/root.crt 

  • Run vecs-cli entry delete command to delete the chain from TRUSTED_ROOTS:

    /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias <alias_name>

  • Run below dir-cli command to publish the two individual certificates:

    /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /certificate/sub_caroot.crt

    /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /certificate/caroot.crt

  • Perform a force refresh of VECS to sync the certificate from VMDIR:

    /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

  • If both the above options fails, then remove the certificate chain using jxplorer with below steps:

    1. Install Jxplorer and connect to vCenter using the steps provided in KB: How to export VMDir information from vCenter Server using the JXplorer utility

      NOTE: vsphere.local is a generic SSO domain name that is commonly used in most environments. If in case your SSO domain name is customized, perform appropriate changes.

      Use below command to confirm your SSO domain name:

      /usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost
       
    2. Once logged in and able to get the LDAP view of vCenter SSO. Follow below click path to access trusted store of vCenter.

      vsphere > Configuration > Certificate-Authorities



    3. Identify the specific certificate chain and right click > Delete.

Additional Information

Note: Jxplorer is not officially supported by Broadcom

  • In case of error similar to below, while trying to unpublish certificates, please follow the modified syntax requirements below:

    /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish -- cert /certificate/root.crt

    Enter password for [email protected]:

    dir-cli failed. Error 2: Possible errors:
    LDAP error: Protocol error
    Win Error: Operation failed with error ERROR_FILE_NOT_FOUND (2)

  • To resolve this, adjust Step 6 of the resolution to include the parameters mentioned below:

    • To unpublish the two separate certificates

      /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert root/certificate/sub_caroot.crt

      /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert root/certificate/root.crt
    • To publish the two individual certificates followed by /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh:

      /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert root/certificate/sub_caroot.crt

      /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert root/certificate/caroot.crt 
Powered by Wolken Software