SDDC Manager Workload domain deployment fails with an error 'Failed to import certificate in vCenter vcenter.example.com trusted root certificates'

Products

VMware SDDC Manager

Issue/Introduction

Deployment of a workload domain fails with an error 'Failed to import certificate in vCenter vcenter.example.com trusted root certificates'
```
FAILED_TO_IMPORT_VC_TRUSTED_ROOT_CERTIFICATE
```

Error in /var/log/vmware/vcf/domainmanager/domainmanager.log

[YYYY-MM-DDTHH:MM:SS] ERROR [vcf_dm,66c9704f6658f49e2c3222be114ab2c0,ad06] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-10]  [9MAVGA] FAILED_TO_IMPORT_VC_TRUSTED_ROOT_CERTIFICATE Failed to import certificate in vCenter vcenter.example.com trusted root certificates
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to import certificate in vcenter.example.com trusted root certificates
        at com.vmware.evo.sddc.vsphere.contract.ImportTrustedRootCertificatesAction.postValidate(ImportTrustedRootCertificatesAction.java:164)
        at com.vmware.evo.sddc.vsphere.contract.ImportTrustedRootCertificatesAction.postValidate(ImportTrustedRootCertificatesAction.java:31)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$1(FsmActionState.java:23)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
..
..
Caused by: java.lang.RuntimeException: java.security.cert.CertificateException: java.security.cert.CertificateException: Can't get single X509 certificate from PEM. More then one certificate in PEM.
        at com.vmware.evo.sddc.vsphere.contract.ImportTrustedRootCertificatesAction.lambda$postValidate$2(ImportTrustedRootCertificatesAction.java:119)

Environment

VMware Cloud Foundation 5.x

Cause

One of the CA certificate is added as Chain of intermediate and root in vCenter's trusted roots store.

Resolution

Identify the certificate from TRUSTED_ROOTS store of mentioned vCenter.

Take an SSH to the vCenter VM and run the below command:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep 'Alias|Entry type|Signature|Issuer|Subject' > certs.txt

For example, when you run the above command we see the below output where one of the CA certificate is added as Chain of intermediate and root in vCenter's trusted roots store:

Alias : cd99bf3912644dc72]0d85f48ae6a052d2842779c
Entry Type:        Trusted Cert
        Signatureignature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Org Application Policy Authority-G1
        Subject: DC=net, DC=organization, DC=exchad, CN=Org Application Identity Authority-2-G1
        Subject Public Key Info:
            X509v3 Subject Key Identifier:
                Digital Signature, Certificate Sign, CRL Sign
                CA Issuers - URI:http://certificateauthorityfqdn/crl/Org\20Application\20Policy$20Authority-G1.crt
                CA Issuers - URI: ldap:///CN=OrgAuthority-G1, CN=AIA, CN=Public\20Key&20Services, CN=Services, CN=Configuration, DC=exchad, DC=organization, DC=net?cACertificate?base?objectClass=certificationAuthority
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: DC=NET, DC=organization, DC=EXCHAD, CN=ROOTCA
        Subject: CN=Org Application Policy Authority-G1
        Subject Public Key Info:
            X509v3 Subject Key Identifier: 
                CA Issuers - URI:http://certificateauthorityfqdn/crl/ROOTCA.crt
                CA Issuers - URI: ldap:///CN=ROOTCA, CN=AIA, CN=Public\20Key\20Services, CN=Services, CN=Configuration, DC=exchad, DC=organization, DC=net?cACertificate?base?objectClass=certificationAuthority
                Digital Signature, Certificate Sign, CRL Sign
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: DC=NET, DC=organization, DC=EXCHAD, CN=ROOTCA
        Subject : DC=NET, DC=organization, DC=EXCHAD, CN=ROOTCA
        Subject Public Key Info:
            Digital Signature, Certificate Sign, CRL Sign
        X509v3 Subject Key Identifier:
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:

Take the backup of the certificate using vecs-cli command and save the file:

SSH to VCenter VM and run the below command

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias  <alias name> --output /certificate/<certificate usage name>.crt

Take the Power off snapshot of all vCenter Servers in ELM (Enhanced Linked Mode)

Delete / remove this certificate from TRUSTED_ROOTS using below command

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /certificate/<certificate usage name>.crt

Create separate certificates using above chain of certificate.
However it may fail for same reason with an error as
```
dir-cli failed error 13
```
an alternative to UNPUBLISH the chain from above [dir-cli failed error 13]
- export the certificate chain which was erroring in the log file to a single cert file.
# /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <alias_name> --output /certificate/<cert_chain>.crt

- as /usr/lib/vmware-vmafd/bin/dir-cli might have an issue unpublishing chains, break the chain into two individual certificates files (the root + intermediate)
use vi/ nano/ sublime etc

- used /usr/lib/vmware-vmafd/bin/dir-cli to unpublish the two separate certificates
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /certificate/sub_caroot.crt
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /certificate/root.crt

- then use /usr/lib/vmware-vmafd/bin/vecs-cli entry delete to delete the chain from TRUSTED_ROOTS
# /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias <alias name>

- lastly /usr/lib/vmware-vmafd/bin/dir-cli to publish the two individual certificates followed by /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /certificate/sub_caroot.crt
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /certificate/caroot.crt # /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
If it BOTH options from above fail, then remove the certificate chain using jxplorer with below steps.
1. Install Jxplorer and connect to vCenter using the steps provided in KB How to export VMDir information from vCenter Server using the JXplorer utility
  
  NOTE: vsphere.local is a generic sso domain name that is commonly used in most environments.. If in case your sso domain name is customized - you would need to make appropriate changes.
  
  Use below command to confirm your SSO domain name.
```
/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost
```
2. Once logged in and able to get the LDAP view of vC SSO. Follow below click path to access trusted store of vCenter,
  - vsphere >> Configuration >> Certificate-Authorities
3. Identify the specific certificate chain and right click >> Delete.

After deleting, we need to import the individual certificates created from the chain using below commands in vCenter SSH

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /certificate/sub-caroot.crt
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /certificate/caroot.crt

Re run the task of deploying WLD.

Additional Information

Note: Jxplorer is not officially supported by Broadcom