"Error occurred while fetching trusted root certificates" occurs while reviewing Trusted Root certificates using the vSphere Client
search cancel

"Error occurred while fetching trusted root certificates" occurs while reviewing Trusted Root certificates using the vSphere Client

book

Article ID: 320771

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

This article provides a way to unpublish and re-publish custom certificates in order to resolve the error in the vSphere Client.

Symptoms:
  • Navigating to vCenter > Menu > Administration > Certificate Management results in an error:
 Error occurred while fetching trusted root certificates
  • As a result, trusted root certificates cannot be viewed in using vSphere Client.
  • The vSphere Client log (/var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log) contains the following entries: 
    [...]
[2021-09-09T14:53:07.184Z] [ERROR] http-nio-5090-exec-124        com.vmware.certificates.util.CertificateHelper                    Exception occurred while generating certificate:  java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Illegal header: -----BEGIN CERTIFICATE-----
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:115)
        at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
        at com.vmware.certificates.util.CertificateHelper.generateCertificate(CertificateHelper.java:115)
        at com.vmware.certificates.util.CertificateHelper.chainToVecsEntry(CertificateHelper.java:53)
        [...]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Illegal header: -----BEGIN CERTIFICATE-----
        at sun.security.provider.X509Factory.checkHeaderFooter(X509Factory.java:651)
        at sun.security.provider.X509Factory.readOneBlock(X509Factory.java:641)
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:99)
        ... 205 common frames omitted

[2021-09-09T14:53:07.188Z] [ERROR] http-nio-5090-exec-124        com.vmware.vise.mvc.exception.GlobalExceptionHandler              Exception handled while processing request for /ui/certificate-ui/ctrl/certificates/trusted-root-list?endPoint=<vCSA_FQDN>:  com.vmware.certificates.exception.CertificateUiException: Unable to proceed due to certificate exception: Could not parse certificate: java.io.IOException: Illegal header: -----BEGIN CERTIFICATE-----
        at com.vmware.certificates.util.CertificateHelper.generateCertificate(CertificateHelper.java:119)
        at com.vmware.certificates.util.CertificateHelper.chainToVecsEntry(CertificateHelper.java:53)
        at com.vmware.certificates.client.controllers.VcCertificateController.getTrustedRoots(VcCertificateController.java:148)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        [...]


Environment

VMware vCenter Server 7.0.x

Cause

This issue is caused by a "space" character in the certificate header for one of the certificates within the vCenter VECS  (vCenter Endpoint Certificate Store). The certificate header should be "-----BEGIN CERTIFICATE-----" without any spaces or other characters before or after.

Examples: 
" -----BEGIN CERTIFICATE-----"
"-----BEGIN CERTIFICATE----- "
" -----BEGIN CERTIFICATE----- "

In most cases, the faulty certificate was previously added to the vCenter using vSphere Client.

Resolution

Use the workaround to unpublish and re-publish the trusted root certificates.

Workaround:
To resolve the issue, you will need to unpublish and re-publish the custom certificates from VMDIR. To unpublish the certificates from VMDIR, you need the certificate files from the TRUSTED_ROOTS VECS store.
Before proceeding, ensure the customer has a valid backup from the vCenter.

1. Connect to the vCenter per SSH
2. Review the TRUSTED_ROOTS certificates: 
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep "Alias|Issuer:|Not Before:|Not After :|Subject:|CA:"
3. The Output will look similar to this: 
Alias : 21eb0eff363fd06c8f04132719ef3f7b29856fb0
        Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=vcsa.example.org, OU=VMware Engineering
            Not Before: Jul 11 14:52:55 2021 GMT
            Not After : Jul  9 14:52:55 2031 GMT
        Subject: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=vcsa.example.org, OU=VMware Engineering
                CA:TRUE, pathlen:0
Alias : 97296d7c3d0d8ee0e0d78146a11180f3de78b592
        Issuer: C=DE, O=Production GmbH, CN=Production Root CA 1
            Not Before: Apr 28 07:49:22 2017 GMT
            Not After : Apr 28 07:59:17 2057 GMT
        Subject: C=DE, O=Production GmbH, CN=Production Root CA 1
                CA:TRUE
## Usually, all certificates that do not contain "VMware Engineering" are custom certificates.

4. Export the custom certificates: 
# /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <<ALIAS>> > /tmp/<<ALIAS>>.crt
5. Once exported, unpublish the certificate and republish it: 
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /tmp/<<ALIAS>>.crt --login administrator
## Confirm with local SSO admin password

6. Now, publish the same certificate again: 
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /tmp/<<ALIAS>>.crt --login administrator
## Confirm with local SSO admin password

7. Repeat steps 4 - 6 for every custom certificate.
8. Verify if the trusted root certificates can now be viewed in vSphere Client.


Powered by Wolken Software