"Error occurred while fetching trusted root certificates" occurs while reviewing Trusted Root certificates using the vSphere Client
search cancel

"Error occurred while fetching trusted root certificates" occurs while reviewing Trusted Root certificates using the vSphere Client

book

Article ID: 320771

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

This article provides a way to unpublish and re-publish custom certificates in order to resolve the error in the vSphere Client:

Navigating to vCenter > Menu > Administration > Certificate Management results in an error:

 Error occurred while fetching trusted root certificates

As a result, trusted root certificates cannot be viewed in using vSphere Client.

The vSphere Client log (/var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log) contains the following entries:

[...]  
[YYYY-MM-DDTHH:MM:SS][ERROR] http-nio-5090-exec-124 com.vmware.certificates.util.CertificateHelper Exception occurred while generating certificate:java.security.cert.CertificateException: Could not parse certificate:java.io.IOException: Illegal header: -----BEGIN CERTIFICATE-----
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:115)
        at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
        at com.vmware.certificates.util.CertificateHelper.generateCertificate(CertificateHelper.java:115)
        at com.vmware.certificates.util.CertificateHelper.chainToVecsEntry(CertificateHelper.java:53)
        [...]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Illegal header: -----BEGIN CERTIFICATE-----
        at sun.security.provider.X509Factory.checkHeaderFooter(X509Factory.java:651)
        at sun.security.provider.X509Factory.readOneBlock(X509Factory.java:641)
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:99)
        ... 205 common frames omitted

[YYYY-MM-DDTHH:MM:SS][ERROR] http-nio-5090-exec-124 com.vmware.vise.mvc.exception.GlobalExceptionHandler Exception handled while processing request for /ui/certificate-ui/ctrl/certificates/trusted-root-list?endPoint=<vCSA_FQDN>:  com.vmware.certificates.exception.CertificateUiException: Unable to proceed due to certificate exception: Could not parse certificate: java.io.IOException: Illegal header: -----BEGIN CERTIFICATE-----
        at com.vmware.certificates.util.CertificateHelper.generateCertificate(CertificateHelper.java:119)
        at com.vmware.certificates.util.CertificateHelper.chainToVecsEntry(CertificateHelper.java:53)
        at com.vmware.certificates.client.controllers.VcCertificateController.getTrustedRoots(VcCertificateController.java:148)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        [...]



Environment

VMware vCenter Server

VMware vSphere ESXi

Cause

This issue is caused by a "space" character in the certificate header for one of the certificates within the vCenter VECS  (vCenter Endpoint Certificate Store). The certificate header should be "-----BEGIN CERTIFICATE-----" without any spaces or other characters before or after.

Examples:

" -----BEGIN CERTIFICATE-----"
"-----BEGIN CERTIFICATE----- "
" -----BEGIN CERTIFICATE----- "


In most cases, the faulty certificate was previously added to the vCenter using vSphere Client.

Resolution

Use the workaround to unpublish and re-publish the trusted root certificates.

Workaround:

To resolve the issue, you will need to unpublish and re-publish the custom certificates from VMDIR. To unpublish the certificates from VMDIR, you need the certificate files from the TRUSTED_ROOTS VECS store.
Before proceeding, ensure the customer has a valid backup from the vCenter.


1. Connect to the vCenter per SSH
2. Review the TRUSTED_ROOTS certificates: 
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep "Alias|Issuer:|Not Before:|Not After :|Subject:|CA:"
3. The Output will look similar to this: 
Alias : #######-#####-#######-#####
        Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=vcsa.example.org, OU=VMware Engineering
            Not Before: [YYYY-MM-DDTHH:MM:SS]
            Not After : [YYYY-MM-DDTHH:MM:SS]
        Subject: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=vcsa.example.org, OU=VMware Engineering
                CA:TRUE, pathlen:0
Alias : #######-#####-#######-#####
        Issuer: C=DE, O=Production GmbH, CN=Production Root CA 1
            Not Before: [YYYY-MM-DDTHH:MM:SS]
            Not After : [YYYY-MM-DDTHH:MM:SS]
        Subject: C=DE, O=Production GmbH, CN=Production Root CA 1
                CA:TRUE
## Usually, all certificates that do not contain "VMware Engineering" are custom certificates.

4. Export the custom certificates: 
# /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <<ALIAS>> > /tmp/<<ALIAS>>.crt
5. Once exported, unpublish the certificate and republish it: 
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /tmp/<<ALIAS>>.crt --login administrator
## Confirm with local SSO admin password

6. Now, publish the same certificate again: 
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /tmp/<<ALIAS>>.crt --login administrator
## Confirm with local SSO admin password

7. Repeat steps 4 - 6 for every custom certificate.
8. Verify if the trusted root certificates can now be viewed in vSphere Client.



Powered by Wolken Software