vCenter login banner "An error occurred while fetching identity providers. Please try again later. If problem persists, contact your administrator."

Products

VMware vCenter Server

Issue/Introduction

vCenter login page reports the following error "An error occurred while fetching identity providers. Please try again later. If problem persists, contact your administrator."

The vCenter Single Sign-On log file is available at /var/log/vmware/sso/websso.log

[YYYY-MM-DDTHH:MM:SS] ERROR sts-default[30:Thread-9] [CorId= OpId=] [com.vmware.identity.util.VcTrustCache] Refresh thread failed to retreive Vctrusts.
java.lang.IllegalStateException: Certificate generation from PEM string failed.[Store: TRUSTED_ROOTS, Server: __localhost__, User: __localuser__]
        at com.vmware.identity.vecs.VecsEntryEnumeration.nextElement(VecsEntryEnumeration.java:55) ~[vmware-endpoint-certificate-store.jar:?]
        at com.vmware.identity.vecs.VecsEntryEnumeration.nextElement(VecsEntryEnumeration.java:12) ~[vmware-endpoint-certificate-store.jar:?]
        at com.vmware.provider.VecsKeyStoreEngine2$CachedStoreEntries.<init>(VecsKeyStoreEngine2.java:478) ~[vmware-endpoint-certificate-store.jar:?]
        at com.vmware.provider.VecsKeyStoreEngine2$StoreManager.getFreshEntriesFromVECS(VecsKeyStoreEngine2.java:417) ~[vmware-endpoint-certificate-store.jar:?]
        at com.vmware.provider.VecsKeyStoreEngine2$StoreManager.snapshotEntries(VecsKeyStoreEngine2.java:383) ~[vmware-endpoint-certificate-store.jar:?]
        at com.vmware.provider.VecsKeyStoreEngine2.engineLoad(VecsKeyStoreEngine2.java:262) ~[vmware-endpoint-certificate-store.jar:?]
        at java.security.KeyStore.load(KeyStore.java:1479) ~[?:1.8.0_422]
        at com.vmware.identity.providers.KeyStoreProviderImpl.loadKeyStore(KeyStoreProviderImpl.java:29) ~[libsamlauthority.jar:?]
        at com.vmware.identity.util.VapiClientConnection.createConnection(VapiClientConnection.java:76) ~[libsamlauthority.jar:?]
        at com.vmware.identity.util.VapiClientConnection.refreshConnection(VapiClientConnection.java:157) ~[libsamlauthority.jar:?]
        at com.vmware.identity.util.VapiClientConnection.invokeStub(VapiClientConnection.java:272) ~[libsamlauthority.jar:?]
        at com.vmware.identity.util.VapiClient.doVcTrustsList(VapiClient.java:51) ~[libsamlauthority.jar:?]
        at com.vmware.identity.util.VcTrustCache.refreshTrustCache(VcTrustCache.java:423) [libsamlauthority.jar:?]
        at com.vmware.identity.util.VcTrustCache$TrustCacheThread.run(VcTrustCache.java:468) [libsamlauthority.jar:?]
Caused by: java.security.cert.CertificateParsingException: java.io.IOException: invalid URI name constraint (should not include scheme):https://example.com

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

The SSO page failed to load due to an invalid certificate in the Trusted Roots Store within VECS. It is important that the required certificates are properly loaded; otherwise, vCenter will be unable to proceed with fetching the configured Identity Sources.

Resolution

To address the issue, identify problematic certificates in the Trusted Roots store and remove the certificate. Below are steps:

Create a snapshot of the vCenter server. If the vCenter is part of an Enhanced Linked Mode (ELM) configuration, ensure that offline snapshots are taken for all vCenters within the ELM setup.

List the certificates using vecs-cli to identify the problematic certificate:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less

Find the certificate that needs to be removed and make a note of the Alias. Example:
```
Alias : 2###################################f
```
Using the Alias ID located in Step 3, run the following command to save the certificate to /root/ folder, adjusting appropriately for the environment.
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias 2###################################f --output /root/<aliasID>.cer
```
Unpublish the expired/expiring CA certificate from VMDIR, it will prompt for SSO Administrator credentials.
```
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /root/<aliasID>.cer
```
Delete the certificate from VECS utilizing the Alias located in Step 3:
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias 2###################################f
```
Notes:
If the alias has special characters, enclose the entire alias in single quotes when using the vecs-cli command for eg. --alias 'https://[IP]:9997/vasa'
If the command is failing with error "Operation failed with error ERROR_OBJECT_NOT_FOUND", ignore the error and proceed further. This error will be logged if the certificate is already removed from the store as part of Step 5.
Perform force refresh of VECS to sync certificate from VMDIR.
```
/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
```
Confirm that the certificate is no longer present.
Note: Output of this command should not be listing the Alias ID that was removed in above steps.
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep Alias
```
Restart all services on the vCenter Servers, ensuring that all services start and respond normally, and that login and management of the environment are functioning properly.
```
service-control --stop --all
service-control --start --all
```

Additional Information

Root certificate expiration alarms in TRUSTED_ROOTS store