Error "[500] An error occurred while fetching identity providers" appears while login to vCenter Server UI

Products

• VMware vCenter Server

Issue/Introduction

Symptoms

•  vCenter Server UI fails with the following error: "[500] An error occurred while fetching identity providers. Try again. If problem persists, contact your administrator."
• In environments using Enhanced Linked Mode (ELM), this error may impact all linked vCenter nodes.
• The following entries appear in logs:

In the /var/log/vmware/vsphere_ui/logs/vsphere_client_virgo.log: 

[ERROR] http-nio-5090-exec-4 com.vmware.vsphere.client.security.oauth2.LoginRequestHandler An error occurred while fetching providers com.vmware.vapi.std.errors.Unauthenticated

In the /var/log/vmware/sso/websso.log:

ERROR websso[63:tomcat-http--16] [CorId=#### [com.vmware.identity.providers.SolutionUserHokTokenProviderImpl] Unable to get SAML HOK token for machine solution user com.vmware.identity.saml.UnsupportedTokenLifetimeException: Signing certificate is not valid at ## ## ## ##:##:## GMT ####

In the /var/log/vmware/sso/vmware-identity-sts.log:

com.vmware.identity.sts.InvalidCredentialsException: Solution user cert is not valid
at org.apache.catalina.core.ApplicationFilterChain. internalDoFilter(ApplicationFilterChain. java: 168) [catalina. jar:9.0.98]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain. java: 144) [catalina. jar:9.0.98]

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x

Cause

Expiration of the Security Token Service (STS) signing certificate, Machine_SSL certificate, or Solution User certificates. In ELM environments, expiration on one node can disrupt the identity provider service across the entire SSO domain.

Resolution

1. Log in to the vCenter Server Appliance via SSH and login as a root user.
2. Verify certificate expiration status by running the following command on the vCenter Server Appliance: for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done 
3. Replace the expired certificates using the vCert - Scripted vCenter expired certificate replacement
4. In Enhanced Linked Mode (ELM), take an offline snapshot of vCenter(s) & perform the certificate replacement on the impacted node(s).
5. Restart all services on all vCenter nodes in the ELM environment to ensure the new certificates are recognized: service-control --stop --all && service-control --start --all