"Error [500] SSO error: com.vmware.vim.vmomi.core.exception...." occurs when VMware VAPI Endpoint service fails to start

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

When trying to connect to vCenter Server using the web client, you may receive a similar error:

[500] SSO error: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint doesn't match
Check the vSphere Web Client server logs for details.

When attempting to check the status of services on VC, you may notice that the vapiEndpoint service is not running.

service-control --status --all
Running:
 EsxAgentManager VMWareAfdService VMWareCertificateService VMWareDirectoryService VMwareComponentManager VMwareDNSService VMwareIdentityMgmtService
onapi vmsyslogcollector vmware-cis-config vmware-license vmware-perfcharts vmware-psc-client vmwareServiceControlAgent vpxd vpxd-svcs vsan-health 
Stopped:
 VMWareCAMService content-library mbcs vapiEndpoint vmware-autodeploy-waiter vmware-imagebuilder vmware-network-coredump

In the endpoint.log you may see similar entries:

YYYY-MM-DDTHH:MM:SS.085-06:00 | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.
com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.
    at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:60)
    at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:349)
    at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:176)
    at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.vmware.vim.sso.admin.exception.CertificateValidationException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint verification is not configured
virgo.logs
YYYY-MM-DDTHH:MM:SS.824-06:00] [INFO ] bio-127.0.0.1-9090-exec-2941 70133207 162400 262092 com.vmware.vise.util.session.SessionUtil                          ClientId 262099 was automatically generated 
[YYYY-MM-DDTHH:MM:SS.824-06:00] [WARN ] bio-127.0.0.1-9090-exec-2941 70133207 162400 262092 com.vmware.vise.util.session.SessionUtil                          Just clobbered an existing httpRequest 
[YYYY-MM-DDTHH:MM:SS.860-06:00] [INFO ] bio-127.0.0.1-9090-exec-2941 70133216 162408 262099 com.vmware.vise.vim.cm.impl.ComponentManagerServiceImpl           Retry won't be attempted for CM https://host.example.com/cm/sdk?hostid=########-####-####-####-########f4d7 for error: com.vmware.vim.vmomi.client.exception.SslException - com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint doesn't match 
[YYYY-MM-DDTHH:MM:SS.889-06:00] [WARN ] bio-127.0.0.1-9090-exec-2941 70133216 162408 262099 com.vmware.vapi.provider.local.LocalProvider                      Implementation method reported unexpected exception: com.vmware.vim.vmomi.client.exception.SslException com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint doesn't match
    at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:250)
    at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
    at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
    at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 6.7.x

Resolution

To resolve the issue, follow below steps:

Open Command Prompt as Administrator.

vCenter Server installed on Windows : cd C:\Program Files\VMware\vCenter Server\vmafdd
vCenter Server Appliance: cd /usr/lib/vmware-vmafd/bin/

Export Trusted Root CRLS to a text file:

vCenter Server installed on Windows :

vecs-cli.exe entry list --store TRUSTED_ROOT_CRLS  | findstr Alias > C:\trusted_root_crl.txt

vCenter Server Appliance:

./vecs-cli entry list --store TRUSTED_ROOT_CRLS |grep Alias > /tmp/trusted_root_crl.txt

Open the trusted_root_crl.txt and Replace "Alias" with "vecs-cli entry delete --store TRUSTED_ROOT_CRLS -y --alias"

Delete all the Alias in this CRLS store using the command:

vCenter Server installed on Windows :

vecs-cli.exe entry delete --store TRUSTED_ROOT_CRLS --Alias <Certificate_Alias> -y  (Here Certificate_Alias is the Alias of each certificate returned by above list command)

vCenter Server Appliance:

./vecs-cli entry delete --store TRUSTED_ROOT_CRLS --Alias <Certificate_Alias> -y

Note: Certificate_Alias is the alias of each certificate returned by the above list command.

Example: ./vecs-cli entry delete --store TRUSTED_ROOT_CRLS -y --alias ########################################

Restart VapiEndpoint Service on vCenter server

service-control --stop vapiEndpoint
service-control --start vapiEndpoint

You can also run the scripts attached under the attachments section in this article for;

Windows: remove_clrs
Appliance: clr-fix

Note: If there are expired Certificates within the BACKUP_STORES please refer to Clearing BACKUP_STORES certificates in the VCSA via shell script

Attachments

remove_clrs get_app

crl-fix get_app