Couldn't establish a connection to the VM web console from vSphere Client

Products

VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

Unable to launch the console for VMs from the vCenter server it gives a blank page with error: Couldn't establish a connection to the VM web console

Able to open RDP sessions to VMs and also can launch the VM console from the ESXi host UI login

There are 2 possible scenarios to encountering this issue.

Scenario A:

Observe Global.Proxy privilege missing error within log:

/var/log/vmware/rhttpproxy/rhttpproxy-xx.log

[YYYY-MM-DDTHH:MM:SS] error rhttpproxy[03983] [Originator@6876 sub=RhttpProxy] [Rhttpproxy JWT] Missing privilege! Global.Proxy is required.
[YYYY-MM-DDTHH:MM:SS] error rhttpproxy[03983] [Originator@6876 sub=RhttpProxy] [Rhttpproxy REST PUT Handler] JWT verification failed
[YYYY-MM-DDTHH:MM:SS] error rhttpproxy[03947] [Originator@6876 sub=RhttpProxy] [Rhttpproxy JWT] Missing privilege! Global.Proxy is required.
[YYYY-MM-DDTHH:MM:SS] error rhttpproxy[03947] [Originator@6876 sub=RhttpProxy] [Rhttpproxy REST PUT Handler] JWT verification failed

/var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log

[YYYY-MM-DDTHH:MM:SS] [ERROR] -nio-127.0.0.1-5090-exec-997 70148618 103212 200173 c.v.v.r.restclient.impl.EnvoyVapiRequestExecutorServiceImpl Couldn't execute request to reverse proxy REST API. Known eTag = 0 java.lang.RuntimeException: Route wasn't added to any listeners.
        at com.vmware.vise.vim.messaging.webconsole.WebconsoleRequestHandler.lambda$handleRequest$0(WebconsoleRequestHandler.java:246)

Scenario B:

/var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log shows error related to Token is in the past:

[YYYY-MM-DDTHH:MM:SS]  [ERROR] nio-127.0.0.1-5090-exec-3672 70124732 109469 200245 c.v.v.r.restclient.impl.EnvoyVapiRequestExecutorServiceImpl       Error obtaining JWT for the vsphere-ui service principal. com.vmware.vcenter.tokenservice.InvalidGrant: InvalidGrant (com.vmware.vcenter.tokenservice.invalid_grant) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.vcenter.tokenservice.exceptions.InvalidGrant,
    defaultMessage = Invalid SUBJECT token: tokenType=SAML2,
    args = [],
    params = <null>,
    localized = <null>
}, LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.identity.saml.InvalidTokenException,
    defaultMessage = Token expiration date: Fri Nov 01 15:19:08 GMT 2024 is in the past.,

/var/log/vmware/sso/tokenservice.log shows similar error that the token is in the past:

[YYYY-MM-DDTHH:MM:SS] INFO tokenservice[61:tomcat-http--23] [CorId=####### OpId=##] [com.vmware.identity.token.impl.SamlTokenImpl] Token expiration date: Fri Nov 08 03:36:46 GMT 2024 is in the past.
[YYYY-MM-DDTHH:MM:SS] ERROR tokenservice[61:tomcat-http--23] [CorId=####### OpId=##] [com.vmware.vcenter.tokenservice.vapi.TokenExchangeProviderImpl] Exchange failed due to invalid grant:
com.vmware.vcenter.tokenservice.exceptions.InvalidGrant: Invalid SUBJECT token: tokenType=SAML2

Environment

vCenter Server 7.x

vCenter Server 8.x

Cause

Scenario A:

Global.Proxy privilege is not enabled for the role named "vSphere Client Service Account".

Scenario B:

Token trustworthiness clock tolerance is set to a larger value from default. The default value is 600000 milliseconds

The vsphere client gets a solution user token from the SSO service when the vsphere-ui service starts. If this value is set to, as an example 600,000 seconds instead of milliseconds, the service does not take this value into account and thus the token can expire.

Resolution

Scenario A:

Enable the missing Global.Proxy privilege for the "vSphere Client Service Account" role by following the steps below to resolve the VM console issue.

Open the browser and login to the vCenter Server webclient using SSO Administrator eg: [email protected]
Navigate to Administration >Access Control > Roles
Select vSphere Client Service Account and click on edit.
Navigate to the Global section and select the Proxy option as mentioned below:

After enabling the privilege, we will be able to launch the VM console from the vCenter Server UI.

Scenario B:

Within vsphere client, modify the Token Trustworthiness Clock Tolerance value to default setting of 600000 milliseconds.

Launch vSphere Client
Menu > Administration
Under Single Sign On > Configuration
Choose Local accounts tab.
Under Token Trustworthiness, Click Edit
Change Clock Tolerance to default value of 600000 milliseconds.
Restart vsphere-ui service
- Open an SSH session to the vCenter appliance
- Once logged in run the command service-control --restart vsphere-ui

The issue in Scenario B is also observed in environments with default value of 600000 milliseconds because of mismatch in the clock tolerances between the SSO Service and the Token Service. This is resolved in VMware vCenter Server 8.0 Update 3e (Build 24674346)

Note: In some cases, a similar issue to Scenario B may be observed in environments that have experienced a recent network outage. A network disruption can cause a temporary state change where the vsphere-ui service's token becomes invalid or expires prematurely, even if the token clock tolerance is correctly set.

Resolution for Outage-Related Issues:

If you are experiencing these symptoms after a network outage, first attempt to resolve the issue by restarting the vsphere-ui service.

Open an SSH session to the vCenter appliance.
To restart only the vsphere-ui service, run the command:

# service-control --stop vsphere-ui

# service-control --start vsphere-ui

If the problem persists, proceed with a full restart of all vCenter services. To restart all vCenter services, run the command:

# service-control --stop --all && service-control --start --all