vSphere Client becomes unresponsive after upgrading the vCenter Server to version 8.0 Update 3b (8.0.3.00200)
search cancel

vSphere Client becomes unresponsive after upgrading the vCenter Server to version 8.0 Update 3b (8.0.3.00200)

book

Article ID: 377734

calendar_today

Updated On:

Products

VMware vCenter Server 8.0

Issue/Introduction

This issue can display in a variety of symptoms, which will usually occur after an extended period of idleness of the session (1-2 hours)

  • After upgrading vCenter Server to 8.0.3.00200, the vSphere Client becomes unstable and session timeouts occur.
  • The loading circle indicator will continually spin.
  • Browsing the inventory may error with errors such as "You have no privileges to view this object or it is deleted" as follows,

  • When attempting to log into the client, the login button may be greyed out.
  • Authentication may fail with the error: An error occurred while fetching identity providers

  • /var/log/vmware/vsphere-ui/vsphere_client_virgo.log shows below events: 

    [YYYY-MM-DDTHH:MM:SSSS] [WARN ] data-service-pool-#################### com.vmware.opsmgmt.client.alarms.impl.AlarmIssueAdapter           Failed to pre-fetch issue data. com.vmware.vcenter.apigw.exception.ServiceClosedException: AsyncTokenProvider has been closed
        at com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider.newServiceClosedException(Unknown Source)
        at com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider.assertIsRunning(Unknown Source)
        at com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider.getSamlToken(Unknown Source)
        at com.vmware.vcenter.apigw.api.sso.tokenmgmt.NonRenewingMutableTokenProvider.getSamlToken(Unknown Source)
        at com.vmware.vise.vim.commons.sso.AuthSessionUtil.getSsoTokenEx(AuthSessionUtil.java:114)
        at com.vmware.vise.data.adapters.core.DataServiceCoreAdapter.requireTokenInSession(DataServiceCoreAdapter.java:391)
        at com.vmware.vise.data.adapters.core.DataServiceCoreAdapter.getAdapterForCurrentSession(DataServiceCoreAdapter.java:214)
        at com.vmware.vise.data.adapters.core.DataServiceCoreAdapter.getData(DataServiceCoreAdapter.java:207)
        at com.vmware.vise.data.query.impl.DataAdapterUtil.getDataFromAdapter(DataAdapterUtil.java:219)
        at com.vmware.vise.data.query.impl.DataAdapterUtil.getResultFromAdapter(DataAdapterUtil.java:176)
        at com.vmware.vise.data.query.impl.DataAdapterUtil.access$000(DataAdapterUtil.java:42)
        at com.vmware.vise.data.query.impl.DataAdapterUtil$1.call(DataAdapterUtil.java:82)
        at com.vmware.vise.data.query.impl.DataAdapterUtil$1.call(DataAdapterUtil.java:79)
        at com.vmware.vise.util.concurrent.ExecutorUtil$2.call(ExecutorUtil.java:826)
        at com.vmware.vise.util.concurrent.ExecutorUtil$ThreadContextPropagatingTask.call(ExecutorUtil.java:1240)
        at io.opentelemetry.context.Context.lambda$wrap$2(Context.java:224)
        at com.vmware.vise.data.query.impl.DataServiceThreadPoolDecorator$1.call(DataServiceThreadPoolDecorator.java:192)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
...
[YYYY-MM-DDTHH:MM:SSSS] [ERROR] p-nio-127.0.0.1-5090-exec-################## c.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl Failed to get the user session com.vmware.vcenter.apigw.exception.ServiceClosedException: AsyncTokenProvider has been closed
        at com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider.newServiceClosedException(Unknown Source)
        at com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider.assertIsRunning(Unknown Source)
        at com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider.getSamlToken(Unknown Source)
        at com.vmware.vcenter.apigw.api.sso.tokenmgmt.NonRenewingMutableTokenProvider.getSamlToken(Unknown Source)
        at com.vmware.vise.vim.commons.sso.AuthSessionUtil.getSsoTokenEx(AuthSessionUtil.java:82)
        at com.vmware.vsphere.client.security.util.Util.getUserSession(Util.java:80)

  • /var/log/vmware/apig/apigw.log shows below events: 

    [YYYY-MM-DDTHH:MM:SSSS] [INFO ] agw-fe-session-maint-1 FrontendSessionManagerImpl [] Invalidating session 200029. Top elements from stack trace:
        at com.vmware.vcenter.apigw.session.frontend.impl.FrontendSessionManagerImpl.doInvalidate(Unknown Source)
        at com.vmware.vcenter.apigw.session.frontend.impl.FrontendSessionManagerImpl.removeExpired(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[YYYY-MM-DDTHH:MM:SSSS] [INFO ] agw-fe-session-maint-1 FrontendSessionManagerImpl [] Closing API GW Session due to logout() request: 200029
[YYYY-MM-DDTHH:MM:SSSS] [INFO ] agw-fe-session-maint-1 PseudoSessionManagerImpl [] Auto-closed pseudo-sessions for dying session '200029' : [p00026, p00029, p00025, p00028]
[YYYY-MM-DDTHH:MM:SSSS] [INFO ] agw-fe-session-maint-1 NotificationServiceImpl [] No subscriptions to unsubscribe for frontend session ####################################
[YYYY-MM-DDTHH:MM:SSSS] [INFO ] agw-fe-session-maint-1 AsyncTokenProvider [] Closing com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider

Environment

vCenter Server 8.0.3.00200 build 24262322

Cause

This issue is due to a change in the default behavior of RECYCLE_FACADES within Tomcat in the release. 

Resolution

This issue is fixed in vCenter Server 8.0U3c release - Release Notes Link 

To work around this issue, use the steps below to disable RECYCLE_FACADES.

  1. Login to the vCenter Server with SSH using root credentials.

  2. Create a backup of the catalina.properties file.

    cp /usr/lib/vmware-vsphere-ui/server/conf/catalina.properties /root/catalina.properties.bak


  3. Run the following command to add a line to the end of the original catalina.properties file to disable RECYCLE_FACADES.

    echo "org.apache.catalina.connector.RECYCLE_FACADES=false" >> /usr/lib/vmware-vsphere-ui/server/conf/catalina.properties


  4. Restart the vsphere-ui service with the below command.

    service-control --restart vsphere-ui
Powered by Wolken Software