vSphere Client becomes unresponsive after upgrading the vCenter Server to version 8.0 Update 3b (8.0.3.00200)

Products

• VMware vCenter Server 8.0

Issue/Introduction

This issue can display in a variety of symptoms, which will usually occur after an extended period of idleness of the session (1-2 hours)

• After upgrading vCenter Server to 8.0.3.00200, the vSphere Client becomes unstable and session timeouts occur.
• The loading circle indicator will continually spin.
• Browsing the inventory may error with errors such as "You have no privileges to view this object or it is deleted" as follows,

• When attempting to log into the client, the login button may be greyed out.
• Authentication may fail with the error: An error occurred while fetching identity providers

• /var/log/vmware/vsphere-ui/vsphere_client_virgo.log shows below events: 
  [YYYY-MM-DDTHH:MM:SSSS] [WARN ] data-service-pool-#################### com.vmware.opsmgmt.client.alarms.impl.AlarmIssueAdapter  Failed to pre-fetch issue data. com.vmware.vcenter.apigw.exception.ServiceClosedException: AsyncTokenProvider has been closed

• /var/log/vmware/apig/apigw.log shows below events:
  [YYYY-MM-DDTHH:MM:SSSS] [ERROR] p-nio-127.0.0.1-5090-exec-################## c.vmware.vsphere.client.usersession.impl.UserSessionServiceImpl Failed to get the user session com.vmware.vcenter.apigw.exception.ServiceClosedException: AsyncTokenProvider has been closed
        at com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider.newServiceClosedException(Unknown Source)
        at com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider.assertIsRunning(Unknown Source)
        at com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider.getSamlToken(Unknown Source)
        at com.vmware.vcenter.apigw.api.sso.tokenmgmt.NonRenewingMutableTokenProvider.getSamlToken(Unknown Source)
        at com.vmware.vise.vim.commons.sso.AuthSessionUtil.getSsoTokenEx(AuthSessionUtil.java:82)
        at com.vmware.vsphere.client.security.util.Util.getUserSession(Util.java

Environment

vCenter 8.0 U3b

Cause

This issue is due to a change in the default behavior of RECYCLE_FACADES within Tomcat in the release. 

Resolution

This issue is fixed in vCenter Server 8.0U3c release .Refer to vCenter 8.0U3c release notes. Log in to the Broadcom Support Portal to download this patch .

Workaround 

Follow the below steps to disable RECYCLE_FACADES.

1. Login to the vCenter Server with SSH using root credentials.
   
   
2. Create a backup of the catalina.properties file.
   cp /usr/lib/vmware-vsphere-ui/server/conf/catalina.properties /root/catalina.properties.bak
   
   
3. Run the following command to add a line to the end of the original catalina.properties file to disable RECYCLE_FACADES.
   echo "org.apache.catalina.connector.RECYCLE_FACADES=false" >> /usr/lib/vmware-vsphere-ui/server/conf/catalina.properties
   
   
4. Restart the vsphere-ui service with the below command.
   service-control --restart vsphere-ui

If this issue is noticed in higher version hide the third party plugin in vCenter to identify the cause

Steps to Hide Client Plug-ins

1. Log in to the vSphere Client with administrator privileges.
2. From the main menu, navigate to Administration.
3. Under the Solutions section, select Client Plugins.
4. In the list of plugins, locate and select the 3rd-party plugin you wish to hide.
5. Depending on the specific plugin and your vCenter version, look for the HIDE button:
   • Hide: Select the plugin server entry and click HIDE. This removes the plugin's visual extensions from the UI while keeping the plugin service operational in the background  vCenter Server summary / Folder summary on vSphere Client display takes a long time