AD users are unable to login into vCenter Servers. Failed with error: Cannot connect to vCenter Single Sign-On server https://FQDN/sts/STSService/vsphere.local

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

AD users are unable to login into vCenter Servers. 

Error: Cannot connect to vCenter Single Sign-On server https://FQDN/sts/STSService/vsphere.local

In /var/log/vmware/vsphere-ui/logs/apigw.log we see entries similar to:

[YYYY-MM-DDTHH:MM:SS.995Z] [WARN ] agw-token-acq22              ######## ###### 200029 AsyncTokenProvider [] Token acquisition took too long: 31568 ms
[YYYY-MM-DDTHH:MM:SS.462Z] [WARN ] agw-token-acq23              ######## ###### 200030 AsyncTokenProvider [] Token acquisition took too long: 37028 ms
[YYYY-MM-DDTHH:MM:SS8Z] [WARN ] agw-token-acq24              ######## ###### 200031 AsyncTokenProvider [] Token acquisition took too long: 31359 ms

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 7.0.3

Cause

This happens because of slow token acquisition from STS service. The STS service is unable to return the token in 30 seconds.

Resolution

The issue has been resolved in vCenter Server version 7.0 P05 (U3d) and later. The fix involves an increased default timeout for token acquisition from the STS service.

Workaround:
In cases where patching the vCenter Server is not possible, the workaround for this issue is to extend the token acquisition timeout from the default 30 seconds to 2 minutes. The steps for implementing this workaround are as follows:

• Take a snapshot of the vCenter server.
• Navigate to the following directory:
  • cd /usr/lib/vmware-vsphere-ui/plugin-packages/cis-data-service-plugin/plugins
• Back up the original file to another directory. For example:
  • cp api-gateway-server.war ~
• Extract the contents of the api-gateway-server.war file:
  • unzip api-gateway-server.war
• Open the bundle-context.xml file using a text editor:
  • vi WEB-INF/spring/bundle-context.xml
• Navigate to the line containing the string "tokenAcquisitionTimeout" (Press '/' type "tokenAcquisitionTimeout" and press 'Return')
  • You should see content like this:
    <constructor-arg index="6" value="30" /> <!-- tokenAcquisitionTimeout -->
<constructor-arg index="7" value="SECONDS" /> <!-- timeUnitForTokenAcquisitionTimeout -->
  • Modify the values to increase the timeout (Press `i` and then type the new values).
    For example:
    <constructor-arg index="6" value="2" /> <!-- tokenAcquisitionTimeout -->
<constructor-arg index="7" value="MINUTES" /> <!-- timeUnitForTokenAcquisitionTimeout -->
• Save the modifications. (Press 'Esc' then type ":x" and then press 'Return')
• Remove the api-gateway-server.war file:
  • rm api-gateway-server.war
• Repackage the api-gateway-server.war file:
  • zip -rm api-gateway-server.war META-INF WEB-INF
• Set the appropriate file permissions on the api-gateway-server.war file:
  • chmod 755 api-gateway-server.war
• Restart the vSphere UI service:
  • service-control --restart vsphere-ui
• Validate the behavior to ensure the changes were applied successfully.

------------------------------------------------------------------------------------------------------------

Note: It is very important the the file permissions of api-gateway-server.war are 755 (rwxr-xr-x). If the permissions are incorrect, the login functionality will not work.