AD users are unable to login into vCenter Servers. Failed with error: Cannot connect to vCenter Single Sign-On server https://FQDN/sts/STSService/vsphere.local

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
AD users are unable to login into vCenter Servers. 
Error: Cannot connect to vCenter Single Sign-On server https://FQDN/sts/STSService/vsphere.local

In /var/log/vmware/vsphere-ui/logs/apigw.log we see entries similar to:

[2022-06-23T13:06:02.995Z] [WARN ] agw-token-acq22              ######## ###### 200029 AsyncTokenProvider [] Token acquisition took too long: 31568 ms
[2022-06-23T13:06:10.462Z] [WARN ] agw-token-acq23              ######## ###### 200030 AsyncTokenProvider [] Token acquisition took too long: 37028 ms
[2022-06-23T13:14:59.978Z] [WARN ] agw-token-acq24              ######## ###### 200031 AsyncTokenProvider [] Token acquisition took too long: 31359 ms

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 7.0.3

Cause

This happens because of slow token acquisition from STS service. The STS service is unable to return the token in 30 seconds.

Resolution

Issue is resolved in vCenter Server 7.0 P05 (U3d) and above.. The fix contains increased default timeout for token acquisition from STS service.

STS team are working on a fix for the root cause of the slow token acquisition.

Workaround:
If we are unable to patch vCenter Server, The workaround for this issue is to increase the 30 seconds timeout for token acquisition to 2 minutes. This can be done as follows,

1. Take a snapshot of vCenter Server.
2. cd /usr/lib/vmware-vsphere-ui/plugin-packages/cis-data-service-plugin/plugins
3. Back-up original file to another directory. For example: cp api-gateway-server.war ~
4. unzip api-gateway-server.war
5. vi WEB-INF/spring/bundle-context.xml

6. Navigate to the line containing the string "tokenAcquisitionTimeout" (Press '/' type "tokenAcquisitionTimeout" and press 'Return')
You should see content like this:

<constructor-arg index="6" value="30" /> <!-- tokenAcquisitionTimeout -->
<constructor-arg index="7" value="SECONDS" /> <!-- timeUnitForTokenAcquisitionTimeout -->

7. Modify the values to increase the timeout (Press `i` and then type the new values).
For example:

<constructor-arg index="6" value="2" /> <!-- tokenAcquisitionTimeout -->
<constructor-arg index="7" value="MINUTES" /> <!-- timeUnitForTokenAcquisitionTimeout -->

8. Save the modifications. (Press 'Esc' then type ":x" and then press 'Return')

9. rm api-gateway-server.war
10. zip -rm api-gateway-server.war META-INF WEB-INF
11. chmod 755 api-gateway-server.war
12. service-control --restart vsphere-ui
13. Validate the behavior.
------------------------------------------------------------------------------------------------------------
Note: It is very important the the file permissions of api-gateway-server.war are 755 (rwxr-xr-x). Otherwise the login will not function at all.

Additional Information