vCenter UI not loading inventory for specific AD User

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:

Specific AD users might take a long time to login
The inventory might fail to load or only load a blank vCenter frame
Errors - “Unable to retrieve hosts information” or “You do not have privileges to view this object”

Environment

VMware vSphere 7.0.x
VMware vCenter Server 7.0.0

Cause

Token Acquisition Timeout has been introduced in vCenter 7.0
The Acquisition Timeout value is set to 60 seconds in vCenter 7.0
If the user takes longer than 60 seconds to acquire the token the request will timeout.

Logging examples:

vpxd log:

Domain:###.com}, {Name: SC17-Web; Domain:####.com}, {Name: Everyone; Domain:vsphere.local}], delegationChain=[{Name: vpxd-4074401a-f99c-4d20-80ca-5ca125c1c385; Domain:vsphere.local} at 2022-07-08T12:59:44.378Z], startTime=2022-07-08 13:01:10.546, expirationTime=2022-07-09 15:01:10.546, renewable=true, delegable=true, isSolution=false,confirmationType=1]

2022-07-08T13:02:37.187Z info vpxd[08184] [Originator@6876 sub=AuthorizeManager] [Auth]: User xxxx.com\user

vsphere-ui.log

[2022-07-08T13:08:58.610Z] [WARN ] linkedVcGroup-pool-9301370108365 102284 200088 c.v.v.vim.commons.vmomi.request.DelegatingRequestContextProvider Failed to generate RequestContext for SessionManager#loginByToken com.vmware.vcenter.apigw.api.sso.tokenmgmt.TokenException: Failed to retrieve token for SSO domain vsphere.local(c59cc29d-1db3-4ceb-aad1-fd7ce7619036)

at com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider.doGetSamlToken(AsyncTokenProvider.java:461)

at com.vmware.vcenter.apigw.sso.tokenmgmt.impl.AsyncTokenProvider.getSamlToken(AsyncTokenProvider.java:432)

at com.vmware.vcenter.apigw.api.sso.tokenmgmt.NonRenewingMutableTokenProvider.getSamlToken(NonRenewingMutableTokenProvider.java:72)

at com.vmware.vise.vim.commons.vmomi.request.SessionManagerContextHandler.handle(SessionManagerContextHandler.java:62)

at com.vmware.vise.vim.commons.vmomi.request.DelegatingRequestContextProvider.getRequestContext(DelegatingRequestContextProvider.java:67)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.getRequestContext(MethodInvocationHandlerImpl.java:352)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:310)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195)

at com.sun.proxy.$Proxy402.loginByToken(Unknown Source)

at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.loginByToken(VcServiceImpl.java:1290)

at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.authenticate(VcServiceImpl.java:1081)

at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.access$300(VcServiceImpl.java:153)

at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl$3.call(VcServiceImpl.java:948)

at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl$3.call(VcServiceImpl.java:938)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.processLogin(VcServiceImpl.java:1056)

at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.doLogin(VcServiceImpl.java:923)

at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl.login(VcServiceImpl.java:913)

at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl$1.call(LinkedVcGroupImpl.java:373)

at com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl$1.call(LinkedVcGroupImpl.java:370)

at com.vmware.vise.util.concurrent.ExecutorUtil$2.call(ExecutorUtil.java:826)

at com.vmware.vise.util.concurrent.ExecutorUtil$ThreadContextPropagatingTask.call(ExecutorUtil.java:1240)

at com.vmware.vise.util.concurrent.DiagnosticThreadPoolExecutor$DiagnosticTask.call(DiagnosticThreadPoolExecutor.java:739)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at com.vmware.vise.util.concurrent.WorkerThreadFactory$1.run(WorkerThreadFactory.java:64)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.util.concurrent.TimeoutException: null

apigw.log

[2022-07-08T13:07:58.492Z] [INFO ] agw-token-acq95######## ###### 200088 SsoServiceImpl [] Acquiring token by token from domain vsphere.local(c59cc29d-1db3-4ceb-aad1-fd7ce7619036) from STS http://localhost:1080/external-vecs/http1/vcsa.com/443/sts/STSService/vsphere.local. actAs={Name: user, Domain: ####.com}, delegateTo=null, authz={Name: vsphere-webclient-4074401a-f99c-4d20-80ca-5ca125c1c385, Domain: vsphere.local}

[2022-07-08T13:07:58.492Z] [INFO ] http-nio-5090-exec-119 70108365 102284 ###### FrontendSessionManagerImpl [] Created API GW session 200088 for user {Name: user, Domain: xxxxx.com} using token _5b522ed0-3b1e-4837-996c-6802f040c6c5

[2022-07-08T13:09:25.179Z] [WARN ] agw-token-acq95 ######## ###### 200088 SsoServiceImpl [] acquireActAsToken() from domain vsphere.local(c59cc29d-1db3-4ceb-aad1-fd7ce7619036) took 86686 ms (authz={Name: vsphere-webclient-4074401a-f99c-4d20-80ca-5ca125c1c385, Domain: vsphere.local} from domain, delegate domain vsphere.local(c59cc29d-1db3-4ceb-aad1-fd7ce7619036), actAs=vsphere.local(c59cc29d-1db3-4ceb-aad1-fd7ce7619036), delegate={Name: user, Domain: ####.com})

[2022-07-08T13:09:25.179Z] [INFO ] agw-token-acq95 ######## ###### 200088 AsyncTokenProvider [] Acquired token _1e504ae8-4306-4b17-be6e-26610fccc5cf from domain vsphere.local(c59cc29d-1db3-4ceb-aad1-fd7ce7619036). Expiration time: Fri Jul 08 13:11:31 UTC 2022

[2022-07-08T13:09:25.179Z] [WARN ] agw-token-acq95 ######## ###### 200088 AsyncTokenProvider [] Token acquisition took too long: 86686 ms --------->>>>> This token acquisition timeout is 60s ... it took 86s ... which caused a timeout exception

Resolution

Increase the timeout value -

1. Create a snapshot of the vCenter

2. cd /usr/lib/vmware-vsphere-ui/plugin-packages/cis-data-service-plugin/plugins

3. Back-up original file to another directory. For example: cp api-gateway-server.war api-gateway-server.war.bck

4. unzip api-gateway-server.war

5. vi WEB-INF/spring/bundle-context.xml

6. Navigate to the line containing the string "tokenAcquisitionTimeout" (Press '/' type "tokenAcquisitionTimeout" and press 'Return')

You should see content like this:

<constructor-arg index="6" value="30" />

<constructor-arg index="7" value="SECONDS" />

7. Modify the values to increase the timeout (Press `i` and then type the new values).

For example:

<constructor-arg index="6" value="2" />

<constructor-arg index="7" value="MINUTES" />

8. Save the modifications. (Press 'Esc' then type ":x" and then press 'Return')

9. rm api-gateway-server.war

10. zip -rm api-gateway-server.war META-INF WEB-INF

11. chmod 755 api-gateway-server.war

12. service-control --restart vsphere-ui

13. Once the service is up and running, the back up of the original file can be removed.

For example: rm api-gateway-server.war

Additional Information

Impact/Risks: