vCenter Server gets unresponsive during backup windows, causing VM backups to fail

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
-  vCenter Server gets unresponsive during backup windows, causing VM backups to fail

- When a login is performed the following are the services that interact for a single login
STS --> TMS --> Authz --> VPX --> Authz --> STS

- in /var/log/vmware/trustmanagement/trustmanagement-svcs.log. we can see TMS makes a hasPrivileges call to vpxd-svcs which times out after 20 mins

YYYY-MM-DDT20:38:05.422Z [tomcat-exec-18  WARN  com.vmware.cis.authorization.client.impl.AuthorizationServiceClientimpl  opId=] Operation: hasPrivileges. Read API execution failed.
java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:10080/invsvc/vmomi/sdk/ invocation failed with "java.net.SocketTimeoutException: Read timed out"
        at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)

- vpxd-svcs logs may have logging gaps at the same timestamp

Environment

VMware vCenter Server 7.0.x

Cause

When a login is performed the following are the services that interact for a single login

STS --> TMS --> Authz ,
VPX --> Authz --> STS

With these many services interacting there was a chance of services getting exhausted with many logins per minute.

Resolution

There were many performance improvements done in 7.0 U2 which reduces the # of calls between these services and even removes the dependencies among them. To resolve this issue, upgrade vCenter Server to 7.0 U2 or later.