vCenter gets unresponsive during backup windows, causing VM backups to fail

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
-  vCenter gets unresponsive during backup windows, causing VM backups to fail

- When a login is performed the following are the services that interact for a single login
STS --> TMS --> Authz --> VPX --> Authz --> STS

- in /var/log/vmware/trustmanagement/trustmanagement-svcs.log. we can see TMS makes a hasPrivileges call to vpxd-svcs which times out after 20 mins

YYYY-MM-DDT20:38:05.422Z [tomcat-exec-18  WARN  com.vmware.cis.authorization.client.impl.AuthorizationServiceClientimpl  opId=] Operation: hasPrivileges. Read API execution failed.
java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:10080/invsvc/vmomi/sdk/ invocation failed with "java.net.SocketTimeoutException: Read timed out"
        at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)

- vpxd-svcs logs may have logging gaps at the same timestamp

Environment

VMware vCenter Server 7.0.x

Cause

1. When a login is performed the following are the services that interact for a single login

STS --> TMS --> Authz ,
VPX --> Authz --> STS

With these many services interacting there was a chance of services getting exhausted with many logins per minute.
There were many performance improvements done in 7.0u2 which reduces the # of calls between these services and even removes the dependencies among them. So  we recommend customers to move to 7.0.u2 if they are facing similar behaviours

Resolution

Upgrade vcenter to 7.0.u2 or later