3rd Party Backup fails when configured in vCenter environment using tags and following error seen in backup solution logs "vAPI Internal Server Error"

Products

• VMware vCenter Server

Issue/Introduction

• 3rd party Backup Solution configured in vCenter environment such a Veeam. 
• May have tags in use for backups 
• In vCenter log /var/log/vmware/vapi/endpoint/endpoint.log may see errors similar to below:

####-##-##T##:##:##.###Z| ERROR | vAPI-I/O dispatcher-0     | SessionFacade                  |                                      | Unexpected error occurred while executing the call with session [email protected] (internal id  #### for method com.vmware.cis.tagging.sessions.session_manager.login.com.vmware.vapi.client.exception.ConnectionException: http://localhost:1080/invsvc/vapi invocation failed with "java.net.SocketTimeoutException: 10,000 milliseconds timeout on connection http-outgoing-#### [ACTIVE]"
        at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheHttpAsyncClientTransport$FutureCallbackImpl.failed(ApacheHttpAsyncClientTransport.java:252) [vapi-runtime.jar:?]

• and/or

####-##-##T##:##:##.###Z | WARN  | vAPI-I/O dispatcher-0     | SessionApiSecurityUtil         |                                      | Failed to create child session with session manager com.vmware.cis.tagging.sessions.session_manager on http://localhost:1080/invsvc/vapi for session  [email protected](internal id ####).com.vmware.vapi.endpoint.vapi.ApiException: null

• and/or

####-##-##T##:##:##.###Z| ERROR | vAPI-I/O dispatcher-0     | SessionFacade                  |                                      | Unexpected error occurred while executing the call with session null
for method com.vmware.vapi.std.introspection.provider.get.com.vmware.vapi.client.exception.TransportProtocolException: HTTP response with status code 503 (enable debug logging for details): no healthy upstream
        at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheHttpUtil.validateHttpResponse(ApacheHttpUtil.java:101) ~[vapi-runtime.jar:?]
     

• You may see errors similar to below in your Backup Solution logs, such as Veeam: 

[####-##-##T##:##:##.###]    <###>   Error (3)    com.vmware.vapi.std.errors.internal_server_error {'messages': [com.vmware.vapi.std.localizable_message {'id': com.vmware.vapi.endpoint.badGateway, 'default_message': The endpoint, while acting as a gateway, received an invalid response from an upstream server it accessed while attempting to fulfill the request., 'args': []}], 'data':} (vmware.vapi.std.errors.InternalServerError)

Environment

vCenter Server 8.0 

 

Cause

Session timeouts occur with vAPI due to possible AD latency issues 

Resolution

To workaround the issue, please follow following KB article workaround steps to increase vapi session timeout value: Aria Suite Lifecycle Manager and Aria Automation fail on Data Collection for Content Library from vCenter Server

 

To resolve, please review and address AD latency in environment