Issue in vAPI “/rest/com/vmware/cis/session” : On sending repeated requests to create multiple sessions with vCenter, using the rest API, after around 12 requests, the vCenter stops responding.

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

1. Version - VMware vCenter Server 6.x 
2. Issue in vAPI “/rest/com/vmware/cis/session” : On sending repeated requests to create multiple sessions with vcenter, using the rest API, after around 12 requests, the vcenter stops responding. No response is received by the backup host.
3. The endpoint log has reported the error – 2021-12-07T08:20:34.974-06:00 | ERROR | sso4 | RequestDurationLogProvider | Slow operation. Request to service: com.vmware.cis.session, operation: create took 14421 ms

Cause

Large size of vCenter’s trust-store combined with simultaneous session creation attempts.       

Resolution

In the endpoint log , errors such as:

 

ERROR | sso1 | RequestDurationLogProvider | Slow operation. Request to service: com.vmware.cis.session, operation:

 

ERROR | sso6 | ApiEndpointServer | Unexpected error on thread Thread[sso6,5,main]

java.lang.OutOfMemoryError: GC overhead limit exceeded

 

increasing vmware-vapi-endpoint service's memory (see https://kb.vmware.com/s/article/2150757) is the only known workaround.

 

Increasing from 160 to 256 will give the service 96 additional megabytes of memory which, with that many certificates in the trust-store, will translate to about 30 more simultaneous session requests on top of the 13 that the service supports currently.
 

Note : The particular number of requests depends on the size of the trust-store (the number and size of certificates that vCenter trusts).
This issue applies to all 6.x versions and it is not present on 7.0.
 

Workaround:
Increasing vmware-vapi-endpoint service's memory (see https://kb.vmware.com/s/article/2150757) is the only known workaround.

Additional Information

Impact/Risks:
The vapi-endpoint service may stop responding to requests rendering the vCenter REST API unusable.