Sending repeated requests to create multiple sessions in vCenter, using the rest API, after several requests (12) vCenter stops responding.

Products

• VMware Aria Suite

Issue/Introduction

No response is received by the backup host.
The endpoint log shows repeated:

2021-12-07T08:20:34.974-06:00 | ERROR | sso4 | RequestDurationLogProvider | Slow operation. Request to service: com.vmware.cis.session, operation: create took 14421 ms
ERROR | sso1 | RequestDurationLogProvider | Slow operation. Request to service: com.vmware.cis.session, operation:
ERROR | sso6 | ApiEndpointServer | Unexpected error on thread Thread[sso6,5,main]
java.lang.OutOfMemoryError: GC overhead limit exceeded

 

Environment

vCenter Servers 6.x

vCenter Server 7.x

vCenter Server 8.x

Cause

The issue is caused by large size of vCenter’s trust-store, combined with simultaneous failed session creation attempts.       

Resolution

To resolve the problem:

1. Increase the cloudvm ram size for vmware-vapi-enpoint

increasing the heap memory of components in vCenter Server

Note: 

1. Increasing from 160 to 256 will give the service 96 additional megabytes of memory which, with that many certificates in the trust-store, will translate to about 30 more simultaneous session requests on top of the 13 that the service supports currently.
 

Note : The particular number of requests depends on the size of the trust-store (the number and size of certificates that vCenter trusts).
This issue applies to all 6.x versions and it is not present on 7.0.
 

Additional Information

If not enough memory is allocated to the vapi-endpoint service, it may stop responding to requests rendering the vCenter REST API unusable.