vCenter cloud account datacollection fails in Aria Automation with "The connection with vAPI endpoint can not be established failedToLoginMaxUserSessionCountReached"

Products

• VMware Aria Suite
• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter cloud account datacollection or provisioning fails in Aria Automation UI with error:

Failed to validate credentials. Error: The connection with vAPI endpoint can not be established. Please make sure the vAPI endpoint is running in the vCenter. ( com.vmware.vapi.endpoint.failedToLoginMaxUserSessionCountReached User session count is limited to 550. Existing session count is 550 for user <Username>.)

• The message is also visible in the provisioning service log located under /var/log/services-logs/prelude/provisioning-service-app/file-logs/provisioning-service-app.log

Environment

VMware vCenter Server 8.0.1
VMware Aria Automation 8.x
VMware vCenter Server 8.0.2

Cause

This issue occurs when the user session count maximum is reached for the vAPI endpoint on the relevant vCenter server.

Resolution

The error is typically indicative of a custom configuration, a workflow or script for example, which makes an excessive amount of calls to the vAPI endpoint with a single username. These should be reviewed alongside the content developers to see if any optimization can be made to reduce the number of calls.

Workaround:

To workaround the issue increase the session.maxSessionsPerUser value on the affected vCenter. 

1. Take a snapshot of the vCenter as a best practice precaution
2. Connect to VCSA via SSH & login with root user:
3. Take a backup of the /etc/vmware-vapi/endpoint.properties file:

cp /etc/vmware-vapi/endpoint.properties /etc/vmware-vapi/endpoint.properties.backup

4. Edit the endpoint.properties file

vi /etc/vmware-vapi/endpoint.properties

5. Remove the comment on the session.maxSessionsPerUser line and increase the value from the default of 550 to 1000.

a) When the maxSessionsPerUser limit is increased to match the total maxSessionCount (=1000) this can open the door to an intended or unintended DoS attack - as one single user can eat up all the available sessions, all other users will not be able to create a session.

b) When the maxSessionsPerUser is increased to 1000, the maxSessionCount should also be increased to 1500.

c) Very important. If the maxSessionCount is increased, then the APIs performance and scaling is no longer guaranteed - it may degrade. Also, the memory that the vAPI Endpoint service will need increases, and thus it is recommended to also increase its memory, following the KB: Manually increasing the heap memory on vCenter Server Components 

6. Restart the vCenter services for changes to take effect:

service-control --restart vapi-endpoint

7. Validate again the vCenter cloud account in Aria Automation

Additional Information

There is an official blogpost which contains further information about the vAPI endpoint concurrency limits .