Error "User session count is limited to 550. Existing session count is 550 for user" on SDDC Manager

Products

• VMware SDDC Manager

Issue/Introduction

Symptoms:

1. Generate CSR Procedure from SDDC manager will fail with error- 

Failed to generate CSR for <hostname.domain> due to: 503 Service Unavailable: 
"{"type":"com.vmware.vapi.std.errors.service_unavailable","value":
{"error_type":"SERVICE_UNAVAILABLE","messages":[{"args":["550","550","user@domain"],"default_message":"User session count is limited to 550. 
Existing session count is 550 for user 
user@domain.","id":"com.vmware.vapi.endpoint.failedToLoginMaxUserSessionCountReached"}]}}".

2. This issue may also happen while loading workload domains fails with error "Failed to get NSX clusters"

In commonsvcs logs, you may notice below errors -

User session count is limited to 550. Existing session count is 550 for user user@domain

In both cases, you will notice below errors in the endpoint logs -

/var/log/vmware/vapi/endpoint.log
yyyy-mm-ddThh-mm-ss | WARN  | sso3                      | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user user@domain. Please retry the login operation

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x
VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

This issue occurs when the user session count maximum is reached on the vCenter Server.

Resolution

IMPORTANT NOTE: Make sure that fresh backup/snapshots of the SDDC manager have been taken prior to any modification.

Workaround 1:

Option 1

• Reboot the affected vCenter Server

Option 2

• Increase the session.maxSessionsPerUser value on the affected vCenter
  
  
  1. SSH to the vCenter via root
     
     
  2. Take a backup of /etc/vmware-vapi/endpoint.properties
     
     cp /etc/vmware-vapi/endpoint.properties /etc/vmware-vapi/endpoint.properties.backup
     
     
  3. Edit the endpoint.properties file
     
     vi /etc/vmware-vapi/endpoint.properties
     
     
  4. Remove the comment on the session.maxSessionsPerUser and increase the value to 1000.
     #session.maxSessionCount=1000
session.maxSessionsPerUser=1000
     
     Note: If the parameter is not present add it manually and the set the value to 1000
     
     
  5. Restart vmware-vapi-endpoint service
     
     service-control --restart vmware-vapi-endpoint

Workaround 2:

Resolution - This issue is caused by the infraprofile service failing to respond to vAPI requests.
This is expected to be resolved in a future vCenter release.

 

Increase the memory of the infraprofile service by completing the following:

1. Check current memory allocated to the infraprofile service:

cloudvm-ram-size -l | grep vmware-infraprofile

2. Double the memory allocated to the infraprofile service:

cloudvm-ram-size -C <DoubleOutputFromStep1> vmware-infraprofile

3. Restart the infraprofile service:

service-control --stop vmware-infraprofile && service-control --start vmware-infraprofile

service-control --restart vmware-vapi-endpoint

Additional Information

Endpoint Limits for Concurrent REST Requests from vCenter APIs