Error: "User session count is limited to 550. Existing session count is 550 for user" during CIS connection via PowerCLI

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• Error while  logging in using PowerCLI :
  

  FullyQualifiedErrorId : An error occurred while establishing a CIS connection for server xxxxxxxxxx.xxxxxx

   .com. For more information: Exception calling "GetCisSessionId" with "0" argument(s): "com.vmware.vapi.std.errors.

    service_unavailable {'messages': [com.vmware.vapi.std.localizable_message {'id': com.vmware.vapi.endpoint.failedTo

  LoginMaxUserSessionCountReached, 'default_message': User session count is limited to 550. Existing session count

• The error displayed is due to the user sessions being exhausted for the user "[e-mail]".
  
  
• /var/log/vmware/vapi/endpoint.log* reports
  
  YYYY-MM-DDTHH:MM:SS | WARN  | sso3                      | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user [e-mail]. Please retry the login operation
  YYYY-MM-DDTHH:MM:SS | WARN  | sso10                     | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user [e-mail]. Please retry the login operation
  YYYY-MM-DDTHH:MM:SS | WARN  | sso6                      | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user [e-mail]. Please retry the login operation
  YYYY-MM-DDTHH:MM:SS | WARN  | sso1                      | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user [e-mail]. Please retry the login operation
  YYYY-MM-DDTHH:MM:SS | WARN  | sso1                      | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user [e-mail]. Please retry the login operation
  
  
• Confirm the idle timeout from the /var/log/vmware/vapi/endpoint/endpoint.log: 
  Keyword: idle
  endpoint.log.XXX:YYYY-MM-DDTHH:MM:SS | INFO  | session-monitor1          | MasterSessionMonitor           | Session  [e-mail] (internal id xxxxxxxx-xxxx-xxxx-xxxxxxxxx...) has been idle for 3,600,389 milliseconds.
  

Troubleshooting:

• grep -r 'User sessions count is limited to 550' endpoint.log* | awk '{print $22}' | sort | uniq -c | sort
• grep -h 'User sessions count is limited to 550' endpoint.log* | grep user | awk '{print $1}' | awk -F : '{print $1}' | sort | uniq -c | sort
• grep " idle for " endpoint.log* | awk '{print $10}' | sort | uniq -c | sort

Environment

• VMware vCenter Server 7.0.X
• VMware vCenter Server 8.0.X

Cause

• This issue occurs when the maximum user session count is reached on the vCenter Server.
• This issue is caused by the infraprofile service failing to respond to vAPI requests.

• maxSessionsPerUser

  The maxSessionsPerUser parameter controls the total number of sessions per user supported by the vAPI endpoint, which is 550 by default. Upon exceeding this limit, additional logins or attempts to create new sessions for that particular user will be rejected.

  • Say user1 has 550 sessions (REST) successfully established on vCenter.
  • If the total sessions of user1 hit the 550 sessions mark, then no further sessions can be created for this user at the vAPI endpoint.
  • Now, when user1 attempts to create a new session (551st session for user1), it leads to a 503 Service Unavailable error.
  • Already existing sessions of user1 will function as usual.

  
  Figure 6. The maximum concurrent sessions per user at the vAPI endpoint is 550; the 551st session errors out with 503 Service Unavailable

• Number in figure 1	Components	Description	Limit
  6	Sessions per user @ vAPI endpoint	Max number of sessions per user for vAPI, enforced by vAPI endpoint	550

Resolution

Workaround 1:

Option 1

• Reboot the affected vCenter Server

Option 2

• Login to the vCenter via SSH and restart following service:
  • service-control --restart vmware-vapi-endpoint
  • service-control --restart observability
  • service-control --restart observability-vapi
    
    

Workaround 2: (if idle events are there)

Increase the memory of the infraprofile service by completing the following:

1. Check current memory allocated to the infraprofile service:
   • cloudvm-ram-size -l | grep vmware-infraprofile
2. Double the memory allocated to the infraprofile service:
   • cloudvm-ram-size -C <DoubleOutputFromStep1> vmware-infraprofile
3. Restart the infraprofile service:
   • service-control --stop vmware-infraprofile && service-control --start vmware-infraprofile
   • service-control --restart vmware-vapi-endpoint

Additional Information

https://blogs.vmware.com/performance/2022/07/endpoint-limits-for-concurrent-rest-requests-from-vcenter-apis.html