Error: "User session count is limited to 550. Existing session count is 550 for user" during CIS connection via PowerCLI
search cancel

Error: "User session count is limited to 550. Existing session count is 550 for user" during CIS connection via PowerCLI

book

Article ID: 376401

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

  • Error while  logging in using PowerCLI :

    FullyQualifiedErrorId : An error occurred while establishing a CIS connection for server xxxxxxxxxx.xxxxxx

       .com. For more information: Exception calling "GetCisSessionId" with "0" argument(s): "com.vmware.vapi.std.errors.

      service_unavailable {'messages': [com.vmware.vapi.std.localizable_message {'id': com.vmware.vapi.endpoint.failedTo

    LoginMaxUserSessionCountReached, 'default_message': User session count is limited to 550. Existing session count

  • The error displayed is due to the user sessions being exhausted for the user "[email protected]".

  • /var/log/vmware/vapi/endpoint.log* reports
    YYYY-MM-DDT16:14:26.488-07:00 | WARN  | sso3                      | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user xxx_xxx@xxx.xxx.xxx.com. Please retry the login operation
    YYYY-MM-DDT16:14:29.660-07:00 | WARN  | sso10                     | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user xxx_xxx@xxx.xxx.xxx.com. Please retry the login operation
    YYYY-MM-DDT16:19:11.729-07:00 | WARN  | sso6                      | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user xxx_xxx@xxx.xxx.xxx.com. Please retry the login operation
    YYYY-MM-DDT16:19:23.855-07:00 | WARN  | sso1                      | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user xxx_xxx@xxx.xxx.xxx.com. Please retry the login operation
    YYYY-MM-DDT16:19:23.855-07:00 | WARN  | sso1                      | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user xxx_xxx@xxx.xxx.xxx.com. Please retry the login operation

  • Confirm the idle timeout from the /var/log/vmware/vapi/endpoint/endpoint.log: 
    Keyword: idle
    endpoint.log.XXX:YYYY-MM-DDT03:52:50.777-07:00 | INFO  | session-monitor1          | MasterSessionMonitor           | Session  xxx_xxx@xxx.xxx.xxx.com (internal id xxxxxxxx-xxxx-xxxx-xxxxxxxxx...) has been idle for 3,600,389 milliseconds.

Troubleshooting:

  • grep -r 'User sessions count is limited to 550' endpoint.log* | awk '{print $22}' | sort | uniq -c | sort
  • grep -h 'User sessions count is limited to 550' endpoint.log* | grep ZS7YVKZ | awk '{print $1}' | awk -F : '{print $1}' | sort | uniq -c | sort
  • grep " idle for " endpoint.log* | awk '{print $10}' | sort | uniq -c | sort
▶

 

Wolken View Helper Settings







◀

 

Wolken View Helper Settings







◀

Environment

  • VMware vCenter Server 7.0.X
  • VMware vCenter Server 8.0.X
▶

Cause

  • This issue occurs when the maximum user session count is reached on the vCenter Server.
  • This issue is caused by the infraprofile service failing to respond to vAPI requests.

  • maxSessionsPerUser

    The maxSessionsPerUser parameter controls the total number of sessions per user supported by the vAPI endpoint, which is 550 by default. Upon exceeding this limit, additional logins or attempts to create new sessions for that particular user will be rejected.

    • Say user1 has 550 sessions (REST) successfully established on vCenter.
    • If the total sessions of user1 hit the 550 sessions mark, then no further sessions can be created for this user at the vAPI endpoint.
    • Now, when user1 attempts to create a new session (551st session for user1), it leads to a 503 Service Unavailable error.
    • Already existing sessions of user1 will function as usual.


    Figure 6. The maximum concurrent sessions per user at the vAPI endpoint is 550; the 551st session errors out with 503 Service Unavailable

  • Number in figure 1 Components Description Limit
    6 Sessions per user @ vAPI endpoint Max number of sessions per user for vAPI, enforced by vAPI endpoint 550
▶

Resolution

Workaround 1:

Option 1

  • Reboot the affected vCenter Server

Option 2

  • Login to the vCenter via SSH and restart following service:

    service-control --restart vmware-vapi-endpoint
    service-control --restart observability 
    service-control --restart observability-vapi

 

Workaround 2: (if idle events are there)

  • Increase the memory of the infraprofile service by completing the following:


1. Check current memory allocated to the infraprofile service: cloudvm-ram-size -l | grep vmware-infraprofile

2. Double the memory allocated to the infraprofile service: cloudvm-ram-size -C <DoubleOutputFromStep1> vmware-infraprofile

3. Restart the infraprofile service:

    • service-control --stop vmware-infraprofile && service-control --start vmware-infraprofile
    • service-control --restart vmware-vapi-endpoint
▶

Additional Information

https://blogs.vmware.com/performance/2022/07/endpoint-limits-for-concurrent-rest-requests-from-vcenter-apis.html

▶

Powered by Wolken Software