"User session count is limited to 550. Existing session count is 550 for user" during SDDC Manager upgrade precheck or Lifecycle Manager

Products

• VMware Cloud Foundation

Issue/Introduction

• Upgrade precheck in SDDC Manager hangs at 3% and then fails after a minute. 

/var/log/vmware/vcf/lcm/lcm-debug.log

YYYY-MM-DDT17:20:07.040+0000 ERROR [vcf_lcm,xxxxxxxxxxxxxx,3b2e] [c.v.e.s.l.a.i.i.HardwareSupportClientImpl,Scheduled-10] Exception occurred while loading the Hardware Support Info for the domain xxxxxxxxx-xxxx-xxxx-xxxxxxxxxx com.vmware.vapi.std.errors.ServiceUnavailable: ServiceUnavailable (com.vmware.vapi.std.errors.service_unavailable) => {     messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {     id = com.vmware.vapi.endpoint.failedToLoginMaxUserSessionCountReached,     defaultMessage = User session count is limited to 550. Existing session count is 550 for user [email protected].,     args = [550, 550, [email protected]],     params = <null>,     localized = <null> }],     data = <null>,     errorType = SERVICE_UNAVAILABLE

• Attempting to import a depot zip into the Lifecycle Manager via Import Updates fails with the error "An unexpected error has occurred."
  
  
•  /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log
  

  com.vmware.vapi.std.errors.service.unavailable => {data=<unset>,error_type=SERVICE_UNAVAILABLE, messages=User session count is limited to 550. Existing session count is 550 for user <Username>., localized=<unset>, id=com.vmware.vapi.endpoint.failedToLoginMaxUserSessionCountReached,

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x
VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

This issue occurs when the user session count maximum is reached on the vCenter Server.

Resolution

Workaround 1:

Option 1

• Reboot the affected vCenter Server

Option 2

• Increase the session.maxSessionsPerUser value on the affected vCenter
  
  1. Take appropriate snapshots of the vCenter(s)
  2. SSH to the vCenter via root
  3. Take a backup of /etc/vmware-vapi/endpoint.properties
     
     cp /etc/vmware-vapi/endpoint.properties /etc/vmware-vapi/endpoint.properties.backup
     
     
  4. Edit the endpoint.properties file
     
     vi /etc/vmware-vapi/endpoint.properties
     
     
  5. Remove the comment on the session.maxSessionsPerUser and increase the value to 1000.  (Note: This parameter is not present in vcenter 8.0.X, need to manually add the set the value to 1000)
     
     #session.maxSessionCount=1000
session.maxSessionsPerUser=1000
# One hour is 3 600 000 millis
#session.maxSessionIdleMilliseconds=3600000
# One day is 86 400 000 millis
#session.minSessionLifespanMilliseconds=86400000
# Two days are 172 800 000 millis
#session.maxSessionLifespanMilliseconds=172800000
     
     
  6. Restart vmware-vapi-endpoint service
     
     service-control --stop vmware-vapi-endpoint && service-control --start vmware-vapi-endpoint
     
     

Workaround 2:

Resolution - This issue is caused by the infraprofile service failing to respond to vAPI requests.
This is expected to be resolved in a future vCenter release.

Confirm the idle timeout from the /var/log/vmware/vapi/endpoint/endpoint.log: 

Keyword: idle

YYYY-MM-DDT16:39:53.554Z | INFO | session-monitor1 | MasterSessionMonitor | Session [email protected] (internal id xxxxxxxx-xxxx-xxxx-xxxxxxxxx, token 3e252...) has been idle for 3,600,907 milliseconds.

Increase the memory of the infraprofile service by completing the following:

1. Check current memory allocated to the infraprofile service:

cloudvm-ram-size -l | grep vmware-infraprofile

2. Double the memory allocated to the infraprofile service:

cloudvm-ram-size -C <DoubleOutputFromStep1> vmware-infraprofile

3. Restart the infraprofile service:

service-control --stop vmware-infraprofile && service-control --start vmware-infraprofile

service-control --restart vmware-vapi-endpoint

Note:

If the message "java.lang.OutOfMemoryError: Compressed class space" is found in any of the following files, it indicates that the memory allocated to the Compressed class space for the infraprofile service was insufficient.
This may also cause the infraprofile service to become unresponsive to vAPI requests.

Log files to check:

  /var/log/vmware/infraprofile/
    infraprofile-svcs.log*
    infraprofile-runtime.log*.stderr
    infraprofile-runtime.log*.stdout

To increase the memory allocation, run the following commands:

  cloudvm-ram-size -K <size in MB> vmware-infraprofile
  service-control --stop vmware-infraprofile && service-control --start vmware-infraprofile

Example:

    cloudvm-ram-size -K 32 vmware-infraprofile

To check the current allocation size of the Compressed class space, use the following command:

  cloudvm-ram-size -J vmware-infraprofile

Example output:

  -Xmx186m
  -XX:CompressedClassSpaceSize=20m
  -Xss256k

In this case, -XX:CompressedClassSpaceSize=20m indicates that the Compressed class space is set to 20 MB.
Please note, 32MB is the default size in vCenter Server 8.0U3.

Additional Information

When the user session count reaches 550, the customer might not be able to delete a workload domain from SDDC Manager.
For more details, please refer to the following KB.
Failed to fetch topology from all known nodes: [FQDN, FQDN, ...] in Validate the Single Sign-On (SSO) Ring Topology task