"User session count is limited to 550. Existing session count is 550 for user" during SDDC Manager upgrade precheck or Lifecycle Manager

Products

• VMware Cloud Foundation

Issue/Introduction

• Upgrade precheck in SDDC Manager hangs at 3% and then fails after a minute. 

/var/log/vmware/vcf/lcm/lcm-debug.log

YYYY-MM-DDT17:20:07.040+0000 ERROR [vcf_lcm,xxxxxxxxxxxxxx,3b2e] [c.v.e.s.l.a.i.i.HardwareSupportClientImpl,Scheduled-10] Exception occurred while loading the Hardware Support Info for the domain xxxxxxxxx-xxxx-xxxx-xxxxxxxxxx com.vmware.vapi.std.errors.ServiceUnavailable: ServiceUnavailable (com.vmware.vapi.std.errors.service_unavailable) => {     messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {     id = com.vmware.vapi.endpoint.failedToLoginMaxUserSessionCountReached,     defaultMessage = User session count is limited to 550. Existing session count is 550 for user [email protected].,     args = [550, 550, [email protected]],     params = <null>,     localized = <null> }],     data = <null>,     errorType = SERVICE_UNAVAILABLE

• Attempting to import a depot zip into the Lifecycle Manager via Import Updates fails with the error "An unexpected error has occurred."
  
  
•  /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log
  

  com.vmware.vapi.std.errors.service.unavailable => {data=<unset>,error_type=SERVICE_UNAVAILABLE, messages=User session count is limited to 550. Existing session count is 550 for user <Username>., localized=<unset>, id=com.vmware.vapi.endpoint.failedToLoginMaxUserSessionCountReached,

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x
VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

This issue occurs when the user session count maximum is reached on the vCenter Server.

Resolution

Workaround 1:

Option 1

• Reboot the affected vCenter Server

Option 2

• Increase the session.maxSessionsPerUser value on the affected vCenter
  
  1. Take appropriate snapshots of the vCenter(s)
  2. SSH to the vCenter via root
  3. Take a backup of /etc/vmware-vapi/endpoint.properties
     
     cp /etc/vmware-vapi/endpoint.properties /etc/vmware-vapi/endpoint.properties.backup
     
     
  4. Edit the endpoint.properties file
     
     vi /etc/vmware-vapi/endpoint.properties
     
     
  5. Remove the comment on the session.maxSessionsPerUser and increase the value to 1000.  (Note: This parameter is not present in vcenter 8.0.X, need to manually add the set the value to 1000)
     
     #session.maxSessionCount=1000
session.maxSessionsPerUser=1000
# One hour is 3 600 000 millis
#session.maxSessionIdleMilliseconds=3600000
# One day is 86 400 000 millis
#session.minSessionLifespanMilliseconds=86400000
# Two days are 172 800 000 millis
#session.maxSessionLifespanMilliseconds=172800000
     
     
  6. Restart vmware-vapi-endpoint service
     
     service-control --stop vmware-vapi-endpoint && service-control --start vmware-vapi-endpoint
     
     

Workaround 2:

Resolution - This issue is caused by the infraprofile service failing to respond to vAPI requests.
This is expected to be resolved in a future vCenter release.

Confirm the idle timeout from the /var/log/vmware/vapi/endpoint/endpoint.log: 

Keyword: idle

YYYY-MM-DDT16:39:53.554Z | INFO | session-monitor1 | MasterSessionMonitor | Session [email protected] (internal id xxxxxxxx-xxxx-xxxx-xxxxxxxxx, token 3e252...) has been idle for 3,600,907 milliseconds.

Increase the memory of the infraprofile service by completing the following:

1. Check current memory allocated to the infraprofile service:

cloudvm-ram-size -l | grep vmware-infraprofile

2. Double the memory allocated to the infraprofile service:

cloudvm-ram-size -C <DoubleOutputFromStep1> vmware-infraprofile

3. Restart the infraprofile service:

service-control --stop vmware-infraprofile && service-control --start vmware-infraprofile

service-control --restart vmware-vapi-endpoint