NSX-T cloud on Avi load balancer goes down due to user session count limit for vCenter service account

Products

• VMware Avi Load Balancer

Issue/Introduction

The NSX-T cloud on Avi load balancer goes down and becomes Red.

We can see the following errors in the controller events -

• "Valid vCenter config doesn't exist in cloud"
  
• 'Get' request to API 'content/library'; Status: "; Code: '401'
  

 

Cause

• This is caused when the user session count limit is reached for the vCenter account that is configured on Avi.
• To confirm the same, we can run the below command towards the vCenter endpoint -

  sudo curl -k -X POST -H "Authorization: Basic <token>" https://<VC Endpoint>/rest/com/vmware/cis/session

  The token value is the base64 encoding of 'username:password' for the vCenter account.
• If the user session count limit has been reached, we should see this error - “User session count is limited to 550."

Resolution

Resolution:

The best practice is to create a dedicated service account on vCenter for Avi load balancer.

After creating the new account, update the new credentials in Avi's cloud configuration.

 

Workarounds on vCenter:

Option 1:

• Reboot the affected vCenter Server.

Option 2:

• Increase the session.maxSessionsPerUser value on the affected vCenter
  
  
  1. SSH to the vCenter via root.
     
     
  2. Take a backup of /etc/vmware-vapi/endpoint.properties
     
     cp /etc/vmware-vapi/endpoint.properties /etc/vmware-vapi/endpoint.properties.backup
     
     
  3. Edit the endpoint.properties file
     
     vi /etc/vmware-vapi/endpoint.properties
     
     
  4. Remove the comment on the session.maxSessionsPerUser and increase the value to 1000.
     #session.maxSessionCount=1000
session.maxSessionsPerUser=1000
     
     Note: If the parameter is not present add it manually and the set the value to 1000
     
     
  5. Restart vmware-vapi-endpoint service
     
     service-control --restart vmware-vapi-endpoint