Could not connect to one or more vCenter Server Systems: https://<vCenter-FQDN>: 443/sdk" error in the vSphere Client due to VPXD session exhaust

Products

• VMware vCenter Server

Issue/Introduction

• vCenter service becomes unresponsive randomly.
  
  
• The log snipped in '/var/log/vmware/vpxd/vpxd.log' looks similar to:
     [YYYY-MM-DDTHH:MM:SS].###-##:## error vpxd[10382] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
   [YYYY-MM-DDTHH:MM:SS].###-##:## error vpxd[10395] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
• Session exhaust could be due to the following reasons:

          a) Any external/internal solution trying to login to vCenter with incorrect credentials.

[YYYY-MM-DDTHH:MM:SS] info vpxd[10394] [Originator@6876 sub=Default opID=faf1555] [VpxLRO] -- ERROR lro-134510 -- SessionManager -- vim.SessionManager.impersonateUser: vim.fault.InvalidLogin:
 --> Result:
 --> (vim.fault.InvalidLogin) {
 --> faultCause = (vmodl.MethodFault) null,
 --> faultMessage = <unset>
 --> msg = ""
 --> }
 --> Args:
 -->
 --> Arg userName:
 --> "USERNAME"
 --> Arg locale:
 --> "en"

b) Any service account user trying to access vCenter with insufficient privileges.

[YYYY-MM-DDTHH:MM:SS] info vpxd[7FE11CC03700] [Originator@6876 sub=Default opID=68209c6d] [VpxLRO] -- ERROR lro-592994 -- SessionManager -- vim.SessionManager.login: vim.fault.NoPermission:
--> Result:
--> (vim.fault.NoPermission) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> object = 'vim.Folder:########-####-####-####-########abc0:group-d1',
--> privilegeId = "System.View"

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x

Cause

This issue can occur due to multiple failed login attempts exhausting the http sessions (Usually Backup client, Monitoring client or any external solution integrated with VC with expired password/account can cause excessive retry attempts which can exhaust the available sessions)

Resolution

To resolve the session exhaust situation, 

1. Identify if the respective solution account which is trying to login to vCenter server with invalid credentials and address the respective account's issue by resetting/updating the password.
2. Assess the session count and if it is required to increase the number of sessions, follow the below steps as a workaround.

Increase <maxSessionCount> value in vpxd.cfg file:

Note: a) vCenter Server will stop working while implementing this action plan
          b) Before the troubleshooting, you must take backup of the vCenter Server Appliance. If the vCenter Servers are in Enhanced Linked Mode (ELM), make sure to take offline snapshot of all the vCenter Servers that are participating in the Enhanced Linked Mode.
Reference article: VMware vCenter in Enhanced Linked Mode pre-changes snapshot best practice

• • Connect to vCenter Server Appliance shell as a root user.
  • Take a backup of the vpxd.cfg from /etc/vmware-vpx/
      cp /etc/vmware-vpx/vpxd.cfg /storage/core/vpxd.cfg
    
    
  • Stop vpxd service by running the command:
      service-control --stop vmware-vpxd
    
    
  • Edit vpxd.cfg file using vi editor to add the soap and MaxSessionCount entries as shown below (In the below example, the maxSessionCount value is set to 6000. The default value is 2000).
    
       <config>
   ...
   <vmacore>
      ...
      <soap>
         ...
         <maxSessionCount>6000</maxSessionCount>
         ...
      </soap>
      ...
   </vmacore>
   ...
 </config


  • Start vpxd service by running the command:
      service-control --start vmware-vpxd

Additional Information

Reviewing the vCenter's journclt logs for "BadUsernameSessionEvent". It should provide the account name and IP the login attempt came from. Please see the example below.

Apr ## ##:##:## #####.####.### vpxd[6847]: Event [34293420] [1-1] [####-##-##T##:##:##.#####Z] [vim.event.BadUsernameSessionEvent] [error] [#######] [TOC] [34293420] [Cannot login #######@###.###.###.###]