vCenter Server service unavailable due to session exhaustion by VMware Live Recovery (vSphere Replication)

Products

• VMware Live Recovery
• VMware vCenter Server 8.0

Issue/Introduction

• vCenter Server experiences constant communication errors and service unavailability.
• Inventory objects, Roles, Global Permissions, or Content Libraries may appear empty in the vSphere Client.
• The vpxd-svcs.log file, located in /var/log/vmware/vpxd-svcs/ in the vCenter Server Appliance (VCSA) file system, contains the following error:

  Too many authenticated sessions (1000), cannot create new session

  or

  com.vmware.vim.vmomi.server.exception.ServiceUnavailableException: Failed to create session

Environment

• VMware vCenter Server 8.0.x
• VMware Live Recovery 9.0.x

Cause

This issue is caused by the VMware Live Recovery (vSphere Replication) service account creating an excessive number of authenticated sessions.
Due to this behavior the default limit of 1,000 sessions for the vCenter Server vpxd-svcs service gets exhaused, preventing new sessions from being created, thus resulting in communication failures.

Resolution

To resolve the session exhaustion and restore service stability, perform the following steps:

1. Identify the Source of Session Leaks:
   Verify the active session count by filtering /var/log/vmware/vpxd-svcs/vpxd-svcs.log for successful token acquisitions. Use the following command in the vCenter Server Appliance (VCSA) to confirm if the vSphere Replication service account is responsible:

   # grep "Successfully acquired token for user" /var/log/vmware/vpxd-svcs/vpxd-svcs.log | awk '{print $NF}' | sort | uniq -c | sort -nr

2. Apply Configuration Fixes:
   Implements the resolution step documented in the following Broadcom Knowledge Base articles to address the underlying session management behavior:
   • Exclude hosts from registration in vSphere Replication
   • ESXi hosts automatically enter and exit maintenance mode post vSphere Replication upgrade
3. Reboot the Replication Appliance:
   Ensure there are no active replication jobs running, then perform a reboot of the VMware Live Recovery (vSphere Replication) Manager appliance. This will clear the existing leaked sessions and resets the communication interface with vCenter Server.
4. Verification:
   Monitor /var/log/vmware/vpxd-svcs/vpxd-svcs.log to confirm that the session count remains within healthy limits and that no more ServiceUnavailableException errors occur.