vCenter Server Session Exhaustion Causing NetWorker Backup Failures

Products

• VMware vCenter Server

Issue/Introduction

 This article addresses intermittent SSL handshake timeouts and authentication failures in VMware vCenter Server environments, specifically when observed in conjunction with NetWorker VM backup failures. This symptom often stems from vCenter HTTP session exhaustion caused by excessive failed login attempts.

Journalctl

HH:MM:SS.MS [FQDN] vpxd[1476556]: Event [572805306] [1-1] [YYYY-MM-DDTHH:MM:SS.MS+HH:MM] [vim.event.BadUsernameSessionEvent] [error] [[USER@DOMAIN]] [] [572805306] [Cannot login [USER@DOMAIN]@##.###.###.##]
A13:17:12 [FQDN] vpxd[1476556]: Event [572805343] [1-1] [YYYY-MM-DDTHH:MM:SS.MS+HH:MM] [vim.event.BadUsernameSessionEvent] [error] [[USER@DOMAIN]] [] [572805343] [Cannot login [USER@DOMAIN]@##.###.###.##]
HH:MM:SS.MS [FQDN] vpxd[1476556]: Event [572805361] [1-1] [YYYY-MM-DDTHH:MM:SS.MS+HH:MM] [vim.event.BadUsernameSessionEvent] [error] [[USER@DOMAIN]] [] [572805361] [Cannot login [USER@DOMAIN]@##.###.###.##]
A HH:MM:SS.MS [FQDN] vpxd[1476556]: Event [572805386] [1-1] [YYYY-MM-DDTHH:MM:SS.MS+HH:MM] [vim.event.BadUsernameSessionEvent] [error] [[USER@DOMAIN]] [] [572805386] [Cannot login [USER@DOMAIN]@##.###.###.##]

Environment

VMware vCenter Server 8.0.x

Dell EMC PowerPath

Cause

 Excessive failed authentication attempts from external service accounts (e.g., [email protected]) consume available HTTP sessions, leading to the error: Out of HTTP sessions: Limited to 2000.

Resolution

1. Identify the Culprit Account: Access the vCenter shell and analyze the vpxd-svcs.log or vpxd.log for failed login patterns. Use the following command to identify the source IP and username:  grep "vim.fault.InvalidLogin" /var/log/vmware/vpxd/vpxd.log | cut -d ' ' -f 1-10 | sort | uniq -c
2.  Mitigate the Issue: Update the credentials for the service account identified in the logs. If immediate service restoration is required, shut down the device or service associated with the failing account.
3. Increase Session Limits (Workaround): If authentication frequency cannot be reduced, increase the MaxSessionCount in vpxd.cfg:

• • Stop the service: service-control --stop vmware-vpxd
  • Make a backup of the vpxd.cfg file
  • Edit /etc/vmware-vpx/vpxd.cfg to increase MaxSessionCount (default 2000).
  • Start the service: service-control --start vmware-vpxd

Additional Information

• Master KB 322837: VPXD Session Exhaustion
• Contact Broadcom Support
• VMware vCenter in Enhanced Linked Mode Snapshot Best Practice