Unable to login to HCX UI with the error message "invalid username or password, or too many active sessions"

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

After upgrading to HCX version 4.10.2, users may experience intermittent authentication failures and web service disruptions. These manifest as "invalid username or password, or too many active sessions" errors when attempting to access the HCX UI, despite having valid credentials. 

The system exhibits multiple concurrent symptoms

• Users receive "invalid username or password, or too many active sessions" error when attempting to access the HCX UI or authenticating site pairs
• "Cloud resources are unavailable" message appears in the UI workload mobility page
• Login failures occur, despite credentials working correctly for SSH and web console access
• Remote Endpoint Failed error when going to Extend Network Extension due to site pair authentication
• Rebooting the HCX Manager temporarily provides relief for an undetermined amount of time
• Below similar logging would be noticed in HCX Manager /common/logs/admin/web.log
  

  2024-##-## UTC [https-jsse-nio-8443-exec-8, , , TxId: ] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet]- Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached] with root cause
  java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

• When attempting to use the HCX Plugin via vCenter Server UI, the error below is displayed:

  Http failure response for https://<vc-ip/fqdn>/plugins/com.vmware.hcx.plugin~4.10.2.24404456~115218814/<hcx-ip/fqdn>-443/vsphere-client/ui/hcx/hcx-ui/rest/hybridity/api/sessions: 500 Internal Server Error

  

Environment

VMware HCX 4.10.2

Cause

• This issue is caused by a memory leak in the process that handles authentication between vCenter and HCX.
• The issue is more likely to be seen with a greater number of active sessions.

Resolution

This issue is resolved in VMware HCX 4.10.3 and later, available at Broadcom downloads.

To recover from the issue follow one of the below options : 

• Restart web service from HCX 9443 Admin UI or from SSH :
  systemctl stop web-engine
systemctl start web-engine
  
  
• Reboot HCX Manager VM

NOTE: Reboot of HCX Manager will cause access to migrations and management of Network Extensions to be temporarily unavailable

If the error persists after following these steps, contact Broadcom Support for further assistance.

Please provide the below information when opening a support request with Broadcom for this issue.
Collect Source and Target HCX log bundles from HCX 9443 page with HCX database dump selected. Refer Gather Technical Support Logs

Additional Information

This issue affects all deployment types in HCX 4.10.2 and can impact multiple services, including

• Authentication and session management
• Service mesh status monitoring
• Log bundle generation capabilities
• General web service operations

The upgrade to 4.10.3 or later versions includes significant improvements to session handling and resource management, providing a permanent resolution to these issues.