Unable to login to HCX UI with the error message "invalid username or password, or too many active sessions"

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

After upgrading to HCX version 4.10.2, the HCX Manager experiences authentication failures and web service disruptions. These manifest as "invalid username or password, or too many active sessions" errors when attempting to access the HCX UI, despite having valid credentials.

Concurrent Symptoms

• HCX Web UI Access Failure: Users receive "invalid username or password, or too many active sessions" error when attempting to login with valid credentials
• vCenter Plugin Failure: Error "Http failure response for https://<vc-ip/fqdn>/plugins/com.vmware.hcx.plugin~4.10.2.24404456~115218814/<hcx-ip/fqdn>-443/vsphere-client/ui/hcx/hcx-ui/rest/hybridity/api/sessions: 500 Internal Server Error" when accessing HCX through vCenter
• Service Availability Issues: "Cloud resources are unavailable" message appears in the UI workload mobility page
• Network Extension Problems: "Remote Endpoint Failed" error when attempting to extend networks
• Site Pair Authentication Failures: Unable to authenticate site pairs despite credentials working correctly for SSH and web console access
• Temporary Resolution: Rebooting the HCX Manager provides temporary relief

Note: It is possible that users may experience some of these symptoms and not others depending on their specific environment and usage patterns. However, the diagnostic log error shown in the Diagnostic Logs section below will always appear in the HCX Manager logs, even if all the concurrent symptoms are not present.

Diagnostic Logs

When experiencing this issue, the following error appears in HCX Manager /common/logs/admin/web.log:

2025-##-## UTC [https-jsse-nio-8443-exec-8, , , TxId: ] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet]- Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached] with root cause
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

When attempting to use the HCX Plugin via vCenter Server UI, the error below is displayed:

Environment

VMware HCX 4.10.2

Cause

This issue is caused by a memory leak in the vCenter to HCX authentication process which increases in severity with a higher number of sessions.

As the number of concurrent sessions on HCX Manager grows, users are significantly more likely to encounter this issue.

Resolution

This issue is resolved in VMware HCX 4.10.3 and later, available at Broadcom downloads.

To recover from the issue follow one of the below options: 

• Restart web service from HCX 9443 Admin UI or from SSH:
  systemctl stop web-engine
systemctl start web-engine
  
  
• Reboot HCX Manager VM

Note: Reboot of HCX Manager will cause access to migrations and management of Network Extensions to be temporarily unavailable

If the error persists after following these steps, contact Broadcom Support for further assistance.

Please provide the below information when opening a support request with Broadcom for this issue.

Collect Source and Target HCX log bundles from HCX 9443 page with HCX database dump selected. Refer to Gather Technical Support Logs

Additional Information

This issue affects all deployment types in HCX 4.10.2 and can impact multiple services, including

• Authentication and session management
• Service mesh status monitoring
• Log bundle generation capabilities
• General web service operations

Beyond user interface access, this issue affects:

• Ongoing migration operations may remain functional but cannot be monitored or managed
• Network extension connectivity remains intact but configuration changes become impossible
• Automated scripts or third-party tools using HCX APIs will experience authentication failures
• Inability to perform any actions across the site pair

The upgrade to 4.10.3 or later versions includes significant improvements to session handling and resource management, providing a permanent resolution to these issues.

For more information on memory leaks please see this external article - Understanding Memory Leak Detection in Software Development