HCX Manager UI slow after upgrade to 9.1.0 due to Service Mesh PMTU probe saturation

Products

• VMware HCX

Issue/Introduction

After upgrading to VMware HCX 9.1.0, the HCX Manager UI becomes extremely slow or unresponsive, particularly when accessing the "Service Mesh" or "Interconnect" tabs.

• Symptoms:
  • Significant delays (up to 10+ minutes) when loading the service-mesh view.
  • UI actions may time out.
  • Performing a HAR trace shows the following requests in cancelled state:
    • How to capture and preserve network requests from a browser - HAR file capture

    • https://<HCX_Manager>/hybridity/api/interconnect/underlay/pmtu/serviceMesh/servicemesh-<uuid>

      
      
  • In HCX Manager /common/logs/httpd/access.log shows multiple API calls for PMTU test triggered when clicking on "Service-Mesh" tab :
    

    [access.log] #.#.#.# - - [##/Aug/###:09:21:56 +0000] "GET /hybridity/api/interconnect/underlay/pmtu/serviceMesh/servicemesh-########-####-####-####-35a6b4f82581?endpointId=############-########-####-#####-####-####e66053ec&vcGuid=########-####-####-#### HTTP/2.0" 200 10 116668

Environment

VMware HCX 9.1

Cause

• The HCX Manager appliance experiences backend thread starvation.
• Accessing the Service Mesh dashboard triggers a burst of parallel PMTU (Path Maximum Transmission Unit) probe API calls in environments with a large number of Service Meshes (e.g., 40-50+).

Resolution

Workaround:

• Scale up HCX Manager appliance resources (vCPU/RAM) to provided immediate relief for the increased concurrent API processing jobs.
  • HCX - Bulk Migration & Replication Assisted vMotion (RAV) scalability guide - for HCX version 4.10 or higher

If you have encountered this behavior, please gather the below details and open an SR with Broadcom Support: Creating and managing Broadcom cases

• Source/Destination HCX Manager log bundle with Database Dump selected.