HCX Service Mesh update stuck at 40%
Article ID: 452542
Updated On:
Products
Issue/Introduction
Primary Product and Component: VMware HCX (Manager)
Exact Error Message from /common/logs/admin/app.log: java.lang.OutOfMemoryError: Java heap space
Symptoms:
- After upgrading both HCX Managers from 4.11.3 to 4.11.4, the HCX Service Mesh update becomes stuck at 40% for an extended duration (e.g., >720 minutes).
- The task progress stalls during the Deploying new appliance phase, specifically at the Allocating Resources for Interconnect Appliances stage for the destination appliance.
- The issue typically manifests following an environment upgrade (e.g., from HCX 4.11.3 to 4.11.4).
Environment
- Product: VMware HCX 4.11.x
- Environment: Google Cloud VMware Engine (GCVE) or On-Premises to Cloud deployments.
- Configuration: Large-scale Mobility Optimized Networking (MON) with a large volume (1,800+) of concurrent Network Extension (NE) jobs.
Cause
The root cause is a java.lang.OutOfMemoryError on the destination HCX Manager. Large-scale MON usage generates a large volume of concurrent NE jobs that exhaust the available Java heap space (approximately 1.7 GB consumed by the NetworkStretch service). Additionally, large heap dump files (.hprof) generated during these OOM events can saturate the /common partition, further impacting manager performance.
Resolution
If you believe you have encountered this issue, open a support case with Broadcom Support and refer to this KB article. For more information, see Creating and managing Broadcom support cases.
Collect log bundles from source and target HCX Managers with database dump included: Collecting Logs and Database Dumps for Ongoing or Intermittent Issues in HCX
- If HCX UI log bundle collection is unavailable or not working: HCX - command line logs collection
Additional Information
Feedback
