VMware HCX - Underlay network speed performance and limitations for HCX-IX migrations over 2Gbps

Products

• VMware HCX

Issue/Introduction

VMware HCX Interconnect service (HCX-IX) appliances provide replication and vMotion-based migration capabilities across diverse environments. While these appliances support strong encryption and traffic engineering, the transfer speed of migrations depends heavily on the underlying network speed and appliance architecture. This article explains the 2Gbps performance boundary of a single HCX-IX appliance and how to scale performance using a Service Mesh scale-out approach.

Customers may observe the following behaviors when migrating large workloads over high-bandwidth underlays:

• Migration transfer speeds do not exceed 2Gbps despite having higher available underlay bandwidth.
• Increasing CPU or Memory resources on the HCX-IX appliance does not improve transfer rates.
• Sporadic slowness or extended migration windows during concurrent OS-Assisted Migrations (OSAM).

Environment

VMware HCX

Cause

The 2Gbps performance limit is an architectural boundary for a single HCX-IX migration appliance. At maximum concurrency, a single IX appliance cannot utilize underlay bandwidth beyond this threshold. Furthermore, transient network congestion or underlay saturation—often caused by multiple concurrent replication jobs—can lead to drops in the effective transfer rate.

Resolution

To utilize network underlays with bandwidth greater than 2Gbps, you must use an appliance scale-out approach to increase concurrency and total utilization.

1. Service Mesh Scale-Out
HCX supports a maximum of 200 concurrent migrations. To distribute this load and maximize bandwidth:

• Deploy additional Service Meshes within your HCX installation.
• Ensure each IX appliance is deployed to a different ESXi host to avoid resource contention.
• Verify that the underlying storage is capable of supporting the aggregated high transfer rates.

2. Manage Migration Concurrency
To prevent underlay saturation and transient congestion:

• Stagger migration schedules to avoid multiple high-bandwidth jobs running simultaneously.
• Limit the number of concurrent OSAM migrations based on your real-time underlay capacity.

3. Performance Validation
If you suspect underlay constraints are impacting migration speed:

• Refer to Steps to run 'Perftest' in HCX to validate current network throughput.
• Review the VMware Configuration Maximums for the latest performance boundaries.
• Fixed in current releases. See Download Broadcom products and software for steps to download the latest HCX updates.
  
  

Additional Information

For other performance outcomes, please refer to: Network Underlay Minimum Requirements