HCX Migration Performance Caps at ~1.4 Gbps Despite Higher Available Network Bandwidth
Article ID: 447245
Updated On:
Products
Issue/Introduction
During large virtual machine migrations using Replication Assisted vMotion (RAV) or Bulk Migration, administrator may observe that total replication throughput maxes out at a specific threshold (e.g., 1.35 to 1.45 Gbps), even when the underlying network is capable of much higher speeds.
For example, when concurrently migrating roughly 7 TB of data (e.g., a 4 TB VM and a 3 TB VM), the process may take approximately 11.5 hours to complete.
Log Evidence of Migration Timeline (11.5 Hours):
Below is an excerpt demonstrating the timeline of a 4 TB VM migration, which began at 11:05 UTC and completed syncing at 22:28 UTC.
/commmon/logs/admin/app.log:
2026-06-29 11:05:09.598 UTC [...] INFO c.v.h.s.m.jobs.MobilityMigrationJob- Migration Requested -> Id: a7####e-####...::#####f###..., Type: RAV
2026-06-29 11:08:50.303 UTC [...] INFO c.v.h.s.r.j.ReplicationTransferMonitor- Source side replication status is: {"state":"syncing","progressInfo":{"progress":0,"bytes_transferred":0...
2026-06-29 11:31:25.531 UTC [...] INFO c.v.h.s.r.j.ReplicationTransferMonitor- Source side replication status is: {"state":"syncing","progressInfo":{"progress":10,"bytes_transferred":155649703936...
2026-06-29 15:08:15.286 UTC [...] INFO c.v.h.s.r.j.ReplicationTransferMonitor- Source side replication status is: {"state":"syncing","progressInfo":{"progress":50,"bytes_transferred":1248406355968...
2026-06-29 21:50:58.023 UTC [...] INFO c.v.h.s.r.j.ReplicationTransferMonitor- Source side replication status is: {"state":"syncing","progressInfo":{"progress":91,"bytes_transferred":3184190611456...
2026-06-29 22:28:12.498 UTC [...] INFO c.v.h.s.r.j.ReplicationTransferMonitor- Source side replication status is: {"state":"idle","progressInfo":{}} for vm: TEST-HCX-RAV-BIG3...(Note: A second 3 TB VM with ID ######-####-####-####-###### was processed concurrently during this same window making it around 7 TB of data taransfer).
Network Performance Test (perftest) Results:
When running a perftest all from the HCX Manager ccli to the IX appliance, the underlying network proves to be highly capable, yet the IPsec tunnel shows standard crypto-overhead limitations:
Raw Link (UPLINK): 9.19 Gbps (indicates a healthy 10 Gbps physical link).
Encapsulation (FOU Tunnel): 7.4 Gbps (normal 18% reduction for transport wrapping overhead).
Encryption (IPSEC Tunnel): 2.39 Gbps (throughput drop due to intensive CPU processing required for encryption/decryption).
Latency: 0.35 to 0.41 milliseconds across all interfaces.
Note: The above perftest results are for explanation, the results may vary from environment to environment. Environment
VMware HCX
Cause
The migration speed is not bottlenecked by the physical network. The perftest confirms the IPsec tunnel can push up to 2.39 Gbps.
However, a single HCX Interconnect (IX) appliance has a hard architectural processing boundary of approximately 2 Gbps for migrations and known architectural limit. This limit exists because IPsec encryption and decryption are highly CPU-intensive tasks. Seeing concurrent migration throughput hovering under this 2 Gbps limit (for example 1.35 to 1.45 Gbps) is normal and expected behavior for a single IX appliance operating under heavy load, as it approaches its architectural compute ceiling.
Resolution
To enhance overall migration throughput and scale beyond the limits of a single IX appliance, implement the following architectural and operational optimizations:
- Set CPU Reservations (Critical): By default, HCX appliances do not have CPU reservations configured in vSphere. Explicitly set full CPU reservations for the HCX-IX appliances. This prevents the ESXi hypervisor from CPU-throttling the appliance's crypto-processing engine during heavy replication waves.
- Scale Out the Service Mesh: Deploy additional Service Meshes within your HCX installation to spin up more HCX-IX appliances. This directly increases your overall concurrent processing and bandwidth capacity.
- Separate the Compute: Ensure that each newly deployed HCX-IX appliance is pinned to a different physical ESXi host. If multiple IX appliances reside on the same host, they will compete for the exact same physical CPU cycles for IPsec encryption, nullifying the scale-out benefits.
- Distribute the Load: Route your migration waves across the multiple IX appliances to effectively load-balance the replication traffic.
- Manage Concurrency: Pushing too many concurrent VMs through a single bulk migration wave can cause random I/O bottlenecks on the destination datastore. Instead of migrating large batches of VMs at once, batch them in smaller waves (e.g., 10-15 at a time) to allow the storage array to process sequential writes more efficiently.
- Evaluate Storage Capabilities: Investigate the write IOPS and throughput limits of the target datastore. If the destination is an older hybrid vSAN or a heavily utilized VMFS LUN, it simply might not be capable of ingesting data faster than the speeds HCX is currently pushing.
Additional Information
Feedback
