HCX Bulk and RAV Migrations Stalled at 0% Base Sync
search cancel

HCX Bulk and RAV Migrations Stalled at 0% Base Sync

book

Article ID: 319757

calendar_today

Updated On:

Products

VMware HCX

Issue/Introduction

  • HCX Bulk and RAV migration workflows may fail to progress beyond 0% during the "Base Sync Initiated" phase. While other migration types like HCX vMotion or Cold Migration may succeed, these replication-based migrations remain in an "Active" state with no progress for extended periods. There is often no visible error in the HCX Manager UI, and the workflows remain actionable (can be halted or cancelled).

    Log Symptoms: Review the following logs for errors related to communication timeouts:

    • Source ESXi (/var/run/log/vmkernel.log): HBR: Failed to establish connection to [IX-IP]:31031: Timeout
    • HCX Manager (/common/logs/admin/app.log): ERROR c.v.v.h.a.hbr.HbrServerInstance- Error running request as communication to HBR server failed INFO c.v.h.s.r.j.ReplicationTransferMonitor- Current bytes transferred value 0 is not greater than the already transferred bytes value 0

Environment

VMware HCX 

Cause

The failure to progress is typically caused by an inability to establish the vSphere Replication handshake between the source ESXi host and the HCX Interconnect (IX) appliance. Primary triggers include:

  1. Network Port Blocks: Required port TCP 31031 (or 44046 for RAV) is closed between the ESXi VMkernel interface and the IX appliance.
  2. MTU Fragmentation: A mismatch in Maximum Transmission Unit (MTU) settings between the ESXi VMkernel, the Distributed Virtual Switch (DVS), or the HCX Network Profile causing silent packet drops.
  3. Interface Tagging Conflicts: Multiple VMkernel adapters on the same ESXi host tagged for "vSphere Replication," causing traffic to egress through an unrouted path.
  4. Subnet/Mask Mismatch: Inconsistent subnet masks between the ESXi host and the HCX-IX appliance (e.g., /24 vs. /25).
  5. Layer 3 Routing Failure on Dedicated Replication Network: If a dedicated network profile/interface is configured for vSphere Replication instead of the default management network, Layer 3 routing or physical connectivity may be broken on that specific path. This results in standard ping test failures and continuous HBR timeouts to the IX appliance on port 31031, even if firewall rules allow the port.

Resolution

Follow these troubleshooting steps in sequence to restore replication connectivity:

1. Validate Port Connectivity

Verify that TCP Port 31031 is open from the source ESXi host to the HCX-IX appliance IP.

  • Log in to the ESXi shell via SSH.
  • Run the command: nc -zv [IX-Appliance-IP] 31031
  • Success: Connection to [IP] 31031 port [tcp/*] succeeded!
  • Failure: Connection refused or timeout. (Check physical firewalls and NSX Distributed Firewall rules).

2. Verify MTU Consistency

Mismatched MTUs will cause large replication packets to be dropped.

  • Perform a vmkping test from the ESXi host to the IX appliance using the specific VMkernel and packet size: 
    • vmkping -I vmk# -d -s 1472 [IX-Appliance-IP] (for 1500 MTU) 
    • vmkping -I vmk# -d -s 8972 [IX-Appliance-IP] (for 9000 MTU)
  • Ensure the Distributed Virtual Switch (DVS) MTU is equal to or higher than the MTU set in the HCX Network Profile.

3. Review VMkernel Tagging

Replication traffic may fail if the host is confused by multiple replication paths.

  • Check ESXi host networking to ensure only one VMkernel adapter is tagged for "vSphere Replication."
  • If multiple interfaces are tagged, remove the tag from the unused interface, cancel the migration, and re-initiate.

4. Check Subnet Mask Alignment

Ensure the subnet mask configured in the HCX Network Profile exactly matches the subnet mask on the ESXi Management/Replication network. A mismatch (e.g., host on /25 and IX on /24) can lead to ARP failures across the underlay.

5. Virtual Machine Configuration Checks

    • Disk Limits: Ensure the VM does not exceed 64 disks (supported only on ESXi 8.0 U1 and later).
    • Stale Files: Check the VM home directory for zero-byte .vmxf files or stale .lck files from previous failed migration attempts. Remove these before retrying.

6. If ICMP/ping tests fail on the dedicated vSphere Replication interface despite the replication port 31031 being open:

  • Workaround: Temporarily untag vSphere Replication on the dedicated interface (vmk#) and enable it on the management interface (vmk0) on the Source ESXi host. Update the IX appliance Network Configuration to match, allowing current migrations to proceed over the management network.

  • Permanent Fix: Engage the network team to investigate and restore proper Layer 3 routing and physical connectivity on the dedicated replication interface. Once routing is restored, revert the replication tags back to the dedicated vmk#.

Additional Resources

Additional Information

If the vMotion network is different from the Management network:

  • Edit Compute profile -> Click CONTINUE to select vMotion Network Profile ->  click ADVANCED CONFIGURATIONS.
  • Add the static route for vMotion Network as the Network and Gateway for the vMotion network as the Next Hop accordingly. 
  • Finish the compute profile and Resync the service mesh to apply the changes.
  • For HCX Health Check, access: HCX - Health Check and Best Practices

If the ESXi host is configured with a separate interface for vSphere Replication, that interface will be used to communicate with the IX Appliance. Ensure that Layer 3 routing is fully functional on this dedicated segment. If migrations stall, verifying which vmk is tagged for vSphere Replication and temporarily failing it back to the management vmk can help isolate routing issues.

Powered by Wolken Software