vMotion fails: migration cancelled due to high memory change rate or low bandwidth

Products

• VMware vSphere ESXi

Issue/Introduction

Performing a vMotion of a virtual machine fails with the following error: "The migration was cancelled because the amount of changing memory for the VM was greater than the available network bandwidth, meaning the migration was not making forward progress. Please attempt the migration again when the VM is not as busy or more network bandwidth is available."

Environment

VMware vSphere

Cause

This issue occurs when the amount of changing memory for the virtual machine is greater than the available network bandwidth.

Note: Changing memory refers to the amount of memory that has changed since the initial copy. In busy workloads, memory contents may change faster than they can be transferred over the network.

Resolution

1. Verify VMotion VMkernel Configuration and Resource Isolation

    Check connectivity and logs:

• Ensure the source and destination hosts can communicate over the vMotion network.
• Review vmkernel.log for any NIC or driver-related errors (e.g., link flaps).

    Validate vMotion Network Interface Settings

• Log in to the vSphere Client and select the host.
• Navigate to Configure > Networking > VMkernel adapters.
• Verify that the vmkernel adapter used for vMotion has vMotion tag Enabled and assigned the correct IP/Subnet.

    Identify TCP/IP Stack Contention:

• Check the TCP/IP Stack assigned to the vMotion adapter.
• Note: If using the Default TCP/IP stack, it shares resources with Management/Backup traffic. For Encrypted VMs, this often causes the contention that triggers bandwidth timeouts.

2. Remediate Netstack: Migrate to Dedicated vMotion TCP/IP Stack
    If the adapter is on the Default stack, migrate it to the dedicated vMotion stack to isolate traffic:

1. Note the IP address and Subnet Mask of the current vMotion adapter (e.g., vmk1).
2. Delete the current vMotion VMkernel adapter. Note: You cannot edit the stack on an existing adapter; it must be recreated.   
3. Recreate the vmkernel adapter using the vMotion stack:

• • Click Add Networking > VMkernel Network Adapter.
  • Select the target Distributed Port Group.
  • In Port properties, select vMotion stack from the TCP/IP stack dropdown.
  • Input the IP/Subnet recorded in the first step(i)

3. Examine Physical Infrastructure

• Physical Connections: Ensure all SFP+ modules and cables are secure and free of physical damage.
• Switch Port Audit: Check for errors (CRC, drops) on the physical switch ports.
  • Verify that the vMotion VLAN is correctly trunked to all uplinks in the teaming policy.
• Firewall Integrity: Verify that the required ports for vMotion (TCP 8000 and 9001 for encrypted vMotion) are open end-to-end.

4. Analyze Network Performance and Compatibility

• Performance Monitoring: Use esxtop or network monitoring tools such as performance charts on the VCenter UI to check for high latency. Refer to Collecting esxtop batch data for ESXi performance troubleshooting to collect extop data at the time of the issue.
• Driver/Firmware Alignment: Check the VMware Compatibility Guide for your specific NIC.
  • Update to the latest recommended Native Driver (e.g., moving from qfle3 to qcnic).
• Workload Timing: For VMs with extremely high memory churn, attempt the vMotion during off-peak hours when the memory change rate is lower.

If the issue persists, kindly collect the logs of both source and destination host immediately along with esxtop batch data and open a case with Broadcom Support Team