vMotion fails with the error: The migration was cancelled because the amount of changing memory for the VM was greater than the available network bandwidth, meaning the migration was not making forward progress

Products

• VMware vSphere ESXi

Issue/Introduction

 

• Cannot vMotion a virtual machine
• Performing a vMotion of a virtual machine fails
• You see the error in the UI:
  
  "The migration was cancelled because the amount of changing memory for the VM was greater than the available network bandwidth, meaning the migration was not making forward progress. Please attempt the migration again when the VM is not as busy or more network bandwidth is available."

Environment

• VMware vSphere vCenter
• vSphere ESXi

Cause

This issue occurs when the amount of changing memory for the virtual machine is greater than the available network bandwidth.

Note: Changing memory refers to the amount of memory that has changed since the initial copy. In busy workloads, memory contents may change faster than they can be transferred over the network.

Resolution

To resolve this issue:

1. Verify VMkernel Configuration and Resource Isolation
    A. Check connectivity and logs:
        * Ensure the source and destination hosts can communicate over the vMotion network.
        * Review vmkernel.log for any NIC or driver-related errors (e.g., link flaps).
    B. Audit Port Configuration:
        * Log in to the vSphere Client and select the host.
        * Navigate to Configure > Networking > VMkernel adapters.
        * Verify the adapter used for vMotion is Enabled and assigned the correct IP/Subnet.
    C. Identify TCP/IP Stack Contention:
        * Check the TCP/IP Stack assigned to the vMotion adapter.
        * Note: If using the Default TCP/IP stack, it shares resources with Management/Backup traffic. For Encrypted VMs, this often causes the contention that triggers bandwidth timeouts.

2. Remediate Netstack: Migrate to Dedicated vMotion TCP/IP Stack
    If the adapter is on the Default stack, migrate it to the dedicated vMotion stack to isolate traffic:
    A. Record current settings:         * Note the IP address and Subnet Mask of the current vMotion adapter (e.g., vmk1).
    B. Remove the existing adapter:         * Delete the current vMotion VMkernel adapter.
        * Note: You cannot edit the stack on an existing adapter; it must be recreated.
    C. Recreate on the vMotion stack:
        * Click Add Networking > VMkernel Network Adapter.
        * Select the target Distributed Port Group.
        * In Port properties, select vMotion stack from the TCP/IP stack dropdown.
        * Input the IP/Subnet recorded in Step 2A.

3. Examine Physical Infrastructure

• A. Physical Connections: Ensure all SFP+ modules and cables are secure and free of physical damage.

• B. Switch Port Audit: * Check for errors (CRC, drops) on the physical switch ports.

  • Verify that the vMotion VLAN is correctly trunked to all uplinks in the teaming policy.

• C. Firewall Integrity: Verify that the required ports for vMotion (TCP 8000 and 9001 for encrypted vMotion) are open end-to-end.

4. Analyze Network Performance and Compatibility

• A. Performance Monitoring: Use esxtop or network monitoring tools to check for high latency or 10%+ packet loss during migration attempts.

• B. Driver/Firmware Alignment: * Consult the VMware Compatibility Guide for your specific NIC.

  • Update to the latest recommended Native Driver (e.g., moving from qfle3 to qcnic).

• C. Workload Timing: For VMs with extremely high memory churn, attempt the vMotion during off-peak hours when the memory change rate is lower.