vMotion is slow and ESXi hosts take a long time to enter maintenance mode due to upstream network rate limit

Products

VMware vSphere ESXi

Issue/Introduction

vMotion migrations run far below the physical link speed of the host NICs. Per-migration sustained throughput sits at a few MB/s on a 10/25/100 GbE network, instead of the hundreds to thousands of MB/s the link can carry.
ESXi hosts take many hours or days to enter maintenance mode, because DRS cannot evacuate the running virtual machines fast enough.

Some migrations are canceled with an error similar to:

The migration was canceled because the amount of changing memory for the
virtual machine was greater than the available network bandwidth. Attempt the
migration again when the virtual machine is not as busy or more network
bandwidth is available.

Migrations also abort with stream-keepalive timeouts and connection resets similar to:

vMotion migration [xxxxxxxxx:xxxxxxxxxxxxx] failed to read stream keepalive:
Connection closed by remote host, possibly due to timeout. Failed waiting for
data. Error xxxxxxxxx. Connection closed by remote host, possibly due to timeout.

In /var/run/log/vmkernel.log on the affected host, the per-migration bandwidth summaries show a very low sustained rate, similar to:
```
VMotion: 2363: <migration-id> S: VMotion bandwidth in last 1s: 3 MB/s, 10s: 4 MB/s, 20s: 3 MB/s, 60s: 3 MB/s
```
Over a long observation window, the affected host logs many migration setup events and zero completions. Live esxtop shows the vMotion VMkernel adapter near saturation at that low absolute rate.

Additional symptoms reported:

"It is taking days for a server to enter maintenance mode."
The delay blocks routine host maintenance, for example a vGPU Manager (host graphics driver) upgrade driven by vSphere Lifecycle Manager.

Environment

VMware vSphere ESXi 8.0 and later
vMotion configured over a 10 GbE or faster network
Hosts distributed across more than one rack or cabinet, each served by its own top-of-rack or aggregation switches

Cause

An upstream network device, not the ESXi host, is limiting the vMotion throughput. A rate limit, traffic policer, or storm-control policy on a switch in the path drops the packets that exceed its configured rate. The policer drops those packets silently, without sending Ethernet pause frames back to the host. vMotion runs over TCP, so TCP reads the silent packet loss as congestion and collapses the flow to a few MB/s. When a stall lasts long enough, the connection times out and resets, which produces the "Connection closed by remote host" message. When a virtual machine dirties memory faster than the throttled link can copy it, vMotion cannot converge and cancels with the "changing memory greater than available network bandwidth" message.

The throttle is usually directional. It limits traffic in one direction, commonly the direction leaving the hosts on the affected rack, while the opposite direction runs at full speed.

This cause is easy to mistake for a host or vMotion configuration problem, because the host is where the error appears. The following are commonly suspected but are not the cause, and each was ruled out where this was first diagnosed:

vMotion sharing the management VMkernel adapter (vmk0). A correctly configured host can still run vMotion at full link rate over the management adapter. Moving vMotion onto its own adapter is good practice, but sharing vmk0 is not what limits throughput here.
The management VLAN, MTU 1500, or the absence of jumbo frames.
Encrypted vMotion.
The totalVmknicLinkspeed value logged per migration (see Internal Notes).
A NIC, cable, transceiver, or duplex fault. A physical-layer fault degrades both directions, whereas this throttle is directional.

Resolution

The goal is to prove whether the limit lives in the network path and not on the host, and if so, then have the network team remove it. Use the following sequence.

Confirm the throughput shortfall. In the vSphere Client, open the slow migration task, or read /var/run/log/vmkernel.log on the affected host and find the VMotion bandwidth in last 60s summaries. A sustained rate of a few MB/s on a 10 GbE or faster link confirms the symptom.
Rule out the host with a control test. Put a host from a different rack or cabinet into maintenance mode. If that host evacuates at full speed while the original host does not, and the two hosts share the same ESXi build and network configuration, the host configuration is not the cause.
Test for directional asymmetry. Run a throughput test in both directions between an affected host and a healthy host, for example with iperf, or compare a vMotion sent from the affected host against one sent to it. Fast in one direction and slow in the other indicates a rate limit or policer on a network device. Slow in both directions points instead to a physical-layer fault.
Rule out Ethernet flow control on the host. On the affected host, check the pause-frame and transmit-error counters on the vMotion uplinks:
```
esxcli network nic stats get -n vmnicX
```
Pause-frame counters and transmit-error counters at zero mean the host is not being flow-controlled and is not dropping on transmit, which is consistent with a silent policer upstream.
Engage the network team with the directional evidence. Ask them to inspect the switches that serve the affected rack (top-of-rack and aggregation) for any rate limit, traffic policer, service policy, or storm-control configuration on the VLAN that carries vMotion, with attention to the direction of traffic leaving the hosts.
Test and work around with a dedicated vMotion network. Create a dedicated vMotion VMkernel adapter on a separate VLAN, using the vMotion TCP/IP stack. If vMotion then runs at line rate, the policy was scoped to the original VLAN and the cluster is unblocked. The change is reversible, so you can test it on one host first.
Unblock urgent maintenance immediately. If the maintenance task only requires the host to be empty, for example a Lifecycle Manager driver or firmware remediation, evacuate the host by powering off or cold-migrating its workloads instead of live vMotion. The host can then enter maintenance mode without depending on vMotion bandwidth.
Map the scope. Test maintenance mode on more hosts and record which rack and switch pair each host uses. This confirms which hosts are affected and gives the network team the boundary of the problem.

Additional Information

For the migration cancellation message itself, see vMotion fails with the error: The migration was cancelled because the amount of changing memory for the VM was greater than the available network bandwidth, meaning the migration was not making forward progress.
Recommendation: place vMotion on its own VMkernel adapter and VLAN using the dedicated vMotion TCP/IP stack, rather than sharing the management VLAN. A dedicated vMotion network is easier to reason about and is less likely to be caught by a policy intended for management or other infrastructure traffic.
Recommendation: keep QoS, rate-limit, policer, and storm-control policy consistent across every rack and switch pair in a cluster. Inconsistent policy between racks produces exactly this kind of per-rack performance difference.
Recommendation: if you apply rate limiting or storm control to a VLAN that carries vMotion, size it for vMotion bursts. vMotion is designed to use as much bandwidth as the link offers during the bulk pre-copy pass.
Recommendation: jumbo frames (MTU 9000) end to end remain a vMotion best practice for throughput and CPU efficiency. Jumbo frames depend on every switch in the path being configured for them, and they are a separate improvement from the policer described here.