vMotion fail due to packet drop

Products

• VMware vSphere ESXi

Issue/Introduction

• vMotion VM from ESXi-A to ESXi-B fails. The packet transfer rate is around 1 MB/s which is very low.

vmkernel.log:

Wa(180) vmkwarning: cpuxx:xxxx)WARNING: VMotionUtil: xxx: xxxx D: failed to read stream keepalive: Connection closed by remote host, possibly due to timeout
In(182) vmkernel: cpuxx:xxxxx)Migrate: xxx: xxxxx D: MigrateState: Failed
Wa(180) vmkwarning: cpuxx:xxxxx)WARNING: Migrate: xxxx: xxxxxxx D: Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.
In(182) vmkernel: cpuxx:xxxxx)VMotion: xxxx: xxxxxxx D: VMotion bandwidth in last 1s: 440 KB/s, 10s: 1 MB/s, 20s: 1 MB/s, 60s: 1 MB/s,

• VM could vMotion from ESXi-B to ESXi-A.
• The vMotion vmk1 on ESXi-A uses vmnic-A. If vmk1 uses vmnic-B does not have vMotion failure.

 

Environment

VMware vSphere ESXi

Cause

There is packet drop from vMotion source to destination ESXi.

Do packet capture on both vMotion source and destination ESXi during vMotion. Some packets are sent by source ESXi but not received by destination ESXi. This causes TCP retransmissions, resulting in significant fluctuations in vMotion traffic, preventing the speed from reaching its maximum, and ultimately causing vMotion to time out.

Resolution

1. Change DRS to manual to avoid auto vMotion.
2. Remove and swap the GBICs along with fibers on vmnic-A and vmnic-B of the problem ESXi. Then test vMotion again. If the problem is resolved, the hardware of vmnic-A is not the issue. If the problem persists, the GBICs, fibers, and physical network are not the problem; therefore, the hardware of vmnic-A needs to be checked.