vMotion -- Failing at 20% -- Errors -- "Failed waiting for data. Error 195887137. Timeout" and/or "Failed to queue disk I/O for unshared storage vMotion: Timeout"

Products

• VMware vSphere ESXi

Issue/Introduction

The following Symptoms do apply:

 

• In the vSphere Client and/or via the /var/log/hostd.log of the Source Host, you observe one or more of the following errors:

Failed waiting for data. Error 195887137. Timeout.

Failed to queue disk I/O for unshared storage vMotion: Timeout

Failed to start migration pre-copy. Error: Connection closed by remote host, possibly due to timeout.

Failed to receive ###/### bytes from remote host <##.##.##.##>: Connection reset by peer

Failed to receive ###/### bytes from remote host <##.##.##.##>: Timeout

Migration transfer failed during receive operation to socket ###: received ### bytes: Connection reset by peer

Failed to read stream keepalive: Connection reset by peer Migration to host <##.##.##> failed with error Connection reset by peer (195887179)

vMotion migration ### failed to send buffer to remote host. ### failed writing stream completion: Timeout 

 

Examples:

 

 

 

 

 

 

The following Symptom might apply:

• In the /var/log/vmkernel.log on the participating Host, you may see messages similar to this example:

Tcpip_Vmk: 97: arp: ##:##:##:##:##:## is using my IP address ##.###.##.##.### on vmk#!

 

Environment

vSphere All Versions

Cause

This issue occurs if there is a Network connectivity issue and/or Network misconfig between the Source and Destination Host participating in the vMotion.

Resolution

Check the Health & Config of the vMotion Network between Source and Target Host participating in the vMotion:

 

 

Preparing Tasks:

A.) Identify the configured MTU used on the participating vMotion VMkernel Ports (= vmks) by following KB 313061
B.) Identify the vmnics (= Network cards) used by the participating vMotion VMKernel Ports by following the steps 1 & 2 in KB 344313 

 

 

Checks: 

• Check for IP address conflicts. Each Host should have a vMotion VMKernel Port (= vmk) assigned a unique IP address
• Check for Packet loss on the participating vmnics (= Network cards) during and/or after the vMotion by following the Troubleshooting Steps of KB 318636
• Check the involved vmnics (= Network cards) of the Source and Target Hosts for Errors/dropped Packets by following KB 341594

• Ensure that each participating vMotion VMKernel Port Group has the same security settings. A Security mismatch can cause a vMotion operation to fail

For example, a failure occurs if the Source VMKernel Port Group is set to allow promiscuous mode and the destination VMKernel Port Group is set to disallow promiscuous mode.

See the following KBs for more info: KB 324553, KB 324520

Additional Information

Understanding and troubleshooting vMotion