vMotion fails with the error: Migration to host failed with error Out of memory

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Cannot perform a vMotion
• vMotion fails with the error:
  
  Migration to host <x.x.x.x> failed with error Out of memory (195887124)
  vMotion migration [-1408236994:1349408191430727] failed to allocate 527876 bytes for stream channel data segment.
  vMotion migration [-1408236994:1349408191430727] timed out waiting 20001 ms for 527876 bytes of heap space. Check for network problems that may be preventing the remote host from reliably receiving vMotion network traffic.
  
  Where x.x.x.x is the destination host vmkernel IP.
  

• In the vmkernel.log file of the source host, you see entries similar to:
  
  WARNING: VMotionUtil: 1000: 1349408191430727 S: timed out waiting 20001 ms for 527876 bytes of heap space. Check for network problems that may be preventing the remote host from reliably receiving vMotion netESC[0$
  WARNING: VMotionUtil: 1000: 1349408191430727 S: timed out waiting 20001 ms for 527876 bytes of heap space. Check for network problems that may be preventing the remote host from reliably receiving vMotion netESC[0$
  WARNING: VMotionUtil: 1848: 1349408191430727 S: failed to allocate 527876 bytes for stream channel data segment.
  WARNING: VMotionUtil: 1848: 1349408191430727 S: failed to allocate 527876 bytes for stream channel data segment.
  WARNING: Migrate: 269: 1349408191430727 S: Failed: Out of memory (0xbad0014) @0x0
  

• In the vmkernel.log file of the destination host, you see entries similar to:
  
  cpu12:8769)MigrateNet: vm 8769: 2061: Accepted connection from <x.x.x.x>
  cpu12:8769)MigrateNet: vm 8769: 2131: dataSocket 0x41002b8299a0 receive buffer size is 563724
  cpu12:8769)MigrateNet: vm 8769: 2061: Accepted connection from <x.x.x.x>
  cpu12:8769)MigrateNet: vm 8769: 2131: dataSocket 0x41002b8e7c00 receive buffer size is 563724
  cpu12:8769)VMotionUtil: 3184: 1349408191430727 D: Stream connection 1 added.
  cpu19:122497)WARNING: Migrate: 269: 1349408191430727 D: Failed: Timeout (0xbad0021) @0x418020e30e17

Note: x.x.x.x refer to source host's vmkernel IP.

Environment

VMware vCenter Server 4.0.x
VMware vCenter Server 5.1.x
VMware ESX 4.0.x
VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 5.5
VMware vCenter Server 5.0.x
VMware vSphere ESXi 5.1
VMware ESXi 4.0.x Installable
VMware ESXi 4.0.x Embedded
VMware vCenter Server 4.1.x
VMware vCenter Server 5.5.x
VMware ESX 4.1.x
VMware vSphere ESXi 5.0

Cause

This issue occurs if jumbo frames (MTU : 9000) is enabled for the vMotion vmkernel port group, but is not enabled on the physical switch.

Resolution

To resolve this issue, verify that the MTU size is set to 9000 on end-to-end (virtual switch and physical switch). If not, you have to change the MTU size to 1500 in the vmkernel port group.

To change the MTU size on the vSphere standard switch:

1. Click the Hosts configuration tab.
2. Click Networking.
3. Click Properties for the vSphere standard switch associated with the VMkernel to be modified.
4. Click the VMkernel interface, and click Edit.
5. Set the MTU to 1500.
6. Click OK.

To change the MTU size for the vNetwork Distributed Switch:

1. Select the Networking inventory view.
2. Select the vNetwork Distributed Switch.
3. Right-click the Distributed Virtual Switch, and click Edit Settings.
4. Click the Properties tab.
5. Click Advance.
6. Set MTU to 1500.
7. Click OK.

Additional Information

iSCSI and Jumbo Frames configuration on VMware ESXi/ESX