Storage vMotion fails at 18% with the vmware.log error: Cannot allocate memory

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Storage vMotion (svmotion) fails at 18%.
• The virtual machine's vmware.log file contains entries similar to:
  
  Worker#0| DISKLIB-VMFS :CopyData '/vmfs/volumes/{datastore}/{vm-name}/{vm-disk-file}-flat.vmdk' : failed to move data (Cannot allocate memory:0xc0009).
  Worker#0| DISKLIB-LIB : DiskLib_Clone: Could not perform clone using vmkernel data mover. Falling back to non-accelerated clone. Cannot allocate memory
  (and subsequent, continuous...)
  Worker#0| DISKLIB-LIB : numIOs = 2450000 numMergedIOs = 0 numSplitIOs = 0
  
  
• The ESX host's /var/log/vmkernel log file contains an entry similar to:
  
  FS3DM: 1595: status Out of memory copying 16777216 bytes from file offset 0 to file offset 0, bytesTransferred = 0 extentsTransferred: 0

Environment

VMware vCenter Server 4.0.x
VMware ESXi 4.0.x Embedded
VMware ESXi 4.0.x Installable
VMware ESX 4.0.x

Resolution

This issue is caused by the VMFS3-DM (Data Mover) having to use contiguous memory space on the host. When the host's kernel memory usage is greater than 50% and memory is fragmented, the destination machine cannot allocate more space and the operation fails. In the example log entries, the host has been up over 161 days.

Note: The 50% memory usage mark is not absolute. Memory use may have to be reduced well below that value.

This issue is resolved in the patch release for ESX/ESXi. For more information, see:

• ESX – VMware ESX 4.0, Patch ESX400-201009401-SG: Updates VMkernel, CIM, Scripts, VMware Tools, VMklinux, and VMX (1023759).
• ESXi – VMware ESXi 4.0, Patch ESXi400-201009401-BG: Updates Firmware (1023768).

To workaround this issue when you do not want to apply the patch, migrate sufficient virtual machines off of the host to reduce memory consumption, or shut down non-critical virtual machines temporarily.

Additional Information

For more information on the location of log files, see Location of log files for VMware products (1021806).