Cannot power on a virtual machine due to COW memory heap exhaustion

Products

• VMware vSphere ESXi

Issue/Introduction

• Cannot power on virtual machine due to COW memory heap exhaustion.
• In the /var/log/hostd.log file, you see entries similar to:
  
  2015-11-02T10:56:09.776Z [2DE80B70 verbose 'Vmsvc.vm:/vmfs/volumes/########-########-####-##########7d/VM/VM.vmx'] Handling message _vmx4: Heap cow already at its maximum size of 201331560. Cannot expand.
  
  2015-11-02T10:56:09.979Z [2C580B70 verbose 'Vmsvc.vm:/vmfs/volumes/########-########-####-##########7d/VM/VM.vmx'] Power On message: Heap cow already at its maximum size of 201331560. Cannot expand.
  
  
• In the /var/log/vmkwarning.log file, you see entries similar to:
  
  WARNING: Heap: 1445: Heap cow could not be grown to accommodate the memory request
  WARNING: Heap: 1522: Heap_Align(cow, 68096/68096 bytes, 4 align) failed. caller: 0xa0155e
  WARNING: Cow: 576: No memory available! Called from 0xa0155e
  WARNING: Cow: 1223: Creation of the device with device lib failed Out of memory
  
  
• Cloning a virtual disk using vmkfstools -i fails.
• In the /var/log/vmkernel.log file, you see entries similar to:
  
  WARNING: Heap: 1435: Heap cow already at its maximumSize. Cannot expand.
  WARNING: Heap: 1645: Heap_Align(cow, 1048576/1048576 bytes, 8 align) failed. caller: 0x418032712212
  WARNING: Cow: 530: No memory available! Called from 0x418032712212 Cow: 1714: Failed on handle 12 (33833182) of 14 with Out of memory
  
  Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware ESX 4.0.x
VMware ESX Server 3.5.x
VMware ESXi 4.0.x Installable
VMware vSphere ESXi 5.1
VMware ESXi 3.5.x Installable
VMware vSphere ESXi 5.0
VMware ESXi 4.0.x Embedded
VMware ESX Server 3.0.x
VMware ESXi 3.5.x Embedded
VMware vSphere ESXi 5.5

Resolution

If you use snapshots on virtual machines running on an ESX host, each snapshot delta disk is a COW (Copy On Write) disk. For each one in use by running virtual machines, their data structures take up ESX kernel memory. This allocation is known as the COW heap. This memory is used to store cached metadata, pointing to where in a VMDK or in a chain of VMDK files disk data to be accessed resides.

In ESX 3.5, the default COW memory heap is 32MB. As more snapshots are used by running virtual machines, particularly by virtual machines with large base disks, this COW memory heap may run low. When exhausted, virtual machines no longer power on.

The following ESX versions can have their COW heap manually increased:

• ESX 3.5.0 Update 1 with the patch ESX350-200803202-UG
• ESX 3.5.0 Update 2 or higher
• ESX 4.0 or higher

Notes:

• You do not automatically have to increase the COW memory heap allocation to its configurable maximum. You can double it to begin with, and see if this resolves the issue, unless you feel the ESX host has a lot of memory to spare.
• The ESX host must be rebooted for the changes to take effect.
  

To increase the COW Memory Heap allocation through VI Client:

1. Open vCenter/VirtualCenter, and click on the relevant host.
2. Click the Configurations tab.
3. Under the Software section click Advanced Settings, and in the dialog that appears, click COW.
4. Change the value of COW.MaxHeapSizeMB.
   

To increase the COW Memory Heap Allocation through the Service Console in ESX 3.5:

1. Ensure that you are logged in as the root user.
2. Run this command to get the current value:
   
   esxcfg-advcfg -g /COW/MaxHeapSizeMB
   
   Value of MaxHeapSizeMB is 32MB.
   
   
3. Run this command to set the new value:
   
   esxcfg-advcfg -s 64 /COW/MaxHeapSizeMB
   
   Value of MaxHeapSizeMB is 64MB.
   
   Note: ESX 3.5 default COW Memory Heap allocation is 32MB and the maximum is 192 MB.

To increase the COW Heap Memory Allocation through the Service Console in ESX 4.0:

1. Ensure that you are logged in as the root user.
2. Run these command to get the current value:
   
   esxcfg-advcfg -g /COW/COWMaxHeapSizeMB
   
   Value of MaxHeapSizeMB is 192 MB.
   
   
3. Run this command to set the new value:
   
   esxcfg-advcfg -s 256 /COW/COWMaxHeapSizeMB
   
   Value of MaxHeapSizeMB is 256 MB.
   
   Note: The maximum COW Memory Heap allocation is 256MB. For more information on COW Heap Value, see Setting COW heap values on Lab Manager (1009144).

For additional information, see VMware Community Out of COW space.

See also, Copy on Write (COW) Heap limitations for ESXi 4.1, and 5.x(2036522).