Copy on Write (COW) Heap limitations for ESXi 4.1, 5.x and 6.0

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information on the limitations of Copy on Write (COW) Heap.

Symptoms:

• Copy on Write (COW) out of memory.
• You see these errors within the vmkernel log files:
  
  WARNING: Cow: 549: No memory available
  
  Mar 14 13:15:29 vmkernel: 7:01:14:51.338 cpu16:2366427)World: vm 2374623: 1540: Starting world vmm0:DLONDBAHLS103_LC61 with flags 4008
  Mar 14 13:15:29 vmkernel: 7:01:14:51.338 cpu16:2366427)Sched: vm 2374623: 6054: Adding world 'vmm0:DLONDBAHLS103_LC61', group 'host/user', cpu: shares=-3 min=0 minLimit=-1 max=-1, mem: shares=-3 min=0 minLimit=-1 max=-1
  Mar 14 13:15:29 vmkernel: 7:01:14:51.338 cpu16:2366427)Sched: vm 2374623: 6073: renamed group 379434 to vm.2366427
  Mar 14 13:15:29 vmkernel: 7:01:14:51.338 cpu16:2366427)Sched: vm 2374623: 6090: group 379434 is located under group 4
  Mar 14 13:15:29 vmkernel: 7:01:14:51.350 cpu16:2366427)Swap: vm 2374623: 2331: extending swap to 4194304 KB
  Mar 14 13:15:29 vmkernel: 7:01:14:51.399 cpu23:2366427)World: vm 2366433: 1540: Starting world vmm1:DLONDBAHLS103_LC61 with flags 4008
  Mar 14 13:15:29 vmkernel: 7:01:14:51.764 cpu17:2366427)WARNING: Cow: 549: No memory available! Called from 0x41800dc39a69
  Mar 14 13:15:29 vmkernel: 7:01:14:51.764 cpu17:2366427)Cow: 1759: Failed on handle 3 (7344784) of 4 with Out of memory
  Mar 14 13:15:29 vmkernel: 7:01:14:51.764 cpu17:2366427)WARNING: Cow: 1329: Opening the cow hierarchy for 46990d32-DLONDBAHLS103_LC61_5-delta.vmdk failed with Out of memory
  Mar 14 13:15:29 vmkernel: 7:01:14:51.955 cpu17:2366427)Sched: vm 2374623: 1487: name='vmm0:DLONDBAHLS103_LC6
  Mar 14 13:15:29 vmkernel: 7:01:14:51.955 cpu17:2366427)CpuSched: vm 2374623: 9990: zombified unscheduled world: runState=NEW
  Mar 14 13:15:29 vmkernel: 7:01:14:51.955 cpu17:2366427)World: vm 2374623: 4024: deathPending set; world not running, scheduling reap
  Mar 14 13:15:29 vmkernel: 7:01:14:51.955 cpu17:2366427)Sched: vm 2366433: 1487: name='vmm1:DLONDBAHLS103_LC61'
  Mar 14 13:15:29 vmkernel: 7:01:14:51.955 cpu17:2366427)CpuSched: vm 2366433: 9990: zombified unscheduled world: runState=NEW
  Mar 14 13:15:29 vmkernel: 7:01:14:51.955 cpu17:2366427)World: vm 2366433: 4024: deathPending set; world not running, scheduling reap

Environment

VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0
VMware ESXi 4.1.x Installable

Resolution

On average, 20% of the COW Heap Memory is used for internal data structure. This means that 80% of the memory can be used for virtual machines. Since root entries mainly occupy the memory, 75% of the memory is used for root entries.

The maximum configurable size for COW Heap Memory is 256 MB. The default configurable size is 192 MB.

For example:

Z = Number of Virtual Machine's
Y = Number of disks
X = Number of snapshotsthat each disk has
W = Size (in bytes)

Z = (75 / 100 * COW_HEAP_SIZE) / ((W / (2 * 1048576) * 4 * X) * Y), where 2 * 1048576 is GDE Coverage and constant 4 is bytes per Root Entry.

If a Virtual Machine has 5 disks, each disk is 80 GB (85899345920 bytes) with 6 levels of snapshots for each disk. The number of Virtual Machines that can be powered-on the ESX host on which the COW Heap size is configured as 256 MB is

= (75 / 100 * 268435456) / ((85899345920 / (2 * 1048576) * 4 * 6) * 5)

Approximately 40 of such Virtual Machines can be powered on.

Note: Use the equation above to calculate your COW requirements and determine your limitations.

For more information, see Cannot power on a virtual machine due to COW memory heap exhaustion (1009086).

Additional Information

Cannot power on a virtual machine due to COW memory heap exhaustion