virtual machine power on failed with error: "vmkernel: Out of memory"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Cannot power on a virtual machine with 850 GB plus memory + PCI Passthrough device
• virtual machine power on failed with 1 TB memory and PCI passthrough device "vmkernel: Out of memory"
• In the var/log/vmkernel.log, you see entries similar to:

cpu9:327411)WARNING: Heap: 3672: Heap mainHeap (30819192/31462264): Maximum allowed growth (643072) too small for size (4198400)
cpu9:327411)WARNING: Heap: 4169: Heap_Align(mainHeap, 4194312/4194312 bytes, 8 align) failed.  caller: 0x41801af5a178
cpu9:327411)VSCSI: 6541: handle 8207(vscsi0:0):Destroying Device for world 327412 (pendCom 0)
cpu46:68106)Config: 706: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)
cpu42:66360)Tcpip_Flow: 129: Failed to pin vmk0 rxDisp 0 to world 328437 (awk): Not initialized

• In the vmware.log, you see entries similar to:

  vmx| I125+ Power on failure messages: Passthrough device 'pciPassthru0' vGPU 'grid_p40-4q' disallowed by vmkernel: Out of memory.
 vmx| I125+ Could not initialize plugin '/usr/lib64/vmware/plugin/libnvidia-vgx.so' for vGPU 'grid_p40-4q'.
 vmx| I125+ Module 'DevicePowerOn' power on failed.
 vmx| I125+ Failed to start the virtual machine.

Cause

This issue occurs when vmkernel is unable to allocate memory from the 'mainHeap' during PCI passthrough device registration. It fails due to not enough memory in mainHeap.

Resolution

This issue is resolved in:

• ESXi600-201807001 available at Broadcom Downloads.
• ESXi 6.5 Update 2, available at Broadcom Downloads.

Workaround:
To workaround the issue if you do not want to upgrade, decrease the virtual machine memory below 850 GB.

Additional Information

Impact/Risks:
Cannot power on VM with 1 TB Memory when PCI passthrough device is attached.