PSOD (purple screen of death) error: “VMKAPI-char-metadata heap at maximum” in ESXi 6.0 host

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• ESXi 6.0 host fails with a purple diagnostic screen
• In the purple diagnostic screen, there are backtraces similar to:
  
  @BlueScreen: #PF Exception 14 in world 33513:tq:tracing IP 0xnnnnnnnn addr 0xnnnnnnnn
  
  0xnnnnnnnn :[ 0xnnnnnnnn ]Panic_Exception@vmkernel#nover+0x258
  0xnnnnnnnn :[ 0xnnnnnnnn ]IDTReturnPrepare@vmkernel#nover+0x174
  0xnnnnnnnn :[ 0xnnnnnnnn ]gate_entry_@vmkernel#nover+0x0
  0xnnnnnnnn :[ 0xnnnnnnnn ]VmkDrv_WakeupAllWaiters@vmkernel#nover+0xc6
  0xnnnnnnnn :[ 0xnnnnnnnn ]VmkTimerQueueWorldFunc@vmkernel#nover+0x21f
  0xnnnnnnnn :[ 0xnnnnnnnn ]CpuSched_StartWorld@vmkernel#nover+0xa2
   
• WARNING: Heap: 3583: Heap VMKAPI-char-metadata already at its maximum size. Cannot expand.
  WARNING: Heap: 3583: Heap VMKAPI-char-metadata already at its maximum size. Cannot expand.
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 6.x

Cause

This issue occurs due to one of these reasons:

• A slow memory leak in the VMKAPI-char-metadata heap.
• When the ESXi host is using the mpt2sas or mptsas drivers.

Resolution

This issue is resolved in VMware ESXi 6.0, Patch Release ESXi600-201608001.

 

To work around this issue:

1. SSH to the host.
2. To check the heap memory usage:
   $ vsish -e get "/system/heaps/`vsish -e ls /system/heaps | grep VMKAPI-char-metadata`stats" | grep "percent free of max size:"
    
3. If the heap memory usage is identified to be above 90%, schedule a planned reboot of the host.