ESXi host is not responding in VC due to Heap globalCartel-1 already at its maximum size

Products

• VMware vSphere ESXi

Issue/Introduction

Impact/Risks:
Requires reboot of ESXi host.

• Resolve ESXi Host not responding in VC.
• ESXi Host is in not responding state due to Heap globalCartel-1 already at its maximum size.
• The ESXi var/run/log/vmkwarning.log will show similar to below

2019-02-13T02:46:39.287Z cpu0:67612)WARNING: LinuxThread: 381: sfcb-ProviderMa: Error cloning thread: -12 (bad0014)
2019-02-13T02:46:47.399Z cpu1:808261)WARNING: LinuxThread: 381: sh: Error cloning thread: -12 (bad0014)
2019-02-13T02:47:01.418Z cpu5:808268)WARNING: Heap: 3534: Heap globalCartel-1 already at its maximum size. Cannot expand.
2019-02-13T02:47:01.418Z cpu5:808268)WARNING: LinuxThread: 381: sh: Error cloning thread: -12 (bad0014)
2019-02-13T10:36:39.878Z cpu5:67612)WARNING: Heap: 4169: Heap_Align(globalCartel-1, 96/96 bytes, 8 align) failed. caller: 0x41801e8db889

Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5

Cause

Global memory pool has ran out of memory.

Resolution

This issue is resolved in versions ESXi 6.5 U3 build 13932383 and ESXi 6.7 U2 build 13006603.

If the issue of globalCartel-1 already at its maximum size is seen in later versions of ESXi, please collect a coredump using hardware NMI before rebooting the host. Please refer to KB Using hardware NMI facilities to troubleshoot unresponsive hosts for more information. 

Collect the ESXi bundle, open a case with VMware by Broadcom and attach the logs for further investigation.

Additional Information

​​​​​​​