"Heap globalCartel-1 already at its maximum size” error in ESXi host

Products

VMware

Issue/Introduction

You must upgrade your host to ESXi 5.5 Update 3 and upgrade AMS to version 10.0.1 or later if you cannot fork.

Symptoms:

Cannot perform a vMotion to and from an ESXi host
vSphere Client reports the error:

Heap globalCartel-1 already at its maximum size of 312160. Cannot expand.
Could not start VMX: msg.vmk.status.VMK_NO_MEMORY

Note: For additional symptoms and log entries, see the Additional Information section.

Cause

This issue occurs with ESXi hosts running on HP hardware with these versions of AMS:

hp-ams 500.9.6.0-12.434156
hp-ams-550.9.6.0-12.1198610
hp-ams 500.10.0.0-18.434156
hp-ams-550.10.0.0-18.1198610

Resolution

This is a known issue affecting ESXi 5.x.

To resolve this issue:

Upgrade to ESXi 5.5 Update 3, available at VMware Downloads.
Upgrade to AMS version 10.0.1 and later.
- For more information on ESXi 5.0 and 5.1, see HP Agentless Management Service Offline Bundle for VMware ESXi 5.0 and vSphere 5.1.
- For more information on ESXi 5.5, see HP Agentless Management Service Offline Bundle for VMware vSphere 5.5.

To work around this issue if you do not want to upgrade, remove the AMS package on all hosts.

Note: In some cases, commands running on the ESXi host fails with cant't fork. In this case, the virtual machines running on the ESXi host needs to be powered off and the ESXi host rebooted.

To verify the installed versions of AMS, run this command:

esxcli software vib list | grep ams

To remove the package on all hosts running on these AMS versions:

Log in to the host using SSH.
Run this command to stop the HP service.

/etc/init.d/hp-ams.sh stop

Note: The changes do not persist across reboot.
Run this command to remove the VIB:

esxcli software vib remove -n hp-ams
Restart the host.

Disclaimer: VMware is not responsible for the reliability of any data, opinions, advice, or statements made on third-party websites. Inclusion of such links does not imply that VMware endorses, recommends, or accepts any responsibility for the content of such sites.

Additional Information

You experience these additional symptoms:

ESXi host becomes unresponsive
When attempting to enable services or perform a vMotion, the ESXi host fails with one of these errors:
- A general system error occurred: Command /bin/sh failed
- A general system error occured: Launch Failure
When connecting to a virtual machine console, you see one of these errors:
- Unable to contact the MKS: Could not connect to pipe\\.\pipe\vmware-authdpipe
- Unable to connect to the MKS: connection terminated by server.
When powering on a virtual machine, you see the error:

VMK_NO_MEMORY
When logging in to the ESXi shell, you see entry:

can't fork
When pressing Alt+F12 at the DCUI, you see the error:

WARNING: Heap: 2677: Heap globalCartel-1 already at its maximum size. Cannot expand.
In the /var/log/vmkwarning file, you see entries similar to:

<YYYY-MM-DD>T<time> cpu18:4287108)WARNING: Heap: 3058: Heap_Align(globalCartel-1, 136/136 bytes, 8 align) failed. caller: 0x41802a2ca2fd
<YYYY-MM-DD>T<time>cpu8:4287111)WARNING: Heap: 2677: Heap globalCartel-1 already at its maximum size. Cannot expand.
<YYYY-MM-DD>T<time>cpu8:4287111)WARNING: Heap: 3058: Heap_Align(globalCartel-1, 136/136 bytes, 8 align) failed. caller: 0x41802a2ca2fd
<YYYY-MM-DD>T<time>cpu1:4287132)WARNING: Heap: 2677: Heap globalCartel-1 already at its maximum size. Cannot expand.
<YYYY-MM-DD>T<time>cpu1:4287132)WARNING: Heap: 3058: Heap_Align(globalCartel-1, 136/136 bytes, 8 align) failed. caller: 0x41802a2ca2fd
<YYYY-MM-DD>T<time>cpu24:4287142)WARNING: Heap: 2677: Heap globalCartel-1 already at its maximum size. Cannot expand.
<YYYY-MM-DD>T<time>cpu24:4287142)WARNING: Heap: 3058: Heap_Align(globalCartel-1, 136/136 bytes, 8 align) failed. caller: 0x41802a2ca2fd
<YYYY-MM-DD>T<time>cpu5:4025029)WARNING: Heap: 2677: Heap globalCartel-1 already at its maximum size. Cannot expand.</time></time></time></time></time></time></time></time>

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Running commands in the ESXi console fail with the error:

-sh: can't set tty process group (Operation not permitted)

Using ESXi Shell in ESXi 5.x and 6.x
How to download and install async drivers in ESXi 5.x/6.x
ESXi ホストの「ヒープ globalCartel-1 はすでに最大サイズです (Heap globalCartel-1 already at its maximum size)」エラー
ESXi 主机中出现 “堆 globalCartel-1 已达到大小上限 (Heap globalCartel-1 already at its maximum size)” 错误

Impact/Risks: