VMFS-6 heap memory exhaustion on vSphere 7.0 ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

The purpose of this article is to let the users know of a VMFS heap related issue present in vSphere 7.0 and 7.0b releases.

Symptoms:

• On vSphere 7.0 and 7.0b, 

 In vmkwarning.log, the following log entries are observed:

[YYYY-MM-DDTHH:MM:SS] cpu48:3479102)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.
[YYYY-MM-DDTHH:MM:SS] cpu62:3479103)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.
[YYYY-MM-DDTHH:MM:SS] cpu92:3909762)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.

In vmkernel.log, the following log entries are observed:

[YYYY-MM-DDTHH:MM:SS] cpu21:5630454)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 5eb9e8f1-f4aeef84-4256-1c34da50d370: Out of memory
[YYYY-MM-DDTHH:MM:SS] cpu21:5630454)Vol3: 4202: Failed to get object 28 type 1 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 0 gen 0 :Out of memory
[YYYY-MM-DDTHH:MM:SS] cpu21:5630454)Vol3: 4202: Failed to get object 28 type 2 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 4 gen 1 :Out of memory
[YYYY-MM-DDTHH:MM:SS] cpu21:5630454)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 5eb9e8f1-f4aeef84-4256-1c34da50d370: Out of memory
[YYYY-MM-DDTHH:MM:SS] cpu21:5630454)Vol3: 4202: Failed to get object 28 type 1 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 0 gen 0 :Out of memory
[YYYY-MM-DDTHH:MM:SS] cpu21:5630454)Vol3: 4202: Failed to get object 28 type 2 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 4 gen 1 :Out of memory

• Datastores are showing "Not consumed" on hosts.
• VMs may fail to vMotion.
• VMs may become orphaned when powered off.
• Snapshot-based backups may fail.
• In vCenter/hosts, snapshot creation and consolidation may fail with: message = "Consolidation failed for disk node 'scsi0:1': 12 (Cannot allocate memory)."

Environment

VMware vSphere ESXi 7.0.0

Cause

This is a known issue in VMFS6. In certain workflows, the memory is allocated but not freed up resulting in VMFS heap exhaustion. 

The memory allocated gets freed during VMFS resource allocation and hence, creating a eagerzeroedthick file on VMFS datastore is an effective workaround. See the workaround section below for more details.

Resolution

This issue is resolved in vSphere ESXi 7.0 Update 1.

To workaround this issue, follow the steps below:

1. Create an Eager zeroed thick disk on all of the mounted VMFS6 datastores.

   vmkfstools -c 10M -d eagerzeroedthick /vmfs/volumes/datastore/eztDisk

2. Delete the Eager zeroed thick disk created in step 1.

     vmkfstools -U /vmfs/volumes/datastore/eztDisk

The workaround has to be implemented for each datastore on each host.

If you need further support, contact Broadcom Technical Support.