VMFS-6 heap memory exhaustion on vSphere 7.0 ESXi hosts
Article ID: 318028
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Purpose of this article is to let the users know of a VMFS heap related issue present in vSphere 7.0 and 7.0b releases.
Symptoms:
Symptoms:
- On vSphere 7.0 and 7.0b user may see message like “Heap vmfs3 already at its maximum size' In the vmkernel.log.
Example In "vmkwarning.*" files user may see messages such as
vmkwarning.0:2020-06-16T13:28:23.291Z cpu48:3479102)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.
vmkwarning.0:2020-06-16T14:20:23.676Z cpu62:3479103)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.
vmkwarning.0:2020-06-17T08:05:54.057Z cpu92:3909762)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.
Example In "vmkernel.*" files user may see messages such as
2020-06-29T14:59:36.351Z cpu21:5630454)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 5eb9e8f1-f4aeef84-4256-1c34da50d370: Out of memory
2020-06-29T14:59:36.351Z cpu21:5630454)Vol3: 4202: Failed to get object 28 type 1 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 0 gen 0 :Out of memory
2020-06-29T14:59:36.351Z cpu21:5630454)Vol3: 4202: Failed to get object 28 type 2 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 4 gen 1 :Out of memory
2020-06-29T14:59:36.356Z cpu21:5630454)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 5eb9e8f1-f4aeef84-4256-1c34da50d370: Out of memory
2020-06-29T14:59:36.356Z cpu21:5630454)Vol3: 4202: Failed to get object 28 type 1 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 0 gen 0 :Out of memory
2020-06-29T14:59:36.356Z cpu21:5630454)Vol3: 4202: Failed to get object 28 type 2 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 4 gen 1 :Out of memory
vmkwarning.0:2020-06-16T13:28:23.291Z cpu48:3479102)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.
vmkwarning.0:2020-06-16T14:20:23.676Z cpu62:3479103)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.
vmkwarning.0:2020-06-17T08:05:54.057Z cpu92:3909762)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.
Example In "vmkernel.*" files user may see messages such as
2020-06-29T14:59:36.351Z cpu21:5630454)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 5eb9e8f1-f4aeef84-4256-1c34da50d370: Out of memory
2020-06-29T14:59:36.351Z cpu21:5630454)Vol3: 4202: Failed to get object 28 type 1 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 0 gen 0 :Out of memory
2020-06-29T14:59:36.351Z cpu21:5630454)Vol3: 4202: Failed to get object 28 type 2 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 4 gen 1 :Out of memory
2020-06-29T14:59:36.356Z cpu21:5630454)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 5eb9e8f1-f4aeef84-4256-1c34da50d370: Out of memory
2020-06-29T14:59:36.356Z cpu21:5630454)Vol3: 4202: Failed to get object 28 type 1 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 0 gen 0 :Out of memory
2020-06-29T14:59:36.356Z cpu21:5630454)Vol3: 4202: Failed to get object 28 type 2 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 4 gen 1 :Out of memory
- Datastores are showing "Not consumed" on hosts.
- VMs may fail to vMotion.
- VMs may become orphaned when powered off.
- Snapshot based backups may fail.
- In vCenter/hosts snapshot creation and consolidation may fail with: message = "Consolidation failed for disk node 'scsi0:1': 12 (Cannot allocate memory)."
Environment
VMware vSphere ESXi 7.0.0
Cause
This is a known issue in VMFS6 where in certain work flows we allocate memory but do not free it up resulting in VMFS heap exhaustion.
The memory allocated gets freed during VMFS resource allocation and hence creating a thick file on VMFS datastore is an effective work around. See work around section below for more details.
The memory allocated gets freed during VMFS resource allocation and hence creating a thick file on VMFS datastore is an effective work around. See work around section below for more details.
Resolution
This issue is resolved in vsphere ESXi 7.0 Update 1.
To workaround this issue follow the steps below:
Workaround has to be done for each datastore on each host.
If you need further support contact VMware Technical Support. For more information see How to file a Support Request in Customer Connect.
To workaround this issue follow the steps below:
- Create Eager zeroed thick disk on all of the mounted VMFS6 datastores.
- Delete the Eager zeroed thick disk created in step 1.
Workaround has to be done for each datastore on each host.
If you need further support contact VMware Technical Support. For more information see How to file a Support Request in Customer Connect.
Additional Information
Feedback
Yes
No
Powered by
