High VMFS heap memory consumption might cause virtual machines to become unresponsive and appear to be in a disconnected state
search cancel

High VMFS heap memory consumption might cause virtual machines to become unresponsive and appear to be in a disconnected state

book

Article ID: 419320

calendar_today

Updated On:

Products

VMware vSphere ESXi 8.0

Issue/Introduction

  • In environments with many large datastores and many VMs on thin-provisioned disks.
  • VMFS resource allocation might consume a high amount of heap memory, proportional to the size of the datastores.
  • As a result, VMFS heap memory might be exhausted.
  • It lead to VMs becoming unresponsive Or vSphere vMotion tasks failing with errors for VM invalid state.

--vmkernel.log--
YYYY-MM-DDTHH:MM:SS cpu48:7481319)WARNING: Heap: 4105: Heap_Align(vmfs3, 1048576/1048576 bytes, 8 align) failed.  caller: 0x42003133c02c
YYYY-MM-DDTHH:MM:SS cpu88:2496333)Admission failure in path: host/user/vm.2496326:vmmanon.2496326
YYYY-MM-DDTHH:MM:SS cpu88:2496333)vmmanon.####### (#######) extraMin/extraFromParent: 476/476, vm.####### (#######) childEmin/eMinLimit: 42902/43008
YYYY-MM-DDTHH:MM:SS cpu158:2102095)FS3J: 3235: Cancelling txn (0x43160f323400) callerID: 0xc1d0000f due to failurepre-committing: Out of memory
YYYY-MM-DDTHH:MM:SS cpu158:2102095)Fil6: 4092: 'Datastore01': Fil6 file IO (<FD c16 r50>) : Out of memory
YYYY-MM-DDTHH:MM:SS cpu158:2102095)Fil6: 4058: ioCtx: 0x45ebf26fc380, world: 2102095, overallStatus: Out of memory, token: 0x0, tokenStatus: N, txnToken: 0x0, txnTokenStatus: N, totalIOSize: 4096, maxIOLength: 4608

Environment

VMware ESXi 8.0

Cause

High VMFS heap memory consumption leads to complete memory exhaustion on the host. This behavior is prevalent in environments running legacy 7.x versions or ESXi versions below 8.0 U3e that feature numerous large datastores and thin-provisioned VMs. Under these conditions, affinity clustering consumes excessive heap memory proportional to the size of the datastores.

Resolution

This issue is resolved in the VMware ESXi 8.0 Update 3e (Build 24674464)

Additional Information

VMware ESXi 8.0 Update 3e Release Notes
https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/release-notes/esxi-update-and-patch-release-notes/vsphere-esxi-80u3e-release-notes.html