ESXi Host Memory Exhaustion and vSAN Datastore Lock Loss Causing VM Hangs

Products

• VMware vSphere ESXi

Issue/Introduction

Virtual machines running on an ESXi host within a vSAN cluster may enter a hung or unresponsive state simultaneously.

• vmkwarning logs: Repeated entries similar to VmAnon: anon mpn allocation failed.
• vSAN storage errors: HBX: 2723: Failed to cleanup registration key on volume ### and Vol3: 4678: Error closing the volume:.
• Symptoms: VMs fail to process traffic, consoles become unresponsive, and they do not recover without  reboot.

 

Environment

VMware vSphere ESXi

 

Cause

1. Memory Exhaustion: The physical ESXi host has reached a critical memory exhaustion point which is often due to fragmentation or high load.
2. vSAN Impact: Because vSAN depends on host memory for caching and I/O management, the hypervisor cannot maintain vSAN object locks (Heartbeat/HBX).
3. Connectivity Loss: When the host fails to maintain storage locks, the VMs lose access to their underlying disks, forcing them into a hung state.

Resolution

1. Quarantine the Host: Immediately place the affected ESXi host in Maintenance Mode. Do not attempt to power on VMs on the host until the memory exhaustion cause is identified.
2. Clear Stale Locks: If VMs remain disconnected or grayed out in vCenter, administrators must manually clear stale MAC address and file locks from the vSAN datastore.
3. Investigate Memory Utilization:
   • Review host memory trends leading up to the anon mpn allocation failed event.
   • Determine if specific VMs or host processes are causing the exhaustion.
4. Validate Workloads: If investigation indicates a potential memory leak or hardware fault (e.g., controller drops), engage the hardware vendor for further diagnostics.

Additional Information

VMs go into an unstable hung state

Troubleshooting a virtual machine that has stopped responding (VM hang/freeze)