Stale VMFS heartbeat locks due to metadata corruption

Products

• VMware vSphere ESXi

Issue/Introduction

Residual virtual machine files (.vmdk, .vmx) remain locked on a datastore following a virtual machine crash or improper host shutdown.

Symptoms

• Unable to power on the virtual machine.
• Unable to delete orphaned VM directories or files.
• vmfsfilelockinfo reports an "Exclusive" lock holder, but lsof or ps on that host return no active processes.
• vmkernel.log contains corruption errors:
  • naa.####:1 has been detected corrupted
  • INTERNAL TARGET FAILURE (0x44 0x0)
  • DID_ABORT (H:0x5)

Environment

• VMware vSphere ESXi 7.x / 8.x / 9.x
• VMware Cloud Foundation (VCF)
• VMFS 5/6 Datastores

Cause

Stale VMFS heartbeat locks occur due to metadata corruption on the physical datastore. A mismatch in the affinity bitmap prevents the ESXi host from releasing the file lock naturally, orphaning the lock despite the absence of an active process. This is frequently exacerbated by hardware failures (e.g., IO module faults) that interfere with metadata write-dependencies.

• Hardware/Fabric Degradation: Often indicated by INTERNAL TARGET FAILURE and DID_ABORT logs. This suggests issues like VMAX IO module failures or fabric/zoning misconfigurations.
• Software-Level: Typically follows a clean VM crash or improper host shutdown without accompanying hardware SCSI error codes.

Resolution

To clear stale metadata locks, use the VMware On-disk Metadata Analyzer (VOMA) in advfix mode.

Warning: Running VOMA with the fix flag is a destructive operation that directly modifies metadata. To prevent data loss or further corruption, you must follow the safety protocol below.

1. Safety Protocol - Snapshot Verification

1. Create a storage-level snapshot of the affected LUN/datastore.
2. Present and mount the snapshot LUN to a standby ESXi host.
3. Run the VOMA advfix command on the snapshot LUN first to verify the fix completes successfully and resolves the inconsistency.

2. Production Remediation

1. Once verified on the snapshot, identify the impacted production device and partition (e.g., naa.####:1).
2. Unmount the production datastore from all ESXi hosts in the cluster. (For steps, see Unmounting a Datastore).
3. Connect to one ESXi host via SSH as root.
4. Run VOMA in advfix mode on the production device: voma -m vmfs -f advfix -a -d /vmfs/devices/disks/naa.####:1
5. Verify the output displays: Found stale lock [Cleared].
6. Re-mount the production datastore and confirm the files are accessible.

Additional Information

• Investigating Virtual Machine file locks on ESXi Host(s)
• Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency
• For defects and enhancements, please Subscribe to this article to be updated on fix status.
• To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.