Error: Failed to lock the file during disk consolidation resulting in VM shutdown

Products

VMware vSphere ESXi VMware vCenter Server 8.0

Issue/Introduction

A virtual machine shut down is observed in the hostd log on ESXi:

Log location: /var/log/run/hostd.log

<YYYY-MM-DD>T<time>Z verbose hostd[2102782] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:<UUID>/<VMname>.vmx opID=lqntbmzn-19795825-auto-bsak2-h5:71916833-1b-01-75-85e9] Handling vmx message 96789784: Failed to lock the file
--> Cannot open the disk 'vm:/vmfs/volumes/<UUID>/<VMname>.vmdk' or one of the snapshot disks it depends on.
--> An operation required the virtual machine to quiesce and the virtual machine was unable to continue running

The vmware.log for the VM and hostd.log on ESXi show a NFC open (nbdmode) occurring while the VM is attempting to lock the base disk.

Log Location: /var/log/run/hostd.log

<YYYY-MM-DD>T<time>Z info hostd[2103955] [Originator@6876 sub=Libs opID=nbdmode-000000c04c570270] OBJLIB-VSANOBJ: VsanObjOpenInt: Device '/vmfs/devices/vsan/<object>' not present, retrying to create one. <YYYY-MM-DD>T<time>Z info hostd[2103955] [Originator@6876 sub=DiskLib opID=nbdmode-000000c04c570270] DISKLIB-LINK :DiskLinkGetSpaceUsedInfo: Failed to get the file system unique id for file vsan://<volume>/<object>, using id as "remote"

Log Location: /vmfs/volumes/<VM's volume>/vmware.log

Note: To find the VM's logging location use Locating virtual machine log files on an ESXi host

<YYYY-MM-DD>T<time>Z In(05) vcpu-0 - Closing disk 'scsi0:1'

...

<YYYY-MM-DD>T<time>Z In(05) vcpu-0 - DISKLIB-LIB : DiskLibUnlinkInt: Disk delete successfully completed { result:0, Msg: 'The operation completed successfully', fileName:'/vmfs/volumes/vsan:<UUID>/<VM Name>-000001.vmdk'}
<YYYY-MM-DD>T<time>Z In(05) vcpu-0 - Closing disk 'scsi0:2'

**NFC open happens at this time. scsi0:1 disk is still closed. At this point, the scsi0:1 delta is deleted and it points to the base disk.

<YYYY-MM-DD>T<time>Z In(05) vcpu-0 - DISK: OPEN scsi0:1 'vmfs/volumes/vsan:<UUID>/<VM Name>_1.vmdk' persistent R[]
<YYYY-MM-DD>T<time>Z Er(02) vcpu-0 - OBJLIB-VSANOBJ: VsanObjLock: Object '<object>' locked, retrying.

Note: The object in the hostd logs will match the object locked in the VM's logs.

Environment

VMware ESXi 7.x / 8.x
VMware vCenter Server 8.0.3
Third-party backup software (e.g., Commvault, NetBackup, Cohesity) using NBD transport mode.

Cause

This issue occurs due to a timing race condition between the VM resuming execution and the backup software releasing its disk lock.

During consolidation, the VM requires an exclusive lock to switch from the snapshot delta back to the base disk. If the backup proxy or a management agent (like hostd) retains a read-only lock during this brief switch-over window, the VM cannot acquire the exclusive lock and terminates to protect data integrity.

Resolution

To resolve the stale lock and restore the VM, perform the following steps:

Identify the Lock Holder: Run the following command on the ESXi host where the VM is registered: vmfsfilelockinfo -p "/vmfs/volumes/[DatastoreName]/[VMName]/[DiskName]-flat.vmdk" -v Note the MAC address or World ID of the process holding the lock.
Release Stale Locks: If the lock is held by hostd-worker, restart the management agents on the affected ESXi host: /etc/init.d/hostd restart /etc/init.d/vpxa restart
Verify Backup Proxy: Ensure the backup proxy VM does not have the target VM's disks still mounted. If found, manually remove the disks from the proxy VM settings (choose Remove from virtual machine; do not delete from disk).
Consolidate Disks: In the vSphere Client, right-click the VM and select Snapshots > Consolidate.
Power On VM: Once consolidation is successful, power on the virtual machine.

Note: If the issue persists, evaluate the timing of access calls with your backup vendor. It is expected that NBD sessions are fully closed before a snapshot delete/consolidation operation begins

For further assistance, see Contact Support.