Stale NFS datastore lock on VM by a different ESXi host causing consolidation failure with error "Unable to access the file since it is locked. Consolidation failed for disk node 'scsi#:#'."

Products

• VMware vCenter Server

Issue/Introduction

When attempting to consolidate disks on NFS datastore, the task fails with below error:

"Unable to access the file since it is locked. Consolidation failed for disk node 'scsi#:#'. Failed to lock the file. An error occurred while consolidating disks. Failed to lock the file."

Symptoms:

• The affected VM is part of NFS datastore.
• The affected VM might fail to power on with same error.
• The affected VM is unable to get consolidated. Manual consolidation also fails.
• Unable to perform backup on the vm. But no identified locked virtual disks by backup proxy vm.
• vmkernel logs show .lck file entries pointing to a different ESXi host.
• Disk consistency check with command "vmkfstools -e <VM-name.vmdk>" command returns error :

"Failed to open disk link /vmfs/volumes/<Datastore-UUID>/<VM-name>/<VM-name>.vmdk :Failed to lock the file (16392)Disk chain is not consistent : Failed to lock the file (16392)".

• vmkernel logs show .lck file entries pointing to a different ESXi host:

YYYY-MM-DDTHH:MM:SS.000Z Wa(180) vmkwarning: cpu35:7302243)WARNING: NFSLock: 4156: File is being locked by a consumer on host esxi1.example.com with read-only lock.
YYYY-MM-DDTHH:MM:SS.000Z In(182) vmkernel: cpu35:7302243)NFSLock: 4160: File <VM-name>-sesparse.vmdk(.lck-d1###0000000000). fromUnlink N
YYYY-MM-DDTHH:MM:SS.000Z In(182) vmkernel: cpu35:7302243)NFSLock: 5141: failed to get lock on file <VM-name>-sesparse.vmdk(.lck-d1###0000000000) 0x4##########c0 on <Datastore-name> 13#################-9#################27 (127.x.x.x): Busy. fromunlink N
YYYY-MM-DDTHH:MM:SS.000Z Wa(180) vmkwarning: cpu30:7302243)WARNING: NFSLock: 4156: File is being locked by a consumer on host esxi1.example.com with read-only lock.
YYYY-MM-DDTHH:MM:SS.000Z In(182) vmkernel: cpu30:7302243)NFSLock: 4160: File <VM-name>-sesparse.vmdk(.lck-d1####0000000000). fromUnlink N
YYYY-MM-DDTHH:MM:SS.000Z In(182) vmkernel: cpu30:7302243)NFSLock: 5141: failed to get lock on file <VM-name>-sesparse.vmdk(.lck-d1####0000000000) 0x4###########80 on <Datastore-name> 13###################-9################27 (127.x.x.x): Busy. fromunlink N

Environment

VMware ESXi 8.x

Cause

This problem occurs when a stale NFS lock on the VMDK files exists on an ESXi host different from the host where the VM is located.

As a result, the consolidation process cannot obtain exclusive access to the VMDK files.

Resolution

1. Identify the locking host:

• SSH into the ESXi host where the vm resides.
• Review vmkernel.log for NFS lock messages.
• Navigate to the appropriate VM directory:
• Check the .lck files in the VM directory.
   # ls -larth
• Use the hexdump command to interrogate the .lck file for the hostname:
  # hexdump -C .lck-d1####0000000000

Output:

00000000  28 ## ## 00 00 00 00 00  ## ## ## ## ## ## ## ##  |(Gn.....#.......|
000000##  02 00 00 00 ## ## ## ##  ## ## ## ## ## ## ## ##  |....esxi2.example.com|

In the above example, we can see this lock was being held on a different host esxi2.example.com

2. Reboot the locking host to release the lock.

3. Retry disk consolidation from the vSphere Client:

• Right-click the VM → Snapshots → Consolidate
• The task should now complete successfully. 
• If not then try the consolidation again but with the VM in a powered-off state.

Additional Information

If the stale NFS lock is on the same host where the vm resides, then follow below KB:

Disk Consolidation Fails with “Unable to access file since it is locked” on NFS Datastore