VM Backup jobs and disks consolidation tasks fail due to stale NFS locks

Products

VMware vSphere ESX 8.x

Issue/Introduction

Symptoms:

Backup jobs fail with an I/O error when attempting to read the VMDK file during a Hot Add backup.
When attempting to consolidate snapshots for a virtual machine hosted on NFS datastore, the task fails with the following error:

Virtual machine disks consolidation failed - "Unable to access file /vmfs/volumes/<Datastore-UUID>/<VM-Name>/<VM-Name>_X.vmdk since it is locked".
Virtual machine vMotion and Clone operations fail to complete.
`vmfsfilelockinfo` or `vmkfstools -D` do not show external locks (as they do not apply to NFS).
The VM may have a deep snapshot chain (approaching the 255-limit)
The issue often occurs following third-party backup operations.
The var/run/log/vmkernal.log file report below errors:
YYYY-MM-DDTHH:MM:SS NFSLock: oid for existing li(lockfile .lck-######### file VMNAME-flat.vmdk) matches for VMNAME-flat.vmdk
YYYY-MM-DDTHH:MM:SS NFSLock: oid for existing li(lockfile .lck-######### file VMNAME_4-flat.vmdk) matches for VMNAME_4-flat.vmdk
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu92:######## opID=########)NFSLock: 2456: oid for existing li(lockfile .lck-######### file VMNAME-flat.vmdk) matches for VMNAME-flat.vmdk
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu91:########opID=########)NFSLock: 2456: oid for existing li(lockfile .lck-######### file VMNAME_4-flat.vmdk) matches for VMNAME_4-flat.vmdk
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu54:######## NFSLock: 2456: oid for existing li(lockfile .lck-######### file VMNAME_4-flat.vmdk) matches for VMNAME_4-flat.vmdk
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu60:######## NFSLock: 2456: oid for existing li(lockfile .lck-######### file VMNAME-flat.vmdk) matches for VMNAME-flat.vmdk

Environment

ESXi 8.x
ESXi 9.x

Cause

Stale NFS locks remain on the VMDK files on the ESXi host where the VM resides. This typically occurs when management agents (`hostd` or `vpxa`) enter a stale state, preventing the release of the `.lck` file handle

Resolution

Identify the locking host:
1. SSH into the ESXi host.
2. Navigate to the VM directory and identify the `.lck` files with command:
  
  ls -larth *.lck*
Use `hexdump` to identify the owner hostname:

hexdump -C .lck-####

*Note: Look for the hostname string in the right-hand column of the output.*
Place the locking ESXi host into maintenance mode with command:

esxcli system maintenanceMode set --enable true
Restart management agents on the locking host with commands:

/etc/init.d/hostd restart/etc/init.d/vpxa restart
Exit maintenance mode after verification with command:

esxcli system maintenanceMode set --enable false

Additional Information

This issue commonly affects VMs hosted on NFS datastores, where locking is managed by the ESXi host rather than the vCenter Server.
If VM logging is disabled (logging = "FALSE" in the .vmx file), enable it to improve traceability: logging = "TRUE"
Regular monitoring of .lck files on NFS datastores can help identify stale locks early.
Ensure ESXi hosts and storage firmware are running the latest versions to avoid known NFS locking bugs.

VM Backup jobs and disks consolidation tasks fail due to stale NFS locks - VMware vSphere

Article ID: 412870

Updated On: