VM on NSF storage failing to power ON due to locked files

Products

• VMware vSphere ESXi

Issue/Introduction

After powering VM down you cannot power it back on.

VM is located on NFS storage.

Message in UI says following 

Task Name
Status
Power On virtual machine
Unable to access file since it is locked
KB 2107795
filePath:
host:
mac:
id: ######
worldName: hostd
lockMode:
Initiator
System

In vmkernel logs located in /var/run/log you see similar entries 

vmkernel: cpu##:######)World: VM ####: ####: Starting World vmm.VMNAME of type 6
vmkernel: cpu##:######)NFSLock: ####: NFS Lock (.lck-#############) already lost for VMNAME-flat.vmdk on DatastoreName (NFS-ServerName)

 

In VM logs located in VM folder you see similar entries 

Db(167) Hostd[2098871]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/######-#######/VMNAME/vVMNAME.vmx] Power On messa
ge: File system specific implementation of Ioctl[file] failed
Db(167) Hostd[2098847]: --> File system specific implementation of Ioctl[file] failed
Db(167) Hostd[2098847]: --> File system specific implementation of Ioctl[file] failed
Db(167) Hostd[2098847]: --> File system specific implementation of LookupAndOpen[file] failed
Db(167) Hostd[2098847]: --> File system specific implementation of LookupAndOpen[file] failed
Db(167) Hostd[2098847]: --> File system specific implementation of LookupAndOpen[file] failed
Db(167) Hostd[2098847]: --> File system specific implementation of LookupAndOpen[file] failed
Db(167) Hostd[2098847]: --> File system specific implementation of LookupAndOpen[file] failed
Db(167) Hostd[2098847]: --> File system specific implementation of LookupAndOpen[file] failed
Db(167) Hostd[2098847]: --> Failed to lock the file
Db(167) Hostd[2098847]: --> Cannot open the disk '/vmfs/volumes/######-#######/VMNAME/VMNAME.vmdk' or one of the snapshot
disks it depends on.
Db(167) Hostd[2098847]: --> Module 'Disk' power on failed.
Db(167) Hostd[2098847]: --> Failed to start the virtual machine.

Snapshot chain is consistent. Above line is misleading in logs.

Checking file sin VM folder yuo don't see any .lck files present.

Environment

vSphere 8.x

Cause

Manually deleted .lck files in VM folder.
 
Due to deleted files host still thinks lock on files is in place hence powering On VM is failing.

Resolution

Full restore from backup.