VMware virtual machine file lock on VMFS datastore

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

Snapshots cannot be committed or consolidated.
Third-party backup application (Veeam, Avamar, Unitrends,...etc) has locked snapshot files, and failed to remove the snapshot after completing backups or failed to initiate backups.
Snapshot commits without errors and Snapshot Manager is no longer populated. However, the snapshot disk is present in the virtual machine directory. (the VM is still running on Snapshot)
The virtual machine summary tab display messages similar to:
- Snapshot consolidation required.
- Virtual machine disks consolidation is needed.
- Virtual machine Consolidation Needed status.

- Consolidation failed with the errors similar to:
- Failed to lock the file or One or more disks are busy.
- Unable to consolidate virtual machine snapshots due to file lock.
- Unable to access filename since it is locked.
- Unable to access file since it is locked

In hostd.log on the ESXi Host where the virtual machine is running, entries are seen similar to the below:
Note: hostd.log can be located in /var/run/log/hostd.log

YYYY-MM-DDTHH:MM:SS.819Z info hostd[1050487] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 361 : Lock(s) held on file Win-test-flat.vmdk by other host(s).
YYYY-MM-DDTHH:MM:SS.822Z info hostd[1051013] [Originator@6876 sub=Vimsvc.TaskManager opID=d88eb069 user=vpxuser] Task Created : haTask--vim.event.EventHistoryCollector.readNext-66249
YYYY-MM-DDTHH:MM:SS.823Z info hostd[1050756] [Originator@6876 sub=Vimsvc.TaskManager opID=d88eb069 user=vpxuser] Task Completed : haTask--vim.event.EventHistoryCollector.readNext-66249 Status success
YYYY-MM-DDTHH:MM:SS.826Z info hostd[1050487] [Originator@6876 sub=Vimsvc.TaskManager opID=d88eb06c user=vpxuser] Task Created : haTask--vim.event.EventHistoryCollector.readNext-66250

Cannot power on virtual machine due to file lock:
- Unable to open Swap File.
- Unable to access a file since it is locked.
- Unable to access a file filename since it is locked.
- Unable to access Virtual machine configuration.
- An unexpected error was received from the ESX host while powering on VM <VM_name>.
  Reason: Cannot open the disk disk_name or one of the snapshot disks it depends on.

In vmkernel.log , on the ESXi Host where the virtual machine is running, entries similar to the below are seen:

Note: vmkernel.log can be located in /var/run/log/vmkernel.log

WARNING: World: VM xxxx: xxx: Failed to open swap file path: Lock was not free
WARNING: World: VM xxxx: xxx: Failed to initialize swap file path

Environment

VMware vSphere 8.0.x

VMware vSphere 7.0.x
VMware vSphere 6.x

Cause

When a virtual machine snapshot fails to consolidate or a virtual machine with snapshots fails to power on, it is usually because one or more of the virtual machine files are abnormally locked by the same ESXi Host the virtual machine is running or any other ESXi Host.
The ESXi Host (either same Host where the VM is running or any other Host) is abnormally locking the virtual machine files due to many reasons and not limited to the below:
- In-completed or interrupted backup task
- Previous storage failure

Resolution

Virtual Machine File Lock most common scenarios:
1) Proxy Back Up Virtual Machine (Veeam, Avamar, Unitrends,...etc) is locking one or more of the virtual machine .vmdk files.
2) An ESXi Host is locking the Virtual Machine files abnormally.

First Scenario

Caution: Make sure there is no backup or snapshot task running on the virtual machine before applying the resolution.

The most common case of virtual machine file lock is when the Backup Proxy Virtual Machine (Veeam, Avamar, Unitrends,...etc) is locking one or more of the virtual machine .vmdk files. This issue can be resolved using the below steps:

Right-click the backup proxy virtual machine.
Click Edit Settings.
Expand all the Hard Disk(s).
Select the Hard Disk(s) which belong to the virtual machine that has the problem.
Click on the X beside the Hard Disk to unmount the Hard Disk from the VM.

Caution: Do NOT select Delete files from the datastore.

Click OK.
Consolidate/Delete the snapshot on the VM.

" Power on the VM if the issue was unable to power on"

Second Scenario

If the first section is not the case, then we need to identify which process is locking the virtual machine files to release.

Connect to the ESXi host the virtual machine is on with an SSH session
Navigate to the VM directory and take a note of where the VM is currently writing.

To know the VM directory, follow the below steps:

a) Run the below command
# vim-cmd vmsvc/getallvms | grep -i "VM_Name"

b) Run the below command:
# cd /vmfs/volumes/datastore_name/virtual_machine_name/ --->From step 1

c) Run the below command:
# grep -i vmdk *vmx

*Now the VM is writing on Test_File_Lock-000002.vmdk in the example

Check which ESXi Host is locking the VM file(s) and clear the lock by running the below script in the VM directory:

# ls | while read x; do vmfsfilelockinfo -p $x| grep -i "is locked"; done

Note: The script will not work if the virtual machine files has spaces. In this case, you will need to run the command vmfsfilelockinfo -p on each and every file listed under the virtual machine directory.
Example: vmfsfilelockinfo -p Test_File_Lock-flat.vmdk

There will be two probabilities: (after running the script)

The VM files are locked by same ESXi Host where the VM is running (owning Host).

Notes:
1. If the VM is powered on, it should normally be locked by the ESXi Host it is running on (referred by the MAC address in the script output)
2. If the VM is powered off then there shouldn't be an output from the script as VM files shouldn't be normally locked by any process (from any ESXi Host)

To know what is locking the VM files and release the lock, follow the below steps:-

Run the command

# lsof | grep -i "VM_Name"

a) It could be vpxa-worker ----> this can be fixed by restart vpxa agent using the below commands:
# /etc/init.d/vpxa stop
# /etc/init.d/vpxa start

b) It could be hostd or hostd-profiler ---> this can be fixed by restarting hostd agent using the below commands:
# /etc/init.d/hostd stop
# /etc/init.d/hostd start

c) The VM files might be stored in a temp directory /dev/deltadisks----> this can be fixed by going to the temp directory and remove the files
# cd /dev/deltadisks

# rm -rf "VM_Name_00000x-delta.vmdk" ---->file name might be slightly different like "8857142-Test_File_Lock-000002-delta.vmdk"

Warning: Make sure that the directory is /dev/deltadisks . This is the only directory where it is safe to remove the VM files without data loss.

Notes:

If the VM is powered off then, this script should show no output as no process should be locking the VM files. So if the script shows an output and the VM is powered off, then there is a stale process that will only be released by rebooting the ESXi Host.
If you got the error "device or resource busy" while trying to run the rm command , you will need to power off the VM, make sure that there is still a VM file in the /dev/deltadisks and perform the remove again.

The VM files are locked by another ESXi Host

All the files are locked by Host that has the same MAC address except one file "Test_File_Lock-flat.vmdk" that is locked by two ESXi Hosts (two MAC addresses)

As the VM is powered on so there should be normal lock from the owing ESXi Host but there is another abnormal lock coming from another Host that has the MAC address "00:50:##:##:##:16" in the example

To release the lock from the other ESXi:

Identify which ESXi Host is locking the file from the vCenter GUI

Expand the Cluster and select any ESXi Host
Click on Configure
Click on Physical adapters and look for the MAC address "Output from the previous step"

Note: You can identify which ESXi Host has a NIC with the MAC address resulted from the script using the below command: (only if vSphere High Availability is enabled)
# esxcli network ip neighbor list

Output should be like the below:
Neighbor Mac Address Vmknic Expiry State Type
-------------- ----------------- ------ -------- ----- -------
192.168.x.xx 00:50:##:##:##:09 vmk0 1190 sec Unknown
192.168.x.xx 00:50:##:##:##:01 vmk0 989 sec Unknown

192.168.x.xx   00:50:##:##:##:16    vmk0     572 sec         Unknown  ----> 192.168.x.xx is the IP address of the locking ESXi Host

192.168.x.xx 00:50:##:##:##:0a vmk0 662 sec Unknown
192.168.x.xx 00:50:##:##:##:1c vmk0 733 sec Unknown
192.168.x.xx 00:50:##:##:##:09 vmk1 122 sec Unknown
192.168.x.xx 00:50:##:##:##:c6 vmk2 833 sec Unknown

Connect to the ESXi Host where the virtual machine is on with an SSH session
To know what is locking the VM files and release the lock , follow the below steps:-

Run the command
# lsof | grep -i "VM_Name"

a) It could be vpxa-worker ----> this can be fixed by restart vpxa agent using the below commands:
# /etc/init.d/vpxa stop
# /etc/init.d/vpxa start

b) It could be hostd or hostd-profiler ---> this can be fixed by restarting hostd agent using the below commands:
# /etc/init.d/hostd stop
# /etc/init.d/hostd start

c) The VM files might be stored in a temp directory /dev/deltadisks----> this can be fixed by going to the temp directory and remove the files
# cd /dev/deltadisks
# rm -rf "VM_Name_00000x-delta.vmdk" ---->file name might be slightly different like "8857142-Test_File_Lock-000002-delta.vmdk"

Warning: Make sure that the directory is /dev/deltadisks . This is the only directory where it is safe to remove the VM files without data loss.

Notes:

If the VM has no clear process locking the files (no output from the command lsof | grep -i "VM_Name") then this is a stale process and the locking ESXi Host has to be rebooted to release the lock.
If you got the error "device or resource busy" while trying to run the rm command, you will need to power off the VM, make sure that there is still a VM file in the /dev/deltadisks and perform the remove again.

Additional Information

For more information , see Troubleshooting issues resulting from locked virtual disks

For more information about Virtual Machine File Lock, see Investigating virtual machine file locks on ESXi hosts