VMware virtual machine file lock on VMFS datastore
search cancel

VMware virtual machine file lock on VMFS datastore

book

Article ID: 313833

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • Snapshots cannot be committed or consolidated.
  • Third-party backup application (Veeam, Avamar, Unitrends,...etc) has locked snapshot files, and failed to remove the snapshot after completing backups or failed to initiate backups. 
  • Snapshot commits without errors and Snapshot Manager is no longer populated. However, the snapshot disk is present in the virtual machine directory. (the VM is still running on Snapshot)
  • The virtual machine summary tab display messages similar to:
    • Snapshot consolidation required.
    • Virtual machine disks consolidation is needed.
    • Virtual machine Consolidation Needed status.
    • Consolidation failed with the errors similar to:
    • Failed to lock the file or One or more disks are busy.
    • Unable to consolidate virtual machine snapshots due to file lock.
    • Unable to access filename  since it is locked.
    • Unable to access file since it is locked
In hostd.log on the ESXi Host where the virtual machine is running, entries are seen similar to the below:
Note: hostd.log can be located in /var/run/log/hostd.log

YYYY-MM-DDTHH:MM:SS.819Z info hostd[1050487] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 361 : Lock(s) held on file Win-test-flat.vmdk by other host(s).
YYYY-MM-DDTHH:MM:SS.822Z info hostd[1051013] [Originator@6876 sub=Vimsvc.TaskManager opID=d88eb069 user=vpxuser] Task Created : haTask--vim.event.EventHistoryCollector.readNext-66249
YYYY-MM-DDTHH:MM:SS.823Z info hostd[1050756] [Originator@6876 sub=Vimsvc.TaskManager opID=d88eb069 user=vpxuser] Task Completed : haTask--vim.event.EventHistoryCollector.readNext-66249 Status success
YYYY-MM-DDTHH:MM:SS.826Z info hostd[1050487] [Originator@6876 sub=Vimsvc.TaskManager opID=d88eb06c user=vpxuser] Task Created : haTask--vim.event.EventHistoryCollector.readNext-66250
  • Can’t power on virtual machine due to file lock:
    • Unable to open Swap File.
    • Unable to access a file since it is locked.
    • Unable to access a file filename since it is locked.
    • Unable to access Virtual machine configuration.
    • An unexpected error was received from the ESX host while powering on VM <VM_name>.
      Reason: Cannot open the disk disk_name or one of the snapshot disks it depends on.
In vmkernel.log , on the ESXi Host where the virtual machine is running, entries similar to the below are seen:
Note: vmkernel.log can be located in  /var/run/log/vmkernel.log
 
WARNING: World: VM xxxx: xxx: Failed to open swap file path: Lock was not free
WARNING: World: VM xxxx: xxx: Failed to initialize swap file path

 
 



Environment

VMware vSphere 8.0.x
VMware vSphere 7.0.x
VMware vSphere 6.x

Cause

  • When a virtual machine snapshot fails to consolidate or a virtual machine with snapshots fails to power on, it is usually because one or more of the virtual machine files are abnormally locked by the same ESXi Host the virtual machine is running or any other ESXi Host.
  • The ESXi Host (either same Host where the VM is running or any other Host) is abnormally locking the virtual machine files due to many reasons and not limited to the below:
    • In-completed or interrupted backup task
    • Previous storage failure

Resolution

Virtual  Machine File Lock most common scenarios:
1-Proxy Back Up Virtual Machine (Veeam, Avamar, Unitrends,...etc) is locking one or more of the virtual machine .vmdk files.
2-An ESXi Host is locking the Virtual Machine files abnormally.

First Scenario

Caution: Make sure there is no backup or snapshot task running on the virtual machine before applying the resolution.

  • The most common case of virtual machine file lock is when the Backup Proxy Virtual Machine (Veeam, Avamar, Unitrends,...etc) is locking one or more of the virtual machine .vmdk files. This issue can be resolved using the below steps:
  1. Right-click the backup proxy virtual machine.
  2. Click Edit Settings.
  3. Expand all the Hard Disk(s).
  4. Select the Hard Disk(s) which belong to the virtual machine that has the problem.
  5. Click on the X beside the Hard Disk to unmount the Hard Disk from the VM.
Caution: Do NOT select Delete files from the datastore.
  1. Click OK.
  2. Consolidate/Delete the snapshot on the VM.
" Power on the VM if the issue was unable to power on"

Second Scenario
  • If the first section is not the case, then we need to identify which process is locking the virtual machine files to release.
  1. Connect to the ESXi host the virtual machine is on with an SSH session
  2. Navigate to the VM directory and take a note of where the VM is currently writing.
To know the VM directory, follow the below steps:

1- Run the below command
# vim-cmd vmsvc/getallvms | grep -i "VM_Name"


2- Run the below command:
# cd /vmfs/volumes/datastore_name/virtual_machine_name/     --->From step 1


3-Run the below command:
# grep -i vmdk *vmx


*Now the VM is writing on Test_File_Lock-000002.vmdk in the example

  1. Check which ESXi Host is locking the VM file(s) and clear the lock by running the below script in the VM directory:
# ls | while read x; do vmfsfilelockinfo -p $x| grep -i "is locked"; done

Note: The script will not work if the virtual machine files has spaces. In this case, you will need to run the command  vmfsfilelockinfo -p on each and every file listed under the virtual machine directory.
Example: vmfsfilelockinfo -p Test_File_Lock-flat.vmdk

There will be two probabilities: (after running the script)
  1. The VM files are locked by same ESXi Host where the VM is running (owning Host).

Notes:
1-If the VM is powered on so it should be normally locked by the ESXi Host it is running on (referred by the MAC address in the script output) 
2-If the VM is powered off then there shouldn't be an output from the script as VM files shouldn't be normally locked by any process (from any ESXi Host)

To know what is locking the VM files and release the lock , follow the below :
# lsof | grep -i "VM_Name"

a- It could be vpxa-worker ----> this can be fixed by restart vpxa agent using the below commands:
# /etc/init.d/vpxa stop
# /etc/init.d/vpxa start

b- It could be hostd or hostd-profiler ---> this can be fixed by restarting hostd agent using the below commands:
# /etc/init.d/hostd stop
# /etc/init.d/hostd start

c- The VM files might be stored in a temp directory /dev/deltadisks----> this can be fixed by going to the temp directory and remove the files
# cd /dev/deltadisks

# rm -rf "VM_Name_00000x-delta.vmdk" ---->file name might be slightly different like "8857142-Test_File_Lock-000002-delta.vmdk"

Warning: Make sure that the directory is /dev/deltadisks . This is the only directory where it is safe to remove the VM files without data loss.

Notes:
  • If the VM is powered off then, this script should show no output as no process should be locking the VM files. So if the script shows and output and the VM is powered off, then there is a stale process that will only be released by rebooting the ESXi Host. 
  • If you got the error "device or resource busy" while trying to run  the rm command , you will need to power off the VM, make sure that there is still a VM file in the /dev/deltadisks and perform the remove again.
  1. The VM files are locked  by another ESXi Host
 
All the files are locked by Host that has the same MAC address except one file "Test_File_Lock-flat.vmdk"  that is locked by two ESXi Hosts (two MAC addresses)

As the VM is powered on so there should be normal lock from the owing ESXi Host but there is another abnormal lock coming from another Host that has the MAC address "00:50:##:##:##:16" in the example

To release the lock from the other ESXi:
  1. Identify which ESXi Host locking the the file from the vCenter GUI
  • Expand the Cluster and select any ESXi Host
  • Click on Configure
  • Click on Physical adapters and look for the MAC address "Output from the previous step"
Note: You can identify which ESXi  Host has a NIC with the MAC address resulted from the script using the below command: (only if vSphere High Availability is enabled)
# esxcli network ip neighbor list

Output should be like the below:
Neighbor        Mac Address        Vmknic    Expiry  State  Type
--------------  -----------------  ------  --------  -----  -------
192.168.x.xx   00:50:##:##:##:09    vmk0    1190 sec         Unknown
192.168.x.xx   00:50:##:##:##:01    vmk0     989 sec         Unknown
192.168.x.xx   00:50:##:##:##:16    vmk0     572 sec         Unknown  ----> 192.168.x.xx is the IP address of the locking ESXi Host
192.168.x.xx   00:50:##:##:##:0a    vmk0     662 sec         Unknown
192.168.x.xx   00:50:##:##:##:1c    vmk0     733 sec         Unknown
192.168.x.xx   00:50:##:##:##:09    vmk1     122 sec         Unknown
192.168.x.xx   00:50:##:##:##:c6    vmk2     833 sec         Unknown
 
  1. Connect to the ESXi Host where the virtual machine is on with an SSH session
  2. To know what is locking the VM files and release the lock , follow the below :
# lsof | grep -i "VM_Name"
a- It could be vpxa-worker ----> this can be fixed by restart vpxa agent using the below commands:
# /etc/init.d/vpxa stop
# /etc/init.d/vpxa start

b- It could be hostd or hostd-profiler ---> this can be fixed by restarting hostd agent using the below commands:
# /etc/init.d/hostd stop
# /etc/init.d/hostd start

c- The VM files might be stored in a temp directory /dev/deltadisks----> this can be fixed by going to the temp directory and remove the files
# cd /dev/deltadisks
# rm -rf "VM_Name_00000x-delta.vmdk" ---->file name might be slightly different like "8857142-Test_File_Lock-000002-delta.vmdk"

Warning: Make sure that the directory is /dev/deltadisks . This is the only directory where it is safe to remove the VM files without data loss.
 
 
Notes:
  • If the VM has no clear process locking the files (no output from the command lsof | grep -i "VM_Name") then this is a stale process and the locking ESXi Host has to be rebooted to release the lock.
  • If you got the error "device or resource busy" while trying to run  the rm command , you will need to power off the VM, make sure that there is still a VM file in the /dev/deltadisks and perform the remove again.

 



Additional Information

For more information , see Troubleshooting issues resulting from locked virtual disks

For more information about Virtual Machine File Lock, see Investigating virtual machine file locks on ESXi hosts



Powered by Wolken Software