Investigating virtual machine file locks on ESXi hosts

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Powering on the virtual machine results in the power on task remaining at 95% indefinitely.
VM in "Invalid" state.
Adding an existing virtual machine disk (VMDK) to a powered on virtual machine fails with the error:
- Failed to add disk scsi0:1. Failed to power on scsi0:1
- Cannot power on the virtual machine after deploying it from a template.
- Powering on a virtual machine fails with an error:
- Unable to open Swap File
- Unable to access a file since it is locked
- Unable to access a file filename since it is locked
- Unable to access Virtual machine configuration
In the /var/log/vmkernel log file, there are entries similar to:

WARNING: World: VM ####: ###: Failed to open swap file path: Lock was not free
WARNING: World: VM ####: ###: Failed to initialize swap file path

When opening a console to the virtual machine, the following error appears:

Error connecting to path/virtual machine/vmx because the VMX is not started

The virtual machine reports conflicting power states between vCenter Server and the ESXi host UI.
Attempting to view or open the .vmx file using a text editor (ex: cat or vi), reports an error similar to:

cat: can't open '[name of vm].vmx': Invalid argument

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

VMware horizon

Resolution

The Purpose of File Locking

To prevent concurrent changes to critical virtual machine files and file systems, ESXi hosts establish locks on these files. In certain circumstances, these locks may not be released when the virtual machine is powered off. As such, the files cannot then be accessed by other ESXi hosts while locked and the virtual machine fails to power on.

Virtual machine files locked during runtime include:

VMNAME.vswp
DISKNAME-flat.vmdk
DISKNAME-ITERATION-delta.vmdk
VMNAME.vmx
VMNAME.vmxf
vmware.log

Initial Quick Test

Set DRS in manual mode and attempt to power on the virtual machine on all hosts in the cluster. If the virtual machine powers on, that host had the lock.

Identifying the Locked File

Power on the virtual machine, this process should fail and display an error message. Make note of the this error.
Connect to the ESXi host the virtual machine is on with an SSH session
Find the IP address of the host holding the lock by running vmfsfilelockinfo on the VMDK flat, delta, or sesparse file for VMFS, or the .UUID.lck file for vSAN.

vmfsfilelockinfo takes these parameters:

File to be tested
Username and password for accessing VMware vCenter Server (when tracing MAC address to ESX host.)

For example:

vmfsfilelockinfo -p /vmfs/volumes/<UUID>/VM1/VM1_1-000001-delta.vmdk -v 192.168.1.### -u [email protected]

vmfsflelockinfo Version 1.0
Looking for lock owners on "VM1_1-000001-delta.vmdk"
"VM1_1-000001-delta.vmdk" is locked in Exclusive mode by host having mac address ['##:##:##:##:##:##']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 0 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Could not get information from Fault domain manager
Connecting to 192.168.1.XXX with user [email protected]
Password: *example*
----------------------------------------------------------------------
Found 3 ESX hosts from Virtual Center Server.
----------------------------------------------------------------------
Searching on Host 192.168.1.###
Searching on Host 192.168.1.###
Searching on Host 192.168.1.###
MAC Address : ##:##:##:##:##:##

Host owning the lock on the vmdk is 192.168.1.###, lockMode : Exclusive

Total time taken : 0.27 seconds.

Note: During the life-cycle of a powered on virtual machine, several of its files transitions between various legitimate lock states. The lock state mode indicates the type of lock that is on the file. The list of lock modes is:

mode 0 = no lock
mode 1 = is an exclusive lock (vmx file of a powered on virtual machine, the currently used disk (flat or delta), *vswp, and so on.)
mode 2 = is a read-only lock (For example on the ..-flat.vmdk of a running virtual machine with snapshots)
mode 3 = is a multi-writer lock (For example used for MSCS clusters disks or FT VMs)

To find the name of the process holding the lock, run lsof 'locked_file_name' on the host holding the lock and filter the output for the file name in question:

For example:

Now display the list of active Cartel IDs by with this command:

esxcli vm process list

This displays information for active virtual machines grouped by virtual machine name and having a format similar to:

Alternate_VM27
World ID: 36205
Process ID: 0
VMX Cartel ID: 36202
UUID: <UUID>
Display Name: Alternate_VM27
Config File: /vmfs/volumes/<UUID>/Alternate_VM27/Alternate_VM27.vmx
………

The virtual machine entry having VMX Cartel ID: 36202 shows the display name of the virtual machine holding the lock on file VM1_1-000001-delta.vmdk, which in this example, is Alternate_VM27.

If no processes are shown, the following script can search for VMs with the locked vmdk mounted.

for i in $(vim-cmd vmsvc/getallvms | grep -v Vmid | awk -F "/" '{print $2}' | awk '{print $1}'); do echo $i && find ./ -iname $i | xargs grep vmdk | grep -Ei VMDKS_TO_LOOK_FOR ; done

Removing the Lock

Power off the virtual machine holding the lock
Reboot the ESXi host that holds the lock

Note: In the event the ESXi host abruptly shut down due to hardware failure, rebooting the host may not be an option. Please refer to KB "Remove VM vmx file lock" in this scenario, to remove lock on "vmx" file.

Removing the .lck file from VM root directory (NFS only)

Power down the virtual machine.

Caution: Only remove the lck files of a powered off virtual machine.
Create directory “bkup” in the VM folder with "mkdir bkup".
Move the "lck-####" lock file(s) with “mv lck-#### bkup”.
Confirm file has been moved and listed in “bkup” folder with “ls -lah bkup”.

Note: VMFS volumes do not have lck files. The locking mechanism for VMFS volumes is handled within VMFS metadata on the volume.

Verify the integrity of the virtual machine configuration file (.vmx)

For more information on checking the integrity of the virtual machine configuration file, see Verifying ESX/ESXi virtual machine file integrity

For related information, see

Opening a Support Request

If the problem persists after completing the steps in this article:

Collecting diagnostic information for VMware products

Additional Information

Note: It is possible for a VM to shutdown after consolidation if a lock is obtained during the switchover period between the initial disk and base disk. See Virtual Machine shuts down after a disk consolidation due to a locked file for more details.