Investigating Virtual Machine file locks on ESXi Host(s)

Products

VMware vSphere ESXi VMware vSphere ESX 8.x VMware vSphere ESX 7.x VMware vSphere ESXi 8.0

Issue/Introduction

Powering on the virtual machine results in the power-on task remaining at 95% indefinitely

Consolidation Task hangs at 0 %
The virtual machine is unable to power on.
The virtual machine is in an 'Invalid' state.
The virtual machine reports conflicting power states between vCenter Server and the ESXi host UI
Adding an existing virtual machine disk (VMDK) to a powered-on virtual machine fails with the following errors:

Failed to add disk scsi0:1. Failed to power on scsi0:1
Cannot power on the virtual machine after deploying it from a template
Powering on a virtual machine fails with an error:
Unable to open Swap File
Unable to access a file since it is locked
Unable to access the virtual machine configuration
Consolidation failing at 0 %

In the /var/log/vmkernel.log file, there are entries similar to

WARNING: World: VM ####: ###: Failed to open swap file path: Lock was not free
WARNING: World: VM ####: ###: Failed to initialize swap file path

When opening a console to the virtual machine, the following error appears

Error connecting to path/virtual machine/vmx because the VMX is not started

Attempting to view or open the .vmx file using a text editor (ex: cat or vi), reports an error similar to:

cat: can't open '[name of vm].vmx': Invalid argument

Environment

ESXi 7.x
ESXi 8.x
ESXi 9.x
VMware Cloud Foundation

Cause

A powered-on virtual machine contains locks on all files in use by the owning ESXi host to facilitate read and write access. Other locks occur by hot-adding disks to snapshot-based backup appliances during the backup process. Failure to create a lock or start a virtual machine occurs if an unsupported disk format is in use or if a lock is already present.
Corruption in the underlying datastore.
A storage hardware failure.

Resolution

The Purpose of File Locking
To prevent concurrent changes to critical virtual machine files and file systems, ESXi hosts establish locks on these files. In certain circumstances, these locks may not be released when the virtual machine is powered off. As such, the files cannot then be accessed by other ESXi hosts while locked and the virtual machine fails to power on.

Virtual machine files locked during runtime include:

VMNAME.vswp
DISKNAME-flat.vmdk
DISKNAME-ITERATION-delta.vmdk
VMNAME.vmx
VMNAME.vmxf
vmware.log

Initial Quick Test:

Set the DRS to manual mode, and attempt to power on the virtual machine on all hosts in the cluster. If the virtual machine powers on, that has the lock.

Identifying the Locked File:

In ESXi 8.0 U2 onwards the filelock owner is shown in the vSphere Client by navigating to VM---> Monitor tab--> Tasks and Events--> Tasks

For pre-ESXi 8.0U2

Power on the virtual machine. This process would fail and display an error message. Make note of this error.
Connect via SSH to the ESXi host on which the virtual machine is currently registered.
Find the IP address of the host holding the lock by running vmfsfilelockinfo on the VMDK flat, delta, or sesparse file for VMFS, or the .UUID.lck file for vSAN.

vmfsfilelockinfo -p /vmfs/volumes/<UUID>/VM_name/VM_name-000001-delta.vmdk

or

vmfsfilelockinfo -p /vmfs/volumes/<UUID>/VM_name/VM_name-000001-delta.vmdk -v <vCenter IP> -u <Admin user/SSO>

vmfsfilelockinfo takes these parameters:

File to be tested
Username and password for accessing VMware vCenter Server (when tracing MAC address to ESX host.)

Example:
vmfsfilelockinfo -p /vmfs/volumes/<UUID>/VM_name/VM_name-000001-delta.vmdk

vmfsfilelockinfo Version 1.0
Looking for lock owners on "VM_name-000001-delta.vmdk"
"VM_name-000001-delta.vmdk" is locked in Exclusive mode by host having mac address ['##:##:##:##:##:##']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 0 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Could not get information from Fault domain manager
Connecting to 192.168.#.### with user administrator@vsphere.local
Password: *example*
----------------------------------------------------------------------
Found 3 ESX hosts from Virtual Center Server.
----------------------------------------------------------------------
Searching on Host ###.###.###.###
Searching on Host ###.###.###.###
Searching on Host ###.###.###.###
MAC Address : ##:##:##:##:##:##
Host owning the lock on the vmdk is ###.###.###.###, lockMode: Exclusive

Note:

During the life-cycle of a powered-on virtual machine, several of its files transitions between various legitimate lock states.

The lock state mode indicates the type of lock that is on the file. The list of lock modes is:

mode 0 = no lock

mode 1 = is an exclusive lock (vmx file of a powered-on virtual machine, the currently used disk (flat or delta), *vswp, and so on.)

mode 2 = is a read-only lock (For example, on the ..-flat.vmdk of a running virtual machine with snapshots)

mode 3 = is a multi-writer lock (For example, used for MSCS clustered disks or FT VMs)

To find the name of the process holding the lock, run lsof 'locked_file_name' on the host holding the lock and filter the output for the file name in question:
Example:
lsof | egrep 'Cartel|VM_name-000001-delta.vmdk'Cartel | World name | Type | fd | Description
362## vmx FILE 80 /vmfs/volumes/<UUID>/VM_name/VM_name-000001-delta.vmdk

This shows that the file is locked by a virtual machine having Cartel ID 362##.

Now display the list of active Cartel IDs with this command:

esxcli vm process list

This displays information for active virtual machines grouped by virtual machine name and having a format similar to:
Alternate_VM_name
World ID: 362##
Process ID: 0
VMX Cartel ID: 362##
UUID: <UUID>
Display Name: Alternate_VM_name
Config File: /vmfs/volumes/<UUID>/Alternate_VM_name/Alternate_VM_name.vmx

The virtual machine entry having VMX Cartel ID: 362## shows the display name of the virtual machine holding the lock on file VM_name-000001-delta.vmdk, which in this example, is Alternate_VM_name.

If no processes are shown, the following script can search for VMs with the locked vmdk mounted

for i in $(vim-cmd vmsvc/getallvms | grep -v Vmid | awk -F "/" '{print $2}' | awk '{print $1}'); do echo $i && find ./ -iname $i | xargs grep vmdk | grep -Ei VMDKS_TO_LOOK_FOR ; done

Note: This script will return all VM_Name.vmx files regardless if the vmdk you're looking for is not found. The reason for this is so you know which VM(s) the vmdks are attached to.
Sample vmdk found output:
vm_name.vmx
vm_name2.vmx
scsi0:0.fileName = "vm_name2-000001.vmdk"
vm_name3.vmx

Sample vmdk not found output:
vm_name.vmx
vm_name2.vmx
vm_name3.vmx

Note:
If your VM name contains spaces, you won't get the full VM name; you'll only get the first word. It's best not to use spaces in VM names.
As of 8.0U3, the vCLS VMs are embedded on the ESXi host and no longer stored on a datastore. As such, the output for the vCLS VMs will return as just /var/..

Sample path to the embedded vCLS VMs:
/var/run/crx/infra/vCLS-28a#####-10b2-4819-37c5-############/vCLS-28a#####-10b2-4819-37c5-############.vmx

Removing the Lock:

A) Reboot Host

1. Power off the virtual machine holding the lock
2. Reboot the ESXi host that holds the lock

B) Removing the .lck file from the virtual machine root directory (NFS only)

Caution: Only remove the .lck files of a powered-off virtual machine. VMFS volumes do not have .lck files; their locking mechanism resides within the VMFS metadata.

1. Power down the virtual machine.
2. Run the command in the virtual machine directory "mkdir bkup" to create a backup directory
3. Move the lock files by running mv lck-#### bkup
4. Confirm the file has been moved and listed in “bkup” folder with the command “ls -lah bkup”

C) Verify the integrity of the virtual machine configuration file (.vmx) - Refer to KB - Validating the .vmx settings of a virtual machine

D) Opening a Support Request
If the problem persists after completing the steps above, Create a Broadcom Support case

Additional Information

Note:

It is possible for a VM to shut down after consolidation if a lock is obtained during the switchover period between the initial disk and base disk. See Virtual Machine shuts down after a disk consolidation due to a locked file for more details.