Investigating Virtual Machine file locks on ESXi Host(s)

Products

• VMware vSphere ESXi

Issue/Introduction

• Powering on the virtual machine results in the power on task remaining at 95% indefinitely
• VM in "Invalid" state
• The virtual machine reports conflicting power states between vCenter Server and the ESXi host UI
• Adding an existing virtual machine disk (VMDK) to a powered on virtual machine fails with the error:
  • Failed to add disk scsi0:1. Failed to power on scsi0:1
  • Cannot power on the virtual machine after deploying it from a template
  • Powering on a virtual machine fails with an error:
  • Unable to open Swap File
  • Unable to access a file since it is locked
  • Unable to access a file filename since it is locked
  • Unable to access Virtual machine configuration

• In the /var/log/vmkernel log file, there are entries similar to:

WARNING: World: VM ####: ###: Failed to open swap file path: Lock was not free
WARNING: World: VM ####: ###: Failed to initialize swap file path

 

• When opening a console to the virtual machine, the following error appears:

Error connecting to path/virtual machine/vmx because the VMX is not started

 

• Attempting to view or open the .vmx file using a text editor (ex: cat or vi), reports an error similar to:

cat: can't open '[name of vm].vmx': Invalid argument

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

 

Resolution

The Purpose of File Locking

To prevent concurrent changes to critical virtual machine files and file systems, ESXi hosts establish locks on these files.

In certain circumstances, these locks may not be released when the virtual machine is powered off.

As such, the files cannot then be accessed by other ESXi hosts while locked and the virtual machine fails to power on.

Virtual machine files locked during runtime include:

• VMNAME.vswp
• DISKNAME-flat.vmdk
• DISKNAME-ITERATION-delta.vmdk
• VMNAME.vmx
• VMNAME.vmxf
• vmware.log

 

Initial Quick Test

• Set DRS in manual mode and attempt to power on the virtual machine on all hosts in the cluster. If the virtual machine powers on, that host had the lock.

 

Identifying the Locked File

1. Power on the virtual machine, this process should fail and display an error message. Make note of the this error.
2. Connect to the ESXi host the virtual machine is on with an SSH session
3. Find the IP address of the host holding the lock by running vmfsfilelockinfo on the VMDK flat, delta, or sesparse file for VMFS, or the .UUID.lck file for vSAN.

vmfsfilelockinfo takes these parameters:

• File to be tested
• Username and password for accessing VMware vCenter Server (when tracing MAC address to ESX host.)

Example:

vmfsfilelockinfo -p /vmfs/volumes/<UUID>/VM1/VM1_1-000001-delta.vmdk -v 192.168.1.### -u [e-mail]

vmfsflelockinfo Version 1.0
Looking for lock owners on "VM1_1-000001-delta.vmdk"
"VM1_1-000001-delta.vmdk" is locked in Exclusive mode by host having mac address ['##:##:##:##:##:##']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 0 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Could not get information from Fault domain manager
Connecting to 192.168.1.XXX with user [e-mail]
Password: *example*
----------------------------------------------------------------------
Found 3 ESX hosts from Virtual Center Server.
----------------------------------------------------------------------
Searching on Host 192.168.1.###
Searching on Host 192.168.1.###
Searching on Host 192.168.1.###
MAC Address : ##:##:##:##:##:##

Host owning the lock on the vmdk is 192.168.1.###, lockMode : Exclusive


Note:

During the life-cycle of a powered on virtual machine, several of its files transitions between various legitimate lock states.

The lock state mode indicates the type of lock that is on the file. The list of lock modes is:

• • mode 0 = no lock
  • mode 1 = is an exclusive lock (vmx file of a powered on virtual machine, the currently used disk (flat or delta), *vswp, and so on.)
  • mode 2 = is a read-only lock (For example on the ..-flat.vmdk of a running virtual machine with snapshots)
  • mode 3 = is a multi-writer lock (For example used for MSCS clusters disks or FT VMs)

 

4. To find the name of the process holding the lock, run lsof 'locked_file_name' on the host holding the lock and filter the output for the file name in question:

Example:

lsof | egrep 'Cartel|VM1_1-000001-delta.vmdk'

Cartel | World name | Type | fd | Description
36202 vmx FILE 80 /vmfs/volumes/<UUID>/VM1/VM1_1-000001-delta.vmdk

This shows that the file is locked by a virtual machine having Cartel ID 36202.

 

5. Now display the list of active Cartel IDs by with this command:

esxcli vm process list

This displays information for active virtual machines grouped by virtual machine name and having a format similar to:

Alternate_VM27
World ID: 36205
Process ID: 0
VMX Cartel ID: 36202
UUID: <UUID>
Display Name: Alternate_VM27
Config File: /vmfs/volumes/<UUID>/Alternate_VM27/Alternate_VM27.vmx

The virtual machine entry having VMX Cartel ID: 36202 shows the display name of the virtual machine holding the lock on file VM1_1-000001-delta.vmdk, which in this example, is Alternate_VM27.
 

6. If no processes are shown, the following script can search for VMs with the locked vmdk mounted

for i in $(vim-cmd vmsvc/getallvms | grep -v Vmid | awk -F "/"  '{print $2}' | awk '{print $1}'); do echo $i && find ./ -iname $i | xargs grep vmdk | grep -Ei VMDKS_TO_LOOK_FOR ; done

 

Removing the Lock - Options:

 

A.) Reboot Host

1.) Power off the virtual machine holding the lock

2.) Reboot the ESXi host that holds the lock

Note: In the event the ESXi host abruptly shut down due to hardware failure, rebooting the host may not be an option. Please refer to KB "Remove VM vmx file lock" in this scenario, to remove lock on "vmx" file.

 

 

B.) Removing the .lck file from VM root directory (NFS only)

1. Power down the virtual machine 
   Caution: Only remove the lck files of a powered off virtual machine.

2. Create directory “bkup” in the VM folder with "mkdir bkup"

3. Move the "lck-####" lock file(s) with “mv lck-#### bkup”

4. Confirm file has been moved and listed in “bkup” folder with “ls -lah bkup”

Note: VMFS volumes do not have lck files. The locking mechanism for VMFS volumes is handled within VMFS metadata on the volume

 

C.) Verify the integrity of the virtual machine configuration file (.vmx)
For more information on checking the integrity of the virtual machine configuration file, see Verifying ESX/ESXi virtual machine file integrity

 

Opening a Support Request

If the Problem persists after completing the steps above, proceed with opening a Ticket with VMware by Broadcom Support

 

Additional Information

• Failed to power on virtual machine
• "Cannot open the disk" errors powering on a VM
• Powering off an unresponsive virtual machine on an ESXi host

 

Note:

It is possible for a VM to shutdown after consolidation if a lock is obtained during the switchover period between the initial disk and base disk.

See Virtual Machine shuts down after a disk consolidation due to a locked file for more details.