Creating a snapshot fails with the error: Failed to lock the file when creating a snapshot

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

Cannot create a snapshot manually
Creating a snapshot from a backup software fails
Cannot delete a snapshot
vCenter Server reports the error:

Failed to lock the file when creating a snapshot
Running the touch command indicates locked files

[root@test-esx1 testvm]# touch *
touch: cannot touch `testvm-000002-delta.vmdk': Device or resource busy
touch: cannot touch `testvm-000003-delta.vmdk': Device or resource busy
touch: cannot touch `testvm-000004-delta.vmdk': Device or resource busy
touch: cannot touch `testvm-000005-delta.vmdk': Device or resource busy
touch: cannot touch `testvm-000006-delta.vmdk': Device or resource busy
touch: cannot touch `testvm-000008-delta.vmdk': Device or resource busy

In this sample, you can see that not all files are locked and there are other snapshot files, such as testvm-000001-delta.vmdk, testvm-000007-delta.vmdk, and testvm-000009-delta.vmdk, that are not locked

The lock file (on the host on which the virtual machine is registered) indicates a process lock with the mac address (in red) for the host on which the lock resides:

[root@test-esx1 testvm]# vmkfstools -D test-000008-delta.vmdk
Lock [type 10c00001 offset 45842432 v 33232, hb offset 4116480
gen 2397, mode 2, owner 00000000-00000000-0000-000000000000 mtime 5436998] <-------------- MAC address of lock owner
RO Owner[0] HB offset 3293184 4f284470-4991d61b-4b28-001a64c335dc <------------------------------ MAC address of read-only lock owner*
Addr <4, 80, 160>, gen 33179, links 1, type reg, flags 0, uid 0, gid 0, mode 100600
len 738242560, nb 353 tbz 0, cow 0, zla 3, bs 2097152

Note: * = This field was added in versions of ESX/ESXi equal or greater than 4.0 U3, 4.1 U1. In earlier versions of ESX/ESXi you will not see this line.

For further information on identifying the type of lock see:
Unable to perform operations on a virtual machine with a locked disk (1003397)
This is not a consistent behavior and does not occur on the same virtual machine each time a snapshot is created or removed

Environment

VMware ESXi 4.0.x Embedded
VMware vSphere ESXi 5.0
VMware ESXi 4.1.x Embedded
VMware ESX 4.1.x
VMware ESX 4.0.x

Resolution

This issue is resolved in ESXi 5.0 Update 1 and ESXi 4.1 Update 3.

For more information on these releases and for download and installation information, see:

ESXi 5.0 Update 1 Release Notes
ESXi 4.1 Update 3 Release Notes

To work around this issue in ESX or earlier ESXi releases, Use the vmkfstools -D command to identify the MAC address of the machine locking the file, then reboot or power off the machine that owns that MAC address to release the lock.

Notes:

If the vmkfstools -D test-000008-delta.vmdk command does not return a a valid MAC address in the top field (returns all zeros), review the RO Owner line below it to see which MAC address owns the read only/multi writer lock on the file.
In some cases, it may be a Service Console-based lock, an NFS lock, or a lock generated by another system or product that can use or read the VMFS file systems. The file is locked by a VMkernel child or cartel world and the offending host running the process/world must be rebooted to clear it.
After you have identified the host or backup tool (machine that owns the MAC) locking the file, power it off or stop the responsible service and then restart the management agents on the host running the virtual machine to release the lock.

Additional Information

For more information, see

Unable to perform operations on a virtual machine with a locked disk
Investigating virtual machine file locks on ESXi
Unable to delete the virtual machine snapshots