Device or resource busy/ file is locked error when making changes to a file on an ESXi datastore

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
When attempting to modify the file by moving, copying or deleting the file on an ESXi datastore, you experience these symptoms:

Modifying the file fails.
You see this error similar to:

device or resource busy

Or

file is locked

Environment

VMware vSphere ESXi 8.0
VMware vSphere ESXi 7.0
VMware vSphere ESXi 6.0

Cause

This issue occurs as the file is locked or is being used by another process or task.

Resolution

You can receive this error when the file you are making changes to is locked by another process or task. When the other process is identified, you must wait for the process to complete or stop it.

Warning: Stopping storage-related processes abruptly may lead to data corruption.

Note: VMware recommends to validate the file locks. For more information, see Investigating virtual machine file locks on ESXi hosts.

To identify processes that are locking the files:

Open an SSH session to the ESXi host and login as the root user.
Run this command:

vmkfstools -D path

where the path is the full path to the file you are attempting to make changes.
You can also see the output of the above on the logs: For ESXi hosts, run this command:

tail /var/log/vmkernel.log
Record the MAC address in the output of the vmkfstools command.

For example:

vmkernel: 11:04:31:50.251 cpu1:1035)FS3: 130: <START New Virtual Machine-flat.vmdk>
vmkernel: 11:04:31:50.251 cpu1:1035)Lock [type 10c00001 offset 60088320 v 160, hb offset 3646464
vmkernel: gen 80, mode 1, owner 604931b4-e9227831-df99-XXXXXXXXXXX mtime 532684]
vmkernel: 11:04:31:50.251 cpu1:1035)Addr <4, 130, 156>, gen 157, links 1, type reg, flags 0x0, uid 0, gid 0, mode 100600
vmkernel: 11:04:31:50.251 cpu1:1035)len 16106127360, nb 15360 tbz 0, zla 3, bs 1048576
vmkernel: 11:04:31:50.251 cpu1:1035)FS3: 132: <END New Virtual Machine-flat.vmdk>

Note: The MAC address 00:50:56:XX:XX:XX is represented as 00505XXXXXX.
Determine which host has created the lock by comparing the MAC address from the output of the vmkfstools command with the MAC address from the management interface of all other ESXi hosts:

a. On the vSphere Client, select the host and click the Configure tab > Networking > VMkernel adapter > Select the vmkernel with a Management tag > Properties > NIC Settings.

b. You can also get this information with the esxcfg-vmknic -l command.
To verify if an existing NIC has the lock use (Check the MAC addresses of the NICS), run this command:

esxcli network nic list or # esxcfg-nics -l
Run this command to verify that there are no virtual machines that could be using the file you are trying to modify.

find -H /vmfs/volumes/*/*/*.vmx -print0 | xargs -0 grep "filename"

Note: In ESXi, remove the "-H" option. By doing so, you would get two lines for every resulting vmx. One with the Datastore easy name and another with the Datastore UUID.
Run this command to verify that there are no service console processes accessing the file:

lsof | grep -i filename
Verify that all processes accessing the file are complete or are stopped.
If these steps do not identify the process that has locked the file, reboot your host or virtual machine.

Warning: Rebooting a host during a storage operation may result in data loss.

Note: If a .vmdk file is locked due to a disk remaining hot-added to a backup appliance or backup proxy VM, such a lock will typically persist after a host reboot. (For example: If the backup appliance/proxy VM is migrated to another host before the host reboot, or if the VM is restarted). Confirm that no active backup job is in progress on the VM disk and manually remove the hot-added .vmdk from the backup appliance/proxy VM. See Unable to consolidate VMware virtual machine snapshots due to file lock