Device or resource busy/ file is locked error when making changes to a file on an ESXi datastore

Products

VMware vSphere ESXi

Issue/Introduction

When attempting to modify the file by moving, copying or deleting the file on an ESXi datastore, the user experiences these symptoms:

Modifying the file fails
User sees this error similar to:

device or resource busy

file is locked

Environment

VMware vSphere ESXi 8.0
VMware vSphere ESXi 7.0
VMware vSphere ESXi 6.0

Cause

This issue occurs as the file is locked or is being used by another process or task.

Resolution

A user can receive this error when the file the user is making changes to is locked by another process or task. When the other process is identified, the user must wait for the process to complete or stop it.

Warning: Stopping storage-related processes abruptly may lead to data corruption.

Note: VMware recommends to validate the file locks. For more information, see Investigating Virtual Machine file locks on ESXi Host(s).

To identify processes that are locking the files:

Open an SSH session to the ESXi host and login as the root user.
Run this command:

vmkfstools -D <path>

where <path> is the full path to the file the user is attempting to make changes.
The user can also see the output of the above within the logs. For ESXi hosts, run:

tail /var/log/vmkernel.log
Record the MAC address in the output of the vmkfstools command.

Example

vmkernel: 11:04:31:50.251 cpu1:1035)FS3: 130: <START New Virtual Machine-flat.vmdk>
vmkernel: 11:04:31:50.251 cpu1:1035)Lock [type 10c00001 offset 60088320 v 160, hb offset 3646464
vmkernel: gen 80, mode 1, owner 604931b4-e9227831-####-### mtime 532684]
vmkernel: 11:04:31:50.251 cpu1:1035)Addr <4, 130, 156>, gen 157, links 1, type reg, flags 0x0, uid 0, gid 0, mode 100600
vmkernel: 11:04:31:50.251 cpu1:1035)len 16106127360, nb 15360 tbz 0, zla 3, bs 1048576
vmkernel: 11:04:31:50.251 cpu1:1035)FS3: 132: <END New Virtual Machine-flat.vmdk>

Note: The MAC address 00:50:56:##:##:## is represented as 00505######.
Determine which host has created the lock by comparing the MAC address from the output of the vmkfstools command with the MAC address from the management interface of all other ESXi hosts:

a. Within the vSphere Client, select the host and click the Configure tab > Networking > VMkernel adapter > Select the vmkernel with a Management tag > Properties > NIC Settings.

b. Alternatively, use the esxcfg-vmknic -l command.
To verify if an existing NIC has the lock use (check the MAC addresses of the NICS), run this command:

esxcli network nic list oresxcfg-nics -l
Run the below command to verify that there are no virtual machines that could be using the file the user is trying to modify.

find -H /vmfs/volumes/*/*/*.vmx -print0 | xargs -0 grep "filename"

Note: In ESXi, remove the "-H" option. By doing so, the user would get two lines for every resulting vmx. One with the Datastore easy name and another with the Datastore UUID.
Run the below command to verify that there are no service console processes accessing the file:

lsof | grep -i filename
Verify that all processes accessing the file are complete or are stopped.

If these steps do not identify the process that has locked the file, reboot the host or virtual machine.

Warning: Rebooting a host during a storage operation may result in data loss.

Note: If a .vmdk file is locked due to a disk remaining hot-added to a backup appliance or backup proxy VM, such a lock will typically persist after a host reboot.

For example, if the backup appliance/proxy VM is migrated to another host before the host reboot, or if the VM is restarted. Confirm that no active backup job is in progress on the VM disk and manually remove the hot-added .vmdk from the backup appliance/proxy VM.