ESXi metadata locking operations may fail with file level lock with the error : "Device or resource busy [type 10c00001] "

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
Batching large numbers of virtual machine (VM) or virtual machine disk (VMDK) creation, cloning, and/or deletion on shared datastores may fail when multiple hosts are operating simultaneously.

Multiple hosts performing VM/VMDK operations on the same shared datastore and accessing file-level locks (type 10c00001) may see this error:

Failed to create virtual disk: Device or resource busy.
Failed to lock the file.

Entries similar to these in the /var/run/log/vmkernel.log file include:

DLX: 3771: vol 'datastore-name', lock at 4325376: [Req mode 1] Checking liveness:[type 10c00001 offset 4325376 v 26266, hb offset 3219456gen 21, mode 1, owner 57579305-35931dbf-05d5000c29923ba3 mtime 62431num 0 gblnum 0 gblgen 0 gblbrk 0].

DLX: 4417: vol 'datastore-name', lock at 4325376: [Req mode: 1] Not free:[type 10c00001 offset 4325376 v 26272, hb offset 3219456gen 21, mode 1, owner 57579305-35931dbf-05d5-000c29923ba3 mtime 62463num 0 gblnum 0 gblgen 0 gblbrk 0].

 

Cause

When virtual machine (VM) and virtual machine directory (VMDK) operations are performed on an ESXi host such as creation, cloning, and deletion, it can lead to contention among multiple hosts accessing shared datastore resources. This can result in delays for other hosts seeking access to the same resource.

The potential for contention may trigger ESXi to issue lock liveness checks of type 10c00001 if the locks are not released when no longer needed. If this occurs, it could potentially prevent the system from responding to certain requests as expected.

Resolution

1. A recommended approach when using vmkfstools involves working at the directory level to separate out access locations on each host. To create a VMDK file under the VM's directory, use the following syntax: . For example, to create vmdk under VM 'vim-cmd /vmsvc/createdummyvm vm1' in the datastore volume, run:
   vmid = vim-cmd /vmsvc/createdummyvm vm1 /vmfs/volumes/datastore
   
   
2. Then, modify the VMDK file using vmkfstools: . For example, to create a 1GB VMDK file for VM 'vm1' in the datastore volume, run:
   vmkfstools -d eagerzerodthick -c 1g /vmfs/volumes/datastore/vm1/vm1_data.vmdk
   
   
3. To truncate a VMDK file before deleting it, use the following syntax: . For example, to delete the vmdk file /vmfs/volumes/datastore/vm1/vm1_data-flat.vmdk, run:
   echo /dev/null > /vmfs/volumes/datastore/vm1/vm1_data-flat.vmdk
   
   
4. To destroy a VM that is no longer needed, use the following syntax: 
   vim-cmd /vmsvc/destroy vmid

Additional Information

For more information, see 

VMware virtual machine file lock on VMFS datastore
Cloning and converting virtual machine disks with vmkfstools