Error "Unable to enumerate all disks. Failed to lock the file" while powering on VM on vSAN Datastore

Products

• VMware vSAN

Issue/Introduction

• vMotion of the VM may fail after 66% and VM may show power off.

• Upon powering on the VM, you may see the error "Unable to enumerate all disks. Failed to lock the file" on vCenter server.

• Some of the disks of the virtual machine VM failed to load. The information present for them in the virtual machine configuration may be incomplete.

• The VMDK is question may also show 0 MB capacity upon checking through VM edit settings.

Environment

• VMware ESXi 8.x
• VMware ESXi 9.x

Cause

• This issue would occur when the VMDK is opened using "cat" or "less" on any of the hosts.  Verify this using the command below.

• On ESXi host command line, look for list of open files and search for VM name or VM folder UUID (if vSAN).

Example as below.

[root@host:~] vmkvsitools lsof | grep  c152f068-a025-d538-ae72-########
64352185    less                  cwd                        -1   /vmfs/volumes/vsan:####-####/c152f068-a025-d538-ae72-########
64352185    less                  FILE                        4   /vmfs/volumes/vsan:####-####/c152f068-a025-d538-ae72-########/VM.vmdk
64351388    sh                    cwd                        -1   /vmfs/volumes/vsan:####-####/c152f068-a025-d538-ae72-########

• Upon querying the VMDK from the VM folder, you may see the below events for the lock.

• On ESXi host, navigate to the VM folder and query the vmdk using command 'vmkfstools -q VM.vmdk -v10'

AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/vsan:524a700766db7fea-####/e37a9166-b2a9-a99e-73e3-####/VM.vmdk' locked (try 0)
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/vsan:524a700766db7fea-####/e37a9166-b2a9-a99e-73e3-####/VM.vmdk' locked (try 1)
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/vsan:524a700766db7fea-####/e37a9166-b2a9-a99e-73e3-####/VM.vmdk' locked (try 2)
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/vsan:524a700766db7fea-####/e37a9166-b2a9-a99e-73e3-####/VM.vmdk' locked (try 3)
AIOMGR: AIOMgr_OpenWithRetry: Descriptor file '/vmfs/volumes/vsan:524a700766db7fea-####/e37a9166-b2a9-a99e-73e3-####/VM.vmdk' locked (try 4)
OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/vsan:524a700766db7fea-####/e37a9166-b2a9-a99e-73e3-####/VM.vmdk' : Failed to lock the file (262146)

• Sometimes you may not see the file opened, but may still see the task process show stale in the vmkernel.

 

Resolution

To resolve the issue, 

• If the file is open in any of the ESXi host, close file or session. This should free up the lock and allow the file to be opened by another task.

• If there are no open sessions and if you still see the output from command for 'vmkvsitools lsof' you can go ahead and kill the process id post verifying the impact.

• If the process cannot be killed, you can put the host into maintenance mode and reboot to resolve the issue.