Removing VMFS Volume locks on datastores - Unable to mount the datastore

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms

• While mounting the datastore, see the error as:"volume on device naa.device_id locked, possibly because remote host Host_IP encountered an error during a volume operation and couldn’t recover"
• For example: If naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:<partition number> represents one of the partitions used in a VMFS volume, you see this message when the VMFS volume is inaccessible:
• Volume on device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:<partition number> locked, possibly because remote host yy.yy.yyy.yy encountered an error during a volume operation and couldn’t recover.
• Virtual machine stored on an inaccessible datastore will appear as unavailable in the vSphere Client.

Validation:

From ESXi, var/log/vmkernel.log, see error message similar to:

WARNING: LVM: 13127: The volume on the device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:<partition number> locked, possibly because some remote host encountered an error during a volume operation and could not recover.

LVM: 11786: Failed to open device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:<partition number> : Lock was not free

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

The datastore remains inaccessible to other hosts because the underlying device is locked by an active host.

Resolution

To resolve this issue, remove the lock on the affected ESXi host, run the below command:

1. Connect to the ESXi host with an SSH session and root credentials.
2. Break the existing LVM lock on the datastore:

vmkfstools –B vmfs_device:<partition number>

Note: You can use the parameter --breaklock instead of -B with the vmkfstools command.

For example: vmkfstools -B /vmfs/devices/disks/naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:<partition number>

3. You will prompted similar to:

VMware ESX Question:

LVM lock on device /vmfs/devices/disks/naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:<partition number> will be forcibly broken. Please consult vmkfstools or ESX documentation to understand the consequences of this.

Please ensure that multiple servers aren't accessing this device.

Continue to break lock?

0) Yes

1) No

Please choose a number [0-1]:

4. Enter 0 to break the lock.
5. Re-read and reload VMFS datastore metadata to memory with:

vmkfstools –V

6. From the vSphere UI, refresh the Storage Datastores View under Configuration tab.
   
   
7. If the above steps, does not release the lock, reboot the ESXi host.