Removing VMFS Volume locks on datastores

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms

• You see the error:

volume on device naa.device_id locked, possibly because remote host Host_IP encountered an error during a volume operation and couldn’t recover.

 

For example:

If naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:<partition number> represents one of the partitions used in a VMFS volume, you see this message when the VMFS volume is inaccessible:

 

volume on device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:<partition number> locked, possibly because remote host yy.yy.yyy.yy encountered an error during a volume operation and couldn’t recover.

 

• Virtual machine stored on an inaccessible datastore will appear as unavailable in the vSphere Client
• In the /var/log/vmkernel.log file, you see similar to:

WARNING: LVM: 13127: The volume on the device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:<partition number> locked, possibly because some remote host encountered an error during a volume operation and could not recover.

LVM: 11786: Failed to open device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:<partition number> : Lock was not free

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Resolution

To resolve this issue, remove the lock on the affected ESXi host, run the below commands or reboot the host:

1. Connect to the ESXi host with an SSH session and root credentials.
2. Break the existing LVM lock on the datastore:

vmkfstools –B vmfs_device:<partition number>

Note: You can use the parameter --breaklock instead of -B with the vmkfstools command.

For example: vmkfstools -B /vmfs/devices/disks/naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:<partition number>

3. You will prompted similar to:

VMware ESX Question:

LVM lock on device /vmfs/devices/disks/naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:<partition number> will be forcibly broken. Please consult vmkfstools or ESX documentation to understand the consequences of this.

 

Please ensure that multiple servers aren't accessing this device.

 

Continue to break lock?

0) Yes

1) No

 

Please choose a number [0-1]:

4. Enter 0 to break the lock.
5. Re-read and reload VMFS datastore metadata to memory with:

vmkfstools –V

6. From the vSphere UI, refresh the Storage Datastores View under Configuration tab.