Virtual Machine cannot be powered on error "Lost previously held disk lock"

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual Machine (VM) power on fails with the following error:

Power On virtual machine, Powering on the new Virtual Machine,#####,System,Module VMMon power on failed.

Failed to start the virtual machine. Failed to power on VM. Could not power on virtual machine: Lost previously held disk lock.

• cat command on file gives input/output error:

# cat /vmfs/volumes/#######-#######-####-###########/VM_folder/vm.vmdk

• Power on process failed with error:

File system specific implementation of OpenFile[file] failed Failed to start the virtual machine. Module Disk power on failed. Cannot open the disk '/vmfs/volumes/#######-#######-####-###########/VM_folder/vm.vmdk' or one of the snapshot disks it depends on. 5 (Input/output error)

• VM vmx file and vmdk files are locked by a host.
  
  
• Host reboot does not clear the lock.
  
  
• VM becomes inaccessible (after rebooting the ESXi host).
  
  
• VMs on VMFS datastore are appearing Invalid.

 

Environment

VMware vSphere 7.x
VMware vSphere 8.x
VMware vSphere 9.x

Cause

• The Virtual Machine files may be locked.
  
  
• Backing storage may have an issue.
  
  
• Datastore may be corrupted.

Resolution

• Follow steps for Investigating Virtual Machine file locks on ESXi Host(s).
  
  
• If the backing storage has an issue, follow these steps to investigate and restore storage connectivity All Paths Down for a storage device and Permanent Device Loss (PDL) and All-Paths-Down (APD) on host.
  
  
• If the above steps do not resolve the issue, follow these steps to check datastore consistency:
  
  
  • Evacuate all working VMs off datastore
    
    
  • Run VOMA on the datastore to identify the lock.
    
    
  • Ensure good backups exist before issuing voma commands
    
    
  • Follow the below steps to run the VOMA (vSphere On-disk Metadata Analyzer) tool:
    
    1. For VMFS datastores, the datastore should be unmounted on all ESXi  hosts.
    
    2. For Identifying disks when working with VMware ESXi, use following command:

# esxcfg-scsidevs -m

3. To check for any metadata inconsistencies or stale lock  on the LUN, run the check command on the partition:

# voma -m vmfs -f check -d /vmfs/devices/disks/naa.######################:1

• • • If "Found stale lock 1c00001 offset ##### v ##### hb offset ##### mode 1 , owner........." messages are observed in the voma check, then contact Broadcom Technical support for assistance in clearing stale lock(s).

Additional Information

Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency highlights any inconsistencies to which you may be required to open a support request.

Prerequisites running VOMA:

• Power off any virtual machines that are running or migrate them to a different datastore.
  
  
• Make sure that the VMFS datastore you analyze does not span multiple extents. You can run the VOMA tool only against a single-extent datastore.