Virtual machine is inaccessible or cannot be powered on due to file lock
search cancel

Virtual machine is inaccessible or cannot be powered on due to file lock

book

Article ID: 392268

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms : 

  • Virtual machines fail to power on.

  • The Migrate option for affected virtual machines is grayed out.

  • Some virtual machines are reported as inaccessible and cannot be powered on.

  • Some virtual machines appear in a powered-on state but are not accessible or fail to boot.

  • When attempting to power off, power on, or reload the VMX file of a virtual machine, command may return an error similar to the following:

    [root@ESXi]vim-cmd vmsvc/reload 14
    (vmodl.fault.SystemError) {
       faultCause = (vim.fault.TaskInProgress) {
          faultCause = (vmodl.MethodFault) null,
          faultMessage = <unset>,
          task = 'vim.Task:haTask-14-vim.VirtualMachine.createSnapshot-98044'
          msg = "Another task is already in progress."
       },
       faultMessage = <unset>,
       reason = "Undeclared fault"
       msg = "Received SOAP response fault from [<<io_obj p:0x0000004f03a0d780, h:5, <TCP '127.0.0.1 : 37616'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: reload
    Method vim.ManagedEntity.reload threw undeclared fault of type vim.fault.TaskInProgress"
    } 

  • .vmx fails to register in vCenter or host inventory, adding it to the inventory will show up as invalid with the full path to the .vmx.

  • The vmware.log file of the affected virtual machine may report disk or snapshot-related errors similar to the following:

 Er(02) vmx - DISKLIB-LIB_CREATE   : DiskLib_NotifySnapshotPrepare: FiltLib failed to notify snapshot prepare: Operation completes asynchronously
 Er(02) vmx - DISKLIB-LIB_CREATE   : DiskLib_NotifySnapshotPrepare: FiltLib failed to notify snapshot prepare: Operation completes asynchronously
 In(05) vcpu-0 - AIOGNRC: Failed to open '/vmfs/devices/vsan/xxxx-5425-e8e1-c157-xxxx' : Could not find the file (60003) (0x2013).
 In(05) vcpu-0 - AIOGNRC: Failed to open '/vmfs/devices/vsan/xxxx-a669-1287-a5c3-xxxx' : Could not find the file (60003) (0x2011).
 Wa(03) vcpu-0 - DISKLIB-LIB_SIDECAR : DiskLibSidecarOpenInt: Failed to get objID associated with disk: One of the parameters supplied is invalid
 Wa(03) vcpu-0 - DISKLIB-LIB_SIDECAR : DiskLibSidecarOpenInt: Failed to get objID associated with disk: One of the parameters supplied is invalid
 In(05) vcpu-0 - AIOGNRC: Failed to open '/vmfs/devices/vsan/xxxx-f26c-acf2-1739-xxxx' : Could not find the file (60003) (0x2013).
 In(05) vcpu-0 - AIOGNRC: Failed to open '/vmfs/devices/vsan/xxxx-5eb5-ae8e-1834-xxxx' : Could not find the file (60003) (0x2011).
 Wa(03) vcpu-0 - DISKLIB-LIB_SIDECAR : DiskLibSidecarOpenInt: Failed to get objID associated with disk: One of the parameters supplied is invalid
 Wa(03) vcpu-0 - DISKLIB-LIB_SIDECAR : DiskLibSidecarOpenInt: Failed to get objID associated with disk: One of the parameters supplied is invalid
 In(05) vmx - OBJDESC-LIB :  Failed to unlink '/vmfs/volumes/vsan:xxxx-28e7f3613a4c53e6/xxxx-cc25-6945-880c-xxxx/Test-Snapshot1751.vmem': No such file or directory (20004)

Environment

vSphere ESXi 7.0.x

vSphere ESXi 8.0.x

vSAN 8.x

Cause

This issue occurs when a virtual machine file is locked. File locks can be created during normal ESXi operations or when third-party, snapshot-based backup solutions hot-add virtual disks to a backup proxy. In some cases, if a backup operation fails or terminates abruptly, the lock may not be released, which prevents the virtual machine from being powered on, migrated, or accessed.

Resolution

Resolution:

  •  Identify the ESXi host holding the file lock.

Run the following command from an ESXi host to determine which host owns the lock on the virtual machine configuration file (.vmx):

vmfsfilelockinfo -p /link/to/vmx -v <vCenterIP/FQDN> -u [email protected]

  • Release the file lock.

After identifying the host holding the lock, ensure that the virtual machine is not actively running on that host. Once confirmed, Follow the below article to release the lock 

Remove the lock

Notes: 

  • Ensure that no backup jobs or snapshot operations are running before attempting to release a lock.

  • Avoid forcibly removing locks unless it is confirmed that the virtual machine is not powered on on another host.

  • If the lock was created by a third-party backup application, verify that the backup job has completed or stopped successfully.

Additional Information

Additional resources:


Powering off an unresponsive virtual machine on an ESXi host

Virtual machines appear as invalid or orphaned or inaccessible

Powered by Wolken Software