VM registration or power-on hangs in vCenter due to orphaned locks after ESXi boot drive failure

Products

• VMware vSAN

Issue/Introduction

When attempting to power on or register a virtual machine via vCenter Server, the task may hang indefinitely at "Registering Virtual Machine on Host" or "Powering on Virtual Machine." This typically occurs when the virtual machine's original host has experienced a critical management failure. Symptoms include:

• vCenter Server UI displays hung tasks for VM registration or power-on.

• Virtual machines appear as "Orphaned" or "Inaccessible" in the inventory.

• Attempts to manually register the .vmx file via the vSphere Client on a different host fail or hang.

• The source ESXi host is unresponsive or shows "Not Responding" in vCenter due to hardware failure (e.g., boot drive/SD card failure).

Environment

• VMware vSphere ESXi 7.x / 8.x

• VMware vCenter Server 7.x / 8.x

• VMware vSAN (OSA/ESA) or Shared Storage

Cause

This issue is caused by a hardware-level failure of the source ESXi host's boot media. When the boot drive fails, the host's management agents (hostd, vpxa) hang. Because the host cannot gracefully shut down or communicate with vCenter, it fails to release the mandatory storage locks (.lck files) on the virtual machine’s home directory.

vCenter Server remains in a loop attempting to coordinate the task with the failed host, while the stale locks prevent other healthy hosts from mounting the VM files.

Resolution

To resolve this issue, you must manually clear the stale locks and bypass the vCenter task queue by registering the VM directly on a healthy host.

1. Identify and Right-click the affected VM in the vCenter inventory and select Remove from Inventory. This clears the hung task in vCenter.

2. Clear Stale Locks:

   • Log into a healthy ESXi host via SSH or use the Browse Datastore function.

   • Navigate to the VM's home directory.

   • Locate any files ending in .lck.

     • These can be hidden files and can be seen by running ls -lah

   • Move these files to a backup folder to release the file lock (this folder can be removed later) 

     • create the folder using the command mkdir <folder name> 
     • Move the .lck files using the command mv <file name.lck> ./<folder name>

3. Manual Register VM via CLI:

   • Connect via SSH to a healthy destination host.

   • Run the following command to register the VM directly to that host:

     vim-cmd solo/registervm /vmfs/volumes/DatastoreName/VMFolderName/VMName.vmx

4. Power On VM via CLI:

   • Retrieve the new VM ID: vim-cmd vmsvc/getallvms | grep <VMNAME>

   • Power on the VM: vim-cmd vmsvc/power.on <VM_ID>

5. Once the VM is powered on, it will automatically reappear in the vCenter Server inventory.
   
   

   Using vim-cmd vmsvc commands are a critical step in this scenario because it allows the host to claim ownership of the VM files without waiting for vCenter to resolve the "hung" state of the failed host.

Additional Information

Related References:

• Investigating virtual machine file locks (314365)