Disk consolidation of a virtual machine on a vSAN datastore fails with an "Unable to access file since it is locked" error.

Products

• VMware vSphere ESXi
• VMware vSAN

Issue/Introduction

When attempting to perform a disk consolidation on a Virtual Machine (VM) residing on a vSAN datastore, the operation fails with errors indicating that the VMDK file is locked. This prevents the consolidation from completing, potentially impacting Virtual Machine performance or preventing further snapshot operations.

Symptoms:
Disk consolidation tasks fail for the affected Virtual Machine, with errors reported in the vSphere Client or vCenter Server tasks, such as:

• Unable to access file since it is locked Consolidation failed for disk node 'scsi0:O': Failed to lock the file. An error occurred while consolidating disks: Failed to lock the file. 


Cause

The disk consolidation failure is directly caused by a persistent, stale lock being held on one or more of the Virtual Machine's virtual disk objects (.vmdk) or their associated lock files. This lock prevents the ESXi host from modifying the disk chain required for consolidation. 

Note: On a vSAN datastore, file locks are validated on the specific virtual disk objects rather than traditional -flat or -delta files as in VMFS, since vSAN is an object-based system.

Resolution

To resolve consolidation issues, identify and release stale locks on the Virtual Machine's virtual disk objects by moving problematic lock files to a temporary location.

1. Identify the affected Virtual Machine and note its name and the corresponding ESXi host.
   
   
2. Enable SSH on ESXi host and connect via SSH client.
   
   
3. Navigate to the Virtual Machine's configuration directory on the vSAN datastore.
   
   

4. Check for Persistent Locks:

   Run the below command to identify any .lck files or associated vSAN object locks:

   for file in .*lck; do echo ${file}; vmfsfilelockinfo -p ${file} |grep -i mode; done

  This command will list any lock files and their respective modes.If an exclusive lock is shown (as in the Symptoms section), it confirms the issue.
   
   Example; ".########-####-####-####-############.lck " is locked in Exclusive mode by host having mac address ['##:##:##:##:##:##']
   Host owning the lock on file is <Hostname>, lockMode : Exclusive
   
          
5. Move Locks: Create a temporary directory and move .lck files:
   
   mkdir /tmp/stale_locks
mv .*lck /tmp/stale_locks/


6. Consolidate and Verify:
   Initiate Consolidation: Right-click Virtual Machine > Snapshots > Consolidate.
   Monitor and Verify: Check consolidation task status and Virtual Machine health.

Additional Information

On a traditional VMFS datastore, file locks are typically associated with .vmdk, -flat.vmdk, or -delta.vmdk files within the VM's directory. However, vSAN operates as an object-based storage system. Virtual disks (vdisks) on vSAN are not stored directly within the Virtual Machine's namespace directory alongside configuration files but as distinct objects. File lock issues for vSAN Virtual Machine's thus refer to locks on these vdisk objects. The vmfsfilelockinfo command provides an abstraction to identify locks on these underlying objects, helping to diagnose consolidation or power-on failures effectively.

Investigating virtual disk file locks on vSAN