Snapshot consolidation failure: Failed to lock the file error
search cancel

Snapshot consolidation failure: Failed to lock the file error

book

Article ID: 374141

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

A virtual machine (VM) displays a "Consolidation needed" warning, and consolidation tasks fail due to file locks held by external processes or stale sessions. In some scenarios, the VM may enter a power-off state and fail to power back on until locks are released

  • vCenter Server displays the alert: Virtual machine disks consolidation is needed.

  • Manual consolidation fails with the error: Unable to access file since it is locked An error occurred while consolidating disks: Failed to lock the file.

  • The VM may power off unexpectedly and remain in a locked state

  • vSAN environments may log: OBJLIB-VSANOBJ: VsanObjLock: Could not lock object

  • The error repeats for every attempts of the consolidation.

  • In the /vmfs/volumes/<datastore_name>/<vm_name>/vmware.log will have the below events logged in: 

YYYY-MM-DDTHH:MM:SS In(05) vmx - SnapshotVMX_Consolidate: Starting online snapshot consolidate operation.
YYYY-MM-DDTHH:MM:SS In(05) vmx - ConsolidateFillSnapDiskTransferArray: Item 0 source: /vmfs/volumes/vsan:<vsan Datastore UUID>/<VSAN OBJECT ID - VM HOME DIR>/Test-VM-000255.vmdk dest: /vmfs/volumes/vsan:<vsan Datastore UUID>/<VSAN OBJECT ID - VM HOME DIR>/Test-VM.vmdk. Cumulative size of redo logs (including meta-data): 6883860480.
YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - [msg.disklib.numLinks.maxReached] This virtual machine has 255 or more redo logs in a single branch of its snapshot tree. The maximum supported limit has been reached, creating new snapshots will not be allowed. To create new snapshots, please delete old snapshots or consolidate the redo logs.
YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - ConsolidateEnd: Snapshot consolidate complete: Failed to lock the file (5).

  • In the /var/run/log/vmkernel.log, below entries can be seen during the consolidation attempt: 

YYYY-MM-DDTHH:MM:SS cpu26:######## opID=74f54281)DLX: 2650: vol '<VSAN OBJECT ID - VM HOME DIR>', lock at 125706240: Lock type: 10C00001. Read Lock(s) held on a file on volume <VSAN OBJECT ID>. numHolders:1 gblNumHolders:0,$
YYYY-MM-DDTHH:MM:SS cpu26:######## opID=74f54281)[type 10c00001 offset 125706240 v 13942, hb offset 3215360
gen 263, mode 2, owner 00000000-00000000-0000-000000000000 mtime 11919512
num 1 gblnum 0 gblgen 0 gblbrk 0] alloc owner 0
YYYY-MM-DDTHH:MM:SS cpu26:######## opID=74f54281)DLX: 2651: vol '<VSAN OBJECT ID - VM HOME DIR>', lock at 125706240: Lock type: 10C00001. owner(s) MAC: ##:##:##:##:##:##:
YYYY-MM-DDTHH:MM:SS cpu26:######## opID=74f54281)[type 10c00001 offset 125706240 v 13942, hb offset 3215360
gen 263, mode 2, owner 00000000-00000000-0000-000000000000 mtime 11919512
num 1 gblnum 0 gblgen 0 gblbrk 0] alloc owner 0
YYYY-MM-DDTHH:MM:SS cpu26:######## opID=74f54281)Fil3: 5033: Lock failed on file: .<VSAN OBJECT ID - VM HOME DIR>.lck on vol '##########' with FD: <FD c288 r4>

Environment

  • ESXi 8.x
  • vCenter 8.x
  • ESX 9.x
  • vCenter 9.x

Cause

One or more snapshot disks in the VM's snapshot chain remain locked by an external process instead of the VM's own .vmx process

This issue occurs when a snapshot-based backup solution fails to release a read-only lock on a VM's snapshot flat file. During the backup process, the proxy VM provisions the snapshot disk in read-only mode, which causes the ESXi host to place a lock on the file.

If the backup job does not release this lock upon completion, snapshot consolidation tasks will fail, as consolidation requires that no process other than the VM's own .vmx process holds a lock on the snapshot files. This leads to the snapshot chain growing continuously until it reaches the maximum limit of 255 snapshots.

Resolution

Determine the locks held on the files by following the below KBs.

Investigating Virtual Machine file locks on ESXi Host(s)

Investigating virtual disk file locks on vSAN

To resolve snapshot consolidation failures caused by stale file locks from backup proxies, follow these steps to identify and release the locked files.

I. Identify the Lock Source

  • Across Multiple Hosts: If your initial investigation indicates that file locks are held by a different host than the one running the impacted VM:

    • Log in to the ESXi host that is not hosting the impacted VM via SSH.
    • Run the lsof command to identify which process is holding the lock.
    • If the lock is held by a vmx process on this secondary host, it confirms that a backup proxy VM (or another VM) has the disk mounted.

  • On the Same Host: If the impacted VM and the backup proxy are residing on the same ESXi host, MAC address filtering will not show a conflict.
    • Run lsof and filter for the specific snapshot file names.
    • Compare the Process IDs (PIDs) holding the locks. Identify any vmx PID that does not belong to the impacted VM.

II. Release the File Lock

  • Identify the Proxy VM: Once the PID holding the lock is identified, determine which VM is associated with that process.
  • De-provision the Disk: If the lock is held by a backup proxy, the disk (VMDK) must be de-provisioned from that proxy VM to release the lock.

IMPORTANT
Before proceeding with de-provisioning:

  • Review the backup server console to ensure there are no active backup jobs currently running for the impacted VM.
  • Always follow your specific backup vendor’s guidelines for manually de-provisioning or "unmounting" disks from a proxy appliance.

III. Workaround: Power Cycle the Proxy

If the lock persists after de-provisioning attempts:

  • A full Power Cycle (Power Off, then Power On) of the backup proxy VM may be required to force the release of the file lock.
  • Note: A standard guest OS reboot may not be sufficient; a complete power-off state is recommended.
  • Warning: Power cycling the proxy should be done at the user's discretion, as it may impact other concurrent backup jobs running on that proxy.
  • Once the power cycle is complete, verify that the VMDK reference has been completely removed from the backup proxy VM configuration (.vmx file).

IV. Consolidate Snapshots

After the lock is released, return to the impacted VM in vCenter and initiate the Consolidate task.

Note: Snapshot consolidation will only succeed when the only process holding a lock on the snapshot files is the vmx process of the impacted VM itself.

Additional Information

Powered by Wolken Software