Snapshot consolidation fails due to locks held by third party backup software - VMware vSphere
search cancel

Snapshot consolidation fails due to locks held by third party backup software - VMware vSphere

book

Article ID: 321365

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

Virtual machine snapshot consolidation may fail in environments utilizing third-party backup solutions (such as Veeam, CommVault, or Cohesity). The snapshot consolidation task fails in the vSphere interface with one of the following error messages:

  • Snapshots cannot be committed.

  • Snapshots commit without errors and Snapshot Manager is no longer populated. However, a snapshot disk remains in the VM directory, and the VM still runs on that snapshot.

  • Consolidation failed with the errors similar to:
    • "Failed to lock the file"
    • "One or more disks are busy"
    • "Unable to consolidate virtual machine snapshots due to file lock"
    • "Unable to access file since it is locked"
  • The VM summary tab displays messages similar to:
    • "Snapshot consolidation required"
    • "Virtual machine disks consolidation is needed"
    • "Virtual machine Consolidation Needed status"

Note: For additional symptoms and log entries, see the Related Information section.

Environment

  • VMware vSphere ESXi 7.x to 9.x 
  • VMware vCenter Server 7.x to 9.x 
  • Third-party backup solutions (e.g., Veeam, Commvault, Cohesity)

Cause

  • A Third-party backup solution's proxy VM (or another VM) holds a stale lock on the base disk or a previous snapshot file. This lock prevents the ESXi host from obtaining the exclusive file access required to merge snapshot delta files back into the base virtual disk. This typically occurs when a backup application fails to cleanly release the disk handle after a backup operation (e.g., failed to follow documented VADP workflows).

Resolution

Note: Before proceeding, ensure no backup jobs are currently running on the affected VM.

  1. Upgrade to latest version of the backup software. 
  2. Investigate locks to locate the third-party backup solution's proxy VM with the disk attached. Refer to Broadcom KB Investigating virtual machine file locks on ESXi hosts

Chose the resolution path below that matches your backup transport mode or underlying issue.

Chose the resolution path below that matches your backup transport mode or underlying issue.


Issue: Locks caused by VDDK HotAdd transport Mode.

To resolve consolidation issues caused by HotAdd transport mode, perform the following steps:
Caution: Make sure there is no backup job running on the VM that has the consolidation issue.

  1. Identify the affected backup proxy VM.
  2. Right-click the backup proxy VM and click Edit Settings.
  3. Expand the Hard Disk list.
  4. Select the Hard Disk belonging to the VM experiencing the consolidation failure.
  5. Click the X icon to unmount the disk.
    Caution: Do NOT select "Delete files from the datastore."
  6. Click OK.
  7. Trigger snapshot consolidation on the affected VM via vSphere Client.

Special note for vSAN environments: If the VM resides on vSAN, move the lock files to a temporary subfolder before attempting consolidation

  1. SSH into the ESXi host via root
  2. cd into the VM namespace/folder on the datastore
  3. Create a subfolder to hold the lock files 
    mkdir tmp
  4. move the lock files to this folder with
    mv .*lck tmp
  5. Proceed with the consolidation

Issue: Locks caused by VDDK NBD, NBDSSL, or SAN transport mode.

To resolve this issue caused by backup with VDDK NBD/NBDSSL/SAN transport mode, terminate the hung backup process in the backup proxy.
Caution: Make sure there is no backup job running on the VM that has the consolidation issue.

  1. Terminate the specific backup process running inside the backup proxy. Engage the 3rd party backup vendor for assistance in identifying the backup process or service.
  2. Alternatively, ensure no backup jobs are running on the backup proxy, then reboot the backup proxy VM to clear any locks.

Issue: Locks caused by improper 3rd-party backup workflows

  1. The backup software should ensure every disk is closed to free the lock on ESX hosts after the disk open.
  2. If the backup process is somehow killed, proper cleanup should be implemented to recover by calling VDDK functions VixDiskLib_Cleanup, VixDiskLib_EndAccess, etc. 
  3. Engage the backup vendor to ensure these mechanisms are properly implemented.

Additional Information

You might experience these additional symptoms:

  • Datastore storage consumption rate is high

    • In the VM's vmware.log file located at /vmfs/volumes/<datastore>/<vmNameDirectory>/vmware.log you see errors similar to:
      vmx| ConsolidateOnlineCB: nextState = 2 uid 3
      vmx| Foundry operation failed with system error: Device or resource busy (16), translated to 5
      vmx| ConsolidateOnlineCB: Done with consolidate

  • When you attempt to remove the datastore, you see this error:

    • The resource '<VMFS-UUID>' is in use.

  • If you are running third party back up software, consolidation might fail with the following errors in /vmfs/volumes/<datastore>/<vmNameDirectory>/vmware.log file:

    • vcpu-0| Vix: [8803 mainDispatch.c:4084]: VMAutomation_ReportPowerOpFinished: statevar=3, newAppState=1881, success=1 additionalError=0
      vcpu-0| Vix: [8803 vigorCommands.c:577]: VigorSnapshotManagerConsolidateCallback: snapshotErr = Failed to lock the file (5:4008)
      vcpu-0| SnapshotVMXConsolidateOnlineCB: Destroying thread 6
      vcpu-0| Turning off snapshot info cache.
      vcpu-0| Turning off snapshot disk cache.
      vcpu-0| SnapshotVMXConsolidateOnlineCB: Done with consolidate

  • In the ESXi /var/run/log/vmkernel.log you see entries similar to:

    • vmkernel: gen 2141, mode 1, owner 4b94bb81-########-3bd1-########### mtime 244622]on volume '<DATASTORE_NAME>'.
      vmkernel: [YYYY-MM-DDTHH:MM:SS] cpu2:4109)FS3: 2890: [Requested mode: 1] Lock [type 10c00001 offset 7505920 v 920, hb offset 3510272
      vmkernel: gen 2141, mode 1, owner 4b94bb81-########-3bd1-###########mtime 244622] is not free on volume '<DATASTORE_NAME>'
      vmkernel: [YYYY-MM-DDTHH:MM:SS] cpu2:4111)FS3: 2798: [Requested mode: 1] Checking liveness of lock holders [type 10c00001 offset 7313408 v 796, hb offset 3510272

  • In the ESXi /var/run/log/hostd.log file, you see entries similar to these during the snapshot delete process:

    • DISKLIB-LIB : Failed to delete disk '/vmfs/volumes/4c5f4b7a-########-32ad-#########/<VMNAME>/<VMNAME>-000001.vmdk' or one of its components: Device or resource busy

  • When you attempt to consolidate by right clicking the virtual machine and clicking Snapshot > Consolidate, you see errors similar to:

    • Consolidate virtual machine disk files <hostname> Unable to access file <unspecified filename> since it is locked
      Consolidation failed for disk node 'scsi0:8': msg.fileio.lock.
Powered by Wolken Software