Snapshot consolidation fails due to locks held by third party backup software
search cancel

Snapshot consolidation fails due to locks held by third party backup software

book

Article ID: 321365

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0

Issue/Introduction

This article covers unwanted file locks caused by backup applications with a proxy VM.

Errors that may be seen:

  • Snapshots cannot be committed.
  • Snapshots commit without errors and Snapshot Manager is no longer populated. However, a snapshot disk remains in the VM directory, and the VM still runs on that snapshot.
  • Consolidation failed with the errors similar to:

          Failed to lock the file

          One or more disks are busy

        Unable to consolidate virtual machine snapshots due to file lock

          Unable to access file since it is locked

  • The VM summary tab displays messages similar to:

          Snapshot consolidation required

 Virtual machine disks consolidation is needed

 Virtual machine Consolidation Needed status

Note: For additional symptoms and log entries, see the Related Information section.

Environment

  • VMware vSphere ESXi
  • VMware vCenter Server
  • Third-party backup solutions (e.g., Veeam, CommVault, Cohesity)

Cause

  • A third-party vendor's backup solution backup proxy VM, or other VM, holds a lock on the base disk, or on a previous snapshot file of the VM that has the consolidation issue. This prevents snapshot consolidation from succeeding.
  • A third-party vendor's backup solution does not complete a backup gracefully or fails to clean up properly.
  • A third-party vendor's backup solution does not follow VADP documented workflow. A typical example: the snapshot disk is not closed properly before snapshot delete and consolidation.

Resolution

Before proceeding with transport-specific resolutions, perform the following general steps:

  1. Upgrade to latest version of the backup software. For example, it's recommended to upgrade to CommVault v11.36 or later, Cohesity v7.3 or later.
  2. Investigate locks to locate the third-party backup solution's proxy VM with the disk attached. Refer to KB Investigating virtual machine file locks on ESXi hosts for additional details

Chose the resolution path below that matches your backup transport mode or underlying issue.

Issue: Locks caused by VDDK HotAdd transport Mode.

To resolve consolidation issues caused by HotAdd transport mode, perform the following steps:

Caution: Make sure there is no backup job running on the VM that has the consolidation issue.

  1. Identify the affected backup proxy VM.
  2. Right-click the backup proxy VM and click Edit Settings.
  3. Expand the Hard Disk list.
  4. Select the Hard Disk belonging to the VM experiencing the consolidation failure.
  5. Click the X icon to unmount the disk.
    Caution: Do NOT select "Delete files from the datastore."
  6. Click OK.
  7. Trigger snapshot consolidation on the affected VM via vSphere Client.

Special note for vSAN environments:

If the VM resides on vSAN, move the lock files to a temporary subfolder before attempting consolidation

  1. SSH into the ESXi host via root
  2. cd into the VM namespace/folder on the datastore
  3. Create a subfolder to hold the lock files 
    mkdir tmp
  4. mv the lock files to this folder with
    mv .*lck tmp
  5. Proceed with the consolidation

Issue: Locks caused by VDDK NBD, NBDSSL, or SAN transport mode.

To resolve this issue caused by backup with VDDK NBD/NBDSSL/SAN transport mode, terminate the hung backup process in the backup proxy.

Caution: Make sure there is no backup job running on the VM that has the consolidation issue.

  • Terminate the specific backup process running inside the backup proxy. Engage the 3rd party backup vendor for assistance in identifying the backup process or service.
  • Alternatively, ensure no backup jobs are running on the backup proxy, then reboot the backup proxy VM to clear any locks.

Issue: Locks caused by improper 3rd-party backup workflows

  • The backup software should ensure every disk is closed to free the lock on ESX hosts after the disk open.
  • If the backup process is somehow killed, proper cleanup should be implemented to recover by calling VDDK functions VixDiskLib_Cleanup, VixDiskLib_EndAccess, etc. 
  • Engage the backup vendor to ensure these mechanisms are properly implemented.

Additional Information

You might experience these additional symptoms:

  • Datastore storage consumption rate is high
  • In the VM's vmware.log file located at /vmfs/volumes/<datastore>/<vmNameDirectory>/vmware.log you see errors similar to:
    vmx| ConsolidateOnlineCB: nextState = 2 uid 3
vmx| Foundry operation failed with system error: Device or resource busy (16), translated to 5
vmx| ConsolidateOnlineCB: Done with consolidate
  • When you attempt to remove the datastore, you see this error:
    The resource '<VMFS-UUID>' is in use.
  • If you are running third party back up software, consolidation might fail with the following errors in vmware.log file:
    vcpu-0| Vix: [8803 mainDispatch.c:4084]: VMAutomation_ReportPowerOpFinished: statevar=3, newAppState=1881, success=1 additionalError=0
vcpu-0| Vix: [8803 vigorCommands.c:577]: VigorSnapshotManagerConsolidateCallback: snapshotErr = Failed to lock the file (5:4008)
vcpu-0| SnapshotVMXConsolidateOnlineCB: Destroying thread 6
vcpu-0| Turning off snapshot info cache.
vcpu-0| Turning off snapshot disk cache.
vcpu-0| SnapshotVMXConsolidateOnlineCB: Done with consolidate
  • In the vmkernel.log file, located on the ESXi at /var/log/vmkernel.log you see entries similar to:
    vmkernel: gen 2141, mode 1, owner 4b94bb81-########-3bd1-########### mtime 244622]on volume '<DATASTORE_NAME>'.
vmkernel: [YYYY-MM-DDTHH:MM:SS] cpu2:4109)FS3: 2890: [Requested mode: 1] Lock [type 10c00001 offset 7505920 v 920, hb offset 3510272
vmkernel: gen 2141, mode 1, owner 4b94bb81-########-3bd1-###########mtime 244622] is not free on volume '<DATASTORE_NAME>'
vmkernel: [YYYY-MM-DDTHH:MM:SS] cpu2:4111)FS3: 2798: [Requested mode: 1] Checking liveness of lock holders [type 10c00001 offset 7313408 v 796, hb offset 3510272
  • In the hostd.log file, located on the ESXi at /var/log/hostd.log you see entries similar to these during the snapshot delete process:
    DISKLIB-LIB : Failed to delete disk '/vmfs/volumes/4c5f4b7a-########-32ad-#########/<VMNAME>/<VMNAME>-000001.vmdk' or one of its components: Device or resource busy
  • When you attempt to consolidate by right clicking the virtual machine and clicking Snapshot > Consolidate, you see errors similar to:
    Consolidate virtual machine disk files <hostname> Unable to access file <unspecified filename> since it is locked
Consolidation failed for disk node 'scsi0:8': msg.fileio.lock.
Powered by Wolken Software