Stale file locks on VMDKs left by 3rd party backup solutions

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• During backup operations, backup software creates a VM snapshot and attaches either the base or previous delta disk to the backup proxy VM for data extraction.
• The problem arises when these disks are not detached from the proxy VM, or the disk handles are not properly closed after the backup completes, leaving a file lock on the original VMDK.
• This lock prevents the source VM from consolidating its snapshots, leading to a buildup of snapshots on storage.

Manifestations:

Scenario 1:

• Backup solution backup proxy VM, or other VM, holds a lock on the base disk, or on a previous snapshot file of the VM that has the consolidation issue
• This prevents snapshot consolidation from succeeding
• Two typical examples:  

1. 1. When a backup job is performed with the HotAdd transport mode feature, the snapshot disk is hot-added into a backup proxy VM. If the snapshot disk is left on the backup proxy VM after a backup failure, it can lock the virtual disk, preventing the source VM from successfully consolidating its snapshots.
   2. When a backup job is performed with NBD transport mode, the disk handles are not closed after backup job, which cause the snapshot disk locked.

Scenario 2:

• Backup solution does not complete a backup gracefully, or fails to clean up properly.  
• A typical example: 

  • The snapshot disk is not closed properly before snapshot delete and consolidation.

Environment

• VMware ESXi 7.x, 8.x, 9.x
• VMware vCenter Server 7.x, 8.x,9.x

Cause

The issue occurs mainly during the incorrect or interrupted workflow of third-party backup solutions.

Resolution

• Backup vendors should review their respective backup workflows to ensure that all VMDKs attached to proxy VMs during backup are reliably detached upon job completion. It is crucial that the backup process prevents any locks from remaining on base or delta disks after the backup.
• The backup solution should ensure every disk is closed after the disk open to free the lock on ESX hosts.
• The backup solution should ensure every disk is closed before deleting the snapshot for backup.
• It is recommended there are no additional snapshot operations (except backup snapshot) during backup .Unexpected issues may be encountered if the backup solution runs backup job for a VM with multiple snapshots.
• If the backup job is somehow killed/crashed, proper cleanup should be implemented to recover by calling the appropriate VixDiskLib_Cleanup, VixDiskLib_EndAccess and other appropriate functions.

Workaround:

If you hit issues with file locks left over by backup solutions, please follow instructions in Investigating Virtual Machine file locks on ESXi Host(s)