Orphaned VM snapshots on vSAN datastore consuming space

Products

VMware vSAN

Issue/Introduction

Virtual machine on vSAN datastore is running on a snapshot disk (VM-Name-000010.vmdk), but several orphaned snapshots for the same vmdk exists.

Consolidating the VM does not remove these orphaned snapshots.

To list the current snapshot the VM is running:
/vmfs/volumes/<vsan:Datastore>/<VM-Folder>] less *.vmx | grep vmdk
scsi0:0.fileName = "VM-name-000010.vmdk"

To list all the snapshots for this disk:
/vmfs/volumes/<vsan:Datastore>/<VM-Folder>] ls -lha | grep vmdk
VM-Name-000010.vmdk
VM-Name-000009.vmdk
VM-Name-000008.vmdk
VM-Name-000007.vmdk
VM-Name-000006.vmdk
VM-Name-000005.vmdk
VM-Name-000004.vmdk
VM-Name-000003.vmdk
VM-Name-000002.vmdk
VM-Name-000001.vmdk
VM-Name.vmdk

Environment

vSAN 8.x
vSAN 7.x
Commvault Backup

Cause

Consolidation failure events can lead to an orphaned snapshot. The cause for the failures need to be identified and fixed if it is a recurring issue.

This is an example of a consolidation failure, where we notice the parent file was removed successfully before the consolidation was marked failed.

2025-02-08T02:29:16.431Z In(05) vcpu-0 - ConsolidateScheduleNext: Combine '/vmfs/volumes/<vsan:datastore>/<VM-Folder>/VM-Name-000010.vmdk' -> '/vmfs/volumes/<vsan:datastore>/<VM-Folder>/VM-Name-V-000009.vmdk' (node = scsi0:0): 2 links starting from 0. usesCurrent = 1, isNative = 1
2025-02-08T02:29:16.431Z In(05) vcpu-0 - Closing disk 'scsi0:0'
2025-02-08T02:29:16.434Z In(05) vcpu-0 - DISKLIB-CBT : Shutting down change tracking for untracked fid 467212521.
2025-02-08T02:29:16.434Z In(05) vcpu-0 - DISKLIB-CBT : Successfully disconnected CBT node.
2025-02-08T02:29:16.457Z In(05) vcpu-0 - DISKLIB-VMFS : "<vsan:datastore>/<vSAN-Disk-UUID>" : closed.
2025-02-08T02:29:16.457Z In(05) vcpu-0 - SNAPSHOT: SnapshotFixDisksAfterCombineNative: remove parent from '/vmfs/volumes/<vsan:datastore>/<VM-Folder>/-000010.vmdk' <<<================== Parent file was removed
2025-02-08T02:29:16.457Z In(05) vcpu-0 - DISKLIB-VMFS : "<vsan:datastore>/<vSAN-Disk-UUID>" : open successful (1114113) size = 0, hd = 0. Type 3
2025-02-08T02:29:16.457Z In(05) vcpu-0 - DISKLIB-VMFS : "<vsan:datastore>/<vSAN-Disk-UUID>::536" : open successful (1114113) size = 0, hd = 0. Type 3
2025-02-08T02:29:16.458Z In(05) vcpu-0 - DISKLIB-VMFS : "<vsan:datastore>/<vSAN-Disk-UUID>::536" : closed.
2025-02-08T02:29:16.458Z In(05) vcpu-0 - DISKLIB-LIB_CHAINMODIFY : DiskLibGetEstimateDeltaBytes: Estimate of delta bytes 107076896384
2025-02-08T02:29:16.672Z In(05) vcpu-0 - DISKLIB-VMFS : "<vsan:datastore>/<vSAN-Disk-UUID>" : closed.
2025-02-08T02:29:16.674Z In(05) vcpu-0 - DISKLIB-VMFS : "<vsan:datastore>/<vSAN-Disk-UUID>::536" : open successful (1114113) size = 0, hd = 0. Type 3
2025-02-08T02:29:16.674Z In(05) vcpu-0 - DISKLIB-LIB_BLOCKTRACK : Resuming from change tracking info file /vmfs/volumes/<vsan:datastore>/<VM-Folder>/VM-Name-000009-ctk.vmdk.
2025-02-08T02:29:16.682Z In(05) vcpu-0 - DISKLIB-CTK : Unlinking change tracking info /vmfs/volumes/<vsan:datastore>/<VM-Folder>/VM-Name-000009-ctk.vmdk.
2025-02-08T02:29:16.792Z In(05) vcpu-0 - DISKLIB-VMFS : "vsan:datastore>/<vSAN-Disk-UUID>:536" : closed.
2025-02-08T02:29:16.792Z In(05) vcpu-0 - OBJLIB-VSANOBJ: VsanObjUnlinkInt: Unlinking '/vmfs/volumes/<vsan:datastore>/<VM-Folder>/' -- '<vsan:datastore>/<vSAN-Disk-UUID>::536' (flag: 256).
2025-02-08T02:29:20.801Z Er(02) vcpu-0 - OBJLIB-VSANOBJ: VsanObjLock: Could not lock object '<vsan:datastore>/<vSAN-Disk-UUID>::536': Failed to lock the file (4).
2025-02-08T02:29:20.801Z Er(02) vcpu-0 - OBJLIB-VSANOBJ: VsanObjUnlinkInt: Could not lock object <vsan:datastore>/<vSAN-Disk-UUID>::536.
2025-02-08T02:29:20.801Z In(05) vcpu-0 - DISKLIB-LIB : Cannot remove extent '<vsan:datastore>/<vSAN-Disk-UUID>::536': Failed to lock the file <<<================== Remove snapshot task fails after failing to lock the snapshot file
2025-02-08T02:29:20.801Z Er(02) vcpu-0 - DISKLIB-LIB : DiskLibUnlinkInt: Disk delete failed to complete { result:16392, errMsg:'Failed to lock the file', fileName:'/vmfs/volumes/<vsan:datastore>/<VM-Folder>/VM-Name-000009.vmdk'}
2025-02-08T02:29:20.801Z In(05) vcpu-0 - SNAPSHOT: SnapshotFixDisksAfterCombineNative /vmfs/volumes/<vsan:datastore>/<VM-Folder>/VM-Name-000009.vmdk unlink fails with Failed to lock the file (16392)
2025-02-08T02:29:20.801Z In(05) vcpu-0 - SNAPSHOT: Snapshot_ConsolidateWorkItemDone failed: Failed to lock the file (5)
2025-02-08T02:29:20.801Z In(05) vcpu-0 - ConsolidateItemComplete: Online consolidation failed for disk node 'scsi0:0': Failed to lock the file (5).

In this particular case, the cause was identified to be a 'auto mount' feature which was enabled with in the backup application server. With the feature disabled, there were no more failures.

Resolution

To resolve the issue clone the VM.
Cloning consolidates all the snapshots into parent disk on the cloned VM.

If cloning cannot be done for any reason, please contact Broadcom support.