VMware vSphere snapshot or third-party backup fails with "Could not create mirror of change tracking file" due to ESXi SVM handle exhaustion

Products

• VMware vSphere ESXi

Issue/Introduction

Third-party backup software (such as Veeam Backup & Replication) fails during the virtual machine backup job.

Manual or automated creation of a VM snapshot (without memory) fails.

In the vmware.log file of the affected virtual machine, can see errors similar to:

YYYY-MM-DDTHH:MM:SS| vmx| opID=#######-##-####| W003: DISKLIB-CBT :ChangeTrackerESX_CreateMirror: Cannot create mirror node /vmfs/devices/svm/########-########-cbtmirror. 
YYYY-MM-DDTHH:MM:SS| vmx| opID=#######-##-####| W003: DISKLIB-CTK :ChangeTracker_Mirror: Failed to create mirror node. 
YYYY-MM-DDTHH:MM:SS| vmx| opID=#######-##-####| W003: DISKLIB-LIB_BLOCKTRACK : DiskLibBlockTrackPrepareChild: Failed to mirror tracking info file /vmfs/volumes/<Datastore_UUID>/<VM Name> /<VM Name>-ctk.vmdk: Could not create mirror of change tracking file (5948). 
YYYY-MM-DDTHH:MM:SS| vmx| opID=#######-##-####| I005: SNAPSHOT: SnapshotPrepareTakeDoneCB: Failed to prepare block track. 
YYYY-MM-DDTHH:MM:SS| vmx| opID=#######-##-####| I005: SNAPSHOT: SnapshotPrepareTakeDoneCB: Prepare phase complete (Could not create mirror of change tracking file). 
YYYY-MM-DDTHH:MM:SS| vmx| opID=#######-##-####| I005: SnapshotVMXPrepareTakeDoneCB: Prepare phase failed: Could not create mirror of change tracking file (5). 
YYYY-MM-DDTHH:MM:SS| vmx| opID=#######-##-####| I005: SnapshotVMXTakeSnapshotComplete: Done with snapshot 'VEEAM BACKUP TEMPORARY SNAPSHOT': 0

In the vmkernel.log file of the ESXi host hosting the VM, can locate the corresponding cbtmirror ID with a No free handles error:

YYYY-MM-DDTHH:MM:SS cpu32:######## opID=########)SVM: 5187: SVM_MakeDev.5187: Creating device ########-########-cbtmirror: No free handles

Environment

VMware ESXi 7.0

Cause

This issue occurs when the host-level device handles (minor numbers) managed by the svmmirror kernel driver (SVM) in the ESXi kernel become exhausted.

When Changed Block Tracking (CBT) is enabled, ESXi attempts to create a virtual device node (cbtmirror) in memory (/vmfs/devices/svm/) for each disk during snapshot preparation. Handle exhaustion generally occurs due to:

1. High Concurrency / Disk Stacking: Machines with a high count of VMDKs or heavy concurrent backup jobs demanding a large number of SVM handles simultaneously.

2. Kernel Handle Leaks: Stale cbtmirror handles fail to be properly reclaimed by the SVM module due to unexpected backup job terminations or kernel race conditions, eventually exhausting the handle pool over time.

When the kernel rejects the device node creation with No free handles, ESXi triggers a snapshot prepare rollback, causing the temporary snapshot and backup task to fail.

Resolution

Workaround 1: Migrate the Virtual Machine (Live Migration)

Migrate the affected VM to another ESXi host in the cluster using vMotion. This offloads the VM to a host with an unexhausted SVM handle pool, allowing immediate snapshot and backup completion.

Workaround 2: Restart Management Agents

Run the following command in the ESXi Command Line Interface (CLI) of the affected host:

/sbin/services.sh restart

CAUTION: If your environment relies on VMware vSAN, NSX, or specific storage clusters, restarting host services may impact running network or storage daemons. Consult Broadcom Support before running /sbin/services.sh restart in hyperconverged or NSX environments.

Resolution: Reboot the Affected ESXi Host

Rebooting the affected ESXi host flushes the vmkernel memory space, clears orphaned SVM devices, and fully resets the device handle pool.