Snapshot failed with Error - "Error occurred while taking a snapshot: Operation failed"

Products

• VMware vSAN
• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

Symptoms:

• Unable to create manual snapshot on the Virtual Machine.
   

         On the vSphere Client, navigate to Virtual Machine in the inventory > Click the Monitor tab> Under Tasks and Events, select Tasks.

Environment

VMware vSphere 7.x

VMware vSphere 8.x

Cause

Issue is due to the third-party I/O filter module, which is blocking and failing the snapshot task during the prepare phase. Snapshot can fail for a multi-disk VM, if the backup appliance does not have enough resources (e.g. attempting to back up too many VMs/disks concurrently).

As a result, some disks of the affected VM may not be snapshotted. In such a case, third-party I/O filters DR restores/returns to a pre-snapshot state for already snapshotted disks. 

Virtual machine Logs: /vmfs/volumes/[datastore-name]/[vm-name]/vmware.log below errors indicates the issue with the third-party I/O filter module

YYYY-MM-DDTHH:MM:SS.Z Er(02):vmx - DISKLIB-LIB_CREATE :DiskLib_NotifySnapshotPrepare: FiltLib failed to notify snapshot prepare: Operation completes asynchronously
YYYY-MM-DDTHH:MM:SS.Z In(05):vmx - DISKLIB-CBT: ChangeTrackerESX_CreateMirror: Created mirror node /vmfs/devices/<vm/****-cbtmirror.
YYYY-MM-DDTHH:MM:SS.Z In(05):vmx - SNAPSHOT: SnapshotPrepareTakeDoneCB: Failed to prepare filter.
YYYY-MM-DDTHH:MM:SS.Z In(05):vmx - SNAPSHOT: SnapshotPrepareTakeDoneCB: Prepare phase complete (Operation failed,
YYYY-MM-DDTHH:MM:SS.Z In(05):vmx - SnapshotVMXPrepareTakeDoneCB: Prepare phase failed: Operation failed (5).
 
YYYY-MM-DDTHH:MM:SS.Z In(05) vmx - DISKLIB-CBT :ChangeTrackerESX_CreateMirror: Created mirror node /vmfs/devices/svm/3f2c8886-371b1c3b-cbtmirror.
YYYY-MM-DDTHH:MM:SS.Z In(05) vmx -I>FLTR_MAIN: FioCbkSnapshot(1824): VMIOF_SNAPSHOT_PREPARE entered for VM.vmdk, opMode=2.
YYYY-MM-DDTHH:MM:SS.Z Wa(03) vmx -I>FLTR_MAIN: FioCbkSnapshot(1945): Unable to take snapshot of VM_1.vmdk as protection start is not completed yet.
YYYY-MM-DDTHH:MM:SS.Z In(05) vmx - FiltLib: jetdr: Notification completed with error: "Operation failed" (1).
YYYY-MM-DDTHH:MM:SS.Z In(05) vmx -FiltLib: jetdr: filter failed notification processing: "Operation failed" (1).

Resolution

Workaround:

Snapshot may fail due to insufficient resource, so wait for a while before attempting to take the snapshot and then retake the snapshot.

If the above operation fails, It is recommended to involve the third-party I/O filter vendor to further investigate the issue.

Additional Information

I/O filters are software components that can be installed on ESXi hosts and can offer additional data services to virtual machines. The filters process I/O requests which move between the guest operating system of a virtual machine and virtual disks. The I/O filters can be offered by VMware or created by third parties through vSphere APIs for I/O Filtering (VAIO).

Information about one of the third party IO filter called: JetStream ESXi host components (IO Filter packages):

JetStream software installed on each ESXi host configured for JetStream DR. The host driver intercepts the vSphere VMs I/O and sends the replication data to the DRVA. The IO filters also monitor relevant events, such as vMotion, Storage vMotion, snapshots, etc.