Error: "An error occurred while taking a snapshot: A digest operation has failed" when taking snapshot of VM

Products

VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0

Issue/Introduction

When creating a snapshot on a Horizon Virtual Machine, these symptoms occur:

In vSphere client, the error "A digest operation has failed. An error occurred while taking a snapshot"

ESXi - within the VMX log of the virtual machine, /vmfs/volumes/<datastoreUUID>/<folder>/vmware.log, there are entries similar to:

vmx| DIGESTLIB-FILE : DigestLib_FlushMetaDataToDisk: digest header of parent digest has CID 0x3f45376c new CID of child digest is 0x529596f4, parent digest may be corrupt, failing meta-data update on child.
vmx| DISKLIB-VMFS  : "/vmfs/volumes/<datastoreUUID>/<folder>/<VMname>-digest-flat.vmdk" : closed.
vmx| DISKLIB-LIB_CREATE   : DiskLibPostCreateChild: could not init child digest disk: Invalid parameter (8).
vmx| DISKLIB-VMFS  : "/vmfs/volumes/<datastoreUUID>/<folder>/<VMname>-digest-sesparse.vmdk" : closed.
vmx| DISKLIB-VMFS  : "/vmfs/volumes/<datastoreUUID>/<folder>/<VMname>-000001-sesparse.vmdk" : closed.
vmx| DISKLIB-LIB_CREATE   : Encountered error in DiskLibPostCreateChild: A digest operation has failed.
..
vmx| DISKLIB-LIB_CREATE   : Failed to create link: A digest operation has failed (66)
vmx|  Er(02) vcpu-X - DISKLIB-LIB_CREATE   : DiskLib_CreateChild: failed to create child disk: A digest operation has failed (66).
..
vmx| DISKLIB-LIB   : DiskLibUnlinkInt: Disk delete successfully completed { result:0, Msg: 'The operation completed successfully', fileName:'/vmfs/volumes/<datastoreUUID>/<folder>/<VMname>-000001-digest.vmdk'}
vmx| SNAPSHOT: SnapshotBranchDisk: Failed to branch disk: '/vmfs/volumes/<datastoreUUID>/<folder>/<VMname>.vmdk' -> '/vmfs/volumes/<datastoreUUID>/<folder>/<VMname>.vmdk' : A digest operation has failed (66)
vmx| DISKLIB-VMFS  : "/vmfs/volumes/<datastoreUUID>/<folder>/<VMname>-digest-flat.vmdk" : closed.
vmx| DISKLIB-VMFS  : "/vmfs/volumes/<datastoreUUID>/<folder>/<VMname>-flat.vmdk" : closed.
vmx| SNAPSHOT: SnapshotBranch failed: A digest operation has failed (5).

ESXi - in the /var/run/log/vpxa.log, there are entries similar to:

YYYY-MM-DDTHH:MM:SS info vpxa[5899216] [Originator@6876 sub=Default opID=<OP-ID>] [VpxLRO] -- ERROR task-#### -- vpxa -- vpxapi.VpxaService.createSnapshot: vim.fault.GenericVmConfigFault:
--> Result:
--> (vim.fault.GenericVmConfigFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "msg.checkpoint.save.fail2.std3",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "1",
-->                value = "msg.disklib.DIGEST"
-->             }
-->          ],
-->          message = "An error occurred while saving the snapshot: A digest operation has failed."
-->       },
-->       (vmodl.LocalizableMessage) {
-->          key = "msg.checkpoint.save.fail2.std",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "1",
-->                value = "/vmfs/volumes/<datastoreUUID>/<folder>/<VMname>-Snapshot.vmsn"
-->             }
-->          ],
-->          message = "An error occurred while saving snapshot file "/vmfs/volumes/<datastoreUUID>/<folder>/<VMname>-Snapshot.vmsn"."
-->       },
-->       (vmodl.LocalizableMessage) {
-->          key = "msg.snapshot.vigor.take.error",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "1",
-->                value = "msg.disklib.DIGEST"
-->             }
-->          ],
-->          message = "An error occurred while taking a snapshot: A digest operation has failed."
-->       }
-->    ],
-->    reason = "An error occurred while saving the snapshot: A digest operation has failed."
-->    msg = "An error occurred while saving the snapshot: A digest operation has failed."
--> }
--> Args:
-->
--> Arg vmid:
--> 179
--> Arg name:
--> "VM Snapshot"
--> Arg description:
--> ""
--> Arg memory:
--> true
--> Arg quiesce:
--> false

Additional symptoms reported:

"unable to take a snapshot of a Horizons vm due to a disk digest error"

Environment

VMware vSphere ESXi 7.0.3
VMware vSphere ESXi 8.0.x

Cause

This issue occurs when there is corruption or metadata inconsistency in the VM's digest files, which are used by VMware's storage acceleration feature. Specifically:

Digest File Corruption: The digest metadata becomes corrupted, causing a mismatch between the parent digest's Chain ID (CID) and the child digest's CID during snapshot operations. This is evidenced by the VMX log entry: "digest header of parent digest has CID 0x3f45376c new CID of child digest is 0x529596f4, parent digest may be corrupt"
Storage Acceleration Conflict: The presence of digest files (indicated by scsi0:0.digest = "TRUE" in the VMX file and digest references in the VMDK descriptor) interferes with the snapshot branching process when the digest metadata is inconsistent
Snapshot Chain Failure: During snapshot creation, VMware attempts to create child disks and update digest metadata, but fails due to the corrupted digest state, preventing the snapshot operation from completing successfully

This issue is most commonly observed on Horizon persistent clone VMs where storage acceleration is enabled, as these VMs frequently utilize digest files for performance optimization. The corruption can occur due to improper VM shutdowns, storage issues, or metadata inconsistencies during VM operations.

Resolution

To workaround the issue:

Power off the VM completely

Navigate to the VM's folder on the datastore:

cd /vmfs/volumes/<datastore-name>/<vm-folder>/

Create backup copies of critical files:

cp <VMname>.vmdk <VMname>.vmdk.backup
cp <VMname>.vmx <VMname>.vmx.backup

Create a temporary folder within the VM directory:
```
mkdir digest_temp
```
Edit the VMDK descriptor file to remove digest references:
```
vi <VMname>.vmdk
```
- Press i to enter insert mode
- Remove these lines:
```
ddb.digestFilename = "<VMname>-digest.vmdk"ddb.digestType = "file"
```
- Press Esc to exit insert mode
- Type :wq! to save and exit
Edit the VMX configuration file to remove digest setting:
```
vi <VMname>.vmx
```
- Press i to enter insert mode
- Remove this line:
```
scsi0:0.digest = "TRUE"
```
- Press Esc to exit insert mode
- Type :wq! to save and exit

Move digest files to the temporary folder:

mv <VMname>-digest.vmdk digest_temp/
mv <VMname>-digest-flat.vmdk digest_temp/

Power on the VM
Attempt to create a new snapshot to verify the issue is resolved