Backup fails causing vm to hang on ESXi 8.x

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

Backup process fails and consolidation doesn't take place, eventually leaving multiple snapshots in the chain.

In dpd.log, You see many logs that cancel snapshot operation returns SyncInProgress fault.

-var/run/log/dpd.log

YYYY-MM-DDTHH:MM:SSZ error dpd[2102314] [Originator@6876 sub=PeRegistry opID=0d52391c-391f] Cannot cancel snapshot 'xxxxxxxxx' of protected entity '31': There is an ongoing sync operation 'XXXXXX-1'

YYYY-MM-DDTHH:MM:SSZ error dpd[2102314] [Originator@6876 sub=Task opID=0d52391c-391f] Task 'Dp.Task.CancelSnapshot.xxxxxxx.xxxxxx' failed with error: (dp.fault.SyncInProgress) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> syncId = "xxxxxxx-1"
--> msg = ""
--> }

 

vmware.log entries show the LWD filter timing out

YYYY-MM-DDTHH:MM:SSZ Wa(03) filtPoll - IOFIPC: c573: IPC client made no request, or could not write a request, before the timeout expired. Connection has 1 open transaction (txn:msg): { 52979:1 }
YYYY-MM-DDTHH:MM:SSZ Er(02) worker-22645658 d11fa3f9 LWD: Failed to get a response for IPC client for request F39EB3F430 for disk #########; error: Connection timed out
YYYY-MM-DDTHH:MM:SSZ Wa(03) vcpu-0 246eafa2 LWD: Error sending CloseDisk IPC to daemon for disk #########: Connection timed out
YYYY-MM-DDTHH:MM:SSZ In(05) vcpu-0 246eafa2 LWD: Closing the SES disk #####-######-######-####.ses

Environment

VMware vSphere ESXi 8.0

Cause

The issue is in the LWD VAIO filter, it is caused when a delta sync is canceled due to a stunned disk.

Resolution

Issue is resolved in 9.x and will be resolved in a future release of 8.x