Backup fails causing vm to hang on ESXi 8.x

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

Backup process fails and consolidation doesn't take place, eventually leaving multiple snapshots in the chain.

In dpd.log, You see many logs that cancel snapshot operation returns SyncInProgress fault.

-var/run/log/dpd.log

YYYY-MM-DDTHH:MM:SSZ error dpd[2102314] [Originator@6876 sub=PeRegistry opID=0d52391c-391f] Cannot cancel snapshot 'xxxxxxxxx' of protected entity '31': There is an ongoing sync operation 'XXXXXX-1'

YYYY-MM-DDTHH:MM:SSZ error dpd[2102314] [Originator@6876 sub=Task opID=0d52391c-391f] Task 'Dp.Task.CancelSnapshot.xxxxxxx.xxxxxx' failed with error: (dp.fault.SyncInProgress) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> syncId = "xxxxxxx-1"
--> msg = ""
--> }

 

vmware.log entries show the LWD filter timing out

YYYY-MM-DDTHH:MM:SSZ Wa(03) filtPoll - IOFIPC: c573: IPC client made no request, or could not write a request, before the timeout expired. Connection has 1 open transaction (txn:msg): { 52979:1 }
YYYY-MM-DDTHH:MM:SSZ Er(02) worker-22645658 d11fa3f9 LWD: Failed to get a response for IPC client for request ########## for disk #########; error: Connection timed out
YYYY-MM-DDTHH:MM:SSZ Wa(03) vcpu-0 246eafa2 LWD: Error sending CloseDisk IPC to daemon for disk #########: Connection timed out
YYYY-MM-DDTHH:MM:SSZ In(05) vcpu-0 246eafa2 LWD: Closing the SES disk #####-######-######-####.ses

Environment

VMware vSphere ESXi 8.0

Cause

The issue is in the LWD VAIO filter, it is caused when a delta sync is canceled due to a stunned disk.

The DPD service will lock the VM. Consequently, if a user attempts to perform snapshot management or any other tasks on this VM, the following error will be displayed: 'Another task is already in progress.

Resolution

Issue is resolved in 9.x and will be resolved in a future release of 8.x

Workaround: SSH to the ESXi host and run the command: /etc/init.d/dpd stop && /etc/init.d/dpd start.

After the DPD service is restarted, the lock on the VM will be released, and its status will recover.

Additional Information

ESXi 8.xでバックアップが失敗し VMがハングアップする