VM hangs during backup sync failure on ESXi 8.x

Products

VMware vSphere ESXi 8.0

Issue/Introduction

Virtual machines running on ESXi 8.x may become unresponsive or "hung" during backup operations. This occurs when a snapshot or sync task fails, leading to management operations being blocked.

Symptoms:

Backup process fails and consolidation does not occur.
VM becomes unresponsive to power operations or snapshot management.
Error message: Another task is already in progress.
Error message on the task section of the VM can show: msg.hbacommon.outofspace: There is no more space for virtual disk ####.vmdk.
In /var/run/log/dpd.log, you see:

YYYY-MM-DDTHH:MM:SSZ error dpd[2102314] [Originator@6876 sub=PeRegistry opID=0d523########] Cannot cancel snapshot '########' of protected entity '31': There is an ongoing sync operation ######-1'
YYYY-MM-DDTHH:MM:SSZ error dpd[2102314] [Originator@6876 sub=Task opID=0d523########] Task 'Dp.Task.CancelSnapshot.#######.#######' failed with error: (dp.fault.SyncInProgress) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> syncId = "######-1"
--> msg = ""
--> }
In vmware.log, you see LWD filter timeouts:

YYYY-MM-DDTHH:MM:SSZ Wa(03) filtPoll - IOFIPC: c573: IPC client made no request, or could not write a request, before the timeout expired. Connection has 1 open transaction (txn:msg): { 52979:1 }
YYYY-MM-DDTHH:MM:SSZ Er(02) worker-22645658 d11fa3f9 LWD: Failed to get a response for IPC client for request ########## for disk #########; error: Connection timed out
YYYY-MM-DDTHH:MM:SSZ Wa(03) vcpu-0 246eafa2 LWD: Error sending CloseDisk IPC to daemon for disk #########: Connection timed out
YYYY-MM-DDTHH:MM:SSZ In(05) vcpu-0 246eafa2 LWD: Closing the SES disk #####-######-######-####.ses

Environment

VMware ESXi 8.0.x
VMware vSphere 8.0 Update 3

Cause

The issue resides in the Light Weight Delta (LWD) VAIO filter. When a delta sync is canceled due to a stunned disk or backup timeout, the Data Protection Daemon (DPD) service incorrectly retains an active SyncInProgress lock on the virtual machine.

Consequently, if a user attempts to perform snapshot management or any other tasks on this VM, the following error will be displayed: 'Another task is already in progress'.

Resolution

Fixed in ESXi 9.0 and higher and will be resolved in a future release of 8.x. See Download Broadcom Products and Software to download the 9.x release.

For defects and enhancements, you may subscribe to this article (reference: https://knowledge.broadcom.com/external/article/275360) to be updated on fix status for ESXi 8.x.

Workaround

To release the lock without rebooting the host or VM, perform the following steps:

Establish an SSH session to the affected ESXi host.
Restart the DPD service by executing: /etc/init.d/dpd stop && /etc/init.d/dpd start
Verify the VM state has recovered and management tasks can be performed.

After the DPD service is restarted, the lock on the VM will be released, and its status will recover.

Additional Information

ESXi 8.xでバックアップが失敗し VMがハングアップする