Failure in LWD snapshot sync causes Virtual machine to hang

Products

VMware vSphere ESXi

Issue/Introduction

Virtual machine may become unresponsive or hung during backup operations with the multiple LWD snapshot failed tasks: "LWD crash-consistent snapshot failed".
This occurs when a snapshot operation or a 'LWD snapshot sync' task fails.
This issue can be observed in a backup mechanism that utilizes LWD snapshot like: 'vSphere Live Recovery Cloud' and 'Dell PowerProtect'.
Virtual machine becomes unresponsive to power operations or snapshot management. Any tasks performed on the virtual machine there after results in an error message: "Another task is already in progress".
One of the failed tasks for the affected virtual machine may show: msg.hbacommon.outofspace: There is no more space for virtual disk ####.vmdk.
DPD service logs in the host (path: /var/run/log/dpd.log) shows "There is an ongoing sync operation" and "dp.fault.SyncInProgress" locks.

YYYY-MM-DDTHH:MM error dpd[2102314] [Originator@6876 sub=PeRegistry opID=0d523########] Cannot cancel snapshot '########' of protected entity '31': There is an ongoing sync operation ######-1'
YYYY-MM-DDTHH:MM error dpd[2102314] [Originator@6876 sub=Task opID=0d523########] Task 'Dp.Task.CancelSnapshot.#######.#######' failed with error: (dp.fault.SyncInProgress) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> syncId = "######-1"
--> msg = ""
--> }

Virtual machine log (path: /vmfs/volumes/<datastore-name>/<vm-name>/vmware.log) reports LWD filter timeouts:

YYYY-MM-DDTHH:MM Wa(03) filtPoll - IOFIPC: c573: IPC client made no request, or could not write a request, before the timeout expired. Connection has 1 open transaction (txn:msg): {52979:1}
YYYY-MM-DDTHH:MM Er(02) worker-22645658 d11fa3f9 LWD: Failed to get a response for IPC client for request ########## for disk #########; error: Connection timed out
YYYY-MM-DDTHH:MM Wa(03) vcpu-0 246eafa2 LWD: Error sending CloseDisk IPC to daemon for disk #########: Connection timed out
YYYY-MM-DDTHH:MM In(05) vcpu-0 246eafa2 LWD: Closing the SES disk #####-######-######-####.ses

Environment

ESXi 8.x
ESX 9.x

Cause

The issue resides in the Light Weight Delta (LWD) VAIO filter. When a delta sync is canceled due to a stunned disk or backup timeout, the Data Protection Daemon (DPD) service incorrectly retains an active SyncInProgress lock on the virtual machine.

Consequently, if a user attempts to perform snapshot management or any other tasks on this VM, the following error will be displayed: 'Another task is already in progress'.

Resolution

Fixed in ESX 9.0.1 and ESXi 8.0 U3i. Refer Download Broadcom Products and Software to download the latest release.

To download the fixed release:

ESXi 8.0 U3i: Log in to the Broadcom Support Portal to download the patch.

ESX 9.0.1: Log in to the Broadcom Support Portal to download the patch, depending on the entitlement, VMware vSphere Foundation or VMware Cloud Foundation.

Workaround

To release the lock without rebooting the host or virtual machine, perform the following steps:

1. Establish an SSH session to the affected ESXi host.
2. Restart the DPD service by executing: /etc/init.d/dpd stop && /etc/init.d/dpd start
3. Verify the virtual machine's state has recovered and management tasks can be performed.

After the DPD service is restarted, the lock on the virtual machine will be released, and its status should recover.

Additional Information

ESXi 8.xでバックアップが失敗し VMがハングアップする

Release notes to the fixed version:

VMware ESX 9.0.1

VMware ESXi 8.0 Update 3i

This issue is fixed in the PR 3502820: Virtual machine migration and Dell Power Protect Data Manager (PPDM) backups might fail after a Dell Transport Snapshot Data Mover (TSDM) panic
A panic event at Dell TSDM during a PPDM-based snapshot synchronization can stop the task in the data protection daemon due to a miscalculation error. The stopped synchronization subsequently affects PPDM backup operations on the VM and results in failure of vSphere vMotion tasks.