LWD based VM snapshot in VMware Live Cyber Recovery fails and crashes the VM

Products

• VMware Live Recovery

Issue/Introduction

• LWD based snapshot fails for some VMs and VM crashes
• VC event : LWD snapshot failed alert.
• VC event : VMware ESX unrecoverable error: (worker-2935733) VERIFY bora/esx/apps/dp/iofilter/lwdSes.c:5134
• The bellow error logs are noticed under vmx.log file:

Wa(03) filtPoll - IOFIPC: c3: Error on '152' while waiting for data to read: Connection reset by peer
Er(02) worker-2934497 - LWD: Failed to get a response for IPC client for request C48#####60 for disk C42#####F0; error: Connection reset by peer
Wa(03) filtPoll - IOFIPC: c4: IPC client made no request, or could not write a request, before the timeout expired
Er(02) worker-2934749 - LWD: Failed to get a response for IPC client for request C48#####60 for disk C42#####F0; error: Connection timed out
In(05) vcpu-0 - DISKLIB-LIB_BLOCKTRACK   : Resuming from change tracking info file /vmfs/volumes/07305f41-214ad5c3/TestVM/TestVM-ctk.vmdk.
Wa(03) filtPoll - IOFIPC: c5: Error on '152' while waiting for data to read: Connection reset by peer
Er(02) worker-2934497 - LWD: Failed to get a response for IPC client for request C48#####60 for disk C42#####F0; error: Connection reset by peer
Wa(03) filtPoll - IOFIPC: c6: IPC client made no request, or could not write a request, before the timeout expired
Er(02) worker-2934497 - LWD: Failed to get a response for IPC client for request C48#####60 for disk C42#####F0; error: Connection timed out
Wa(03) vcpu-0 - LWD: Error sending CloseDisk IPC to daemon for disk C42#####F0: Connection timed out
In(05) vcpu-0 - LWD: Closed disk C42#####F0

• You may see critical alerts for the protection groups. 
• Below errors may be seen in da/data/var/log/drcMgr_svc.log

errorMsg: "Failed to backup VM (/mnt/datrium/Ingest/uuid/examplevm.vmx) (examplevm): High-frequency snapshot failed. Internal error: Non-BaseSnapshotMismatch and non-FullSyncRequired fault (DpFaultDpsFault)."
stringVal: "High-frequency snapshot failed. Internal error: Non-BaseSnapshotMismatch and non-FullSyncRequired fault (DpFaultDpsFault)."

Note: This requires root login to the connector appliance.

Environment

VMware Live Cyber Recovery 7.27.x

VMware Live Cyber Recovery 7.26.x

Cause

The IO Filter crashes while SES is half created, subsequent LWD operations fail when the filter attempts to open the old SES.

Resolution

The Issue has fixed on ESXi version 7.0u3q build 23794027 and above versions. Refer ESXI Build Number and Version: Build numbers and versions of VMware ESXi/ESX (316595)

Workaround : 

1) Restart dpd service either from vSphere client or command line. 

UI option :

Activate or deactivate an esxi service

Command line :

/etc/init.d/dpd stop && /etc/init.d/dpd start

2) Manually remove the ses disks from the VM by running below command 

vmkfstools -U examplevm-ses.vmdk