High-Frequency snapshot fails with error "DPS deltaSync failed" or "errorCodes: 6900" in VMware Live Cyber Recovery

Products

VMware Live Recovery

Issue/Introduction

Following error is seen in VLCR UI

Severity Warning Timestamp pm11h ago Target VMCAWS Linux - VMDR2 Description Failed to backup VM 'VM NAME' (/VM NAME/VM NAME.vmx) during stage: finishing backup, for task: drc-UUID. High-frequency snapshot failed. Internal error: DPS deltaSync failed (Sync call failed on DpFaultSnapshotTransportFault).

/da/data/var/log/drcMgr-svc.log

[.drc-backup-restore-threads-5] DrcLoadBalancer: Selected DRC Agent service UUID-domain-UUID (0 running tasks) for new DRC task .

 [.drc-backup-restore-threads-5] DrcLoadBalancer: adding drcMgr task drc-mgr-UUID to DRCAgent service UUID-domain-UUID with 0 active tasks.

 [.drc-backup-restore-threads-5] [BKUP ] drcAgentDvxId UUID from drcAgentSvcId UUID-domain-UUID, siteId UUID, clusterMoId domain-cxxxx

[.drc-backup-restore-threads-5] [BKUP ] Forwarding request to UUID-domain-UUID

[.drc-backup-restore-threads-5] [BKUP ] Task on remote agent on siteId UUID: drc-agent-UUID ......

[.drc-backup-restore-threads-5] [BKUP ] Task drc-agent-UUID status is RUNNING (progress 0.0)

[.drc-backup-restore-threads-5] [BKUP ] Task drc-agent-UUID status is ERROR (progress 0.0)

 [.drc-backup-restore-threads-5] DrcLoadBalancer: removing drcMgr task drc-mgr-UUID from DRCAgent service UUID-domain-UUID with 1 active tasks. @cee{"timestamp":"TIMESTAMP", "priority":"ERROR", "threadName":"drc-backup-restore-threads-5", "codeLocation":"DRCMgrSvc.java:434", "msg":"

[BKUP ] Task drc-agent-UUID failed: errorCodes: 6900 @cee{"timestamp":"TIMESTAMP", "priority":"ERROR", "threadName":"drc-backup-restore-threads-5", "codeLocation":"DRCMgrSvc.java:628", "msg":"

[BKUP ] Failed during backup: errorCodes: 6900 @cee{"timestamp":"TIMESTAMP", "priority":"INFO", "threadName":"drc-backup-restore-threads-5", "codeLocation":"DrcTaskManager.java:325", "msg":"[DRCTASK 42b1bc8b] Task drc-mgr-UUID aborted with an error errorCodes: 6900

[.drc-backup-restore-threads-5] [BKUP 42b1bc8b] Disconnecting connHandle for cleanup (END)

vmware.log file for the affected vm may show LWD snapshot being aborted.

/var/run/log/dpd.log in the esxi host may shows below errors, when searched with the snapshot ID

2025-03-04T19:59:25.880Z Er(163) dpd[9507694]: [Originator@6876 sub=Task opID=83d77d40-7d42] Task 'Dp.Task.Sync.UUID' failed with error: (dp.fault.SnapshotTransportFault) {
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> faultCause = (vmodl.MethodFault) null,
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> faultMessage = <unset>,
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> transportProviderResults = (vim.KeyValue) [
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> (vim.KeyValue) {
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> key = "vltErrSourceLine",
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> value = "287"
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> },
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> (vim.KeyValue) {
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> key = "vltErrMessage",
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> value = "Vltd was unable to connect to mux.psk.ssl.tcp:<Connector IP>:1492."
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> },
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> (vim.KeyValue) {
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> key = "vltErrBacktrace",

2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> },
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> (vim.KeyValue) {
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> key = "vltErrNumber",
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> value = "1116"
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> },
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> (vim.KeyValue) {
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> key = "vltErrName",
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> value = "CONNECT_FAILED"
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> },
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> (vim.KeyValue) {
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> key = "vltErrSourceFile",
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> value = "DRC/VLT/Vlt.c"
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> }
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> ]
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> msg = ""
2025-03-04T19:59:25.880Z Er(163) dpd[9507684]: --> }

Cause

This is caused if the LWD snapshot call from the ESXi host does reach the VLCR connector for various reasons.

Resolution

Ensure that the Virtual Machine is not backed up by any other backup solutions which uses high frequency snapshots. refer Caveats for High-Frequency snapshots

High-frequency snapshots, reading/writing of vdisks requires TCP port 1492 to be open from the ESXi host to the VLCR Connector. Refer Connector Networking Requirements

For On-premise or Cloud protected sites, ensure all required ports are enabled in On premise firewall and ESXi hosts to reach the connector on TCP port 1492.
For NXS-T environments using DFW firewall, ensure TCP port 1492 is allowed from ESXi hosts to the Connector along with the other required ports.
Run # nc -z <connector IP> 1492 to test the connectivity. Refer KB341078 to test network port connectivity in esxi.

Once all the ports requirements are checked, and the snapshots still fail, Please follow steps below

SSH to ESXi hosts
Run below commands to check the status of the deamons

/etc/init.d/vltd status
/etc/init.d/dpd status

Run below commands to restart the deamons

/etc/init.d/vltd restart
/etc/init.d/dpd restart
You can also restart these services from UI refer "Activate or Deactivate an ESXi Service "

If the snapshots still fail after service restart, Engage VMware by Broadcom support for further assistance.