[VCDR] High-Frequency snapshot fails with error "DPS deltaSync failed" or "errorCodes: 6900"

Products

• VMware Live Recovery

Issue/Introduction

To provide cause and steps to troubleshoot snapshot failure

Symptoms:
Following error is seen in VCDR UI 

Severity Warning Timestamp  pm11h ago Target VMCAWS Linux - VMDR2 Description Failed to backup VM 'VM NAME' (/VM NAME/VM NAME.vmx) during stage: finishing backup, for task: drc-UUID. High-frequency snapshot failed. Internal error: DPS deltaSync failed (Sync call failed on DpFaultSnapshotTransportFault).

drcMgr.log

 [.drc-backup-restore-threads-5] DrcLoadBalancer: Selected DRC Agent service UUID-domain-UUID (0 running tasks) for new DRC task 42b1bc8b. 
 [.drc-backup-restore-threads-5] DrcLoadBalancer: adding drcMgr task drc-mgr-UUID to DRCAgent service UUID-domain-UUID with 0 active tasks. 
 [.drc-backup-restore-threads-5] [BKUP 42b1bc8b] drcAgentDvxId UUID from drcAgentSvcId UUID-domain-UUID, siteId UUID, clusterMoId domain-c23871 
 [.drc-backup-restore-threads-5] [BKUP 42b1bc8b] Forwarding request to UUID-domain-UUID 
 [.drc-backup-restore-threads-5] [BKUP 42b1bc8b] Task on remote agent on siteId UUID: drc-agent-UUID ...... 
 [.drc-backup-restore-threads-5] [BKUP 42b1bc8b] Task drc-agent-UUID status is RUNNING (progress 0.0) 
 [.drc-backup-restore-threads-5] [BKUP 42b1bc8b] Task drc-agent-UUID status is ERROR (progress 0.0) 
 [.drc-backup-restore-threads-5] DrcLoadBalancer: removing drcMgr task drc-mgr-UUID from DRCAgent service UUID-domain-UUID with 1 active tasks. @cee{"timestamp":"TIMESTAMP", "priority":"ERROR", "threadName":"drc-backup-restore-threads-5", "codeLocation":"DRCMgrSvc.java:434", "msg":"[BKUP 42b1bc8b] Task drc-agent-UUID failed: errorCodes: 6900 @cee{"timestamp":"TIMESTAMP", "priority":"ERROR", "threadName":"drc-backup-restore-threads-5", "codeLocation":"DRCMgrSvc.java:628", "msg":"[BKUP 42b1bc8b] Failed during backup: errorCodes: 6900 @cee{"timestamp":"TIMESTAMP", "priority":"INFO", "threadName":"drc-backup-restore-threads-5", "codeLocation":"DrcTaskManager.java:325", "msg":"[DRCTASK 42b1bc8b] Task drc-mgr-UUID aborted with an error errorCodes: 6900 2023-11-09T03:02:27.411+0000 [.drc-backup-restore-threads-5] [BKUP 42b1bc8b] Disconnecting connHandle for cleanup (END)  

Cause

This is caused if the LWD snapshot call from the ESXi host does reach the DR connector.

Resolution

Ensure that the Virtual Machine is not backed up by any other backup solutions like Commvault, Veeam etc. 
High-frequency snapshots, reading/writing of vdisks requires TCP port 1492 to be open from the ESXi host to the DR Connector. Refer DRaaS Connector Networking Requirements

• For On-premise, ensure all required ports are enabled in On premise firewall and ESXi hosts to reach the DR connector on TCP port 1492. 
• For VMC on AWS Clouds, ensure the MGW has outbound rule from ESXi to the CloudDR-Connector on TCP port 1492.
• For NXS-T environments using DFW firewall, ensure TCP port 1492 is allowed from ESXi hosts to the DR-Connector along with the other required ports. 

Once all the ports requirements are checked, and the snapshots still fail, Please follow steps below 

• SSH to ESXi hosts 
• Run below commands to check the status of the deamons 

                /etc/init.d/vltd status 
         /etc/init.d/dpd status 

• Run below commands to restart the deamons 

                  /etc/init.d/vltd restart
                  /etc/init.d/dpd restart
You can also restart these services from UI refer "Activate or Deactivate an ESXi Service "

If the snapshot still fail after service restart, Engage VMware Support  for further assistance.

Additional Information

Impact/Risks:
Snapshot fails and VM is not protected