Error “Sync Timeout for replication 'C4 ID'” when synchronizing a replication in VMware Cloud Director Availability

Products

• VMware Cloud Director

Issue/Introduction

• For existing replications, the replication status shows the status as “Unknown” and the Overall health of the replication shows status as “Red”.
• Attempt to perform a manual synchronization fails with the Sync timeout error.
• On the replicator appliance, in the file /opt/vmware/h4/replicator/replicator.log you can see entries similar to:

DEBUG - [UI-########-####-####-####-############-#####-##-##-##-##] [###-##] c.v.h.r.replication.SyncSourceJob        : Requesting manual sync for H4-########-####-####-####-############

ERROR - [UI-########-####-####-####-############-#####-##-##-##-##] [###-##] com.vmware.h4.jobengine.JobExecution     : Task ########-####-####-####-############ (WorkflowInfo{type='sync', resourceType='replication', resourceId='H4-########-####-####-####-############', isPrivate=false, resourceName='null'}) has failed

com.vmware.vim.binding.vmodl.fault.ManagedObjectNotFound: The object 'vim.VirtualMachine:vm-####' has already been deleted or has not been completely created

        at jdk.internal.reflect.GeneratedConstructorAccessor428.newInstance(Unknown Source)

• New replications can be configured and synchronized without any issues.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x

Cause

This issues can occur if a virtual machine's (VM) unique identifier (VMID) changes. This typically happens when a VM is moved, copied, or re-registered with vCenter Server. VMware Cloud Director Availability (VCDA) relies on these VMIDs remaining consistent; therefore, any mismatch can disrupt and ultimately break the replication process.

Resolution

To resolve this issue:

• Delete and recreate the replications which observe the sync error.
• To recreate replications for a larger number of VMs (e.g., more than 100) use automation and use API to re-create the replication. Find more details in the document regarding the VCDA API : VMware Cloud Director Availability API

Additional Information

Note:

Deleting and recreating replications is a disruptive process with important implications:

• Temporary Loss of Protection: During the time between deleting the old replication and the initial synchronization of the new one, the VM or vApp is unprotected. In the event of a disaster at the source site during this window, recovery might not be possible, or data loss could be significant.
• Full Initial Synchronization: Recreating a replication always triggers a full initial synchronization. This means all the data from the source VM will be copied to the destination again.
• RPO Violation: The Recovery Point Objective (RPO) will effectively be reset. Any point-in-time instances from the previous replication will be lost, and new ones will only start being created after the new initial synchronization completes.
• Seed VMs: If you have used a seed VM for the original replication (pre-existing copy of the VM at the destination), you might be able to reuse it to expedite the initial sync of the new replication. However, this requires careful management and might not always be applicable if the issue was related to VMID changes or corrupted data.