Sync Timeout for replication 'C4 ID' error when synchronizing a replication to a cloud site in vCloud Availability 4.x.x

Products

VMware Cloud Director

Issue/Introduction

Symptoms:

Replications to a cloud site are unable to synchronize and in the Replication Tasks view the synchronize task fails with the error:

Sync Timeout for replication C4-xxxxxx-cxxd-4xx8-8xx9-9xxxxxxxg'.

In /opt/vmware/h4/cloud/log/cloud.log on the vApp Replication Manager, you see similar entries:

WARN - [UI__6efabf67-a697-4bcc-a8f6-9595dd46b68d_lX] [c4-scheduler-1] com.vmware.h4.cloud.job.VmSyncJob        : Replication C4-xxxxxx-cxxd-4xx8-8xx9-9xxxxxxxg'. can't be synchronized.
ERROR - [UI__6efabf67-a697-4bcc-a8f6-9595dd46b68d_lX] [c4-scheduler-1] com.vmware.h4.jobengine.JobExecution     : Task e0f049f9-96d7-4365-be4b-672fee17a655 (WorkflowInfo{type='sync', resourceType='vmReplication', resourceId='C4-xxxxxx-cxxd-4xx8-8xx9-9xxxxxxxg'.', isPrivate=false, resourceName='KXXXX1_New'}) has failed

com.vmware.h4.manager.api.exceptions.SyncTimeoutException: Sync timeout for replication ' C4-xxxxxx-cxxd-4xx8-8xx9-9xxxxxxxg'.
   at com.vmware.h4.cloud.job.VmSyncJob.syncFailed(VmSyncJob.java:110)
   at com.vmware.h4.cloud.job.VmSyncJob.lambda$execute$0(VmSyncJob.java:62)
   at com.vmware.h4.common.monitoring.ReplicationListener.onComplete(ReplicationListener.java:53)
   at com.vmware.task.rest.client.TaskMonitor.lambda$workImpl$0(TaskMonitor.java:189)
   at com.vmware.task.rest.client.TaskMonitor.notifyListener(TaskMonitor.java:211)
   at com.vmware.task.rest.client.TaskMonitor.workImpl(TaskMonitor.java:188)
   at com.vmware.task.rest.client.TaskMonitor.work(TaskMonitor.java:120)
   at com.vmware.h4.cloud.service.ManagerTaskMonitorService.lambda$taskMonitor$0(ManagerTaskMonitorService.java:102)
   at com.vmware.h4.common.mdc.MDCRunnableWrapper.run(MDCRunnableWrapper.java:30)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
   at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
   at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCloud Director Availability 4.x.x

Cause

This issue occurs when the source vCloud Availability On-Premises appliance or vCloud Availability Tunnel appliance, depending on the type of source site, is unable to communicate with the vCloud Availability Tunnel appliance in the destination cloud site.

Resolution

To resolve this issue:

Ensure that the the source vCloud Availability appliance is able to communicate with the public API endpoint in the destination cloud site directly over port 8048/TCP or 443/TCP if DNAT is configured for the public endpoint.
Ensure that we check and allow all the necessary ports required for communication for the vCDA applaince as per the Network requirements and prerequisites.
Use the curl command to verify the connectivity between the appliances.
Example: For the network Communication to complete we need the ESXi host and the VMware Cloud Director Availability On-Premises Appliance to communicate over the port 44046/TCP. Port 44046/TCP is used for transferring replication data traffic from the ESXi hosts to the VMware Cloud Director Availability On-Premises Appliance. We can use the curl command from the esxi host to on prem : curl -kv https://ON-PREM_FQDN_or IP:44046

Additional Information

For more information on port requirements in cloud deployments, please refer to the "Network Architecture" section and ensure that the necessary ports are enabled in VMware vCloud Availability.

For detailed information on VMware ports and protocols for vCDA in cloud environments, please consult the following document: VMware Ports and Protocols for vCDA Cloud.

For on-premises deployments, check the "Network Architecture" section for your on-premises setup and ensure that the required ports are enabled accordingly. Refer to the following document for information on ports for vCDA on-premises appliances: VMware Ports and Protocols.