Error: 'Sync Timeout for replication '####'' in UI when attempting initial synchronization due to insufficient target datastore space

Products

• VMware Cloud Director
• VMware Live Recovery

Issue/Introduction

• Initial replication synchronization for a virtual machine fails within the VMware Cloud Director Availability (VCDA) user interface, resulting in a SyncTimeoutException
• The replication workflow stalls during disk allocation or temporary recovery file creation phases.
• The /opt/vmware/h4/manager/log/manager.log records a replication failure due to an InsufficientStorageSpace fault. Concurrently, the /var/log/vmware/hbrsrv.log on the destination Cloud Director Replication Management Appliance captures the underlying ESXi-level NFC_NO_DISKSPACE fault on the target datastore volume:
  
  <Time_Stamp> info hbrsrv[<process id>] [Originator@6876 sub=Recover groupID=<replication-group-uuid>] Removing recovery file /vmfs/volumes/<datastore-uuid>/<vm-uuid>/hbrgrp.<replication-group-uuid>.txt for group <replication-group-uuid>

<Time_Stamp> error hbrsrv[<process id>] [Originator@6876 sub=Main groupID=<replication-group-uuid>] [1] NFC error: NFC_NO_DISKSPACE


• Following the failure, the system automatically triggers a StopReplicationJob to cancel the workflow, clean up recovery files, and release associated replication locks:
  
  <Time_Stamp> DEBUG - [...] c.v.h.m.replication.StopReplicationJob   : Stopping replication ReplicationInfo{id='<replication-group-uuid>', ...} with settings StopSettings{diskCleanup=AUTO}

<Time_Stamp> DEBUG - [...] com.vmware.h4.jobengine.JobExecution     : Task <job-id> (...) completed with result null

Environment

VMware Cloud Director Availability 4.7

Cause

This issue can occur when the target ESXi datastore runs out of available free space while attempting to allocate preallocated disk space or temporary recovery files for a virtual machine during the initial synchronization phase, causing Network File Copy (NFC) write rejections, workflow timeouts, and subsequent task cancellations.

Resolution

1. Verify Storage Utilization in vCenter:

   • Log into vCenter Server and locate the target datastore associated with the failing replication.

   • Check the storage capacity and available headroom to ensure it meets or exceeds the virtual machine's disk size requirements.

2. Free Up Capacity or Re-target Replication:

   • If the datastore lacks sufficient free space, either free up capacity on the current target volume or modify the replication settings from the VCDA or cloud portal UI to target an alternative datastore with adequate headroom.

3. Re-trigger Initial Synchronization:

   • Once valid storage capacity is confirmed or a new target datastore is assigned, log into the VCDA interface and re-attempt the initial synchronization workflow.