Failover started and stuck synchronizing

Products

• VMware Cloud Director

Issue/Introduction

• VM failover started and the status is stuck on "Synchronizing".
• On the replicator on the destination site, the failover status is stuck on an higher percentage.
• In /var/log/vmware/hbrsrv.log on the destination Replicator the below entry can be seen:
  
  DATE TIME info hbrsrv[20305] [Originator@6876 sub=Misc groupID=H4-########-####-####-########be10 opID=hsl-#####bef] Exceeded maximum number of complete instances at start of an instance. (Current: 4 Max: 3) Forcing sync prune.
  
  AND
  
  DATE TIME info hbrsrv[01090] [Originator@6876 sub=StatsLog] HbrEvent: {"eventID":"consolidateProgress","groupID":"H4-########-####-####-########be10","diskID":"H4D-########-####-####-########bfbe","percentCompleted":24,"ETA":"DATE TIME","serverID":"########-####-####-########42ad","hbrEvent":1}

Environment

VMware Cloud Director availability 4.6.x

VMware Cloud Director availability 4.7.x

Cause

This behavior can occur when a failover task is initiated while a synchronization is currently ongoing for the replication.
Due to the synchronization, the number of instances for the replication may exceed the retention limit requiring the instance chain to be pruned. This results in a disk consolidation during which the replica disk chain is locked, stalling progress of the failover until the consolidation is completed

Resolution

This is an expected behavior if a failover is initiated while a synchronization is ongoing that requires a sync prune.

The disk consolidation tasks need to complete before the failover can resume progress.

Additional Information

The disk consolidation progress can be followed on the /var/log/vmware/hbrsrv.log on the destination Replicator searching for the failover H4 ID running the command below:

grep "H4-########-####-####-#########be10" hbrsrv.log | grep "consolidateProgress"