Replicator maintenance mode failed for VMs in VMware Cloud Director Availability

Products

• VMware Cloud Director

Issue/Introduction

• After attempting to place a replicator in maintenance mode, a subset of replications failed to transfer and are now unhealthy and unmanageable.
• Following the start of the maintenance mode task, the items fail to attach to the new replicator across multiple attempts, as in the following example:
  
  2026-06-11 17:55:57.791 ERROR - [UI-########-####-####-####-############-####-##-###] [task-poller-4] c.v.h.m.r.SwitchDestinationReplicatorJob : Failed to attach replication H4-########-####-####-####-############ to replicator ########-####-####-####-############.

com.vmware.h4.api.error.exceptions.GenericServerException: Operation aborted due to an unexpected error.
...
  
  
• The items then fail to re-attach to the original replicator multiple times, similar to the following:
  
  2026-06-11 17:56:34.652 ERROR - [UI-########-####-####-####-############-####-##-###] [task-poller-1] c.v.h.m.r.SwitchDestinationReplicatorJob : Re-attach of replication H4-########-####-####-####-############ to replicator ########-####-####-####-############ failed.

com.vmware.h4.api.error.exceptions.GenericServerException: Operation aborted due to an unexpected error.
...
  
  
• In the replicator logs on the replicator appliance (/opt/vmware/h4/replicator/log/replicator.log), the failures are due to a mismatch between the expected location of the vmdk file in the underlying hbr (vSphere Replication) service and the actual location:
  
  2026-06-11 17:55:57.754 DEBUG - [########-####-####-####-############] [pc-task-monitor-4] c.v.h.r.monitoring.hbr.HbrTaskMonitor    : The state of task Hbr.Replica.Task.########-####-####-####-############ is error
2026-06-11 17:55:57.754 DEBUG - [########-####-####-####-############] [pc-task-monitor-4] c.v.h.r.monitoring.hbr.HbrTaskListener   : The HBR task failed (hbr.replica.TaskInfo) {
   dynamicType = null,
   dynamicProperty = null,
   task = ManagedObjectReference: type = HbrReplicaTask, value = Hbr.Replica.Task.########-####-####-####-############, serverGuid = null,
   operation = replicationManagerRecoverGroup,
   method = LoadReplicationGroup_Task,
   object = ManagedObjectReference: type = HbrReplicationManager, value = HbrServer, serverGuid = null,
   extraInfo = null,
   state = error,
   canceled = false,
   cancelable = false,
   error = (hbr.replica.fault.GroupSpecMismatch) {
      faultCause = null,
      faultMessage = null,
      groupId =
   },
   result = null,
   progress = 100,
   totalTransferSizeKb = null
}
2026-06-11 17:55:57.754 ERROR - [UI-########-####-####-####-############-####-##-###-##] [pc-task-monitor-4] c.v.h.r.r.AttachDestinationJob           : Failed to load group in HBR for replication H4-########-####-####-####-############

com.vmware.vim.binding.hbr.replica.fault.GroupSpecMismatch: Recovery file base path for disk H4D-########-####-####-####-############ is /vmfs/volumes/########-########/C4-########-####-####-####-############/#####.vmdk, but database says /vmfs/volumes/########-########/C4-########-####-####-####-############/####.vmdk; Code set to: Group recovery file and group info mismatch.; Validating recovery disk state; Loading recovery file: /vmfs/volumes/########-########/C4-########-####-####-####-############/hbrgrp.H4-########-####-####-####-############.txt; Recovering group H4-########-####-####-####-############
...
  
  

• This same mismatch prevents the items from being successfully re-attached to the original replicator:

  
  2026-06-11 17:56:34.596 DEBUG - [########-####-####-####-############] [pc-task-monitor-2] c.v.h.r.monitoring.hbr.HbrTaskMonitor    : The state of task Hbr.Replica.Task.########-####-####-####-############ is error
  2026-06-11 17:56:34.596 DEBUG - ########-####-####-####-############] [pc-task-monitor-2] c.v.h.r.monitoring.hbr.HbrTaskListener   : The HBR task failed (hbr.replica.TaskInfo) {
     dynamicType = null,
     dynamicProperty = null,
     task = ManagedObjectReference: type = HbrReplicaTask, value = Hbr.Replica.Task.########-####-####-####-############, serverGuid = null,
     operation = replicationManagerRecoverGroup,
     method = LoadReplicationGroup_Task,
     object = ManagedObjectReference: type = HbrReplicationManager, value = HbrServer, serverGuid = null,
     extraInfo = null,
     state = error,
     canceled = false,
     cancelable = false,
     error = (hbr.replica.fault.GroupSpecMismatch) {
        faultCause = null,
        faultMessage = null,
        groupId =
     },
     result = null,
     progress = 100,
     totalTransferSizeKb = null
  }
  2026-06-11 17:56:34.596 ERROR - [UI-########-####-####-####-############-####-##-###-##] [pc-task-monitor-2] c.v.h.r.r.AttachDestinationJob           : Failed to load group in HBR for replication H4-########-####-####-####-############

  com.vmware.vim.binding.hbr.replica.fault.GroupSpecMismatch: Recovery file base path for disk H4D-########-####-####-####-############ is /vmfs/volumes/########-########/C4-########-####-####-####-############/####.vmdk, but database says /vmfs/volumes/########-########/C4-########-####-####-####-############/####.vmdk; Code set to: Group recovery file and group info mismatch.; Validating recovery disk state; Loading recovery file: /vmfs/volumes/########-########/C4-########-####-####-####-############/hbrgrp.H4-########-####-####-####-############.txt; Recovering group H4-########-####-####-####-############
  ...

Environment

VMware Cloud Director Availability 4.7.3.x

Cause

The replication files for the failed replications were not in the expected location.

Resolution

After these failures, the replications in this state can no longer be corrected, so they must be deleted and recreated.

Additional Information

As discussed in the following document, the only supported method for moving replication files at the destination is to update the storage policy within VMware Cloud Director Availability:

Replicating workloads