Configuring new replications time out while waiting for the setup destination task to complete in Cloud Director Availability 4.x

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Creating a new protection or migration fails, and you see an error similar to: 

Assuming task '########-####-####-####-########ce3f' failed, because its status did not update in a timely fashion.

• In /opt/vmware/h4/manager/log/manager.log on the destination Cloud Director Replication Management Appliance, you see the setup destination task is initiated but then times out after 60 seconds with entries similar to:

2023-07-17 13:59:04.994 DEBUG - [UI-########-####-####-####-########9148-r241-tJ-eA-Bu1-T1] [job-64] c.v.h.m.replication.StartReplicationJob  : Setup destination task: ApiTask{id=########-####-####-####-########ce3f, user='admin@mydomain', progress=0, state=RUNNING, lastUpdated=1689602344952, startTime=1689602344950, endTime=-1, warnings=[], workflowInfo=WorkflowInfo{type='setupDestination', resourceType='replication', resourceId='H4-########-####-####-####-########3b39', isPrivate=false, resourceName='null'}, result=null, error=null}
...
2023-07-17 14:00:06.492 ERROR - [UI-########-####-####-####-########9148-r241-tJ-eA-Bu1-T1] [health-check-3] com.vmware.h4.jobengine.JobExecution     : Task ########-####-####-####-########3db3 (WorkflowInfo{type='start', resourceType='replication', resourceId='H4-########-####-####-####-########3b39', isPrivate=false, resourceName='null'}) has failed

com.vmware.h4.api.error.exceptions.TaskMonitoringTimeOutException: Assuming task '########-####-####-####-########ce3f' failed, because its status did not update in a timely fashion.
...

• In /opt/vmware/h4/replicator/log/replicator.log on the destination Replicator Appliance, you see the setup destination task complete after the task on the Manager Service has already timed out with an entry similar to:

2023-07-17 14:01:37.485 DEBUG - [UI-########-####-####-####-########9148-r241-tJ-eA-Bu1-T1-cn] [job-3] com.vmware.h4.jobengine.JobExecution     : Task ########-####-####-####-########ce3f (WorkflowInfo{type='setupDestination', resourceType='replication', resourceId='H4-########-####-####-####-########3b39', isPrivate=false, resourceName='null'}) completed with result SetupDestinationResponse{disks=[SourceDiskInfo{key=2000, uuid=########-####-####-####-########7d83, replicationId='H4D-231f7dc9-########-####-####-####-########7d83'}], placement=ReplicationPlacement{datastoreId='datastore-133579', datastoreFolder='C4-########-####-####-####-########9610', vcId='########-####-####-####-########b4de', disks=[], validateSeedUuids=true, storageProfileId='########-####-####-####-########4b2b'}, destinationReplicationState=DestinationReplicationState{id='H4-########-####-####-####-########3b39', currentRpoViolation=-1, latestInstance=null, state=passive, recoveryInfo=RecoveryInfo{recoveryState=NOT_STARTED, vcId='null', vmId='null', vmName='null', optimizeUntil=null}, lastError=null, replicatedDisks=[ReplicatedDiskInfo{diskKey=2000, uuid=########-####-####-####-########7d83, diskType='thin', baseName='test.vmdk', capacityBytes=42949672960, isSeed=false, spaceRequirement=-1}], spaceRequirement=10485760, isMovingReplica=false}}

• This issue occurs when you are  creating a protection or migration to a destination site that is backed by Cloud Director.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue occurs when there is a delay in Cloud Director Availability interacting with and processing API calls to the destination vSphere environment, which causes the setup destination task on the Manager Service to timeout.

Resolution

Improvements have been made that are designed to mitigate this issue in Cloud Director Availability 4.6, available at Broadcom Downloads.

Additional Information

This error can also be generated when the Cloud Director Availability appliances and virtual infrastructure do not have their time synchronized or when there are multiple offline or inaccessible Replicator appliances.

For more information, see:

• "Assuming task failed, because it's status did not update in a timely fashion" error when configuring replications in Cloud Director Availability 4.x
• Migration and replication tasks fail because the task status did not update in a timely fashion in Cloud Director Availability 4.x