Error: : "No connection to VR Server: Unknown" on a replicated VM in VMware Cloud Director Availability

Products

• VMware Cloud Director

Issue/Introduction

• The following alert appears in vCenter for replicated encrypted VM:
  
  No connection to VR Server: Unknown
  
  
• The synchronization fails for the affected encrypted VMs
• In the /opt/vmware/h4/manager/log/manager.log on the destination site the below error is present:
  
  DATE TIME ERROR - [UI-########-####-###-########86f5-###14-##-##-#8] [sync-2] com.vmware.h4.jobengine.JobExecution     : Task ########-####-###-########68f3 (WorkflowInfo{type='syn
c', resourceType='replication', resourceId='H4-########-####-###-########1ff8', isPrivate=false, resourceName='VM NAME'}) has failed com.vmware.h4.manager.api.exceptions.SyncTimeoutException: Sync timeout for replication 'H4-########-####-###-########1ff8'.
        at com.vmware.h4.manager.replication.sync.SyncManager.checkSyncHealthImpl(SyncManager.java:542)
        at com.vmware.h4.manager.replication.sync.SyncManager.checkSyncHealth(SyncManager.java:485)
        at com.vmware.h4.manager.replication.sync.SyncManager.lambda$startHealthMonitor$8(SyncManager.java:636)
        at com.vmware.h4.common.mdc.MDCRunnableWrapper.run(MDCRunnableWrapper.java:32)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)


• In the /var/run/log/vmkernel.log on the host where the affected VM is running on below error is present:
  
  DATE TIME cpu36:22011138)WARNING: Hbr: 893: Failed to receive from ##.##.##.## (groupID=########-####-###-########1ff8): Broken pipe
DATE TIME cpu36:22011138)WARNING: Hbr: 6530: Failed to receive session init response.
DATE TIME cpu36:22011138)WARNING: Hbr: 5093: Failed to establish connection to [##.##.##.##]:32032 (groupID=########-####-###-########1ff8): Broken pipe

 

• Reapplying the replication settings on the affected encrypted VMs allows the replication to resume correctly.

Environment

VMware Cloud Director Availability  4.7.x

Cause

The replication for the encrypted VM was configured when the HBR agent VIB was not installed on all the ESX hosts in the cluster. 

Resolution

Installing the HBR agent VIB on all the ESX hosts in the cluster. For additional information visit the documentation Create a replication for encrypted virtual machines

Recreate the replication for the affected encrypted VMs.

Additional Information

For more information about VIBs and how to install them, see VIBs, Image Profiles, and Software Depots in the VMware ESXi Upgrade Guide