vSphere Replication intermittently disconnecting due to hbrsrv service issues

Products

• VMware Live Recovery

Issue/Introduction

vSphere Replication (VR) instances may experience intermittent disconnections leading to replication failures. Symptoms observed include:

• In the vSphere GUI, replication errors for specific VMs displaying: <VM Name> replication error: A replication error occurred at the vSphere Replication Server for replication 'VM Name'. Details: 'No connection to VR Server for virtual machine <VM Name> on host <ESXi Host Name> in cluster <Cluster Name> in <vDC>: Unknown'.
• Some configured replications not working properly.
• Connection errors observed in vSphere Replication logs.

These issues indicate a problem with the replication data path between the source and target environments.

Environment

vSphere Replication (9.x)

Cause

The underlying cause was identified as a communication issue between the source ESXi hosts and the target ESXi hosts on port 32032, even though the port was confirmed to be open at the network level. This indicated a service-level problem rather than a network blockage. Specifically, the hbrsrv services on the target ESXi hosts were in a state that prevented proper replication communication.

Resolution

To resolve this issue, the hbrsrv service on the affected target ESXi hosts needs to be restarted:

1. Within the target vCenter Server, navigate to each affected target ESXi host.
2. Select the ESXi host.
3. Click on the Configure tab.
4. In the left navigation pane, under 'System', select Services.
5. Locate the service named hbrsrv (Host Based Replication Service).
6. Right-click on hbrsrv or use the action button, and select Restart.
7. Repeat this process for all target ESXi hosts located in the target cluster.

Additional Information

After restarting the hbrsrv service on each target ESXi host, monitor the replication status in the vSphere Replication interface. If a replication is still showing as "Error" or "Not Syncing," you may need to manually trigger a "Sync Now" operation for the affected VMs to re-establish the replication. This resolution typically re-establishes the necessary communication channels for vSphere Replication to function correctly.