Reprotect operation fails with error: ‘VR synchronization failed for VRM group. Synchronization monitoring has stopped’

Products

VMware Live Recovery

Issue/Introduction

Symptoms

The vSphere Replication (VR) appliances are running version 9.0, and the ESXi hosts are on version 8.0 Update 2
Virtual Machines are configured with legacy replication and have been successfully failed over to the Disaster Recovery (DR) site by initiating a planned migration.
A Reprotect operation is initiated. During this process, the replication configuration of the VMs automatically switches to enhanced replication.
The reprotect operation fails with the error message:
“VR synchronization failed for VRM group. Synchronization monitoring has stopped.”
When checking the replication status, the following error is observed:
“No connection to VR server.”

Note: If you are using vSphere Replication 9.0 and Site Recovery Manager 9.0 and perform reprotect on a replication without enhanced capabilities, the replication will be automatically converted to enhanced replication, if the enhanced setup is supported by both sites. Even if you reconfigure the replication to go back to a replication without enhanced capabilities, when you perform another reprotect operation, it will be automatically converted to enhanced replication. If you are using vSphere Replication 9.0.1 and Site Recovery Manager 9.0.1 and perform reprotect on a replication without enhanced capabilities, the replication type does not change.

For more details, refer to the official documentation: How Site Recovery Manager Reprotects Virtual Machines with vSphere Replication

Environment

vSphere Replication 9.0

Cause

The Reprotect operation automatically transitions the replication type from legacy to enhanced since we are using vSphere Replication 9.0. In this scenario, since enhanced replications are experiencing issues, the replication fails, and the reprotect operation ends with a warning.

Note: Enhanced replications require TCP network connectivity on ports 32032 from the ESXi hosts on which the replicated VMs are running to the ESXi hosts of the cluster containing the target datastore. Make sure your firewall settings are adjusted accordingly.

The root cause for enhanced replication failing is due to the ESXi hosts hbr-agent service being unable to authenticate with the target ESXi hosts and vSphere Replication appliance over port 32032. This failure is caused by an invalid broker certificate, which leads to unsuccessful login attempts from the hbr-agent.

Cause Validation

From the /var/run/log/hbr-agent log file of the ESXi host where the vm is powered on, the following errors confirm failed login attempts due to certificate issues:

2025-04-15T09:02:05.569Z In(166) hbr-agent-bin[2137855]: 2025-04-15T09:02:05.569567 hbr-agent-bin [2137855] [0x0000009b78237700] error: [Proxy [Group: GID-a54e080a-xxxx-xxxx-xxxx-xxxxxxxxxxxx] -> [172.##.##.##:32032]] Failed to login to brokered server additional error info: Login request is denied .; ClientConnection (client=[172.##.##.##]:59126 ) failed login attempt
2025-04-15T09:02:05.569Z In(166) hbr-agent-bin[2137855]: 2025-04-15T09:02:05.569594 hbr-agent-bin [2137855] [0x0000009b78237700] error: [Proxy [Group: GID-a54e080a-xxxx-xxxx-xxxx-xxxxxxxxxxxx] -> [172.##.##.##:32032]] Exhausted all server endpoints reported by broker.

These errors indicate that the hbr-agent cannot establish a connection due to authentication failures from the invalid certificate used by the brokered server.

Resolution

To resolve the issue, restart the hbrsrv and hms services on both vSphere Replication appliances. Restarting these services generates and pushes out a new broker certificate, allowing the ESXi hosts to successfully establish a connection with the vSphere Replication appliance.

SSH into each vSphere Replication appliance.

Run the following commands:

systemctl restart hbrsrv
systemctl restart hms

Note: For permanent fix for the reprotect failures caused by transition of replications from legacy to enhanced during reprotect operation, upgrade the vsphere replication appliances to 9.0.1 or later