All of the enhanced replications have stopped

Products

• VMware Live Recovery

Issue/Introduction

You have replications using Enhanced vSphere Replication showing status, "Not Active (RPO Violation)" with the Last error showing as, "A replication error occurred at the vSphere Replication Server for replication : '[VM name]'. Details: 'No connection to VR Server for virtual machine [VM name] on host [hostname] in cluster. Unknown'.

Environment

vSphere Replication 8.8 
vSphere Replication 9.x

ESXI 8.0 u2 and above

Cause

The vSphere replication appliance is not able to successfully log into any of the target hosts to broker a connection due to an old or faulty certificate created during the last start up of the HMS service. 

Verifying Enhanced Replication mappings will show an error with status, "Login: Login denied. Login request is denied"

The /var/run/log/hbr-agent.log on the source ESXI host will show the following errors:

Resolution

To clear the issue from the Browser:

1. Log into VAMI interface of both Protected and Recovery Replication Appliances
2. Navigate to services tab and restart "hms" service and "hbrsrv" service
3. Close any open tabs with vSphere Replication
4. Launch the Live Site Recovery DR interface and navigate to Enhanced Replication mappings tab. Click the "Run Tests" button
5. Errors will clear after the mappings have run
6. Verify Replications are now performing Sync operations.

To clear the issue from an SSH:

1. SSH into both Protected and Recovery Replication Appliances with Admin account and perform su to root
2. Run the following commands:
   service hms restart
service hbrsrv restart
3. Launch the Live Site Recovery DR interface and navigate to Enhanced Replication mappings tab. Click the "Run Tests" button
4. Errors will clear after the mappings have run
5. Verify Replications are now performing Sync operations.

Restarting the services on the management appliances will create new brokerage certificates and push them to the ESXI hosts to allow for login.

Additional Information

If running into further issues you can Verify network connectivity for Enhanced Replication