vSphere replication is attempting to establish connection to ESXI hosts that have been decommissioned

Products

• VMware Live Recovery

Issue/Introduction

ESXI hosts have been decommissioned from environment and are no longer present in the vCenter server.
vSphere replication is presenting the error message, "vSphere Replication Management Server could not establish connection to vSphere Replication Server at '##.##.##.##:443'.

ESXI hosts still appear in list of Replication Servers and show up as "Disconnected"

Environment

vSphere Replication 8.x and above
ESXI 8.x and above

Cause

vCenter Server removes the ESXI host faster than the vSphere Replication appliance can process the calls to disconnect and remove the ESXI host. This results in stale entries being left behind. 

Resolution

The stale hosts entries will need to be removed from the vSphere replication appliance database.

To find the Stale Host Entries:
1. Snapshot Replication appliance on the site in which you have decommissioned ESXI host
2. Take note of ESXI host IP address from error message
3. SSH into vSphere replication appliance using the Admin login and sudo to root account with command:
sudo -i

4. Stop the vSphere replication appliance management service:
service hms stop

5. Access the postgres database:
/opt/vmware/hms/bin/embedded_db_connect.sh

6. Make the tables easier to read with the following command
/x

7. Run the following command to verify if ESXI host is still present in list of Replication Servers. Replace ## symbols with host IP address.
select * from hbrserverentity where vsrv_address = '##.##.##.##' AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue);

One record will be returned for the ESXI host. 
Example:


8. Find the Entity ID for the entry that corresponds to ip address of the ESXI host in the Fault Entity table
select * from faultentity;

9. Contact Broadcom Support to remove entries from database safely.