vSphere replication is attempting to establish connection to ESXI hosts that have been decommissioned

Products

VMware Live Recovery

Issue/Introduction

ESXI hosts have been decommissioned from environment and are no longer present in the vCenter server.

vSphere replication is presenting the error message, "vSphere Replication Management Server could not establish connection to vSphere Replication Server at '##.##.##.##:443'.
ESXI hosts still appear in list of Replication Servers and show up as "Disconnected"
Site Recovery manager UI displays error as below,

Error - Unable to retrieve pairs from extension server at https://##.##.#.###:8043.Unable to login to 'HBR Management Server at https://##.##.#.###:8043'.

or

Error - Unable to retrieve pairs from extension server at https://##.##.#.###:8043.A generic error occurred in the vSphere Replication Management Server. Exception details: 'Error creating bean with name 'com.vmware.hms.remote.HmsServerImpl': Unsatisfied dependency expressed through field 'repositoryFactory'; nested exception is org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'repositoryFactory': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)'.

Environment

VMware Live Recovery

vSphere Replication 8.x

Cause

The current issue arises when stale entries of disconnected or discontinued hosts still exist in the vSphere replication server database.
The ESXi hosts removed by the vCenter Server is faster than the vSphere Replication appliance, This leads to the retention of stale entries.

From the logs /opt/vmware/hms/logs/hms.log,

####-##-## 02:43:33.940 ERROR com.vmware.hms.net.hbr.ping.svr.4c4c4544-0042-4d10-8033-############ [hms-main-thread-3] (..net.impl.PersistentConnection) [] | Failed to connect to server ##.##.##.###:443/hbr
####-##-## 02:47:45.837 ERROR com.vmware.hms.net.hbr.ping.svr.4c4c4544-0039-4d10-8033-############ [hms-main-thread-2] (..net.impl.PersistentConnection) [] | Failed to connect to s erver ##.##.##.###:443/hbr
####-##-## 02:48:15.833 ERROR com.vmware.hms.net.hbr.ping.svr.4c4c4544-0042-4d10-8033-############ [hms-main-thread-1] (..net.impl.PersistentConnection) [] | Failed to connect to s erver ##.##.##.###:443/hbr
####-##-## 02:48:45.833 ERROR com.vmware.hms.net.hbr.ping.svr.4c4c4544-0042-4d10-8033-############ [hms-main-scheduled-thread-14] (..net.impl.PersistentConnection) [operationID=152 fcdc8-ec22-4321-aa36-c55b8185e597-HMSINT-2] | Failed to connect to server ##.##.##.###:443/hbr

The vSphere replication server is unable to connect because the IPs are stale entries.

Resolution

The stale hosts entries will need to be removed from the vSphere replication appliance database.

To find the Stale Host Entries:

Snapshot Replication appliance on the site in which you have decommissioned ESXI host.
Take note of ESXI host IP address from error message.
SSH into vSphere replication appliance using Username - admin and change access to root account with command:$sudo -i
Stop the vSphere replication appliance management service:$service hms stop
Access the postgres database:

$cd /opt/vmware/vpostgres/current/bin

$./psql -U vrmsdb

Enter vrmsdb Password :

Note: To access vrmsdb password run below command

/opt/vmware/hms/bin/embedded_db_connect.sh
Make the tables easier to read with the following command
/x
Run the following command to verify if ESXI host is still present in list of Replication Servers. Replace ## symbols with host IP address.
select * from hbrserverentity where vsrv_address = '##.##.##.##' AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue);

One record will be returned for the ESXI host.
Example:
Find the Entity ID for the entry that corresponds to ip address of the ESXI host in the Fault Entity table,
select * from faultentity;
Contact Broadcom Support to remove entries from database safely.