vSphere Replication in "Not Connected" State in UI with Login error

Products

• VMware Live Recovery

Issue/Introduction

Symptoms :

- Customer cannot add new Replication for Virtual Machines. All Replications are in Error state.

- vSphere Replication appliances on SRM Site Pair UI page shows as "Not Connected" and "Unknown" :


- Site Pair error "Unable to retrieve pairs from extension server at https://<VR>:8043. Unable to log in to 'HBR Management Server at https://<VR>:8043"

- All required ports are already open and there is no DNS resolution issue.

- Connecting VRMS  throws error "Error creating bean with name 'com.vmware.hms.remote.HmsServerImpl': Unsatisfied dependency expressed through field 'repositoryFactory'; nested exception is org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'repositoryFactory': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)'"

- hbrsrv log :
2025-05-31T12:42:02.148+01:00 info hbrsrv[] [Originator@6876 sub=Host opID=hs-init-] Heartbeat handler detected dead connection for agent: host-102036/hostd
2025-05-31T12:42:02.148+01:00 info hbrsrv[] [Originator@6876 sub=Main opID=hs-init-] HbrError stack:
2025-05-31T12:42:02.148+01:00 info hbrsrv[] [Originator@6876 sub=Main opID=hs-init-]    [0] Exception Vmacore::InvalidStateException: No connection (host-102036/hostd)
2025-05-31T12:42:02.148+01:00 info hbrsrv[] [Originator@6876 sub=Main opID=hs-init-]    [1] Heartbeat failed (host-102036/hostd)

Environment

VMware vSphere Replication 9.x

Cause

vSphere replication stores a list of host IP's and connects with each one with extremely persistent approach. In some cases, if there is any unresponsive Host, hbr service will take alot of time to connect or remains in "Not Connected" state 

Resolution

1. Take a snapshot of the VR appliance which has the Host connection error

2. Stop hbrsrv service -
# systemctl stop hbrsrv 

3. Backup the latest hbrsrv.***.db (Where * represents the highest DB number)

   root [ ~ ]# cd /etc/vmware 
   root [ /etc/vmware ]# mkdir backup 
   root [ /etc/vmware ]# cp hbrsrv.***.db /backup 
   root [ /etc/vmware ]# sqlite3 hbrsrv.***.db (Login to the latest hbrsrv DB)

   root [ /etc/vmware ]# sqlite3 hbrsrv.103.db
    SQLite version 3.22.0 2025-05-31 18:45:57
    Enter ".help" for usage hints.
    sqlite> select * from hostinfo; (Displays the contents of hostinfo table)
    sqlite> delete from hostinfo;   (Deletes the contents of hostinfo table)

     The commands below can be used if you'd like to remove a specific host IP/s from the table instead of clearing the entire table. 

     sqlite3 hbrsrv.100.db "DELETE FROM HostInfo WHERE addresses in ('xx');"
   sqlite3 hbrsrv.100.db "DELETE FROM HostInfo WHERE addresses in ('<ip1>', '<ip2>');"
   sqlite3 hbrsrv.100.db "DELETE from HostInfo where addresses in ('192.X.X.11','192.X.X.12');"  

      sqlite> CTRL+D (Exits sqlite login)

 

4. Start hbrsrv service -
 # systemctl start hbrsrv 

5. Reboot the appliance