vSphere Replication in "Not Connected" State in UI with Login error

Products

• VMware Live Recovery

Issue/Introduction

• vSphere Replication appliances on SRM Site Pair UI page shows as "Not Connected" and "Unknown" :
  
  
  
  
• Site Pair error "Unable to retrieve pairs from extension server at https://<VR>:8043. Unable to log in to 'HBR Management Server at https://<VR>:8043"
  
  
• While connecting to VRMS, it may report below error:
  
  Error creating bean with name 'com.vmware.hms.remote.HmsServerImpl': Unsatisfied dependency expressed through field 'repositoryFactory'; nested exception is org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'repositoryFactory': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)'
  
  
• /var/log/vmware/hbrsrv.log of the vSphere replication will report below error:
  
  2025-05-31T12:42:02.148+01:00 info hbrsrv[] [Originator@6876 sub=Host opID=hs-init-] Heartbeat handler detected dead connection for agent: host-102036/hostd
2025-05-31T12:42:02.148+01:00 info hbrsrv[] [Originator@6876 sub=Main opID=hs-init-] HbrError stack:
2025-05-31T12:42:02.148+01:00 info hbrsrv[] [Originator@6876 sub=Main opID=hs-init-]    [0] Exception Vmacore::InvalidStateException: No connection (host-102036/hostd)
2025-05-31T12:42:02.148+01:00 info hbrsrv[] [Originator@6876 sub=Main opID=hs-init-]    [1] Heartbeat failed (host-102036/hostd)
  

Environment

VMware vSphere Replication 9.x

Cause

This is reported if any ESXi host in the environment becomes unresponsive. vSphere Replication maintains a list of host IP addresses and attempts to connect to each one using a highly persistent approach. Because of this, if any host in the environment becomes unresponsive, the hbr service may take an extended amount of time to connect or get stuck in a 'Not Connected' state.

• From /opt/vmware/hms/logs/hms.log in vSphere replication appliance shows it fails to connect to ESXi hosts,

YYYY-MM-DD HH:MM:SS.ZZZ ERROR com.vmware.hms.net.hbr.ping.svr.920765b6-3t8e-1led-a0c2-f3df5ac7a391 [hms-main-thread-2] ( .. net.impl.PersistentConnectiom) [] | Failed to connect to server: ##.##.##.52:443/hbr
YYYY-MM-DD HH:MM:SS.ZZZ ERROR com.vmware.hms.net.hbr.ping.svr.55d6f092-ef01-1lec-9cle-Ja6ldd7d36d3 [HbrBrokerMappingsSynchronizer-thread-0] ( .. net.impl.PersistentComnectiom) [operation ID=HbrBrokerMappingsSync#l, operationID=HbrBrokerMappingsSync#2] | Failed to connect to server ##.##.##.68:443/hbr
YYYY-MM-DD HH:MM:SS.ZZZ ERROR com.vmware.hms.net.hbr.ping.svr.5a2bd5el-ef01-11ec-834d-Ja68dd7d3707 [hms-main-thread-3] ( .. net.impl.PersistentConnectiom) [] | Failed to connect to server ##.##.##.61:443/hbr
YYYY-MM-DD HH:MM:SS.ZZZ ERROR com.vmware.hms.net.hbr.ping.svr.92c614cl-ef01-11ec-ab02-Ja68dd7d36cb [hms-main-thread-2] ( .. net.impl.PersistentConnectiom) [] | Failed to connect to server:##.##.##.54:443/hbr

Resolution

Fix the unresponsive host(s) in the environment.

Note: Please contact the Broadcom Support if any assistance required to remove the unresponsive/Unused ESXi host the hbrsrv DB.