Site recovery UI reports "Failed to retrieve pairs from extension server" and "Failed to connect to HBR Management Server"

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

 

• Following error is observed in Site Recovery UI:
  
  Failed to retrieve pairs from extension server at https://<VR-FQDN>:8043.Failed to connect to HBR Management Server at https://<VR-FQDN>:8043
  
  
  
• Runtime errors as stated below are also observed while trying to access Replication jobs:
  

  A generic error occurred in the vSphere Replication Management Server. Exception details: 'Error creating bean with name 'com.vmware.hms.remote.HmsServerImpl': Unsatisfied dependency expressed

  2025-03-17 04:39:53.936 INFO  com.vmware.hms.i18n.class com.vmware.hms.response.filter.I18nActivationResponseFilter [tcweb-15] (..response.filter.I18nActivationResponseFilter) [operationID=4cc89d25-757a-473c-a2d4-855e85e6bd8d-HMS-14,sessionID=B557BD19] | The localized message is: A runtime error occurred in the vSphere Replication Management Server. Exception details: 'Task java.util.concurrent.FutureTask@4587d1d8[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@1d9e8d77[Wrapped task = com.vmware.jvsl.sessions.net.impl.TlsPreservingWrapper$2@14e60004]] rejected from java.util.concurrent.ThreadPoolExecutor@2bfbffb2[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 49]'.

  2025-03-17 04:39:53.936 INFO  com.vmware.hms.i18n.class com.vmware.hms.response.filter.I18nActivationResponseFilter [tcweb-15] (..response.filter.I18nActivationResponseFilter) [operationID=4cc89d25-757a-473c-a2d4-855e85e6bd8d-HMS-14,sessionID=B557BD19] | The localized message is: A runtime error occurred in the vSphere Replication Management Server. Exception details: 'Task java.util.concurrent.FutureTask@4587d1d8[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@1d9e8d77[Wrapped task = com.vmware.jvsl.sessions.net.impl.TlsPreservingWrapper$2@14e60004]] rejected from java.util.concurrent.ThreadPoolExecutor@2bfbffb2[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 49]'.

  2025-03-17 04:39:53.937 DEBUG com.vmware.hms.vlsi.step.InvocationStep [tcweb-15] (..vlsi.step.InvocationStep) [] | vmomiOp-finish [method: HmsSessionManager.HmsSessionManagerLogin; target: session-manager; user: N/A; client: x.x.x.x:xxxxx; operationID=4cc89d25-757a-473c-a2d4-855e85e6bd8d-HMS-14; sessionID=B557BD19]; time: 17 ms 

• HMS service on target vSphere Replication appliance crashes frequently.
  (Note : This frequency could be as high as once every 2-3 minutes)
  
  Monitored using command : watch "systemctl status hms"
  
  

• /opt/vmware/support/logs/hbrsrv.log reported following errors for these 'Not responding' or 'Disconnected' ESXi hosts in the inventory of target vCenter :

  2025-03-18T03:28:08.206Z error hbrsrv[01320] [Originator@6876 sub=AgentConnection opID=hs-init-245ee9cc] Connection failed to agent host-477562/hostd (10.144.4.122): Can't login to the host
2025-03-18T03:28:18.446Z error hbrsrv[12433] [Originator@6876 sub=AgentConnection opID=hs-init-245ee9cc] Connection failed to agent host-476096/hostd (10.144.4.120): Can't login to the host
2025-03-18T03:28:28.686Z error hbrsrv[01775] [Originator@6876 sub=AgentConnection opID=hs-init-245ee9cc] Connection failed to agent host-158453/hostd (10.144.4.231): Can't login to the host

Environment

vSphere Replication 9.x

Cause

• The issue has been identified as a result of the VR management service crashing, caused by the execution of multiple threads to connect to hosts that are either "Not Responding" or "Disconnected."
  
  
• This leads to the exhaustion of service resources

Resolution

1. Remove 'Not responding' hosts from vCenter inventory. 
   
   
   
2. Restart hms and hbrsrv services using following command:
   # systemctl restart hms
# systemctl restart hbrsrv 
   
   
   
3. Re-pair the two VR appliances using 'Reconnect' option in Site Recovery UI.