vSphere Replication Management Server could not establish connection to vSphere Replication Server at '127.0.0.1:8123'

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• • The SRM console displays the following error: Error: vsr-sdr : vSphere Replication Management Server could not establish connection to vSphere Replication Server at '127.0.0.1:8123'
  • Recent tasks show: Reconnect vSphere Replication Server | #### | The object 'vim.VirtualMachine:vm-####' has already been deleted or has not been completely created.
  • The Replication server tab under the SRM Site Pair shows the status as Disconnected.
  • The hbrsrv.log indicates that an SSL thumbprint mismatch or initialization error is causing HMS login to fail: warning hbrsrv[####] [Originator@6876 sub=Main opID=####-HMS-PING] [0] Invalid login warning hbrsrv[####] [Originator@6876 sub=Main opID=####-HMS-PING] [1] Unknown client SSL thumbprint ####
  • Database object not found and Couldn't read cached value for key guestinfo.hbr.hbrsrv-certificate-revoked

• Recent tasks show:

Reconnect vSphere Replication Server | vCenter_FQDN_Name | The object'vim. VirtualMachine:vm-####' has already been deleted or has not been completely created.

• In the Replication server tab under the SRM Site Pair, the Replication server shows as Disconnected.

• The hbrsrv.log indicates that the SSL thumbprint mismatch is causing HMS login to fail.

YYYY-MM-DDThh:mm:ss.ms.094+07:00 warning hbrsrv[01004] [Originator@6876 sub=Main opID=####-####-####-####-####-HMS-PING---Ping Thread for session key: N/A and vmomi session: null and server: 127.0.0.1:8123] HbrError stack:
YYYY-MM-DDThh:mm:ss.ms.094+07:00 warning hbrsrv[01004] [Originator@6876 sub=Main opID=####-####-####-####-####-HMS-PING---Ping Thread for session key: N/A and vmomi session: null and server: 127.0.0.1:8123]    [0] Invalid login
YYYY-MM-DDThh:mm:ss.ms.094+07:00 warning hbrsrv[01004] [Originator@6876 sub=Main opID=########-####-####-####-####-HMS-PING---Ping Thread for session key: N/A and vmomi session: null and server: 127.0.0.1:8123]    [1] Unknown client SSL thumbprint 06:FE:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:E1
YYYY-MM-DDThh:mm:ss.ms.094+07:00 warning hbrsrv[01004] [Originator@6876 sub=Main opID=e4601195-0405-431d-85aa-5aebd7671725-HMS-PING---Ping Thread for session key: N/A and vmomi session: null and server: 127.0.0.1:8123]    [2] Error converted to Vmomi fault hbr.replica.fault.InvalidLogin

• The SSL thumbprint used by hbrsrv does not match the one configured for hms.

• Run the following command to compare the thumbprint between hbrsrv and HMS.

root [ /opt/vmware/support/logs ]# /usr/bin/hbrsrv-guestinfo.sh get guestinfo.hbr.hbrsrv-thumbprint
06:FE:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:E1

root [ /opt/vmware/support/logs ]# /usr/bin/hbrsrv-guestinfo.sh get guestinfo.hbr.hms-thumbprint
6F:4F:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:####:E1

Environment

VMware Site Recovery Manager 8.x

VMware Live Site Recovery 9.x

 

Cause

• This issue can occur due to one of the following reasons:

  1. Trust Initialization Failure: The hbrsrv (Replication Service) trust settings are not fully initialized in memory, even if SSL thumbprints match at the OS level.
  2. Thumbprint Mismatch: A certificate change on the VR appliance has caused a mismatch between the hms and hbrsrv services.
  3. Stale Host Data: ESXi hosts with link-local IP addresses (169.254.x.x) or stale entries in the HostInfo database table are preventing the service from starting correctly.

Resolution

Follow these steps to resolve the connection issue:

1. Log in to the vSphere Replication appliance via SSH as root.
2. Restart the hbrsrv service to force the process to load verified thumbprint configurations into memory: systemctl restart hbrsrv
3. Wait 2–5 minutes and check the Site Recovery UI to see if the server status returns to Connected.
4. If the service restart does not resolve the issue, reboot the affected vSphere Replication appliance(s) on both sites to re-initialize all management services.
5. If the status remains Disconnected, verify the SSL thumbprints:
   • Run /usr/bin/hbrsrv-guestinfo.sh get guestinfo.hbr.hbrsrv-thumbprint
   • Run /usr/bin/hbrsrv-guestinfo.sh get guestinfo.hbr.hms-thumbprint
   • If they do not match, update the HMS thumbprint to match the HBR thumbprint: /usr/bin/hbrsrv-guestinfo.sh set guestinfo.hbr.hms-thumbprint [CORRECT_THUMBPRINT]
   • Restart the service again: systemctl restart hbrsrv

If you require a software update or patch to resolve persistent connectivity issues, please see Download Broadcom Products and Software.

Additional Information

Troubleshooting a disconnected vSphere Replication Appliance OR vSphere replication server randomly connects and disconnects in SRM UI

To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.