vSphere Replication cannot establish a TCP connection to server at 127.0.0.1:8123 - Connection refused

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• vSphere Replication events in the vCenter client fails
• You see the error:

ERROR
Operation Failed
Cannot establish a TCP connection to server at '10.X.X.X:8123'. Details: 'https://10.X.X.X:8123/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to 10.X.X.X:8123 [/10.X.X.X] failed: Connection refused (Connection refused)"'.
7/16/23, 12:32:32 AM +0530

• In the VRM(HMS) server logs, located at /opt/vmware/hms/logs/hms*.log, you see entries similar to:
  
  ---
  <YYYY-MM-DD><time> ERROR hms.net.hbr.ping.svr.52708a4f-e992-0321-91ee-1cbaf1d5ff8f [Ping Thread for server 127.0.0.1:8123] (..net.impl.PersistentConnection) | Ping for server 127.0.0.1:8123 failed: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connection to https://192.168.2.100:8123 refused : org.apache.http.conn.HttpHostConnectException: Connection to https://192.168.2.100:8123 refused.
  ---
   
• In the VR (HBR) server logs, located at /var/log/vmware/hbrsrv*.log, you see entries similar to:
  
  ---
  <YYYY-MM-DD><time> [7FF0C3378700 info 'HostCreds' opID=hs-init-4f74cc43] Ignoring link-local address for host-17: "fe80::20a:f7ff:fe36:c8f0"
  :
  <YYYY-MM-DD><time> [7FF0C3378700 info 'HostCreds' opID=hs-init-4f74cc43] Ignoring link-local address for host-826: "169.254.95.101"
  ---</time></time>

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere Replication 8.x

Cause

This issue occurs when one or more ESX servers registered in the vSphere Replication database has an IPv4 or IPv6 Link Local IP address.

 

IPv4 Link Local address is defined in the range 169.254.0.0/16 and IPv6 Link Local address is assigned with the fe80::/10 prefix.

Resolution

The issue can be avoided in vSphere Replication 8.7 by adding com.vmware.vr.disallowed Host tag in the ESXi host. 

To work around this issue:

Try these common steps first. 

1. Reconnect hosts in unresponsive state or remove them from vCenter inventory 
2. Power OFF & Power ON VR appliance (Don't shutdown)
3. Reboot vCenter 

1. Take a snapshot of the VR appliance you intend to apply this fix on. 
2. systemctl stop hbrsrv 
3. Backup the latest hbrsrv.***.db (Where * represents the highest DB number)

root@vrmspr [ ~ ]# cd /etc/vmware 

root@vrmspr [ /etc/vmware ]# mkdir backup 
root@vrmspr [ /etc/vmware ]# cp hbrsrv.***.db /backup 
root@vrmspr [ /etc/vmware ]# sqlite3 hbrsrv.***.db (Login to the latest hbrsrv DB)

root@vrmspr [ /etc/vmware ]# sqlite3 hbrsrv.100.db
SQLite version 3.22.0 2024-02-02 18:45:57
Enter ".help" for usage hints.
sqlite> select * from hostinfo; (Displays the contents of hostinfo table)
sqlite> delete from hostinfo;   (Deletes the contents of hostinfo table)

The commands below can be used if you'd like to remove a specific host IP/s from the table instead of clearing the entire table. 

sqlite3 hbrsrv.100.db "DELETE FROM HostInfo WHERE addresses in ('xx');"
sqlite3 hbrsrv.100.db "DELETE FROM HostInfo WHERE addresses in ('<ip1>', '<ip2>');"
sqlite3 hbrsrv.100.db "DELETE from HostInfo where addresses in ('192.X.X.X','192.X.X.X','192.X.X.X','192.X.X.X');"

4. systemctl start hbrsrv 
5. Reboot the appliance