Error: No connection to VR Server for virtual machine on host in cluster : Unknown'

Products

VMware Live Recovery

Issue/Introduction

This article addresses replication failures where the vSphere Client reports "No connection to VR Server for virtual machine on host in cluster : Unknown." This typically indicates a network communication failure along the replication data path.

Symptoms :

Replication tasks complete, but synchronization fails.
vCenter UI displays: "No connection to VR Server for virtual machine: Not responding".
Site Recovery UI reports:

"A replication error occurred at the vSphere Replication Server for replication [vm_name]. Details: 'No connection to VR Server for virtual machine [vm_name] on host [esx_name] in cluster [cluster_name]: Unknown"

Environment

VMware Live Site Recovery 9.0.2, 9.0.5.1

Cause

Bidirectional communication on TCP ports 31031 and 32032 is blocked between the source ESXi host, target ESXi host, and the target vSphere Replication (VR) appliance.

From ESXI, var/run/log/hbr-agent.log, gives Failed to bind to any of the specified VMKs

2025-01-30T06:43:32.974Z In(166) hbr-agent-bin[3431566]: [0x0000002296080700] info: [ProxyConnection] Setting up secure tunnel to broker on ##.##.##.## :32032
2025-01-30T06:43:32.974Z In(166) hbr-agent-bin[3431566]: [0x0000002296080700] info: [Proxy [Group: ] -> [######### :32032] Bound to vmk: vmk1 for connection to ######### :32032
2025-01-30T06:43:32.974Z In(166) hbr-agent-bin[3431566]: [0x0000002295e3c700] error: [Proxy [Group: ] -> [#########:32032] Failed to connect to ######### :32032. Using nic 'vmk1'. Error: Network is unreachable
2025-01-30T06:43:32.974Z In(166) hbr-agent-bin[3431566]: [0x0000002295e3c700] error: [Proxy [Group: ] -> [#########:32032] Failed to bind to any of the specified VMKs for connection to ######### :32032
2025-01-30T06:43:32.974Z In(166) hbr-agent-bin[3431566]: [0x0000002295e3c700] error: [Proxy [Group: ] -> [######### :32032] Failed to connect to broker on ######### :32032: Input/output error
2025-01-30T06:43:32.974Z In(166) hbr-agent-bin[3431566]: [0x0000002295e3c700] error: [Proxy [Group: ] -> [######### :32032] Failed to connect to broker: Input/output error

When there is a communication and connectivity issues between ESXi hosts and vSphere replication appliance which is used for replication network, the VM replication failures are expected. If port 31031/32032 is closed or blocked below error appears and VM replication fails.

From ESXi, /var/run/log/vmkwarning.log there is an event with "Connection refused"

YYYY-MM-DDT06:23:37.479Z Wa(180) vmkwarning: cpu31:22158964) WARNING: Hbr: 571: Connection failed to ######### (groupID=GID-#######-####-####-####-############) : Connection refused
YYYY-MM-DDT06:23:37.479Z Wa(180) vmkwarning: cpu31:22158964) WARNING: Hbr: 5362: Failed to establish connection to [##########] :31031 (#######-####-####-####-############) : Connection refused

YYYY-MM-DDT06:23:43 Wa (180) vmkwarning: cpu18:12646058) WARNING: Hbr: 5362: Failed to establish connection to [##########]:32032 (groupID-GID-#######-####-####-####-############) : Broken pipe
YYYY-MM-DDT06:23:43 Wa (180)vinkwarning: cpul7: 12646058) WARNING: Hbr: 5362: Failed to establish connection to [##########]:32032 (groupID=#######-####-####-####-############) Connection refused.
YYYY-MM-DDT06:23:43 Wa (180)vmkwarning: cpu17:12646058)WARNING: Hbr: 788: Failed to receive from ##########(groupID=GID-#######-####-####-####-############) : Broken pipe

Ping fails for port 31031 and 32032

vmkping -I vmk0 -s 1472 ##.##.##.##
PING ##.##.##.## (##.##.##.##) : 1472 data bytes

--- ##.##.##.## ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

[root@ESX01:~ ] nc -zv ######### 31031
nc: connect to ######### port 31031 (tcp) failed: Connection refused[root@ESX01 :~ ] nc -zv ######### 32032
nc: connect to ######### port 32032 (tcp) failed: Connection refused

Resolution

Verify Network Connectivity: Ensure bidirectional communication for TCP ports 31031 and 32032 is allowed between all source ESX hosts, target ESX hosts, and the target VR appliance.
Firewall Configuration: Open ports 31031 and 32032 on all intermediate firewalls for both inbound and outbound traffic.
VAMI Configuration: If a dedicated replication network is in use, ensure the IP Address for Incoming Storage Traffic is correctly configured in the vSphere Replication Appliance VAMI.
Validate Connectivity: Run the following commands from the ESX host CLI:

nc -zv [Target_Host_IP] 32032

nc -zv [Target_VR_IP] 31031

To isolate replication traffic from the management network, you must configure static routes on both the vSphere Replication appliance and the ESX hosts.

5. Restart Services: Restart the HBR agent on the affected ESXi hosts if the ports are confirmed open but the error persists:

For legacy; Restart "hms" service and "hbrsrv" service on Target VR

# systemctl stop hbrsrv ; # systemctl stop hms
# systemctl start hbrsrv; # systemctl start hms

Leave for several minutes .. * Reconfigure VM replication job.

For Enhanced stop/stat hbrsrv and hbr-agent on Source ESX host

# /etc/init.d/hbrsrv stop ........ {fyi: it auto-restarts}
# /etc/init.d/hbr-agent stop
# /etc/init.d/hbr-agent start

Leave for several minutes .. * Reconfigure VM replication job.

If reconfigure fails restart vcenter service

Note: To restart all services, first stop then start all the services:

service-control --stop --all && service-control --start --all

Additional Information

Port numbers that must be open for vSphere Replication

Configure a Static Route on an Additional VM Network Adapter