Error: No connection to VR Server for virtual machine on host in cluster : Unknown'. while configuring replication for VM

Products

VMware Live Recovery

Issue/Introduction

All the connectivity between vCenter, SRM and vSphere replication servers look healthy and connected.

When enable the replication, it completes the task and gives an error on UI during the sync.

There is a dedicated vmkernel adapter on hosts for vSphere replication network and also vSphere replication appliance has a dedicated network card and adapter.

Environment

VMware vSphere Replication 8.x

VMware vSphere Replication 9.x

Cause

It is observed that the replication was failing with connectivity issues. "Failed to bind to any of the specified VMKs for connection to"

In this scenario, the management network is on 162.x.x.x series IP and dedicated network is 192.x.x.x series IP.

This can be verified on ESXi host logs under path /var/run/log/hbr-agent.log and the below events can be found.

2025-01-30T06:34:32.967Z In(166) hbr-agent-bin[3431566]: [0x0000002295ebd700] error: [ConfigManager] Failed to get config store object. Comp: esx, Grp: services, Key: hbrsvc_target_info, Id: 162.xx.xx.xx, Prop: certificate
2025-01-30T06:34:32.967Z In(166) hbr-agent-bin[3431566]: [0x0000002295ebd700] info: [ProxyConnection] Setting up secure tunnel to broker on 162.xx.xx.xx:32032
2025-01-30T06:34:32.967Z In(166) hbr-agent-bin[3431566]: [0x0000002295ebd700] info: [Proxy [Group: ] -> [162.xx.xx.xx:32032]] Bound to vmk: vmk1 for connection to 162.xx.xx.xx:32032
2025-01-30T06:34:32.967Z In(166) hbr-agent-bin[3431566]: [0x0000002295e3c700] error: [Proxy [Group: ] -> [162.xx.xx.xx:32032]] Failed to connect to 162.xx.xx.xx:32032. Using nic 'vmk1'. Error: Network is unreachable

2025-01-30T06:43:32.974Z In(166) hbr-agent-bin[3431566]: [0x0000002296080700] info: [ProxyConnection] Setting up secure tunnel to broker on 162.xx.xx.xx:32032
2025-01-30T06:43:32.974Z In(166) hbr-agent-bin[3431566]: [0x0000002296080700] info: [Proxy [Group: ] -> [162.xx.xx.xx:32032]] Bound to vmk: vmk1 for connection to 162.xx.xx.xx:32032
2025-01-30T06:43:32.974Z In(166) hbr-agent-bin[3431566]: [0x0000002295e3c700] error: [Proxy [Group: ] -> [162.xx.xx.xx:32032]] Failed to connect to 162.xx.xx.xx:32032. Using nic 'vmk1'. Error: Network is unreachable
2025-01-30T06:43:32.974Z In(166) hbr-agent-bin[3431566]: [0x0000002295e3c700] error: [Proxy [Group: ] -> [162.xx.xx.xx:32032]] Failed to bind to any of the specified VMKs for connection to 162.xx.xx.xx:32032
2025-01-30T06:43:32.974Z In(166) hbr-agent-bin[3431566]: [0x0000002295e3c700] error: [Proxy [Group: ] -> [162.xx.xx.xx:32032]] Failed to connect to broker on 162.xx.xx.xx:32032: Input/output error
2025-01-30T06:43:32.974Z In(166) hbr-agent-bin[3431566]: [0x0000002295e3c700] error: [Proxy [Group: ] -> [162.xx.xx.xx:32032]] Failed to connect to broker: Input/output error

When there is a communication and connectivity issues between ESXi hosts and vSphere replication appliance which is used for replication network, the VM replication failures are expected.

Resolution

When there is a dedicated network configured for vSphere replication from both ESXi hosts and vSphere replication appliance, we need to configure the "IP Address for Incoming Storage Traffic" to enable the communication to work with defined/dedicated network. Refer the below link for instructions.

Create a VM Network Adapter to Use for Incoming Replication Traffic on the Combined vSphere Replication Appliance

When there is a dedicated network configured for replication purpose and this network not to be interrupted from management network, it is required to set the "IP Address for Incoming Storage Traffic" on appliance to address the issue.

After making the change we see the traffic established using dedicated network adapter and sync started successfully.

This can be verified on ESXi host logs under path /var/run/log/hbr-agent.log and the below events can be found.

2025-01-30T06:43:38.474Z In(166) hbr-agent-bin[3431566]: [0x0000002295ebd700] info: [Proxy [Group: GID-9d365f20-1808-47ec-881d-92373b3451c4] -> [192.xx.xx.xx:32032]] Setting up secure tunnel to server on 192.xx.xx.xx:32032
2025-01-30T06:43:38.474Z In(166) hbr-agent-bin[3431566]: [0x0000002295ebd700] info: [Proxy [Group: GID-9d365f20-1808-47ec-881d-92373b3451c4] -> [192.xx.xx.xx:32032]] Bound to vmk: vmk1 for connection to 192.xx.xx.xx:32032