Failure to Isolate vSphere Replication Traffic on custom TCP/IP Stack

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Enhanced Replication Mapping in Site Recovery UI reports following error:
  "Fault occurred while performing health check. Details: 'Connect: Input/output error'."

       

• Enhanced replication fails for VMs failed with
  "A replication error occurred at the vSphere Replication Server for replication '###'. Details: 'No connection to VR Server for virtual machine ### on host ### in cluster ### in ###: Unknown'"


• Replication traffic was configured to utilize a customized TCP/IP stack, with vmkX designated as the VMkernel adapter for vSphere Replication traffic
  

• Port connectivity from source ESXi host to destination broker IP fails with error: “bind failed: Cannot assign requested address.”
# nc -z [Target-VRMS IP/FQDN] 32032 -s [designated VMK IP for vsphere replication traffic]

       

• Validate the "Failed to bind" errors and "Connection reset" messages in /var/run/log/hbr-agent.log.

  In(166) hbr-agent-bin[2103047] [0x00000055ed71b700] info: [ProxyConnection] Setting up secure tunnel to broker on [replication_broker_ip]:32032
 In(166) hbr-agent-bin[2103047] [0x00000055ed71b700] error: [Proxy [Group: ] -> [replication_broker_ip:32032]] Failed to bind to vmkX. Error:22
 In(166) hbr-agent-bin[2103047] [0x00000055ed71b700] error: [Proxy [Group: ] -> [replication_broker_ip:32032]] Failed to bind to any of the specified VMKs for connection to replication_broker_ip:32032
 In(166) hbr-agent-bin[2103047] [0x00000055ed79c700] error: [Proxy [Group: ] -> [replication_broker_ip:32032]] Failed to connect to broker on replication_broker_ip:32032: Input/output error
 In(166) hbr-agent-bin[2103047] [0x00000055ed79c700] error: [Proxy [Group: ] -> [replication_broker_ip:32032]] Failed to connect to broker: Input/output error
 In(166) hbr-agent-bin[2103047] [0x00000055ed79c700] info: [ConfigManager] No user configuration for key=hbrsvc_target_info in ConfigStore.
 In(166) hbr-agent-bin[2103047] [0x00000055ed79c700] error: [ConfigManager] Failed to get config store object. Comp: esx, Grp: services, Key: hbrsvc_target_info, Id: replication_broker_ip, Prop: certificate

Environment

VMware Live Recovery 9.X

Cause

The issue is caused by the lack of support for enabling or routing vSphere Replication traffic over a custom TCP/IP stack on ESXi hosts. vSphere Replication communication is supported exclusively over the default TCP/IP stack. Consequently, attempts to bind replication traffic to a VMkernel adapter associated with a custom stack result in connectivity failures during health checks and replication broker communication.

Resolution

vSphere Replication traffic isolation can be achieved through the following methods

1.Tagging a VMkernel adapter on the default TCP/IP stack for vSphere Replication traffic 

2. Configuring static routes to direct replication traffic through the intended network path.

Additional Information

Refer the Broadcom tech document for more information on isoltaing vsphere replication traffic :

Isolating the Network Traffic of vSphere Replication