Enhanced replication mapping test from source to target ESXi hosts fails with Input/Output error.

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• Enhanced replication mapping test from source to target hosts fails with the error "Fault occurred while performing health check. Details: 'Connect: Input/output error."
• A dedicated vmkernel adapter configured for replication on both the source and the target ESXi hosts.
• VMkernel adapter configuration appear as:
  • Source Host:
    VMK     IP              Tagging 
VMK0    ###.##.A.123   Management network
vmk3    ###.##.A.456   vSphereReplicationNFC, vSphereReplication
• • Target Host:
    VMK     IP             Tagging
VMK0    ###.##.B.123  Management network
VMK3    ###.##.B.456  vSphereReplicationNFC, vSphereReplication
    
    
• Verifying the routing table on the source vSphere replication appliance confirms that the static routes are configured from destination (target ESXi replication network) to source esxi replication network gateway. 
  netstat -r 
Kernel IP routing table
Destination             Gateway           Genmask            Flags MSS  Window irtt    Iface
default                 _gateway          0.0.0.0            UG     0   0      0       eth0
###.##.B.###            ###.##.A.###      ###.###.###.###    UG     0   0      0       eth1

Environment

VMware Live Site Recovery 9.x

Cause

• The replication traffic coming from source ESXi host are routed to management vmkernel adapter on the target ESXi host. 
• The target vSphere replication appliance lacked a static route to the replication vmkernel adapter (vmk3) on the target ESXi hosts due to which the mapping failed with Input/Output error.

Cause Validation:

• Review of the /var/run/log/hbr-agent.log file from the source ESXi host confirms that VMK3 on source ESXi is trying to reach management IP (###.##.B.123) on vmk0 on the target host and VMK3 is failing to establish a connection to port 32032 on the target ESXi hosts.
  YYYY-MM-DDTHH:MM.SSSZ In(166) hbr-agent-bin[3085521]: [0x000000f4d2cea700] error: [Proxy [Group: PING-GID-d4f0da32-eda7-4199-9a5c-525faef0d5b9] -> [###.##.B.123:32032]] [########-####-####-####-############] Failed to connect to ###.##.B.123:32032. Using nic 'vmk3'. Error: Connection timed out
YYYY-MM-DDTHH:MM.SSSZ In(166) hbr-agent-bin[3085521]: [0x000000f4d2cea700] error: [Proxy [Group: PING-GID-d4f0da32-eda7-4199-9a5c-525faef0d5b9] -> [###.##.B.123:32032]] [########-####-####-####-############] Failed to bind to any of the specified VMKs for connection to ###.##.B.123:32032
YYYY-MM-DDTHH:MM.SSSZ In(166) hbr-agent-bin[3085521]: [0x000000f4d2cea700] error: [Proxy [Group: PING-GID-d4f0da32-eda7-4199-9a5c-525faef0d5b9] -> [###.##.B.123:32032]] [########-####-####-####-############] Failed to connect to server ###.##.B.123:32032 using broker info: Input/output error
YYYY-MM-DDTHH:MM.SSSZ In(166) hbr-agent-bin[3085521]: [0x000000f4d2dec700] error: [Proxy [Group: PING-GID-d4f0da32-eda7-4199-9a5c-525faef0d5b9] -> [###.##.B.123:32032]] [########-####-####-####-############] Exhausted all server endpoints reported by broker.
• Verifying the routing table on the target vSphere replication appliance confirms that the static routes are not configured from destination (source ESXi replication network) to target esxi replication network gateway. Therefore, the target vSphere replication appliance routing the replication traffic coming from source host to management vmkernel adapter (vmk0) on the target host and the connection is getting rejected. 
  netstat -r 
Kernel IP routing table
Destination             Gateway           Genmask            Flags MSS  Window irtt    Iface
default                 _gateway          0.0.0.0            UG     0   0      0       eth0

Resolution

• Log in to the target vSphere replication appliance via SSH and navigate to the network directory:
  cd /etc/systemd/network/
• List the contents of the directory:
  ls -l
  Files appear as eth0.network and eth1.network.
• Take a backup of the respective network file by copying it:
  example: cp 10-eth1.network 10-eth1.networkduplicate
• Edit the file 
  vi 10-eth1.network
• Press "i" to enter into insert mode. Update the Gateway and Destination under "[Route]" section:
  Example:
  

  [Match]
Name=ethl
[Network]
Gateway=###.##.##.###
Address=###.##.##.###/##
DHCP=no
Domains=####.###.###.##
[DHCP]
UseDNS=false

  [Route]
Gateway=###.##.##.###
Destination=###.##.##.###/##

• Restart "systemd-networkd.service" on the target vSphere replication appliance. 
  systemctl restart systemd-networkd.service
• Restart the hbrsrv and hbr-agent services on source and destination ESXi hosts:
  /etc/init.d/hbr-agent restart
/etc/init.d/hbrsrv restart