Enhanced replication mapping failed with an error " Fault occured while performing health check. Details :Connect :Input/ouput error".

Products

• VMware Live Recovery

Issue/Introduction

Symptoms :

    Enhanced replication failed with an error below in vSphere--> Open site Recovery -->Site recovery  --> Enhanced replication mappings. 

     

     

Validation Steps: 

• Verify that vSphere Replication services are enabled on only one vmkernel adapter, as required.

  
  This can be checked by navigating to: Hosts > Configure > Networking > VMkernel adapters

• Confirm that the hbr-agent service is updated and running on the ESXi host:
  
  To check the service status navigate to:
  Hosts > Configure > System > Services > hbr-agent
• To verify the version of the hbr VIB installed, run the following command on the ESXi host:
  
  esxcli software vib list | grep hbr
  
  Sample output:
  esxcli software vib list | grep hbr
vmware-hbr-agent               8.0.3-0.0.24299506                    VMware  VMwareCertified   2025-01-02    host
vmware-hbrsrv                  8.0.3-0.0.23305546                    VMware  VMwareCertified   2025-01-02    host


• Validate hbrsrv-agent.log from esxi host and found below error 
  
  Log path :  /var/run/log/hbr-agent.log

2025-05-09T04:59:13.128Z In(166) hbr-agent-bin[2100295]: [0x0000004f82cf4700] info: [Proxy [Group: ] -> [1##.##.7.##:32032]] Bound to vmk: vmk3 for connection to 172.##.7.##:32032
2025-05-09T04:59:43.296Z In(166) hbr-agent-bin[2100295]: [0x0000004f82d75700] info: [ConfigManager] No user configuration for key=hbrsvc_target_info in ConfigStore.
2025-05-09T04:59:43.296Z In(166) hbr-agent-bin[2100295]: [0x0000004f82d75700] error: [ConfigManager] Failed to get config store object. Comp: esx, Grp: services, Key: hbrsvc_target_info, Id: 172.##.7.63, Prop: certificate
2025-05-09T04:59:43.296Z In(166) hbr-agent-bin[2100295]: [0x0000004f82d75700] info: [ProxyConnection] Setting up secure tunnel to broker on 1##.##.7.##:32032
2025-05-09T04:59:43.296Z In(166) hbr-agent-bin[2101419] : [0x000000c6ceedd700] error: [Proxy [Group: PING-GID-92xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] -> [10.##.##.##: 32032]] [d####-###-4da4-###-#######] Failed to connect to 10.##.##.##:32032. Using nic 'vmkl', Error: Connection timed out

Environment

VMware vSphere ESXi 8.x 

VMware vSphere Replication 9.x 

Cause

• The ESXi host is unable to connect to the target ESXi host on TCP port 32032, which blocks communication between the hosts and causes replication mapping errors during enhanced replication mapping test.

Cause Validation 

• Confirm that port 32032 is open bidirectionally on the affected ESXi host by running nc -zv <target host IP> 32032


•  vmkping does not work from the source host to the destination host, indicating no connectivity over port 32032
  
   root [esxi01] -vmkping -I vmk1 [target host IP]  -d -s 1472
 PING 1##.20.#.## (1##.##.7.##): 1472 data bytes
 sendto() failed (Network is unreachable)
 root[esxi01] nc -zv [target host IP] 32032
 nc: connect to 1##.##.3.#1 port 32032 (tcp) failed: Connection refused
   
• Check for connectivity error in hms.log 

          Log path : less /opt/vmware/hms/logs/ hms.log

                 (..hbrsrvuw.healthmonitor.HealthChecksWorkflow) [operationID=###-###-4cb2-####-############-HMS-237,sessionID=####,
         operationID=####-7048-###-###-####-HMS-237,sessionID=###] | Ping test for peer site '###-##-4906-b457-###' and               replication mapping '(hms.ReplicationMapping) {
         dynamicType = null,
         dynamicProperty = null,
         localClusterMoId = domain-###34,
         peerClusterMoId = domain-##
         }' completed.
         }' is: '{"group":"PING-GID-#######-899a-#####-###-#####","endpoints":{"broker":                        {"address":"##.##.##.##","port":32032,"connectivity": {"tcp":false,"ssl":false},"latency":{"tcp":   {"value":##,"units":"us"}},"failReason":"Connect: Input/output error"},"targets":[]},"sourceHostId":"host-###","sourceHostName":"###-###.###.###","targetHostId":"host-112","targetHostName":"###-##.idsco###.##"}'.

Resolution

• Enhanced replications require TCP network connectivity on ports 31031 and 32032 from the ESXi hosts on which the replicated VMs are running to the ESXi hosts of the cluster containing the target datastore. Make sure firewall settings are adjusted accordingly.
  
  
• Ensure below ports are open by following the document 
  
  Services, Ports, and External Interfaces that the  vSphere Replication Virtual Appliance Uses
  
  

• Perform a traceroute from source ESXI to target ESXI and ensure a complete route. If not route is available : Create a static route on the ESXI host

Additional Information

Related KB : 379468