Enhanced Replication Mappings report error : "Fault occurred while performing health check. Details: 'Connect: Input/output error'."
search cancel

Enhanced Replication Mappings report error : "Fault occurred while performing health check. Details: 'Connect: Input/output error'."

book

Article ID: 387837

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

Symptoms:

  • Enhanced Replication Mapping in Site Recovery UI reports following error:
    "Fault occurred while performing health check. Details: 'Connect: Input/output error'."

  • Replication traffic may fail in the VMware Live Recovery Appliance environment with error messages indicating "Connection refused", "Connection Timed Out" or "Broken pipe" when attempting to connect to the target server on TCP port 32032.

  • Enhanced replication for VMs fails with error:

    "A replication error occurred at the vSphere Replication Server for replication '####'. Details: 'No connection to VR Server for virtual machine #### on host ### in cluster #### in ####: Unknown'"
  • hbr-agent.log (on source ESXi host) reports Input/output error while trying to connect to a target ESXi vmkernel adapter over port 32032:

    • (in /var/run/log/hbr-agent.log)

      YYYY-MM-DDTHH:mm:ss.sssZ In(166) hbr-agent-bin[2172222]: [0x000000e886cf1700] info: [Proxy [Group: GID-#######-a742-####-8f01-#######a3a8b] -> [##.##.##.#6:32032]] Setting up secure tunnel to brokered server ##.##.##.##:32032 (2 of 2)
      YYYY-MM-DDTHH:mm:ss.sssZ In(166) hbr-agent-bin[2172222]: [0x000000e886cf1700] info: [Proxy [Group: GID-#######-a742-####-8f01-#######a3a8b] -> [##.##.##.#2:32032]] Bound to vmk: vmk## for connection to ##.##.##.##:32032
      YYYY-MM-DDTHH:mm:ss.sssZ In(166) hbr-agent-bin[2172222]: [0x000000e886f80700] error: [Proxy [Group: GID-#######-a742-####-8f01-#######a3a8b] -> [##.##.##.#3:32032]] Failed to connect to ##.##.##.##:32032. Using nic 'vmk##'. Error: Connection timed out
      YYYY-MM-DDTHH:mm:ss.sssZ In(166) hbr-agent-bin[2172222]: [0x000000e886f80700] error: [Proxy [Group: GID-#######-a742-####-8f01-#######a3a8b] -> [##.##.##.#3:32032]] Failed to bind to any of the specified VMKs for connection to ##.##.##.##:32032
      YYYY-MM-DDTHH:mm:ss.sssZ In(166) hbr-agent-bin[2172222]: [0x000000e886f80700] error: [Proxy [Group: GID-#######-a742-####-8f01-#######a3a8b] -> [##.##.##.#3:32032]] Failed to connect to server ##.##.##.##:32032 using broker info: Input/output error
      YYYY-MM-DDTHH:mm:ss.sssZ In(166) hbr-agent-bin[2172222]: [0x000000e886f80700] error: [Proxy [Group: GID-#######-a742-####-8f01-#######a3a8b] -> [##.##.##.#3:32032]] Exhausted all server endpoints reported by broker.

  • Validating via SSH from the source ESXi host confirms that port 32032 to the target ESXi host's replication IP is closed:

    root@esx_source~] nc -zv ##.##.##.## 32032
    nc: connect to ###.###.###.### port 32032 (tcp) failed: Connection refused

Environment

VMware vSphere Replication 8.x/9.x
VMware Live Recovery Appliance 

Cause

  • Either multiple vmkernel adapters on target ESXi hosts is enabled with the vSphere Replication and vSphere Replication NFC services or incorrect vmkernel adapter is enabled with these services
  • Network issues between the source and target host (such as unconfigured routes, port 32032 blocked, or an MTU size mismatch causing packet fragmentation/drops).
  • If network issues are resolved and the connection is still failing, replication fails due to application-layer and service-state lockups on the ESXi hosts during the Lightweight Delta (LWD) handshake. Specifically, previous network drops or storage latency leave lingering locks on the target host’s internal hbrsrv SQLite tracking database (SQLite Error 14), while stale Group ID (GID) session contexts in the source host’s hbr-agent lead to protocol-level payload rejections. Consequently, the target host dynamically closes firewall rule hbrsrv-lwds (TCP port 32032) or encounters 30-second I/O timeouts when freezing delta logs on latent target datastores, surfacing as an Input/output error.

Resolution

  1. Verify network connectivity between DC and DR ESXi sites on TCP port 32032. Ensure firewalls are not blocking this traffic.

  2. On the destination ESXi host, navigate to the networking configuration and ensure the "vSphere Replication" traffic type is enabled on the VMkernel port.

  3. If multiple adapters are enabled for these services, disable them on adapters where replication traffic is not intended to flow.

  4. Check for MTU size mismatches across the network path.

  5. Run the 'Run Tests' option under Enhanced Replication Mapping in the Site Recovery UI to validate the connection.

After fixing the network issue if still face the issue follow the following:

Step 1: Service Restarts
  • VR Appliances: Restart hms and hbrsrv services on both source and target appliances.
  • ESXi Hosts: Restart hbr-agent, envoy, and hbrsrv services on all DC and DR ESXi hosts.
Step 2: Clear hbrsrv Database (If issue persists)
  1. SSH into the Replication appliance and go to /etc/vmware/.
  2. Find and back up the latest database: hbrsrv.X.db.
  3. Stop service: service hbrsrv stop
  4. Clear table & verify:
    • sqlite3 /etc/vmware/hbrsrv.X.db "DELETE FROM HostInfo;"
    • sqlite3 /etc/vmware/hbrsrv.X.db "SELECT * FROM HostInfo;"
  5. Start service: service hbrsrv start

Additional Information

KB reference exhibiting similar connection or replication errors, refer to this article:

Powered by Wolken Software