vSphere Replication Goes into "Not Active" State with Connection Timeout Errors Due to Physical Routing Misconfigurations and Non-Uniform LAG Settings
search cancel

vSphere Replication Goes into "Not Active" State with Connection Timeout Errors Due to Physical Routing Misconfigurations and Non-Uniform LAG Settings

book

Article ID: 444037

calendar_today

Updated On:

Products

VMware Site Recovery Manager

Issue/Introduction

Symptoms:

When using vSphere Replication version 9.0.5, replication tasks occasionally fail across various virtual machines and hosts within a cluster. The following symptoms are observed:

  • Affected virtual machines enter a Not Active state.

  • The Site Recovery client displays the following error for impacted VM replications:

    A replication error occurred at the vSphere Replication Server for replication '<vm-name>'. Details: 'No connection to VR Server for virtual machine <vm-name> on host <host-fqdn> in cluster <cluster-name> in <datacenter>: Unknown'.

  • During recovery plan execution, VMs fail to recover because synchronization cannot complete.

  • The issue is intermittent and not isolated to specific VMs or hosts; it occurs randomly across different workloads.

  • Enhanced replication mappings show error states such as:

    There is no network connectivity between the source host (id: 'host-#', name: '<source-host fqdn>') and the target broker '<target VLR IP>'. Details: 'Connect: Input/output error'.
    There is no network connectivity between the source host (id: 'host-#', name: '<source-host fqdn>') and the target broker '<target VLR IP>'. Details: '401 Unauthorized from GET https://<target host IP>/hbragent/api/v1.1/appPing'."

     

Environment

VMware Live Recovery 9.x

Cause

This issue is caused by a dual network misconfiguration across both the physical and virtual infrastructure:

  1. Physical Switch Routing Misconfiguration: An incorrect IP routing table configuration on the physical network switches blocks Layer 3 traffic between the source environment and the target Virtual Landscape Recovery (VLR) appliance.

  2. Non-Uniform Host Uplink Configurations (LAG Asymmetry): There is inconsistent uplink configuration across ESXi hosts within the same cluster and Distributed Virtual Switch (DVS). Some hosts utilize Link Aggregation Groups (LAG), while others rely on individual standard uplinks (vmnics). Because a DVS cannot concurrently support a mix of LAG and standard uplinks on the same distributed port group, hosts without LAG configurations drop packets and fail to reach their default gateways.

Cause Validation:

1. Network Connectivity & Ping Failures

Attempts to ping the target VLR appliance from the source ESXi host via vmkping fail entirely:

vmkping -I vmk3 -s 1472 10.#.#.#
PING 10.#.#.# (10.#.#.#): 1472 data bytes
--- 10.#.#.# ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

2. Host Log Analysis

Review of /var/run/log/hbr-agent.log on the source ESXi hosts reveals connection timeouts to the target VLR on port 32032:

2026-05-21T11:13:14.640Z In(166) hbr-agent-bin[91725930]: [0x0000003d29640700] info: [Proxy [Group: PING-GID-bad9f05b-####-####-####-############] -> [10.#.#.#:32032]] [948eb2d3-####-####-####-############] Bound to vmk: vmk3 for connection to 10.#.#.#:32032
2026-05-21T11:14:29.649Z In(166) hbr-agent-bin[91725930]: [0x0000003d295bf700] error: [Proxy [Group: PING-GID-bad9f05b-####-####-####-############] -> [10.#.#.#:32032]] [948eb2d3-####-####-####-############] Failed to connect to 10.#.#.#:32032. Using nic 'vmk3'. Error: Connection timed out
2026-05-21T11:14:29.649Z In(166) hbr-agent-bin[91725930]: [0x0000003d295bf700] error: [Proxy [Group: PING-GID-bad9f05b-####-####-####-############] -> [10.#.#.#:32032]] [948eb2d3-####-####-####-############] Failed to bind to any of the specified VMKs for connection to 10.#.#.#:32032
2026-05-21T11:14:29.649Z In(166) hbr-agent-bin[91725930]: [0x0000003d295bf700] error: [Proxy [Group: PING-GID-bad9f05b-####-####-####-############] -> [10.#.#.#:32032]] [948eb2d3-####-####-####-############] Failed to connect to broker on 10.#.#.#:32032: Input/output error
2026-05-21T11:14:29.649Z In(166) hbr-agent-bin[91725930]: [0x0000003d295bf700] error: [Proxy [Group: PING-GID-bad9f05b-####-####-####-############] -> [10.#.#.#:32032]] [948eb2d3-####-####-####-############] Failed to connect to broker: Input/output error

3. Traceroute Diagnostics

Traceroute results isolate the two distinct points of failure:

  • Physical Route Failure (Hosts with LAG): Traffic leaves the host but drops consistently after a specific physical network hop (10.25.#.#), confirming a core routing table issue.

     
    traceroute -n 10.#.#.# -p 32032 -s 10.#.#.#
    traceroute to 10.#.#.# (10.#.#.#), 30 hops max, 60 byte packets
     1  10.#.#.29  0.245 ms  0.218 ms  0.217 ms
     2  10.#.#.99  0.196 ms  0.202 ms  0.233 ms
     3  10.#.#.108  0.226 ms  0.224 ms  0.220 ms
     4  10.25.#.#  0.183 ms  0.194 ms  0.220 ms
     5  * * *
     6  * * *
     7  * * *
     8  * * *
     9  * * *
    10  * * *
    11  * * *
    12  * * *
    13  * * *
    14  * * *
    15  * * *
    16  * * *
    17  * * *
    18  * * *
    19  * * *
    20  * * *
    21  * * *
    22  * * *
    23  * * *
    24  * * *
    25  * * *
    26  * * *
    27  * * *
    28  * * *
    29  * * *
    30  * * *

  • Local Network Failure (Hosts without LAG): For hosts missing the LAG configuration, traffic fails immediately at the first hop, meaning the host cannot communicate with its own default gateway.

     
    traceroute -n 10.#.#.# -p 32032 -s 10.#.#.#
    traceroute to 10.#.#.# (10.#.#.#), 30 hops max, 60 byte packets
     1  * * *
     2  * * *
     3  * * *
     4  * * *
     5  * * *
     6  * * *
     7  * * *
     8  * * *
     9  * * *
    10  * * *
    11  * * *
    12  * * *
    13  * * *
    14  * * *
    15  * * *
    16  * * *
    17  * * *
    18  * * *
    19  * * *
    20  * * *
    21  * * *
    22  * * *
    23  * * *
    24  * * *
    25  * * *
    26  * * *
    27  * * *
    28  * * *
    29  * * *
    30  * * *

Resolution

To resolve the replication dropouts, both the physical network routing and virtual switch configurations must be remediated.

Step 1: Correct Physical Switch Routing

  1. Engage your Physical Network Team to review the IP routing tables on the upstream switches.

  2. Correct the missing or broken routes to guarantee a clean Layer 3 path between the source ESXi cluster and the target VLR appliance network.

  3. Verify the fix by running a traceroute from a properly configured host to ensure it reaches the target destination successfully.

Step 2: Standardize ESXi Host Uplink Configurations

  1. Audit all ESXi hosts within the affected cluster to identify which hosts are missing the LAG configuration.

  2. Standardize the Distributed Virtual Switch (DVS) uplink configurations across all cluster members.

    • Note: A distributed port group cannot mix active LAGs and standalone vmnics natively without causing asymmetric packet loss.

  3. Reconfigure the mismatched hosts to use LAG uniformly, or migrate the entire cluster to a unified standalone uplink strategy depending on your organizational standards.

Step 3: Verify and Resume Replication

  1. Navigate to the Site Recovery Client and rerun the replication mapping tests.

  2. Ensure that the mapping statuses transition out of the Error state.

  3. Once the networking is uniform and physical routing is restored, replication traffic will automatically resume, allowing initial and delta synchronizations to finish successfully.

Powered by Wolken Software