Application HA Failover During vMotion Due to NSX Overlay BFD Tunnel Convergence Delay
search cancel

Application HA Failover During vMotion Due to NSX Overlay BFD Tunnel Convergence Delay

book

Article ID: 445794

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • A Solace workload VM connected to NSX Overlay Segment experiences an HA failover/cluster election due to missed heartbeats immediately following a vMotion.

  • vMotion execution completes, and datapath traffic is resumed on the destination host port. However, the overlay topology's BFD session does not transition to UP for several seconds. During this convergence gap, the guest application encounters a heartbeat timeout and initiates an election.

    ESXi logs (vmkernel.log):

    06:25:08.242Z vdl2: VDL2PortOutputUplinkChangeCB:1481 - Output Uplink change ignored for portID: 4#####b  ← VM port activated

06:25:16.742Z vdl2: VDL2BFDCheckAndUpdateSessionMac:1545 - BFD add remote gateway for 192.##.##.21 ← first VTEP session
06:25:16.742Z BFD_CreateSession:440 - local: 192.##.##.42, remote: 192.##.##.37, type: overlay ← critical session
06:25:16.908Z BFD_HandleStatusChange:851 - local: 192.##.##.42, remote: 192.##.##.37, oldState: down, newState: up

    The 8 seconds gap is measured between the time when vMotion completes (06:25:08) and when BFD session comes up(06:25:16).

  • The Solace VM kernel.log shows a very aggressive heartbeat interval. It takes just three seconds (e.g., 06:25:09 to 06:25:12) for the HA peer to hit a 'heartbeat timeout reached' state over <Peer-IP>:8300.

    2026-06-04T06:25:09.935+00:00 <local5.notice> consul[710]: agent.server.memberlist.lan: memberlist: Suspect hk#####3monitor has failed, no acks received
2026-06-04T06:25:09.939+00:00 <local5.warning> consul[710]: agent: error getting server health from server: server=hk#####3primary error="context deadline exceeded"
2026-06-04T06:25:09.939+00:00 <local5.warning> consul[710]: agent: error getting server health from server: server=hk#####3monitor error="context deadline exceeded"
2026-06-04T06:25:11.936+00:00 <local5.notice> consul[710]: agent.server.memberlist.lan: memberlist: Suspect hk#####3primary has failed, no acks received
2026-06-04T06:25:11.939+00:00 <local5.warning> consul[710]: agent: error getting server health from server: server=hk#####3monitor error="context deadline exceeded"
2026-06-04T06:25:11.939+00:00 <local5.warning> consul[710]: agent: error getting server health from server: server=hk#####3primary error="context deadline exceeded"
2026-06-04T06:25:12.076+00:00 <local5.warning> consul[710]: agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr=<Peer-IP>:8300 last-leader-id=17######-b##a-9##c-0##5-04########5e

     

  • Packet captures on the destination host show TCP retransmissions on the HA heartbeat ports. This confirms the heartbeat probes are failing until the tunnel establishes.

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX 4.2.x
VMware vSphere ESXi

Cause

Transient network latency caused by dynamic NSX Overlay BFD tunnel establishment coupled with aggressively tuned application heartbeat thresholds. In NSX 4.2.x, the destination host's VNI membership ((VTEP table entry for the migrated VM's segment)) is not pre-provisioned before vMotion completes. The configuration is pushed and processed reactively, resulting in a post-vMotion tunnel convergence delay of approximately 8-9 seconds. Applications with aggressive timeouts (e.g., ~3 seconds) will trigger false-positive failovers during this normal infrastructure tunnel negotiation window. Please note that this is not a software bug, and the NSX system is functioning exactly as designed.

Resolution

Enhancements are targeted for a future NSX/VMware Cloud Foundation (VCF) release to optimize vMotion performance by enabling proactive early NSX configuration provisioning on the destination host before VM migration completes.

To work around the issue in current NSX 4.2.x environments, implement one of the following methods:

Method 1: Adjust Application Timeout

  1. Access the affected guest application's HA configuration.

  2. Increase the HA heartbeat timeout beyond 10 seconds to account for the NSX 4.2.x tunnel convergence delay.

Method 2: Pre-warm the Destination Host

  1. Identify the target ESXi host for the vMotion operation.

  2. Deploy or identify a lightweight, dummy VM residing on the same NSX segment as the sensitive application VM.

  3. Migrate the dummy VM to the destination host before migrating the sensitive application VM.

  4. Note: This action forces the Central Control Plane (CCP) to pre-provision the VNI membership and establish the TEP BFD sessions (laying down the overlay network plumbing) while the impact is negligible.

  5. Proceed with migrating the sensitive application VM. Because the tunnel is already active on the host, the VM will bypass the establishment latency, preventing the HA failover from triggering.

Additional Information

For more information regarding network latency during vMotion, refer to the Broadcom Knowledge Base article: Network Latency observed during vMotion of time sensitive VM

Powered by Wolken Software