North-South Connectivity Loss After OpenShift IP Failover Due to Incorrect IP-MAC Binding
Article ID: 451955
Updated On:
Products
Issue/Introduction
In an OpenShift environment running on VMware NSX, when an IP failover occurs and an IP address migrates to a new node, North-South connectivity is lost.
Traffic Scope: The issue exclusively impacts North-South traffic crossing the NSX Edge nodes. East-West traffic traversing local hypervisors remains entirely unaffected.
Reproducibility: The failure occurs even when "Trust On First Use" (TOFU) is disabled within the IP Discovery Profile.
Behavioral Trigger: Connectivity is only successfully maintained during a failover if Duplicate Address Detection (DAD) and VMTools are enabled simultaneously.
Environment
VMware NSX
Cause
The root cause is a behaviour within the NSX Edge Distributed Router (DR), where it dynamically accepts IP-MAC bindings from either Gratuitous ARP (GARP) broadcasts when a new node becomes active or Central Control Plane (CCP) updates, prioritising whichever arrives latest.
When an OpenShift IP failover occurs:
The new active node broadcasts a GARP.
The old active node's kernel responds to this GARP with a unicast ARP reply. This reply comes from kernel as the node itself is down which initially triggered the failover.
Because this unicast reply occurs milliseconds later, the CCP records the standby node's MAC address as the "latest" valid binding.
The Edge DR imports this CCP update and routes all North-South return traffic (SYN+ACK) to the standby VM instead of the active one.
The Distributed Firewall (DFW) on the standby VM rejects this misrouted packet with a TCP RST (Reset). This RST packet triggers another binding refresh, trapping the CCP in a routing loop.
(Note: East-West traffic is unaffected because the ESXi DR is designed to strictly honor GARPs from the new node and ignore subsequent conflicting CCP updates.)
Resolution
This is a known issue impacting VMware NSX. Subscribe to this KB article to be updated if a fix version becomes available.
Workaround
If you cannot upgrade immediately, apply the following configuration change to the IP Discovery Profile applied to the impacted segments:
Log in to the NSX Manager UI.
Navigate to Networking > Segment Profiles > IP Discovery.
Edit the applicable profile and modify the settings to:
Trust On First Use (TOFU): Disabled
VMTools: Enabled
Duplicate Address Detection (DAD): Enabled
Save the profile changes.
Enabling DAD and VMTools simultaneously ensures that the correct active binding is retained during failover events.
Additional Information
If you are contacting Broadcom support about this issue, please provide the following:
NSX Manager support bundles.
Packet captures taken simultaneously from the Edge uplink and the application/database VM interfaces during an active failover test.
Current configuration export of the IP Discovery Profiles applied to the segments.
Handling Log Bundles for offline review with Broadcom support:
Feedback
