Openshift HA VIP failover results in indefinite datapath disruption

Products

• VMware NSX

Issue/Introduction

• OpenShift HA cluster VMs run on NSX Segments
• Node shutdown or failure triggers a VIP/EgressIP failover as expected
• After failover the VIP is no longer reachable
• VMs on some hosts can reach the VIP, VMs on other hosts cannot
• When communication doesn't work, traffic is found to be sent to the host of the former active OS node
• CLI command shows a duplicate MAC for the VIP IP
  
  nsxcli -c get service nsx-cfgagent cache-table l2 remote | grep -B 2 -A 2 "VIP IP"

UUID LOG_SWITCH_FIB 1 L2_VM_IP None {
  "mac": "MAC1", <<< Old VIP owner VM
  "ip": "VIP",

UUID LOG_SWITCH_FIB 1 L2_VM_IP None {
  "mac": "MAC2",  <<< Current VIP owner VM
  "ip": "VIP"

• All of the hosts with the issue are connected to the same controller
  
   nsxcli -c get controllers
 Controller Port     SSL         Status       Is Physical Master   Session State  Controller FQDN           Failure Reason
  <Mgr1>    1235   enabled     connected             true               up               NA                       NA
  <Mgr2>    1235   enabled      not used            false              null              NA                       NA
  <Mgr3>    1235   enabled      not used            false              null              NA                       NA

Environment

VMware NSX

Cause

A race condition may occur resulting in one Controller retaining a stale IP/MAC binding. VMs on ESX hosts which connect to this Controller will have their traffic misrouted to the former active Openshift VM. VMs on hosts connected to the other Controllers will have their traffic routed correctly. The issue may persist indefinitely.
Note it is possible the same behaviour may be seen with other clustering software as well.

Resolution

This is a known issue impacting VMware NSX 4.x and 9.x. Subscribe to this KB article to be updated if a fix version becomes available.

If the problem is already present:

As admin cli user, restart the controller service on all 3 Managers, one at a time:

restart service controller

To prevent the issue from occurring:

On the NSX UI,  Networking > Segments > Segment Profiles, create a new IP Discovery Profile and enable Duplicate IP Detection

Note: The best practice profile settings for HA VMs should also be used per VIP failover on VMs connected to NSX segment results in traffic disruption