Routes pointing to HA VIP gets installed on T0 gateway

Products

• VMware NSX

Issue/Introduction

Routes pointing to HA VIP of the uplink interface can get installed in the T0 routing table, resulting in the traffic blackholing.
This can happen when the below combination of configuration is present in the setup:

• T0 has a Virtual IP (VIP) configured.
• The BGP peer is advertising routes to the T0 gateway, with the VIP address as the nexthop.
• The routes are not installed in the ACTIVE node's routing table.
• But after the T0 HA failover, the routes get installed in the routing table of the new active node, despite the deny prefix-list.

Environment

VMware NSX

VMware NSX-T Data Center

Cause

• The nexthop of the BGP routes is the same as a connected IP ie the VIP address. When we have our own IP as the nexthop, the routes are not installed in the bgp table as this is "self nexthop". This is to avoid routing loops.
  The same can be validated with the below logs in frr file:
  
  2023/10/07 22:06:32.176957 BGP: #.#.#.# rcvd UPDATE about #.#.#.# IPv4 unicast -- DENIED due to: martian or self next-hop;


• However, since the HA VIP belongs only to the Active Edge's uplink interface, standby node still installs such routes in the routing table since the next hop is not self for the Standby node.
• If failover occurs between the edges, the already learned routes on the standby node do not get removed and stay in the routing table even if the next hop is 'self', since the same got learned when HA VIP was not assigned to the standby uplink interface.

Resolution

This is an expected behavior and no checks are done for self nexthop when a new IP gets added to an interface.
When failover happens and the VIP address moves to another edge, the edge does not check all routes to see if there is any martian/self nexthop after the IP address move.

This issue can be avoided by configuring IP Prefix list to avoid installation of routes pointing to HA VIP on T0:
https://knowledge.broadcom.com/external/article/406568/routes-installed-on-t0-gateway-despite-t.html