BGP flaps exactly every 3 mins to all peers from NSX edge nodes
search cancel

BGP flaps exactly every 3 mins to all peers from NSX edge nodes

book

Article ID: 443513

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • You observe BGP flapping exactly every 3 minutes to all peers from NSX edge nodes with error - Hold Timer Expired

  • Ping tests to BGP peers from the edge nodes succeed using standard frames but ping tests using jumbo frames fail.

    Below is the screenshot from ping tests showing regular pings work and jumbo frames failing:



  • Packet captures on ESXi hosting the edge node, show that ICMP replies for jumbo frames are received from the peer (as seen at capture points - UplinkRcvKernel) but these packets are dropped before reaching the edge nodes (as seen at capture point - VnicRx)

    Below is the screenshot from packet captures taken at UplinkRcvKernel of the uplink used by the edge node showing both ICMP requests from the edge node and replies from the peer:




    Below is the screenshot from packet captures taken at VnicTx,VnicRx of the edge node where only ICMP requests are seen from the edge but no replies from the peer confirming the packets were received at the uplink level but not delivered to the edge node:



  • The nic cards in use are ENS and INTR capable and driven/enabled.

    Below is the sample screenshot of ENS status:



  • Further investigation via packet capture, reveals that the packets are entering the ENS stack but not exiting the stack (via capture points ENSInput and ENSOuput)

  • This issue can also be confirmed by disabling route advertisement for few minutes (for testing) and the BGP peer stays stable during this period and when the routes are advertised again, the stability is lost confirming that there are BGP updates sent with the size > 1500

  • It is observed that there are 2 different nic cards used in active active setup (with 2 different compatible versions of driver and firmware versions and both the cards (though using same driver) have different compatible versions of driver and firmware depending on the ESXi version used, while both cards don't align with any particular driver version as compatible)

    For Example:

    ESXi is of 9.0

    One nic card of Broadcom 57454 10GBASE-T 4-port OCP Ethernet Adapter with highest compatible driver and firmware versions - bnxtnet 230.0.136.0-1OEM (firmware - 230.1.116.0)
    Other nic card of Broadcom NX-E PCIe 10Gb 2-Port Base-T Ethernet Adapter with minimum compatible driver and firmware versions - bnxtnet 232.0.154.0-1OEM (firmware - 232.1.132.8)

    As observed above, both these nic cards don't align with any particular driver version as compatible for both causing a mismatch and combination of these cards as active-active adapters on ESXi is not preferrable.c

Environment

VMware NSX

Resolution

As major step of troubleshooting, you must align the driver and firmware combinations or the nic cards to the supported/compatible versions.

If you still face the issue even after aligning the nic card combination to have compatible driver and firmware versions, feel free to open a case with Broadcom Support Team for further investigation.

Additional Information

Powered by Wolken Software