BGP Session Down due to Hold Timer Expiration on External Neighbor.

Products

VMware NSX

Issue/Introduction

A BGP neighbor adjacency flaps or goes down unexpectedly on a specific NSX Edge node within a Tier-0 Gateway.
NSX Alarms: "BGP Neighbor Down" alerts are generated for a specific Edge Transport Node.
In a multi-node Tier-0 cluster, the alarm may be isolated to a single Edge node while others remain stable.
The session moves from Established to Clearing after receiving a hold timer expired notification from the neighbor.

Log lines similar to the below are encountered in the Edge frr.log:

YYYY/MM/DD HH:MM:SS BGP: 10.#.#.# [Event] BGP connection closed fd 32
YYYY/MM/DD HH:MM:SS BGP: 10.#.#.#BGP connection fd 32, recieved fatal error, wipe off ringbuf
YYYY/MM/DD HH:MM:SS BGP: %NOTIFICATION: received from neighbor 10.#.#.# 4/0 (Hold Timer Expired) 0 bytes
YYYY/MM/DD HH:MM:SS BGP: 10.#.#.# [FSM] Receive_NOTIFICATION_message (Established->Clearing), fd 32
YYYY/MM/DD HH:MM:SS BGP: %ADJCHANGE: neighbor 10.#.#.#(Unknown) in vrf default Down BGP Notification received
YYYY/MM/DD HH:MM:SS BGP: 10.#.#.#: peer keepalive being removed, acquiring lock
YYYY/MM/DD HH:MM:SS BGP: 10.#.#.#: peer keepalive removed
YYYY/MM/DD HH:MM:SS BGP: 10.#.#.#(0x1d51d676bff0): close file descriptor
YYYY/MM/DD HH:MM:SS BGP: bgp_fsm_change_status : vrf default(0), Status: Clearing established_peers 0
YYYY/MM/DD HH:MM:SS BGP: 10.#.#.# (0x1d51d676bff0 -1) went from Established to Clearing
YYYY/MM/DD HH:MM:SS BGP: Peer 10.#.#.# fd -1 send BGP_DOWN message to BGP adapter
YYYY/MM/DD HH:MM:SS BGP: BGP Adapter: peer 10.#.#.# (vrf: default) send update, old UP, new DOWN state

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Evidence from the Edge logs (var/log/debug/routing/bgp-flap****) , which are available from NSX version 4.2.x confirms that the NSX Edge node was successfully generating and transmitting Keepalives every 30 seconds leading up to the flap:

Down at: YYYY/MM/DD HH:MM:SS
    Session Reset Due To: BGP Notification received
      Down events:
      YYYY/MM/DD HH:MM:SS: Receive_NOTIFICATION_message [Established -> Clearing]
      YYYY/MM/DD HH:MM:SS: ConnectRetry_timer_expired [OpenConfirm -> Established]
      YYYY/MM/DD HH:MM:SS: BGP_Start [Idle -> Connect]
      YYYY/MM/DD HH:MM:SS: Clearing_Completed [Clearing -> Idle]
      YYYY/MM/DD HH:MM:SS: Receive_OPEN_message [OpenSent -> OpenConfirm]
      YYYY/MM/DD HH:MM:SS: TCP_connection_open [Active -> OpenSent]
     YYYY/MM/DD HH:MM:SS: New_bgp_connection [Idle -> Active]
      Keepalives sent before flap:
        YYYY/MM/DD HH:MM:SS
        YYYY/MM/DD HH:MM:SS
        YYYY/MM/DD HH:MM:SS
        YYYY/MM/DD HH:MM:SS
        YYYY/MM/DD HH:MM:SS
(Keepalives being sent every 30 seconds)

Environment

VMware NSX

Cause

The BGP session is torn down because the external neighbor's hold timer expired. This indicates that the neighbor did not receive or process BGP Keepalive packets from the NSX Edge within the configured hold-down interval.
Evidence from the Edge logs confirms that the NSX Edge node successfully generates and transmits Keepalives every 30 seconds leading up to the flap. This behavior can also be verified by capturing packets at the uplink used by the Edge (reference:Troubleshooting Packet Captures for VMware NSX)
Despite these Keepalive packets being actively sent by the Edge, the neighbor reports a timeout. This demonstrates that the packets are either being dropped in the physical underlay network, or the external neighbor is unable to process them.

Resolution

To resolve this issue, engage your physical network team or hardware vendor to review the underlay infrastructure.

Troubleshooting should focus on the physical path and the neighboring router to identify:

1. Potential packet drops or configuration issues in the physical network fabric.
2. Resource constraints (such as high CPU or memory utilization) on the physical neighbor that may be preventing it from processing the received Keepalive packets in a timely manner.

Additional Information

If you are contacting Broadcom support about this issue, provide the following:

NSX Manager support bundles.
NSX Edge support bundles from the node reporting the alarm.
Text of any error messages seen in NSX GUI or command lines pertinent to the investigation.

Handling Log Bundles for offline review with Broadcom support: