BGP and BFD flaps observed between Edge nodes and external routers due to mismatch of MTU size

Products

• VMware NSX

Issue/Introduction

The BGP session is established only for 180 secs between NSX edge VM with external peer. The BGP session goes down due to "Hold timer expired" on NSX edge nodes.

Below is part of the command output of get bgp neighbor  on NSX Edge VM which shows the "Hold Timer Expired" on NSX Edge node

Environment

4.2.0

Cause

The issue is caused since the MTU on vswitch is set at 1500 (on DVS in vcenter) connected to vnics (uplinks) of Edge VM as shown below

So bulk route update from peer network got failed to pushed to NSX Edge VM as the MTU size was set as 1500 which is small for jumbo frames. As a result the route update got failed and keep alive between peers were not exchanged, so BGP/BFD session did not get established between peer networks. 

 

Resolution

Update MTU size on vswitch to 9000 and validate if MTU size is set consistently on all interfaces from tier-0 of Edge VM to physical infrastructure.

 

Additional Information

Check if ICMP (ping) from tier-0 vrf to external BGP peer with jumbo frames are successful. 

Additionally if the BGP routing is between Edge VMs and any application VMs in overlay segment, check the size/length of the BGP update message when routing updates are shared between the peers. If there are more BGP updates to be sent and if the size is > 1500, then check if the MTU of the DVS where the segment port of the Edge VMs and application VMs are configured to carry jumbo frames (ie., 9000). 

Set the MTU of the DVS to 9000 via vcenter if not configured already to allow BGP updates between peers. Else allow only the required BGP updates to be sent to the application by filtering them on Edge tier-0.