BFD and BGP Flap observed when NSX Edge VMs are vMotioned

Products

• VMware NSX

Issue/Introduction

• When Active Edge VM is vMotioned from one ESXi host to another, BFD and BGP flap is observed, which recovers within few seconds.
• You may see alerts for BGP session down.
• This is a known behavior and expected of Edge VMs during vMotion.
• On the Edge, check the log file /var/log/frr/frr.log. As per the logs, the BFD goes down and comes up automatically within seconds.
  

  ZEBRA: zebra_ptm_handle_bfd_msg: Recv Port [uplink-##] bfd status [Down] vrf [default] peer [##.##.##.##] local [##.##.##.##]
  ZEBRA: MESSAGE: ZEBRA_INTERFACE_BFD_DEST_UPDATE ##.##.##.##/32 on uplink-885 Down event
  BGP: [##.##.##.##]: BFD Down
  ....
  
  BGP: [##.##.##.##]: BFD Up
  BGP: BFD status for peer ##.##.##.## changed from Down -> Up
  BGP: ##.##.##.## [FSM] Timer (start timer expire).

• On the Edge, check the log file /var/log/syslog. As per the logs, the BFD goes down and comes up automatically within seconds.

  BGP Down:
  EventFeatureName="routing" eventSev="error" eventType="bgp_down"] In Router ########-####-####-####-############, BGP neighbor ########-####-####-####-############ (##.##.##.##) is down, reason: Network or config error.
  ...
  
  BGP UP.
  
  NSX 4928 - [nsx@6876 comp="nsx-edge" s2comp="nsx-monitoring" entId="########-####-####-####-############" tid="4965" level="ERROR" eventState="Off" eventFeatureName="routing" eventSev="error" eventType="bgp_down"] Context report: {"entity_id":"########-####-####-####-############","sr_id":"########-####-####-####-############","lr_id":"########-####-####-####-############","bgp_neighbor_ip":"##.##.##.##","failure_reason":"BGP Established"}
  
  bgpd 11191 - - %ADJCHANGE: neighbor ##.##.##.##(Unknown) in vrf default Up

• Login to the ESXi host (where the Edge VM is deployed) as user root, navigate to the path /vmfs/volumes/<datastore-of-the-Edge-Node>/<Edge-VM-Name>/ and check vmware.log, entries similar to the below will be observed showing the vMotion took place:

  vmx| I125: MigrateVMXdrToSpec: type: 1 srcIp=<##.##.##.##> dstIp=<##.##.##.##> mid=<Migration-ID> uuid=########-####-####-####-############ priority=yes checksumMemory=no maxDowntime=0 encrypted=0 resumeDuringPageIn=no latencyAware=yes diskOpFile= srcLogIp=<<unknown>> dstLogIp=<<unknown>> ftPrimaryIp=<<unknown>> ftSecondaryIp=<<unknown>>
  
  ...
  
  vmx| I125: MigrateSetInfo: state=8 srcIp=<##.##.##.##> dstIp=<##.##.##.##> mid=<Migration-ID> uuid=########-####-####-####-############ priority=high

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

VMware NSX-T Data Center

Cause

This is an expected behavior when the Edge VMs are vMotioned, and if BFD timers are aggressive, a BGP flap can occur.

Resolution

This is a condition that may occur in a VMware NSX environment, for detailed information see NSX Edge vMotion best practices.

Recommendation 1:

• Avoid vMotioning Edge VMs.
  
  Note: vMotion of Edge VMs should not happen under normal operations and should be done only when necessary (during Host upgrades etc.). Also, as a best practice, vMotion of Edge VMs should only happen during a maintenance window.

Recommendation 2:

• Increase the BFD timers  to avoid flaps. Below is the document which has the details (Manager mode). Refer to this documentation to know more details :- Configure BFD on a Tier-0 Logical Router in Manager Mode

For any further assistance, kindly open a support case with Broadcom. Refer to the KB Creating and managing Broadcom cases

Additional Information

Impact/Risks:

• Since BFD flaps will lead to a BGP flap, and in scenarios where there is only one uplink on the edge, this will impact the Datapath until the connectivity is restored between the BGP peers on the uplink and BGP has re-converged.
• If there are multiple uplinks on the Edge, the Datapath will failover to the next best path and impact may be minimal.
• It is possible that all uplinks experience a BFD flap during a vMotion, in that case there will be data plane impact.