BFD and BGP Flap observed when NSX Edge VMs are vMotioned
Article ID: 319087
Updated On:
Products
Issue/Introduction
- When Active Edge VM is vMotioned from one ESXi host to another, BFD and BGP flap is observed, which recovers within few seconds.
- You may see alerts for BGP session down.
- This is a known behavior and expected of Edge VMs during vMotion.
On the Edge, check the log file /var/log/frr/frr.log. As per the logs, the BFD goes down and comes up automatically within seconds.
:30.725986 ZEBRA: zebra_ptm_handle_bfd_msg: Recv Port [uplink-885] bfd status [Down] vrf [default] peer [xx:xx:xx:xx] local [xx:xx:xx:xx] 1/1/1 01:01:01 ZEBRA: MESSAGE: ZEBRA_INTERFACE_BFD_DEST_UPDATE xx:xx:xx:xx/32 on uplink-885 Down event 1/1/1 01:01:01 BGP: [xx:xx:xx:xx]: BFD Down .... 1/1/1 01:01:01 BGP: [xx:xx:xx:xx]: BFD Up 1/1/1 01:01:01 BGP: BFD status for peer xx:xx:xx:xx changed from Down -> Up 1/1/1 01:01:01 BGP: xx:xx:xx:xx [FSM] Timer (start timer expire).
On the Edge, check the log file /var/log/syslog. As per the logs, the BFD goes down and comes up automatically within seconds.
BGP Down:
EventFeatureName="routing" eventSev="error" eventType="bgp_down"] In Router ########-####-####-####-############, BGP neighbor ########-####-####-####-############ (xx:xx:xx:xx) is down, reason: Network or config error.
...
BGP UP.
1/1/1 01:01:01 NSX 4928 - [nsx@6876 comp="nsx-edge" s2comp="nsx-monitoring" entId="########-####-####-####-############" tid="4965" level="ERROR" eventState="Off" eventFeatureName="routing" eventSev="error" eventType="bgp_down"] Context report: {"entity_id":"########-####-####-####-############","sr_id":"########-####-####-####-############","lr_id":"########-####-####-####-############","bgp_neighbor_ip":"xx:xx:xx:xx","failure_reason":"BGP Established"}
1/1/1 01:01:01 bgpd 11191 - - %ADJCHANGE: neighbor xx:xx:xx:xx(Unknown) in vrf default Up
Login to the ESXi host (where the Edge VM is deployed) as user root, navigate to the path /vmfs/volumes/<datastore-of-the-Edge-Node>/<Edge-VM-Name>/ and check vmware.log
- The logs below show that the VM was vMotioned
1/1/1 01:01:01| vmx| I125: MigrateVMXdrToSpec: type: 1 srcIp=<xx:xx:xx:xx> dstIp=<xx:xx:xx:xx> mid=<Migration-ID> uuid=########-####-####-####-############ priority=yes checksumMemory=no maxDowntime=0 encrypted=0 resumeDuringPageIn=no latencyAware=yes diskOpFile= srcLogIp=<<unknown>> dstLogIp=<<unknown>> ftPrimaryIp=<<unknown>> ftSecondaryIp=<<unknown>>
1/1/1 01:01:01| vmx| I125: MigrateSetInfo: state=8 srcIp=<xx:xx:xx:xx> dstIp=<xx:xx:xx:xx> mid=<Migration-ID> uuid=########-####-####-####-############ priority=high
Cause
This is an expected behavior when the Edge VMs are vMotioned, and if BFD timers are aggressive, a BGP flap can occur.
Resolution
Recommendation 1:
- Avoid vMotioning Edge VMs.
Note: vMotion of Edge VMs should not happen under normal operations and should be done only when necessary (during Host upgrades etc.). Also, as a best practice, vMotion of Edge VMs should only happen during a maintenance window.
Recommendation 2:
- Increase the BFD timers to avoid flaps. Below is the document which has the details (Manager mode). Refer to this documentation to know more details :- Configure BFD on a Tier-0 Logical Router in Manager Mode
For any further assistance, kindly open a support case with Broadcom. Refer to the KB Creating and managing Broadcom cases
Additional Information
- Since BFD flaps will lead to a BGP flap, and in scenarios where there is only one uplink on the edge, this will impact the Datapath until the connectivity is restored between the BGP peers on the uplink and BGP has re-converged.
- If there are multiple uplinks on the Edge, the Datapath will failover to the next best path and impact may be minimal.
- It is possible that all uplinks experience a BFD flap during a vMotion, in that case there will be data plane impact.
Feedback
