NSX Edge BGP Peering Flaps During Datastore Connectivity Loss

Products

VMware NSX

Issue/Introduction

When the NSX Edge node experiences BGP flapping, ESXi logs [/var/run/log/hostd.log] indicate a correlation with storage connectivity events.
Specifically, the Edge VM reports BGP hold timer expiration or connection drops simultaneously with errors logged on the ESXi host, such as "Lost access to volume" regarding the datastore hosting the Edge node.

<timestamps> In(166) Hostd[2099671]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 66186 : Lost access to volume <Datastore-UUID> (<datastore-name>) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
<timestamps> In(166) Hostd[2099625]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 66187 : Successfully restored access to volume <Datastore-UUID> (<datastore-name>) following connectivity issues.
<timestamps> In(166) Hostd[2099675]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 66190 : Lost access to volume <Datastore-UUID> (<datastore-name>) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
<timestamps> In(166) Hostd[2099225]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 66191 : Successfully restored access to volume <Datastore-UUID> (<datastore-name>) following connectivity issues.

NSX Edge frr [/var/log/frr/frr.log] logs shows BGP flaps.

less frr.log | grep "BFD status for peer"
<timestamps> BGP: BFD status for peer <BGP-neighbor-ip> changed from Down -> Up
<timestamps> BGP: BFD status for peer <BGP-neighbor-ip> changed from Up -> Down
<timestamps> BGP: BFD status for peer <BGP-neighbor-ip> changed from Down -> Up
<timestamps> BGP: BFD status for peer <BGP-neighbor-ip> changed from Up -> Down

NSX Edge syslog [/var/log/syslog] logs shows BGP state changes.

less syslog | grep "state=BGP"
<timestamps> edge-node NSX 9755 FABRIC [nsx@6876 comp="nsx-edge" subcomp="rcpm" s2comp="routing-service-realization" level="INFO"] Alarm for BGP <BGP-neighbor-ip>, peer_uuid: <####-uuid-####> in SR: <####-uuid-####>, state=BGP_DOWN
<timestamps> edge-node NSX 9755 FABRIC [nsx@6876 comp="nsx-edge" subcomp="rcpm" s2comp="routing-service-realization" level="INFO"] Alarm for BGP <BGP-neighbor-ip>, peer_uuid: <####-uuid-####> in SR: <####-uuid-####>, state=BGP_UP
<timestamps> edge-node NSX 9755 FABRIC [nsx@6876 comp="nsx-edge" subcomp="rcpm" s2comp="routing-service-realization" level="INFO"] Alarm for BGP <BGP-neighbor-ip>, peer_uuid: <####-uuid-####> in SR: <####-uuid-####>, state=BGP_DOWN
<timestamps> edge-node NSX 9755 FABRIC [nsx@6876 comp="nsx-edge" subcomp="rcpm" s2comp="routing-service-realization" level="INFO"] Alarm for BGP <BGP-neighbor-ip>, peer_uuid: <####-uuid-####> in SR: <####-uuid-####>, state=BGP_UP

Environment

VMware NSX
VMware ESXi

Cause

The root cause is underlying storage instability or intermittent loss of connectivity to the datastore hosting the NSX Edge VM. When the ESXi host loses access to the datastore (the backing volume), I/O operations are paused or queued.
This storage deadlock prevents the guest OS of the Edge VM from processing time-sensitive BGP and BFD keepalive packets. Once the BGP hold timer expires, the BGP session resets, leading to the observed flapping behavior.

Resolution

To resolve this issue, perform the following steps to verify storage stability and relocate the affected Edge node.

Identify the affected datastore:
Check the ESXi host logs (`/var/log/hostd.log` and `/var/log/vmkernel.log`) for storage access errors corresponding to the timestamps of the BGP flaps.
Look for entries similar to: `Lost access to volume <datastore_naa_id> (<datastore_name>)`
Engage storage experts to fix the underlying issues.
Refer >> Understanding lost access to volume messages in ESXi
Migrate the NSX Edge VM :
If the datastore hosting the Edge VM is confirmed to be experiencing intermittent connectivity issues, perform a live migration of the Edge VM to a known stable datastore.
* In the **vSphere Client**, right-click the Edge VM.
* Select **Migrate**.
* Choose **Change storage only**.
* Select a stable datastore and complete the migration.
Monitor BGP stability:
After migration, monitor the BGP peering status in the NSX Manager UI under Networking > Tier-0 Gateways > BGP.