NSX Edge BGP and BFD Sessions Drop Due to ESXi Host Storage Latency

Products

• VMware vSphere ESXi

Issue/Introduction

NSX Edge VMs experience complete northbound routing failures and dropped dynamic routing (BGP) sessions.

Symptoms include:

• NSX Manager generates the alarm: No northbound connectivity from the edge. (errorCode="MP701099").

• The nsx-event.log on the affected Edge node reports: {"event_state":0,"event_external_reason":"All BGP/BFD sessions DOWN","event_src_comp_id":"####"}.

• Edge Support Bundle CPU statistics reveal severe processing delays: example values: (e.g., "max_tx_intv": 103780, "max_rx_enq_intv": 99523, "max_wakeup_intv": 103779), indicating top runs delayed for multiple seconds. 

• The underlying ESXi host exhibits extreme Device Average Latency (DAVG/cmd > 2,000 ms) in esxtop for the datastores backing the Edge VM. The disk latency can also be verified from vCenter UI for this specific host under the Monitor tab

• vCenter performance charts show ESXi host disk latency spiking, an example shown below shows disk latency going above 50000ms:

• Storage array statistics show a massive amount of physical layer failures, such as Invalid Tx Word Count reaching tens of millions on specific physical adapters, example shown below:

Environment

VMware NSX 4.2.1.4

VMware vSphere 8.x

Cause

A physical layer signaling fault on an ESXi host Fibre Channel adapter causes severe storage command timeouts. This storage deadlock induces physical CPU scheduling delays (PCPU locks) on the hypervisor, which starves the Edge VM of compute cycles. The resulting CPU starvation prevents the Edge VM from processing time-sensitive Bidirectional Forwarding Detection (BFD) keepalives, directly triggering the BFD and BGP session drops and resulting in the highly elevated max_tx_intv metrics.

Resolution

 

1. Log in to the ESXi host via SSH and identify the failing Fibre Channel adapter by checking for a high Invalid Tx Word Count: esxcli storage san fc stats get -A vmhbaX

2. Determine the runtime names of the logical paths mapping to the degraded adapter: esxcli storage core path list | grep vmhbaX

3. Mask the degraded paths to force the ESXi Native Multipathing Plugin (NMP) to route storage I/O over healthy adapters: esxcli storage core path set --state=off -p <PATH_RUNTIME_NAME>

4. Verify via esxtop (Disk Device view 'u') that DAVG/cmd for the affected LUNs drops to a healthy baseline (<15 ms).

5. Replace the failing SFP transceiver module and/or fiber patch cable connected to the degraded HBA port.

6. Following hardware replacement, re-enable the logical paths: esxcli storage core path set --state=active -p <PATH_RUNTIME_NAME>

7. Clear the physical error counters to establish a new baseline: esxcli storage san fc stats reset -A vmhbaX

 

Additional Information

Viewing and Managing StoragePaths-On Esxi hosts