NSX Edge Health, Routing, TEP Health,and Transport Node Health Alarms in the NSX Manager simultaneously

Products

• VMware NSX

Issue/Introduction

VMware NSX environments experience concurrent overlay network disruptions characterized by the following alarms:

• Faulty TEP

• BGP Down (Hold Timer Expired)

• Edge NIC Out Of Transmit Buffer

• Monitoring Framework Unhealthy

Reviewing the underlying ESXi host vobd logs reveals recurring physical network interface disconnects reporting a loss of carrier signal:

[vob.net.vmnic.linkstate.down] vmnic vmnic<x> linkstate down Uplink: vmnic<x> is down. Affected dvPort: <SECRETS>/<MAC_ADDRESSES>. 1 uplinks up. Failed criteria: 128

Concurrently, the upstream Top-of-Rack (ToR) physical switch syslogs display severe memory buffer warnings (e.g., %TAHUSD-SLOT1-4-BUFFER_THRESHOLD_EXCEEDED: Pool-group buffer 90 percent threshold is exceeded!).

Environment

VMware NSX 4.2.x
VMware ESXi 8.x

Cause

Physical Layer 2 data center fabric congestion (Top-of-Rack switch memory buffer exhaustion) forces Ethernet flow control or packet drops, which stalls the ESXi physical NIC transmit queues. This physical layer stalling propagates backpressure to the NSX Edge VM, instantly filling its virtual DPDK transmit rings to capacity and causing critical BFD/BGP keepalive packets to be dropped.

Resolution

1. Identify the physical Top-of-Rack (ToR) switch ports connected to the affected ESXi host uplinks (vmnics) that are reporting criteria 128 disconnects.

2. Review the physical switch syslogs for memory buffer exhaustion, port flaps, optical Tx/Rx light-level drops, or excessive flow control (PAUSE frames) events matching the exact timestamps of the NSX Edge alarms.

3. Engage the physical network team to remediate hardware buffer allocations on the physical fabric. This includes reviewing QoS policies, traffic microburst patterns, and isolating potential "slow drain" devices exhausting the shared memory pools.

4. Validate strict alignment of port speed (disabling Auto-Negotiation if required) and Forward Error Correction (FEC) configurations between the ESXi hosts and the physical switchports to prevent Layer 1 signaling drops.

5. As a temporary mitigation, vMotion the affected NSX Edge VMs to stable ESXi hosts connected to non-congested physical switches.

Additional Information

Network adapter (vmnic) is down or fails with a Failed Criteria Code

BGP flapping and Edge NIC Transmit Queue Overflow

25Gbps Intel E810-XXV vmnic down after switch port flap