NSX-T Edge Packet drops in a setup using Bridge Firewall configuration due to exhaustion of mbuf_pool_socket_0

Products

• VMware NSX

Issue/Introduction

Symptoms:
1) Intermittent packet drops are observed in an environment with High number of fragmented packets.
 2) From the Edge node support bundle, users observe rx_nombufs on one or several physical ports
 less ./edge/physical-ports-stats | grep -iE "name|rx_misses|rx_nombufs"
         "name": "fp-eth0",
         "rx_misses": 29221, <<
         "rx_nombufs": 205794159, <<

AND
Users would observe that a lot of packets sitting in the fragmentation queue.
 less ./edge/fw-if-total-stats | grep frag
             "reason-queued-frag": 54917, << These 2 counters key identifiers of the issue
             "reason-other-frag": 54381, <<

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

The fragmented packets share the same mempool as the other packets. In NSX-T 3.1.X and earlier versions, the fragmented packets may stay in reassembly queues, in which case the buffer is not returned to the pool. In an environment with a high fragmented packet count, this can deplete the mempool, resulting in packet drop due to exhaustion on mbufs.

Resolution

The Edge Datapath is optimized in the NSX-T 3.2.x and above to efficiently handle the fragmented packets.

Workaround:
Currently, there is no workaround.

Additional Information

Impact/Risks:
Intermittent packet drops.