Edge node experiencing high Datapath Mempool alarm for pfstatepl3 and pf_snat_pl3 leading to packet loss

Products

• VMware NSX

Issue/Introduction

• In NSX, an active edge node shows alarm for "Edge Datapath Mempool High".
• A NAT rule is configured for the T1/T0 gateway.
• However, the edge VM memory usage is under 70% and CPU usage is also low.
• This issue can lead to packet loss and slow traffic for T1/T0 gateways associated with the affected Edge node.
• Upon checking the edge details, a few datapath service showing high memory usages, specifically pfstatepl3 and pf_snat_pl3.

• This issue may be intermittent and traffic may flow properly for a short period of time throughout the day.  
• One or more logical routers may show high connection counts with the following command from root shell of affected NSX Edge node:

  root#: edge-appctl -t /var/run/vmware/edge/dpd.ctl fw/lr/show total-stats
  [
      {
          "uuid": "<UUID>",
          "vrf": 1,
          "pvi": 3,
          "config-loaded": true,
          "active": true,
          "name": "SR-<Gateway-Name>",
          "type": "SERVICE_ROUTER_TIER0",
          "mp-router-id": "<UUID>",
          "sync-enabled": true,
          "connection-count": 4194124,                <=========== High number of connections

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

This is caused by VMs on the segment that is establishing large amount of connections which caused the datapath services to run out of memory to handle these connections. 

A common scenario is when a Virtual Machine performing excessive network scanning are exhausting connection limits. 

 

Resolution

This is not a NSX issue. However, a workaround can be implemented to prevent a certain T1 from taking over too many connections and exhaust the mempool. 

Workaround: Implement reflexive NAT on a gateway

A reflexive NAT does not consume connection entries and therefore shall not deplete the resources.

Additional Information

If this article did not help resolve your issue, you can review the following article for further information about Edge Datapath mempool usage high alarm