"Edge datapath memory usage is over 85%" event generated for pool_name: pfa_attrconn_pl3

Products

• VMware NSX

Issue/Introduction

- Edge node having high memory usage of over 85% for the event source: pfa_attrconn_pl3

- If this Edge is active in T0 or T1, there are data path issues seen as Edge will not be processing the packets

In edge /var/log/syslog:

026-01-27T10:16:23Z datapathd 10717 stats tname="stats21" [ERROR] {"event_state":100,"event_external_reason":"Edge datapath memory usage is over 85%","event_src_comp_id":"#########","event_sources":{"pool_name":"pfa_attrconn_pl3"}}  eventId="vmwNSXEdgeMpoolUsage"

- NSX manager also alerts the alarm relating to this error: "The datapath mempool usage for pfa_attrconn_pl3on Edge node X has reached 85% which is at or above the high threshold value of 85%"

Environment

NSX 4.2.3.0

Cause

The pfa_attrconn_pl3 (similar to pfstatepl3) is a mempool managed by the Gateway Firewall for the Stateful Service State Pool. High usage in this specific pool typically occurs due to following possible causes: 

• Connections not properly cleared from the firewall state pool.
• If the current Edge form factor has reached its maximum concurrent connection limit for stateful services like NAT or Firewall.
• A specific VM or application is performing excessive network scanning or establishing a high volume of simultaneous connections.

Resolution

Workaround:

As admin user on the affected Edge Node, restart the datapathd process using the following command, which will clear the memory pool:

edge> restart service dataplane

Note: When restarting the dataplane service, no traffic will be processed by the Edge node for a short time. Either move services to another Edge node before restarting the service, or plan this activity for a maintenance window if you wish to avoid all dataplane impact.