NSX Manager reports alarms "The datapath mempool usage for pfstatepl3 on Edge node X has reached 85% which is at or above the high threshold value of 85%"

Products

• VMware NSX

Issue/Introduction

- NSX Manager reports the following alarm : The datapath mempool usage for pfstatepl3 on Edge node X has reached 85% which is at or above the high threshold value of 85%

- From the impacted edge node, " syslogs"  logs were reporting the below errors.

2024-12-09T05:27:26.504Z edge02.####.### NSX 4709 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="stats" tname="stats17" level="ERROR" eventId="vmwNSXEdgeMpoolUsage"] {"event_state":100,"event_external_reason":"Edge datapath memory usage is over 85%","event_src_comp_id":"c######a-f###-4###-b###-4b########1e","event_sources":{"pool_name":"pfstatepl3"}}

2024-12-09T05:27:26.505425+00:00 edge02.####.### 16a#######a6 3### - -  2024-12-09T05:27:26Z datapathd 4709 stats tname="stats17" [ERROR] {"event_state":100,"event_external_reason":"Edge datapath memory usage is over 85%","event_src_comp_id":"c######a-f###-4###-b###-4b########1e","event_sources":{"pool_name":"pfstatepl3"}}  eventId="vmwNSXEdgeMpoolUsage"

 

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

pfstatepl3 is the mempool owned by the Gateway Firewall for  "Stateful Service State Pool".
NAT rules are also implemented with firewall.
This alarm occurred because of a memory leak in the firewall for terminated connections.

NSX UI reports with below alarm:

 

 

To check DPDK memory usage from Edge root mode -- edge-appctl -t /var/run/vmware/edge/dpd.ctl mempool/show

 

- The Gateway Firewall count is nearing to the max as below.

 

"available_entries": 581,
    "available_entries_in_cache": 581,
    "cache_size_per_core": 512,
    "description": "Stateful Service State Pool",
    "in_use_count": 523707,
    "name": "pfstatepl3",
    "per_core_cache": [
      {
        "available_entries": 581,
        "core_id": 1}],
    "size": 524288,
    "socket_id": 0},

 

 

Resolution

This issue is fixed in NSX-T 3.2.1 and later versions

Workaround:
Failing the Edge over will resolve the issue

Additional Information

Impact/Risks:
If there is insufficient memory available, new connections would not be accepted.