Edge mempool sess_priv_mp_0 is exhausted causing continuous VPN disconnect or regular reports of "BFD Down On External Interface" alarms

Products

• VMware NSX

Issue/Introduction

• The Edge the T0 router is running on is also running an IPSEC service.
• You see in the Edge /var/log/syslog repeated logs similar to the following about sync attempts for a particular rule or failed IPsec SA negotiations:
  
  NSX 3707688 VPN [nsx@6876 comp="nsx-edge" subcomp="iked" s2comp="iked-sync-handler" level="INFO"] More that max ipsec sync records exist for rule <RULE ID>, discarding old entries
NSX 3707688 VPN [nsx@6876 comp="nsx-edge" subcomp="iked" s2comp="ike-stack" level="INFO"] IPsec SA negotiations: 75114 done, 40500 successful, 34614 failed
  
  
• If you check on the Edge /var/log/syslog you see logs like the following being logged regularly, which indicate that there is BGP flapping happening between the T0 router on the Edge and all its peers:

<Edge VM Name> bgpd 1347073 - -  %NOTIFICATION: sent to neighbor x.x.x.x 6/2 (Cease/Administratively Shutdown) 0 bytes
<Edge VM Name> bgpd 1347073 - -  %ADJCHANGE: neighbor x.x.x.x(Unknown) in vrf default Down Admin. shutdown
<Edge VM Name> bgpd 1347073 - -  %NOTIFICATION: sent to neighbor x.x.x.x 6/2 (Cease/Administratively Shutdown) 0 bytes
<Edge VM Name> bgpd 1347073 - -  %ADJCHANGE: neighbor x.x.x.x(Unknown) in vrf default Down Admin. shutdown
<Edge VM Name> bgpd 1347073 - -  %NOTIFICATION: sent to neighbor x.x.x.x 6/2 (Cease/Administratively Shutdown) 0 bytes

• If you check the BGP summary table on the Edge CLI within the T0 context using the command 'get bgp neighbor summary', then you can see from checking the Up/Down column that the BGP connection was restored at the same time for each neighbor.
• Around the times of the alarms, you see in the /var/log/syslog on the Edge similar log messages like the example below about the Edge entering and exiting maintenance mode:
  
  NSX 1347340 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="stats" tname="stats21" level="INFO"] trigger enter and exit maintenance mode
  
  
• The specific memory pool sess_priv_mp_0 is repeatedly running out of memory, as per edge /var/log/syslog
  
  NSX 3709127 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="stats" level="INFO"] mempool exhausted, usage: 88, threshold: 85, pool: sess_priv_mp_0
59857:2025-05-21T18:29:38Z datapathd 163922 stats tname="stats33" [ERROR] mempool sess_priv_mp_0 is exhausted (100% is used) errorCode="EDG0400710"
  
  
• You can monitor the mempool usage using the following command, and you see a value close to 12000 for the field in_use_count. This indicates memory exhaustion since the limit for in_use_count is 12000:
  
  root@edge##:~# edge-appctl -t /var/run/vmware/edge/dpd.ctl mempool/show sess_priv_mp_0
  {
    "available_entries": 12000,
    "available_entries_in_cache": 0,
    "cache_size_per_core": 64,
    "description": "session mempool for crypto device used by ipsec",
    "in_use_count": 11567,
    "name": "sess_priv_mp_0",
    "size": 12000,
    "socket_id": 0}

Environment

NSX-T 3.x, 4.x

Cause

• Duplicate IPsec SA's entries cause exhaustion of mempool on NSX Edge. The peer VPN device is incorrectly requesting multiple IPSec SAs, and NSX Edge honors these requests to remain RFC compliant.

Resolution

• The issue is fixed in NSX version 9.0, where the sess_priv_mp mempool size has been increased from 12K to 48K, and a change has been introduced to limit multiple IPsec SAs if the peer misbehaves by initiating duplicate SAs.

Workaround: 

• Configure the peer VPN device to responder-only mode, ensuring that the peer does not initiate multiple SAs with the NSX Edge.