An NSX-T Edge may experience a datapathd memory leak if it receives invalid DHCP traffic

Products

• VMware NSX

Issue/Introduction

Symptoms:

• A dataplane restart is observed which will result in failovers in A/S environments or disruption in A/A environments
• Monitoring the available memory in mbuf_pool_socket_0 shows a continuous fall in available memory

edge01> get dataplane memory stats
Tue May 11 2021 UTC 15:48:09.928
Memory Usage

Available_entries             : 51851  <<<< monitor available mbuf memory
Available_entries_in_cache    : 204
Cache_size_per_core           : 128
Name                          : mbuf_pool_socket_0
Per_core_cache
    Available_entries         : 75
    Core_id                   : 0
    Available_entries         : 129
    Core_id                   : 1
Size                          : 77690

• In /var/log/syslog, memory stats show the fall in available memory, monitor the second numeric field after Mempool

2021-04-16T11:39:29.120Z edge1 NSX 19492 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="stats" tname="stats22" level="INFO"] Mempool 1024 28564 24000 53236 16383 2061548 524288 524288 524288 524288 262144 18737 49152 8096 131072 1048576 4194304 1048576 1048576 1048576 4194304 34399
2021-04-16T11:49:29.120Z edge1 NSX 19492 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="stats" tname="stats22" level="INFO"] Mempool 1024 26168 24000 53236 16383 2063474 524288 524288 524288 524288 262144 18738 49152 8096 131072 1048576 4194304 1048576 1048576 1048576 4194304 34399
2021-04-16T11:59:29.120Z edge1 NSX 19492 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="stats" tname="stats22" level="INFO"] Mempool 1024 23675 24000 53245 16383 2063767 524288 524288 524288 524288 262144 18738 49152 8096 131072 1048576 4194304 1048576 1048576 1048576 4194304 34399
2021-04-16T12:09:29.121Z edge1 NSX 19492 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="stats" tname="stats22" level="INFO"] Mempool 1024 21293 24000 53241 16383 2063727 524288 524288 524288 524288 262144 18739 49152 8096 131072 1048576 4194304 1048576 1048576 1048576 4194304 34399
2021-04-16T12:19:29.121Z edge1 NSX 19492 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="stats" tname="stats22" level="INFO"] Mempool 1024 19195 24000 53246 16383 2065737 524288 524288 524288 524288 262144 18739 49152 8096 131072 1048576 4194304 1048576 1048576 1048576 4194304 34399

 

• In /var/log/syslog logging indicates that datapathd has experienced an out of memory condition

2021-04-16T17:04:10.738Z edge1 NSX 28886 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" level="WARN"] KNI: Out of memory

• There is DHCP Request traffic in the environment which has a destination IP on the NSX-T Gateway however no DHCP service is configured on the Gateway.

Environment

VMware NSX-T Data Center

Cause

A memory leak condition can occur on an Edge when invalid DHCP traffic targets an IP on the Edge but no DHCP services are configured on the Edge.

Resolution

This issue is resolved in NSX-T Data Center 3.1.2, available at VMware Downloads .

Workaround:
To workaround this issue, the invalid configuration that results in these DHCP requests should be removed.
Until the source of that traffic can be identified, a firewall rule can be used to immediately drop it.