NSX 6.4.6 QLarge Edge memory utilization goes above 90% when active interface count is 6 or above

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• Edge continuously reporting high memory utilization
• User space memory shows normal when checked via “top -H” output
• With “slabtop -o” output, we see object count for skbuff_head_cache is above 100k

[root@Edge1-0 ~]# free -m

total        used        free      shared  buff/cache   available

Mem:           1999        1573         135           0         291         215                   <<<  Used memory is high

Swap:           510           0         510

 

[root@Edge1-0 ~]# slabtop -o

 Active / Total Objects (% used)    : 366017 / 370905 (98.7%)

 Active / Total Slabs (% used)      : 18523 / 18523 (100.0%)

 Active / Total Caches (% used)     : 123 / 172 (71.5%)

 Active / Total Size (% used)       : 197142.33K / 199005.53K (99.1%)

 Minimum / Average / Maximum Object : 0.01K / 0.54K / 8.00K

 

 OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   

166944 164293  98%    0.25K  10434       16     41736K skbuff_head_cache             <<<<<  Number of objects goes above 100k

64192  63856  99%    2.00K   4012       16    128384K kmalloc-2048                         <<<<<< Large number of kmalloc-2048 allocations (used by packet sk_buffs)

25228  25228 100%    0.12K    742       34      2968K kernfs_node_cache      

22347  22347 100%    0.10K    573       39      2292K buffer_head            

14070  14070 100%    0.19K    670       21      2680K dentry

Environment

VMware NSX Data Center for vSphere 6.4.x

Cause

In 6.4.6 we have changed the Rx ring buffer size to 4096 for QL/XL edges.

The vmxnet3 driver will allocate 1 Rx queue per vCPU. The QL has 4 vCPU, so 4 Rx rings are allocated, each with 4096 sk_buff allocations.

Resolution

This issue is planned to be fixed in a future release and currently there is no resolution.

Additional Information

Impact/Risks:
This is a know issue with 'NSX Data Center for vSphere' when the number of interfaces increased to 6 or above, the overall memory consumption will go beyond the limit.