[NSX-ALB] VMTOOLSD using more SE memory
search cancel

[NSX-ALB] VMTOOLSD using more SE memory

book

Article ID: 312807

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Symptoms:
Issue:
- High Memory on SE's

Symptoms:
- There may be varied symptoms noticed when memory utilization is high:

Symptom1:

- SE
events regarding connection drops due to SE resources [ CONN_DROP_NO_CONN_MEM, CONN_THROTTLED_MEMFAIL_FLOW_TBL and CONN_DROP_NO_PKT_BUFF ]

image.png

Symptom2

- SE crash at se_malloc
SE will mark down and generate a crash report

image.png

Symptom3

- Pool will go down with this failure-code "NO_RESOURCES"

image.png


Note:
- The issue occurs in scenarios where vmtoolsd is consuming more than 100 M of memory.

How to verify that the issue is related to VMTOOLSD:
-
you can verify that the VMTOOLSD is consuming the SE memory by the following:

1- command: top -b -n1 in the SE bash mode to check the VMTOOLSD mem consupmption
$ top -b -n1
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
315 root 20 0 538528 302856 260 S 0.0 14.9 182:14.60 vmtoolsd
As shown in the above example, the VMTOOLSD is consuming more than 100M.

2- using command show serviceengine <SE Name> mallocstats to check if there is any Mem Failure


image.png


Cause

In 22.1.x,  there was a feature introduced in VMware to monitor the IP addresses across namespaces. This is being tracked via vmtoolsd service on ServiceEngine. It is seen to be consuming more memory, which can impact other SE processes

Resolution

Upgrade to version 22.1.3-2p3 patch, as the fix is merged to this patch version

Workaround:
Workaround1:
- disable the VMTOOLSD using "systemctl stop vmtoolsd".

Note: during upgrades, we should make sure that the service is started again.


Workaround2:
- Increase SE memory resources.

Additional Information

Impact/Risks:
This issue causes SE Production impact as the SE may mark down or causes connection drops for the traffic.