[NSX-ALB] VMTOOLSD using more SE memory

Products

• VMware

Issue/Introduction

Symptoms:
Issue:
- High Memory on SE's

Symptoms:
- There may be varied symptoms noticed when memory utilization is high:

Symptom1:

- SE events regarding connection drops due to SE resources [ CONN_DROP_NO_CONN_MEM, CONN_THROTTLED_MEMFAIL_FLOW_TBL and CONN_DROP_NO_PKT_BUFF ]

Symptom2

- SE crash at se_malloc
SE will mark down and generate a crash report



Symptom3

- Pool will go down with this failure-code "NO_RESOURCES"




Note:
- The issue occurs in scenarios where vmtoolsd is consuming more than 100 M of memory.

How to verify that the issue is related to VMTOOLSD:
- you can verify that the VMTOOLSD is consuming the SE memory by the following:

1- command: top -b -n1 in the SE bash mode to check the VMTOOLSD mem consupmption

$ top -b -n1
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
315 root 20 0 538528 302856 260 S 0.0 14.9 182:14.60 vmtoolsd

As shown in the above example, the VMTOOLSD is consuming more than 100M.

2- using command show serviceengine <SE Name> mallocstats to check if there is any Mem Failure

Cause

In 22.1.x,  there was a feature introduced in VMware to monitor the IP addresses across namespaces. This is being tracked via vmtoolsd service on ServiceEngine. It is seen to be consuming more memory, which can impact other SE processes

Resolution

Upgrade to version 22.1.3-2p3 patch, as the fix is merged to this patch version

Workaround:
Workaround1:
- disable the VMTOOLSD using "systemctl stop vmtoolsd".

Note: during upgrades, we should make sure that the service is started again.

Workaround2:
- Increase SE memory resources.

Additional Information

Impact/Risks:
This issue causes SE Production impact as the SE may mark down or causes connection drops for the traffic.