[AVI LOAD BALANCER] VMTOOLSD using more SE memory

Products

• VMware Avi Load Balancer

Issue/Introduction

- There may be varied symptoms noticed when memory utilization is high:

Symptom1:

- SE events regarding connection drops due to SE resources [ CONN_DROP_NO_CONN_MEM, CONN_THROTTLED_MEMFAIL_FLOW_TBL and CONN_DROP_NO_PKT_BUFF ]

Symptom2

- SE crash at se_malloc
SE will mark down and generate a crash report



Symptom3

- Pool will go down with this failure-code "NO_RESOURCES"

Cause

In 22.1.x,  there was a feature introduced in VMware to monitor the IP addresses across namespaces. This is being tracked via vmtoolsd service on ServiceEngine. It is seen to be consuming more memory, which can impact other SE processes

 

How to Verify the Issue is Related to vmtoolsd

1. Check Memory Usage via top

• SSH into the SE 

• Run the following command to view memory usage:

 

        top -b -n1 | grep vmtoolsd

• Sample output:

   

  315 root 20 0 538528 302856 260 S 0.0 14.9 182:14.60 vmtoolsd

         As shown in the above example, the VMTOOLSD is consuming more than 100M.(RES memory)

2. Check for Memory Allocation Failures

Use the Avi CLI on the Controller to inspect SE memory failure stats:

show serviceengine <SE-Name> mallocstats

 

Resolution

• Upgrade to the recommended release with the latest 2px patch branch. Release notifications can be found in the below KB: 
  Avi Release Notifications 
  
  Workaround:
  Workaround1:
  - disable the VMTOOLSD using the below command on Avi SE CLI : 
     sudo systemctl stop vmtoolsd
  
  Note: during upgrades, we should make sure that the service is started again.
  
  Workaround2:
  - Increase SE memory resources.

Additional Information

Impact/Risks:
This issue causes SE Production impact as the SE may mark down or causes connection drops for the traffic.