False positive of High Memory on SE.

Products

• VMware Avi Load Balancer

Issue/Introduction

Service Engines (SE) in AVI Networks may report high memory usage in the AVI UI, even after the memory for the se_dp process is freed. This issue is observed particularly after the SE becomes a secondary SE. Despite the memory being released by the se_dp process, the glibc library does not return the freed memory back to the operating system. Consequently, Linux shows high memory usage by se_dp, although the actual memory usage by the process has decreased. The cached memory is not excluded from the memory calculations displayed in SE analytics, leading to false-positive high memory alerts.

Environment

vCenter

Cause

1. glibc Memory Behavior: The glibc library does not release freed memory back to the OS; instead, it retains it for future allocations. This cached memory can accumulate to several GBs, causing the Linux system to report high memory usage for the se_dp process.

2. Analytics Reporting: The SE analytics calculations do not currently exclude this cached memory, leading to overestimation of actual memory usage.

Resolution

Reboot the Service engine

Additional Information

This issue is planned to be addressed in future AVI versions, where cached memory consumed by glibc processes will be excluded from memory calculations in SE analytics.

For now, administrators can manually verify memory usage and confirm if alerts are false positives by using the steps outlined below.

Monitoring tools and alerts should be reviewed with the understanding that some reported memory usage might be attributed to glibc's memory caching behavior and does not necessarily indicate a real issue.

To verify if the high memory alert is a false positive:

1. Log into the affected SE.

2. Run the top or htop command to identify processes consuming high memory. Typically, the processes consuming memory will be:

   • se_dp

   • se_log_agent

3. Execute the command:

   show serviceengine <SE_NAME> meminfo

   • Review the output, particularly the following fields:

     • heap_conn_usage

     • heap_config_usage

     • heap_config_memory_status These fields provide insight into the actual memory usage of the SE.