SE connection drops (CONN_DROP_NO_PKT_BUFF / CONN_DROP_NO_CONN_MEM) caused by frequent Analytics Profile updates (memory growth in se_log_agent)

Products

• VMware Avi Load Balancer

Issue/Introduction

• Service Engines (SEs) may report CONN_DROP_NO_PKT_BUFF and CONN_DROP_NO_CONN_MEM and, in some cases, crash or restart due to high memory consumption. This can occur when the Analytics Profile is updated frequently - especially via repeated PATCH API calls - while log streaming is enabled. The repeated updates can trigger incremental memory growth in the SE process se_log_agent, eventually leading to memory pressure and connection drops.
• Potential SE crash/restart under sustained memory pressure.
• Most commonly observed in environments performing frequent Analytics Profile changes (often automated)

Environment

Avi version <= 30.2.5, 31.1.x

Cause

• Check whether there is any automation performing repeated PATCH calls against the Analytics Profile (for example, periodic toggling of streaming/log settings).
• On the Service Engine, observe memory utilization and identify whether se_log_agent memory steadily increases over time.
• Correlate timing:
  • Increased frequency of Analytics Profile updates
  • Rising SE memory usage
  • Connection drop events and/or SE restarts

Resolution

Workaround:

• Stop or reduce the frequency of Analytics Profile PATCH API calls, especially any automation that continuously “refreshes” the profile.
• If operationally required, avoid repeated toggling/updates of streaming-related settings.
• Monitor SE memory and connection drop counters until the system can be upgraded.

For fix, upgrade to Avi Load Balancer version 30.2.6:

• Refer to AV-252799 (Updating or changing the analytics profiles with streaming enabled results in a memory leak on Service Engine, which causes a drop in connections) release notes Release Notes for VMware Avi Load Balancer Version 30.2.6 "Updating or changing the Analytics Profiles with streaming enabled results in a memory leak on Service Engine, which causes a drop in connections.” Updating or changing the Analytics Profiles with streaming enabled results in a memory leak on Service Engine, which causes a drop in connections.”