Avi SE high memory and CPU due to external client log streaming

Products

• VMware Avi Load Balancer

Issue/Introduction

During application traffic load, high CPU and Memory utilization may be observed with se_log_agent progress(s). 

Example HTOP command output from the SE Linux bash indicating the % usage of CPU under column CPU%

You may also observe SE_CPU_HIGH events.

 

Environment

Affects Version(s):

22.1.1 through  22.1.4-2p6 

22.1.5 through 22.1.5-2p1

Base version  30.1.2

Cause

This issue is caused by a known issue with external client log streaming.  External client log streaming is a feature that sends the virtual service significant and non-significant logs to any Linux appliance or services such as Splunk via the Syslog protocol.

You can find more information on external client log streaming in the Documentation link below: 

Streaming NSX Advanced Load Balancer Client Logs to an External Server

Bug ID: AV-195595

Details: External log streaming to a servers or load balancers which erroneously responds to simplex log stream causes Service Engine memory growth, eventually leading to SE crash.

Resolution

Please upgrade your system to the fix version for the permanent solution.

Fix Version: 30.2.1, 22.1.6, 22.1.5-2p2, 30.1.2-2p1, 22.1.4-2p7

 

Workaround(s)

1. Reboot the Service Engine(s).  The amount of time this workaround will mitigate this issue depends on the traffic load on the SE.
   
   OR
   
   
2. Disable external client log streaming on all virtual services hosted on the affected SE Group(s).  Navigate to Templates > Profiles > Analytics > select profile used by the virtual services > uncheck box "Stream Logs to an External Server"