Service Engine Crash due to SE Agent out of memory
search cancel

Service Engine Crash due to SE Agent out of memory

book

Article ID: 413358

calendar_today

Updated On:

Products

VMware Avi Load Balancer

Issue/Introduction

There could be varied symptoms of the issue

  • Avi Service Engine Failed event noticed for Service engine.
  • SE crash file may get generated. The stacktrace points to crash at ipstk_dns_tcp_read_client_so function. 
  • There can be events related to connection drops due to SE being low on resources such as  CONN_DROP_NO_CONN_MEM, CONN_THROTTLED_MEMFAIL_FLOW_TBL

 

Cause

  • Detection of an DoS attack may cause the Service Engine to experience memory leakage for the SE_agent process. 
  • We can identify the same by looking at the footprint of SE_agent memory utilization along with "num_dos_msgs_sent" counter in the output of the command : "show serviceengine <se-name> metrics debug". An increment in this counter points towards the system being prone to hitting this issue.
  • Below is a sample screenshot for reference from Service Engine tech-support logs to identify the issue:  

  • Another way to identify the issue is by looking at DOS related logs in SE under /var/lib/avi/log/se_trace* . High value for such logs can point towards SE hitting this issue.

    zgrep "DOS" se_trace.* | wc -l
    500281

Resolution

The issue is fixed in the latest 22.1.x, 30.2.x releases

 

Additional Information

Below Techdoc also documents the bug under AV-185506

Avi Load Balancer Release Notes