High Connection Memory Usage on Service Engine due to abrupt client connection closure

Products

• VMware Avi Load Balancer

Issue/Introduction

• Customers may observe a continuous surge in Connection Memory Usage on one or more Service Engines (SEs), potentially reaching 80% or higher. This growth is indicative of connection entries not being properly cleaned up, which can eventually lead to resource exhaustion if not addressed.

Environment

 

• Avi Load Balancer Service Engines (SEs).

• Virtual Services (VSs) utilizing Layer 7 (L7) Application Profiles

• Scenarios where client HTTP connections are abruptly closed (e.g., via TCP RST) during the response transfer phase.

 

Cause

• The high connection memory usage is caused by an issue in how the Layer 7 (L7) proxy handles abrupt client-side connection closures (e.g., a client sending a TCP Reset (RST)) while the Avi Load Balancer is transferring the response back to the client.

• In the default streaming mode (no request body buffering), the SE proxy's logic for handling this specific abrupt closure state is incorrect. It causes the proxy to ignore and not process the connection closure notification effectively.

• A significant discrepancy is observed in the SE mallocstate between the number of M_SOCKET count entries and M_FLOW_ENTRY count entries. Ideally, these numbers should be close when connections are cleaned up gracefully. The imbalance indicates orphaned or uncleaned connection-related entries.
  

  [admin:avi-controller]: > show serviceengine <SE-Name> mallocstats
  +-------------------------------------+-------+----------+------+-----------+----------------+
  | Buffer type                         | Count | Size     | Fail | Free List | Free List Size |
  +-------------------------------------+-------+----------+------+-----------+----------------+
  M_SOCKET                            305022      124448976   	0     	1315       	504960        
  M_FLOW_ENTRY                        30119       7469512     	0     	847       	189728        

• This failure to process the closure prevents the corresponding connection memory entries (flow entries, sockets) from being fully cleaned up, leading to a build-up of uncleaned connection entries and the resulting surge in connection memory usage.

Resolution

The permanent fix for this issue is provided in the following maintenance releases and requires an upgrade of the Avi Controller and Service Engines:

• 22.1.7-2p10

• 30.2.4

• 31.1.2

• 31.2.1
  
  

Workaround (Temporary): As a temporary measure to stop the memory buildup before an upgrade can be scheduled, enable Request Body Buffering in L7 Application Profile.

1. Navigate to the affected Virtual Service's L7 Application Profile.

2. Enable Request Body Buffering.

• When buffering is enabled, the SE holds the entire HTTP request body in memory (up to 32MB) before sending the request to the backend. In this mode, when a client abruptly closes the front-end connection during the response transfer, the connection closure is handled more gracefully, ensuring the connection memory is cleaned up.
  
  

• Enabling request body buffering will inherently increase the Service Engine's overall memory consumption due to the data buffering (up to 32MB per request). This workaround should only be implemented if the current overall SE memory consumption is low enough to safely absorb this increase.

Additional Information

Release Notes:

AV-242180: Service Engine connection memory may not be released when clients abort the connection during server to client response transmission, leading to high memory consumption.

https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/avi-load-balancer/avi-load-balancer/22-1/vmware-avi-load-balancer-release-notes/release-notes-22-1-7.html