Steady SE connection memory growth after upgrade from 22.1.x to 30.2.x or 31.2.x

Products

• VMware Avi Load Balancer

Issue/Introduction

SE connection memory grows after upgrade to 30.2.x/31.2.x from 22.1.x when Service-Engine Group is configured with Hardware Security Module (HSM)

Please refer to the below doc to verify if HSM is configured for a SE-Group
https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/avi-load-balancer/avi-load-balancer/30-2/vmware-avi-load-balancer-configuration-guide/hardware-security-module-hsm-/thales-luna-formerly-safenet-luna-hsm/enabling-hsm-support-in-nsx-advanced-load-balancer.html

Connection Memory graph under SE Analytics should show a steady increase post-upgrade



Mallocstats output for a SE should show very high values for SE_MTYPE_OPENSSL_CONN

[admin:10-x-x-x]: >show serviceengine <SE-name> mallocstats | grep -i openssl_conn
| SE_MTYPE_OPENSSL_CONN               | 37577852 | 3557708819 | 0    | 579       | 4632           |

 

Environment

Versions with HSM enabled
30.2.x <= 30.2.6 
31.2.x < 31.2.3

Cause

The root cause of the memory exhaustion is a memory leak within the engine used for Hardware Security Module (HSM) integration in Avi. 

A defect was identified in the TLS handshake path during session teardown where the HSM engine fails to actually free the memory back to the SE. 

Because the memory is not correctly deallocated, every TLS handshake involving the HSM leaks a small amount of resources. Over time, this cumulative leak leads to high memory utilization and eventual service instability.

Resolution

The memory leak has been resolved by updating the HSM engine used for integration. The fix ensures that memory is correctly deallocated following the completion of each TLS handshake.

Bug ID: AV-262053

This fix will be included in next maintenance release i.e 30.2.7 and 31.2.3