Virtual Service enters fault state with reason "Insufficient memory to apply configuration" due to shared memory Leak on Service Engines
Article ID: 444191
Updated On:
Products
Issue/Introduction
VS may enter fault state with reason
Fault in SE[se-<uuid>] Reason[Insufficient memory to apply configuration] Obj: VirtualServiceSe[<VS-name>] Recommendation[Free up resources on this SE[se-<uuid>] and then disable/enable the Virtual Service or migrate the Virtual Service to a different SE]Environment
Versions affected:
- 22.1.x: All versions impacted.
- 30.2.x: All versions less than 30.2.7.
- 31.2.x: Only 31.2.1 is affected.
- 32.x is not affected
Cause
This issue is caused by a memory leak of server-related objects in the Service Engine's shared memory. The following are contributing factors to the memory leak:
- Shared Pool Members: The same backend server endpoint (identical IP and Port combination) is configured as a member across multiple pools.
Server Lifecycle Deletion: One of these duplicate backend server endpoints is deleted from a pool, while the other matching backend server endpoints still maintain active client connections in other pools.
Object Persistence: The shared memory objects associated with the deleted server are improperly held in memory and will continue to exist until all active connections across all identical backend endpoints completely die down.
Shared Memory Buildup: If server deletions occur faster than connections terminate, a progressive shared memory buildup occurs.
Diagnostic Verification via CLI:
- Reviewing the shmallocstats output on the affected Service Engine reveals a disproportionately high count for SE_SHM_MTYPE_SO_SERVER when contrasted against the actual total server count configured on the cluster.
[admin:<IP>]: > show serviceengine <SE-name> shmallocstats | grep SO_SERVER SE_SHM_MTYPE_SO_SERVER | 51567 | 318065256 | 0 | - When the shmallocstats allocation count (51567 in the above o/p) is significantly higher than the actual number of servers present in the Avi configuration database, it confirms a steady accumulation of leaked server memory objects.
Resolution
This memory leak issue has been formally fixed by optimizing object garbage collection within the Service Engine runtime environment.
The fix is integrated into the following Avi Load Balancer releases (and all subsequent patch versions):
32.1.1
31.2.2
30.2.7
Workaround:
If an immediate upgrade to a fixed release is not viable, you can temporarily mitigate the memory exhaustion state using one of the following methods:
Reboot the SE: This will cause datapath impact, so it is recommended to perform rolling reboots by disabling the SE first so that the existing VSes get migrated to a different SE. If you do not have enough capacity on other SEs for the VSes to be migrated, then it is recommended to perform the reboots during low traffic hours.
Manually Migrate Virtual Services: Migrate affected Virtual Services away from the exhausted SE to another available SE with sufficient memory resources, as suggested in the Controller fault recommendations.
Additional Information
Bug ID: AV-257307
Feedback
