Adding WAF policy on VS fails with the error "Insufficient memory to apply configuration"

Products

• VMware Avi Load Balancer

Issue/Introduction

• Unable to add new WAF policy to the VS as it prompts " Error " Reason: Fault in SE[se-************] Reason[Insufficient memory to apply configuration] Free up resources on this SE[se-************] and then disable/enable the Virtual Service or migrate the Virtual Service to a different SE
• This issue occurs when shared or heap memory soft limits are exceeded
  • Heap: The memory allocation is dynamic. When a program needs memory at runtime (dynamically), it requests space from the heap. This memory is allocated and deallocated as needed during the program's execution in general
  • Shared: The memory allocation is fixed, and as the name implies, shared among multiple processes.

Cause

• Check meminfo output to see which memory is exceeding the limit
  • Login to the controller's CLI (shell mode) and execute the command "show serviceengine <se-name> meminfo" 
  • In the below example, we can see that the limit for Heap memory is getting a hit. Default mem for config is 400MB and the current mem usage is 415MB

Resolution

• As the soft limit for the heap memory is getting a hit, we need to allocate additional heap config memory
• This can be achieved by configuring "extra_config_multiplier" at the SE group level
• This value can be configured based on the requirement. Configuring multiplier 1 will add 100MB per core, multiplier 2 will add 200MB per core, and so on.
• In this scenario, we have set the "extra_config_multiplier" to 1 and allocated an extra 400MB(SE has 4 cores)
• Login to the controller's CLI (shell mode) and execute the below commands
  • configure serviceenginegroup <SEG_NAME>
  • extra_config_multiplier 1
  • save
• After setting the "extra_config_multiplier" to 1 and rebooting the service engine, we can see that the default heap config memory has increased from 400MB to 800MB and is now below the soft limit and the new WAF policy can be applied.
• If the shared_memory configuration limit is reached, the "extra shared memory" must be increased. Steps to increase the extra_shared memory. can be found from the below link

https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/avi-load-balancer/avi-load-balancer/31-1/vmware-avi-load-balancer-configuration-guide/load-balancing-overview/service-engine-group/extra-shared-memory/configuring-extra-shared-memory.html

NOTE: Both these changes require SE to be rebooted.