Frequent healthmonitor flaps on pool servers and high connection memory on SEs observed in AWS Cloud

Products

• VMware Avi Load Balancer

Issue/Introduction

In AWS Cloud, the connection memory on the Service Engines could rise signficantly reaching close to 100% and causing health-monitor probes to the pool servers to fail. 

Frequent HM flaps may be observed under Pool > Events, with the HM failures due to NO_RESOURCES

 

Memory failures can be observed in the serviceengine mallocstats with the below command.

> show serviceengine <name> mallocstats

Environment

Avi Versions:

30.2.1 - 30.2.6

31.2.1 - 31.2.2

Cause

This is due to a known issue in AWS Cloud for Service Engines configured with RSS enabled with max_queues_per_vnic or when configured with more than 1 dispatcher cores configured. 

Ref: Configuring Maximum Queues per vNIC

Resolution

This issue will be fixed in future releases.

Workaround:

Recommend to perform the following configuration change in SEG properties using the CLI, which will help solve the connection build up issue in the affected version.

1. SSH to controller
2. Type shell to access the CLI
3. Run below commands to set the number of dispatcher cores to 1 in the affected SEG.

> configure serviceenginegroup <name>
> num_dispatcher_cores 1
> save

Note: This requires the SEs to be rebooted after this configuration. 

Additional Information

Multiple dispatcher cores can be configured after upgrading to the fix version.