Packet Drops in Multi-Dispatcher L2 Scaleout Environments

Products

• VMware Avi Load Balancer

Issue/Introduction

SEs might drop packets intermittently when all the 3 conditions are met:
1. VIP is L2 scaleout (not advertised via BGP).
2. Remove Listening port when VS down is enabled. 

The above 2 config knobs would like below on a VS



3. Multiple dispatchers are configured on the SE-Group (value for num_dispatcher_cpu is greater than 1)

[admin:cntrl]:> show serviceengine <SE-name> seagent | grep dispatcher_cpu
|num_dispatcher_cpu                   | 8

Environment

vCenter Cloud with Avi versions 22.1.x, <30.2.7, 31.1.x and <31.2.3

Cause

When an incoming packet arrives, the receiving vNIC is "owned" by a specific CPU core. However, with multiple dispatchers active, a flow may be picked up and processed by a non-vNIC owner core.

If this flow requires to be punted to a Secondary SE, there exists a bug when the packet is incorrectly dropped at the primary SE. 

Resolution

Workaround: Disable Remove Listening Port when VS Down setting on the VS experiencing this issue. 

This issue will be fixed in the upcoming 30.2.7, 31.2.3 and 32.1.1 versions.