Packet Buffer Usage increasing on Avi Service Engine consistently - NAT usecase (for non-udp traffic)

Products

• VMware Avi Load Balancer

Issue/Introduction

Symptoms:

Pool servers are marked down.

On Avi UI, we can see events related to Packet buffer usage being high

SE analytics will suggest buffer usage is growing consistently.

Cause

A misconfiguration on Avi is leading to a buffer memory leak.

The issue is seen when non-UDP traffic is configured for NAT policy action type :

NAT_POLICY_ACTION_TYPE_DYNAMIC_IP_PRESERVE_PORT

Resolution

Fix:

The memory leak bug will be fixed in 31.1.1

Workaround:

Make sure the configuration for the NAT policy is a supported one.

Avi supports NAT policy action type: "Dynamic IP and preserve port" only for UDP traffic.  For non-UDP traffic, please set the action type to "Dynamic IP and Port".

Below is a screenshot to configure the protocol in the NAT Rule.

Please note that after applying the workaround, the existing SEs must be rebooted to recover the mbuf leaks that already happened.

 

Additional Information

Document for NAT policy configuration on Avi : 

https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/avi-load-balancer/avi-load-balancer/30-2/vmware-avi-load-balancer-configuration-guide/se-advanced-networking/routing/nat-configuration-on-nsx-advanced-load-balancer-service-engine.html