Packet Drops Due to Asymmetric Traffic Flow in Active/Active SE Setup when DFW is enabled in the environment

Products

• VMware vDefend Firewall
• VMware Avi Load Balancer

Issue/Introduction

• The user is intermittently facing issues with the application.

• The issue is specific when the connection is going through the secondary SE

• 408 responses seen in Avi VS logs

 

 

Cause

ALB Autoscale Overview:

• When connection lands on Secondary SE, the traffic flow in a L2 scaled-out setup is :
  Client traffic
  Client to Server:
  Client -> Primary SE -> Secondary SE -> Server
  Server response traffic
  Server to Client:
  Server -> Secondary SE -> Client

• While the client-to-server flow always passes through the Primary SE, the return flow does not.
• As a result, the Primary SE does not receive the Acknowledgment (ACK) from the server.

The issue happens because NSXT DFW has stateful firewall capability. It monitors flows on a per-VM vNIC basis.

Since responses from server to client bypass the Primary SE vNIC, DFW assumes the server does not send back Ack to the client, and assumes the permitted max data transfer from client to server will be ~64KB.

When the amount of data sent from the client to the server exceeds 64k in this state, DFW determines that the limit has been exceeded, and the packet should be dropped.

Resolution

There are two potential solutions:

1. Enable SE Tunnel Mode

With tunnel mode enabled, traffic symmetry is maintained during Scaleout. The response to the client is sent via the Primary SE.
Below is a kb explaining the packet flow when tunnel mode is enabled:

Packet-flow-in-vmware-avi-load-balancer

Tunnel mode configuration can be done via controller CLI using below commands:

>configure serviceenginegroup Default-Group

>serviceenginegroup> se_tunnel_mode 1

>serviceenginegroup> save

Tunnel mode values are:

0 (default) — Automatic, based on customer environment

1 — Enable tunnel mode

2 — Disable tunnel mode

Reference document for tunnel mode configuration: SE tunnel Mode  

2. Add Service Engines to DFW exclusion List.

By excluding Avi Service Engine VMs from DFW, asymmetric packet flows due to autoscaling do not trigger packet drops by the stateful feature of NSX DFW.

Below document mentions that any load balancers must be in DFW exclusion list:

Manage-a-firewall-exclusion-list

 

Additional Information

Impact/Risks:
In an environment using NSX-T DFW and ALB Autoscale, communication between Client to Server is interrupted during data transfer.