TCP/UDP Fast Path Configuration Impact on Client Connectivity During Firewall Failover

Products

• VMware Avi Load Balancer

Issue/Introduction

In a VMware AVI Load Balancer deployment where a Virtual Service is configured with TCP/UDP Fast path network profiles, Existing client connections may time out following an uplink firewall failover.

The issue is observed under the following conditions:

• Traffic flow: Client → Firewall → Virtual Service → Backend Server
• The default gateway is an uplink firewall configured in active/standby mode. 
• The firewall shares a single IP address but uses different MAC addresses during failover.
• Only Existing Traffic flows times out after the firewall failover.
• New requests from the same client using a different source port succeed.

The following behavior can be observed in the VS PCAP:

• Prior to failover, client traffic arrives from MAC address MAC-A, and return traffic is sent to the same MAC address.
• After firewall failover, client traffic arrives from a new MAC address MAC-B, while AVI continues to send return traffic to the previously learned MAC address MAC-A causing the uplink firewall to drop the return traffic thereby timing out the client connection.

MAC-A: MAC address of the firewall prior to the uplink firewall failover

MAC-B: MAC address of the firewall after the uplink firewall failover

Environment

• VMware Avi Load Balancer

Cause

This behavior is by design and is related to Fast Path flow optimization in VMware Avi Load Balancer.

For performance reasons, AVI caches the client MAC address when a flow is established. This cached MAC address is used for all subsequent packets belonging to that flow to avoid repeated route lookups.

To preserve Fast Path performance:

• The source MAC address is not revalidated for every packet in an existing flow.
• MAC address updates are not supported mid-flow.

As a result, if the uplink firewall changes its MAC address during an active flow, AVI continues to forward the return traffic using the cached MAC address.

Resolution

To prevent traffic disruption during firewall failover, use one of the following solutions:

• Use System-UDP-Per-Pkt or TCP-Proxy Network profiles: 

Configure the Virtual Service to use the System-UDP-Per-Pkt profile for UDP and TCP-Proxy profile for TCP connections.

To check or change the network profile on a Virtual Service, go to Applications > Virtual Services > [VS Name] > Settings > Network Profile

• Configure VMAC on the Uplink Firewall:

Enable a Virtual MAC (VMAC) feature on the uplink firewall so that the MAC address remains consistent during failover events. This ensures AVI LB does not observe a MAC address change and existing UDP flows remain unaffected

This ensures Avi does not detect a MAC address change, and existing TCP and UDP flows remain unaffected.