SD-WAN - TCP traffic fails when there is asymmetric routing/route change and stateful firewall is enabled

Products

• VMware VeloCloud SD-WAN Edge Appliance
• VMware VeloCloud SD-WAN

Issue/Introduction

TCP traffic will fail and get dropped by the SD-WAN edge when there is asymmetric routing/route change and stateful firewall is enabled.

Environment

Velocloud SDWAN, SDWAN, Stateful Firewall 

Cause

There are two possible cases.

3-way handshake incomplete due to asymmetric routing

As described in the Firewall Overview doc, the stateful firewall checks for valid TCP sessions based on traffic traversing it.  When traffic is routed asymmetrically, as in only one direction traverses the edge and the other direction of traffic does not, this can cause the edge to only see the first SYN in the TCP three-way handshake and will not see the SYN/ACK reply.  Since it doesn't see the complete TCP handshake, it does not recognize subsequent packets for this flow to be part of a valid TCP session, dropping the traffic.  The same would occur if the first SYN did not traverse the edge while the returning SYN/ACK does traverse the edge, it will get dropped for the same reason.

Additionally, If the TCP SYN packet traverses through Edge-1 and the corresponding SYN-ACK packet is returned through Edge-2, the packet drop may also occur on Edge-2

One way to confirm if this could be happening is to generate non-TCP traffic between the same two hosts, such as ping or UDP traffic, and confirm it gets through.

 

3-way handshake incomplete due to route change

Even if traffic is not routed asymmetrically, there are multiple routes between the client and server across different VeloCloud SD-WAN Edges, existing TCP flows will be dropped if a route failover occurs.

For example, let's look at the following case.

• There is a TCP traffic from client to server. 
• All Velocloud SD-WAN Edges (Edge-0, Edge-1, Edge-2) enabled the Stateful Firewall. 
• Edge-1 is prioritized using dynamic routing. Traffic is sent through Edge-1.

 

1. If communication is working correctly, the TCP State on Edge-1 should be as ESTABLISHED on Remote Diagnostics > List Active Firewall Sessions.
   This indicates that the TCP 3-way handshake has been successfully exchanged through Edge-1.
   
   
   
   
   
2. In this situation, a route change occurs when Edge-1 goes down. Existing TCP traffic is sent through Edge-2, but the Edge-2 Stateful Firewall blocks the flows.
   
   
   
   This is because the TCP State of existing flows on Edge-2 is as SYN_SENT on Remote Diagnostics > List Active Firewall Sessions. 
   This means that the TCP 3-way handshake via Edge-2 has not started and the Stateful Firewall is not in a state where it can forward the flows.
   This down will continue until the TCP traffic is disconnected (normally around 5 minutes).
   
   
   

   *Of course, if a new TCP traffic will start via Edge-2, the Stateful Firewall can forward it.

3. In addition, there is another route change when Edge-1 goes up. If existing TCP traffic is sent through Edge-1, the Edge-1 Stateful Firewall will block the flows for the same reason.
   Asymmetric routing is also blocked in the same reason. This down will continue until the TCP traffic is disconnected (normally around 5 minutes).
   
   

Resolution

If the stateful firewall is not required in this environment, disabling the stateful firewall will prevent traffic from being dropped.

If the stateful firewall is required, routing will need to be fixed so that both directions of the flow traverse the edge. However, as previously explained, the same issue arises when a route change occurs.

Additional Information

Reference KB "TCP traffic will go down after certificate renewal(386480)"