Intermittent packet drops with Active/Active T0 and Stateful Firewall Rules.

Products

• VMware NSX

Issue/Introduction

VMware introduced stateful services for Tier-0 (T0) gateways in Active-Active High Availability (HA) mode with the release of NSX version 4.0.1.1.
This feature allows the use of stateful service when T0 routers are configure as Active Active.  Prior to 4.0.1.1 the use of stateful services required that the T0 routers be configure as Active/Standby.
The feature "Stateful" feature addresses the to handle stateful flows when both T0 routers are active.


This feature activates an internal process that tracks flow information.  Along with this it also enables two "Punt Logical Switches" to handle the redirection of packets as needed to maintain statefulness.  If flow A used T0-A it records this fact. Then when this flow has a response and it comes into T0-B it will forward the  packet to T0-A thus maintaining the stateful nature of the flow.  This allows stateful firewall rules to work correctly.

These two punt switches are used to punt packet to each T0 base on the recorded data of their flow.  Packets are punted to the specific originating T0 via these punt switches.
For more detail concerning this action refer to VMware NXS Reference Design Guide 4.2  page 133 Stateful active/active gateways (Document Attached to this Article).

Environment

NSX 4.0.1.1 and newer

Cause

If stateful firewall rules are configure where T0 routers are setup as active/active HA the stateful feature must be enabled.  If this is not configure flows egressing and ingress can leave via one T0 and return through another T0.  Stateful firewall rules will expect ingress and egress via the same T0.   When a packet returns or leaves via a different T0 the stateful firewall rule will drop it. 
These are the support topologies to use when configuring T0 Active/Active HA with stateful firewall rules.

NOTE:   There is no warning that will be thrown stating the stateful rules are being applied to a stateless topology and will fail to work correctly.

Resolution

To correct the issue of stateful firewall rules employed with T0 Active/Active HA choose one of the following:

1. Convert the firewall rules to be stateless.
2. Configure the T0 routers to use "Stateful" option and validate that the T0 and T1 configuration match the supported topologies listed in the design guide.

Additional Information

The NSX 4.2 design guide has been attached to this article.