Traffic disruption when Preserved Client IP is enabled on NSX Advanced Load Balanced

Products

• VMware Cloud Director
• VMware NSX Networking

Issue/Introduction

Symptoms:

• You have VMware vCloud Director, VMware NSX Advanced Load Balanced (Avi) and VMware NSX deployed in the environment.
• You are configuring NAT rules on VMware NSX via VMware vCloud Director.
• You are using Preserved Client IP feature from NSX Advanced Load Balanced.
• Both NSX Advanced Load Balanced Service Engine and Backend server are on the downlink of the same VMware NSX Tier-1 Gateway.
• You have DNAT rule and SNAT rule on the same Tier-1 Gateway.
• SNAT or DNAT are not applied as expected.
• Traffic is restored if you disabled either the SNAT or DNAT rules.

Environment

VMware NSX-T Data Center

Cause

NAT processing on the downlinks is usually avoided, however the exception to this is that when any DNAT rule is configured on the logical route. The DNAT has to be processed on the incoming direction, i.e. coming from outside to into the logical router, even when it is downlink. The code checks to see that if there is any DNAT rule configured on that logical router, it will enforce NAT (SNAT or DNAT) on the downlink.
As a result, any generic SNAT rule that is configured on the logical_router scope will get applied.
But if this SNAT rule is applied on an interface level, then the SNAT rule will be applied only on that specified interface.
However, if you configure NAT rules from vCloud Director, you won't be able to select a specific interface and the only option available is to apply rule on logical router rather than specific interface.

Resolution

This is a known issue impacting VMware NSX.

Workaround:
Disable the distributed routing feature from VMware vCloud Director, refer to the following links:

• Activate or Deactivate Distributed Routing on an Edge Gateway
• Using Non-Distributed Routing with NSX

When we disable the distributed routing, the segment which is connected to the SE VMs will be disconnected from Tier1 gateway, then a service interface will be created on the Tier 1 gateway and connects to the SE VMs segment.

This way the traffic is not passing through a downlink interface, rather it passes through a service interface.
When the traffic first enters the logical router on a service interface and then exits out on a downlink interface, the NAT processing will not happen on the downlink interface because of double-lookup optimization.