Edge VM causing ESXi host's uplinks used unevenly leading to tenant slowness

Products

• VMware NSX

Issue/Introduction

- Tier-1 is in Active/Standby setup (Edge cluster is associated with this Tier-1) with stateful services being used

- Same Edges are also used in the Active/Active configuration on the Tier-0

- You are seeing uneven traffic between both uplinks of the ESXi host where Edge VM is present

- Two TEP interfaces for Edges and Hosts, multiple tunnels are formed between Host TEPs and Edge TEPs

Environment

VMware NSX

Cause

- On a distributed network using Edge Transport Nodes, traffic for stateful services is not load balanced across all uplinks for a given flow because it must be processed by a single, active Edge TEP (Tunnel Endpoint) to maintain the session state.

- Uneven uplink usage on ESXi hosts supporting NSX Edge VMs with multiple TEPs and Active/Standby Tier-1 Gateways is a common scenario, especially with stateful services.

Resolution

- In a multi-TEP configuration, the Edge maps traffic for overlay segment to individual TEPs

- For stateful services like a firewall, NAT, or load balancer, every packet belonging to the same flow must be sent through the same logical service instance

- This unevenness arises due to how traffic is handled in an Active/Standby configuration of Tier-1 Gateway and the nature of stateful services.

- Load balancing at the physical uplink level typically uses stateless policies, such as hashing based on source/destination IP and port. For stateful services, this approach is not suitable because it cannot guarantee that all packets from a single session will arrive at the correct active Edge Node.

• • Dependence on TEP groups and BFD: For failover, NSX use TEP groups and BFD (Bidirectional Forwarding Detection). These features monitor the health of the TEPs and their associated links. If an active TEP fails, traffic is failed over to the standby TEP. This process ensures resilience but does not involve active load balancing of a single flow across multiple uplinks. 

- While TEP traffic can be load balanced for stateless services, stateful services specifically require session persistence, which is maintained by ensuring that all packets belonging to a flow are processed by the same active Edge TEP instance. 

Addressing Uneven Uplink Usage (Mitigation Strategies):

While complete equalization of uplink usage in an Active/Standby setup with stateful services is difficult, several strategies can mitigate the imbalance:

• Active/Active Tier-1 Gateways (with ECMP): 

  If your design allows and services are stateless or can be distributed, consider using Active/Active Tier-1 Gateways with Equal-Cost Multi-Pathing (ECMP). This distributes traffic across multiple active Edges, leading to better uplink utilization.

• Stateless Services: Where possible, design services to be stateless to allow for more flexible traffic distribution and potentially leverage Active/Active configurations.

• Configuring Distributed only gateway for stateless tenant configurations