Loopback traffic behaviors on ESXi hosts between NSX versions- Pre 4.1.1 and Post 4.1.1

Products

• VMware NSX

Issue/Introduction

1. What is categorized as loopback traffic?

A packet generated from a VM exits via one vmnic of the host and enters the host via another vmnic since destination VM is also residing on that same host. In another context, from vmnic's perspective, if it receives a packet whose source mac address is present on that host (net-stats -l) then it declares that packet as a loopback packet.

      2. Scenarios where we can typically see loopback traffic and traffic behavior of that in respective NSX versions:

a. Third-party load balancer (LB) appliance and NSX-T edge node hosting T0/T1 GWs having VLAN backed connectivity with LB in same broadcast domain:

For example consider the below topology,

 

Architecture

• • Here Edge T0 Uplink is connected to a NSX-T VLAN segment - VL-100 part of VDS.
  • Edge Uplink is currently pinned to vmnic3.
  • F5 appliance is connected to a DVPG in VL-100  part of same VDS.
  • DVPG in VL-100 is configured with 'Promiscuous mode' enabled so that packets coming on this VLAN gets forwarded to the LB appliance.
  • LB has a VRRP MAC (virtual MAC) configured inside the appliance. This MAC is not present in 'net-stats -l' on the host.
  • Reachability to VRRP MAC of LB is currently pinned with vmnic5 from outside (LB switchport pinned to vmnic5).

 

Traffic pattern getting used here:

• • Assume we are pinging LB VRRP IP from T0-SR sourcing uplink interface.
  • Here ICMP echo request (seq-1) leaves edge vnic and lands on switchport on NSX-T VLAN segment VL-100 on VDS. This traffic is tagged with VLAN 100.
  • Since on VLAN100 we have DVPG-VL-100 enabled with promiscuous mode, a copy of this packet gets (ICMP req seq 1) moved internally towards the LB VM switchport.
  • For the original copy of this packet, the destination MAC here is VMAC of VRRP sitting inside the LB. ESXi host isn't aware of this MAC. So for ESXI host this ICMP echo request is a 'Unknown unicast' packet. So it sends this packet via it's pinned uplink, vmnic3 and lands on switch-01.
  • Since LB switchport is pinned to vmnic5 connected to switch-02, the LB VRRP MAC is being learnt from that interface. Switch-01 is learning that mac from switch-02 via VPC/MLAG peer link.
  • So, switch-01 will forward this ICMP echo request to switch-02 and switch-02 will push this packet via vmnic-5 inside the host.

Traffic behavior Pre 4.1.1:

• • Once this ICMP request packet (seq-1) lands on the vmnic-5 it will get forwarded to the destination LB VM.
  • F5 switchport already received an ICMP echo request (seq-1) due to promiscuous mode earlier. That means 2 copies of same ICMP echo request seq-1 landed on the f5 switchport.
  • Hence LB will generate 2 ICMP echo response for seq-1 and send it towards T0-SR uplink.
  • Edge will receive 2 echo response packets for same sequence number and going to report DUP packet.
  • This behavior can cause significant disturbance when happening with production traffic

 

Traffic behavior Post 4.1.1:

• • Here the copy of ICMP request seq-1 due to promiscuous mode between edge switchport and LB switchport happens as previous.
  • When original copy (unknown unicast packet) of the ICMP echo request seq-1 lands on the vmnic5 from switch-02, vmnic notices that the source MAC of this packet is the MAC of edge switchport interface, this MAC belongs to this host only and present in 'net-stats -l'. So vmnic considers this packet as 'loopback packet'.
  • There is a RPF (reverse path filter) enhancement introduced from 4.1.1 onwards which will drop/block these type of loopback packet. So, this packet will be dropped and won't be forwarded to LB switchport.
  • On LB switchport we will only notice one instance of ICMP echo request seq-1 (promiscuous copy one) and LB will only respond with 1 echo reply.
  • This eliminates the DUP packet issues observed in earlier versions.

b. In NSX-T Edge bridge setup, when active edge bridge and a VLAN backed VM present on the same host:

In this setup also, when active edge bridge bridging between overlay segment and VLAN, if we have a VLAN backed VM for the same bridged VLAN present on the same host and if the architecture represented on the previous example matches, we will see DUP packets between edge bridge and the VLAN backed VM.

Environment

VMware NSX-T DataCenter
VMware NSX 4.1.0

Resolution

• In Pre-4.1.1 versions, the feasible workaround is to keep f5/VLAN backed VM and edge nodes on separate ESXI hosts.

• In Post-4.1.1 version, if the esxi host is NSX-T prepared, it will have the RPF check enabled for loopback traffic to prevent DUP packet issue.