Duplicate packets egressing from two different uplink interfaces on the same Edge node, despite ECMP being enabled.

Products

• VMware NSX

Issue/Introduction

The same packet appearing in two separate capture sessions taken on different interfaces of the same NSX Edge Node.

Uplink Interface 1 = de####89-b##e-4##a-9##6-2d49####2db9 

EDGE> start capture interface de####89-b##e-4##a-9##6-2d49####2db9 direction dual count 100 expression host 1#.8#.1#.9#
10:05:17.490484 00:##:##:##:##:56 > 48:##:##:##:##:27, ethertype IPv4 (0x0800), length 74: 1#.1#.1#.6.4###5 > 1#.8#.1#.9#.##3: Flags [S], seq 27######84, win 29###, options [mss 1460,sackOK,TS val 641### ecr 0,nop,wscale 9], length 0
<base64>SC5##########################################################################################################Awk=</base64>
10:05:18.492151 00:##:##:##:##:56 > 48:##:##:##:##:27, ethertype IPv4 (0x0800), length 74: 1#.1#.1#.6.4###5 > 1#.8#.1#.9#.##3: Flags [S], seq 27######84, win 29###, options [mss 1460,sackOK,TS val 641### ecr 0,nop,wscale 9], length 0
<base64>SC5##########################################################################################################Awk=</base64>
10:05:20.492171 00:##:##:##:##:56 > 48:##:##:##:##:27, ethertype IPv4 (0x0800), length 74: 1#.1#.1#.6.4###5 > 1#.8#.1#.9#.##3: Flags [S], seq 27######84, win 29###, options [mss 1460,sackOK,TS val 641### ecr 0,nop,wscale 9], length 0
<base64>SC5##########################################################################################################Awk=</base64>
10:05:24.512319 00:##:##:##:##:56 > 48:##:##:##:##:27, ethertype IPv4 (0x0800), length 74: 1#.1#.1#.6.4###5 > 1#.8#.1#.9#.##3: Flags [S], seq 27######84, win 29###, options [mss 1460,sackOK,TS val 642### ecr 0,nop,wscale 9], length 0
<base64>SC5##########################################################################################################Awk=</base64>
10:05:32.532614 00:##:##:##:##:56 > 48:##:##:##:##:27, ethertype IPv4 (0x0800), length 74: 1#.1#.1#.6.4###5 > 1#.8#.1#.9#.##3: Flags [S], seq 27######84, win 29###, options [mss 1460,sackOK,TS val 642### ecr 0,nop,wscale 9], length 0
<base64>SC5##########################################################################################################Awk=</base64>
10:05:48.553189 00:##:##:##:##:56 > 48:##:##:##:##:27, ethertype IPv4 (0x0800), length 74: 1#.1#.1#.6.4###5 > 1#.8#.1#.9#.##3: Flags [S], seq 27######84, win 29###, options [mss 1460,sackOK,TS val 644### ecr 0,nop,wscale 9], length 0
<base64>SC5##########################################################################################################Awk=</base64>

Uplink Interface 2 = e1####82-b##b-4##7-9##4-f444####3714

EDGE> start capture interface e1####82-b##b-4##7-9##4-f444####3714 direction dual count 100 expression host 1#.8#.1#.9#
10:05:17.490484 00:##:##:##:##:56 > 48:##:##:##:##:27, ethertype IPv4 (0x0800), length 74: 1#.1#.1#.6.4###5 > 1#.8#.1#.9#.##3: Flags [S], seq 27######84, win 29###, options [mss 1460,sackOK,TS val 641### ecr 0,nop,wscale 9], length 0
<base64>SC5##########################################################################################################Awk=</base64>
10:05:18.492151 00:##:##:##:##:56 > 48:##:##:##:##:27, ethertype IPv4 (0x0800), length 74: 1#.1#.1#.6.4###5 > 1#.8#.1#.9#.##3: Flags [S], seq 27######84, win 29###, options [mss 1460,sackOK,TS val 641### ecr 0,nop,wscale 9], length 0
<base64>SC5##########################################################################################################Awk=</base64>
10:05:20.492171 00:##:##:##:##:56 > 48:##:##:##:##:27, ethertype IPv4 (0x0800), length 74: 1#.1#.1#.6.4###5 > 1#.8#.1#.9#.##3: Flags [S], seq 27######84, win 29###, options [mss 1460,sackOK,TS val 641### ecr 0,nop,wscale 9], length 0
<base64>SC5##########################################################################################################Awk=</base64>
10:05:24.512319 00:##:##:##:##:56 > 48:##:##:##:##:27, ethertype IPv4 (0x0800), length 74: 1#.1#.1#.6.4###5 > 1#.8#.1#.9#.##3: Flags [S], seq 27######84, win 29###, options [mss 1460,sackOK,TS val 642### ecr 0,nop,wscale 9], length 0
<base64>SC5##########################################################################################################Awk=</base64>
10:05:32.532614 00:##:##:##:##:56 > 48:##:##:##:##:27, ethertype IPv4 (0x0800), length 74: 1#.1#.1#.6.4###5 > 1#.8#.1#.9#.##3: Flags [S], seq 27######84, win 29###, options [mss 1460,sackOK,TS val 642### ecr 0,nop,wscale 9], length 0
<base64>SC5##########################################################################################################Awk=</base64>
10:05:48.553189 00:##:##:##:##:56 > 48:##:##:##:##:27, ethertype IPv4 (0x0800), length 74: 1#.1#.1#.6.4###5 > 1#.8#.1#.9#.##3: Flags [S], seq 27######84, win 29###, options [mss 1460,sackOK,TS val 644### ecr 0,nop,wscale 9], length 0
<base64>SC5##########################################################################################################Awk=</base64>

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX
VMware NSX-T Datacenter

Cause

This behavior is expected and does not indicate actual network traffic duplication. It is a capture artifact caused by the same packet being sent to the span interface, resulting in both tcpdump sessions displaying the same frame.

When we run the commands 'start capture interface <Interface-1> direction dual' and 'start capture interface <Interface-2> direction dual' in two different SSH sessions, the backend or datapath of the system responds by creating a Linux span interface called span0. This span interface is specifically used for packet captures initiated by users.

The purpose of span0 is to capture and mirror network traffic from other interfaces within the system, enabling monitoring and analysis of the network traffic. During the two user SSH sessions, tcpdump is running and capturing packets on the span0. Since the backend mirrors or spans the packets from the source interface <Interface-1> to span0, and tcpdump captures packets from span0, both tcpdump sessions see and display the same packets from <Interface-1>. Similarly, if packets are spanned from <Interface-2> to span0, both tcpdump sessions will display the same packets from <Interface-2>.

However, it's important to note that while tcpdump on span0 shows the packets from <Interface-1> and <Interface-2>, the packets are not actually transmitted from the interface <Interface-1> or <Interface-2> themselves. Instead, the packets are only copied or mirrored to span0 for monitoring purposes.

To check if both interfaces <Interface-1> and <Interface-2> are on span0, run the command 'get capture session'.

Edge> get capture session
Packet Capture Session
ID : 0
PORTS : ['Interface-1 UUID', 'Interface-2 UUID']

Packet Capture Session
ID : 1
PORTS : []

Packet Capture Session
ID : 2
PORTS : []

Packet Capture Session
ID : 3
PORTS : []

Packet Capture Session
ID : 4
PORTS : []

Packet Capture Session
ID : 5
PORTS : []

When <Interface-1> and <Interface-2> are configured under the same capture session, they both utilize the same capture buffer (e.g., span0). Consequently, the traffic from both interfaces is aggregated, causing them to mirror each other's packets in the output.

Resolution

To isolate traffic per interface and avoid capturing mirrored packets, please assign a unique session ID (e.g., 1, 2) to each capture command.

As demonstrated in the following SSH session, using distinct session numbers ensures that each interface has a dedicated capture stream, allowing you to monitor the traffic separately without overlap.

In the first SSH session, run the following command to start capturing packets on <Interface-1> and direct them to span-1:
set capture session 1 <Interface-1> direction dual

In the second SSH session, run the following command to start capturing packets on <Interface-2> and direct them to span-2:
set capture session 2 <Interface-2> direction dual

This configuration creates separate span interfaces for each session, enabling simultaneous packet captures across multiple uplinks or other interfaces.

Edge> get capture session
Packet Capture Session
ID : 0
PORTS : []

Packet Capture Session
ID : 1
PORTS : ['Interface-1 UUID']

Packet Capture Session
ID : 2
PORTS : ['Interface-2 UUID']

Packet Capture Session
ID : 3
PORTS : []

Packet Capture Session
ID : 4
PORTS : []

Packet Capture Session
ID : 5
PORTS : []