Packet loss at the upstream link when 3rd party VXLAN packets use same destination port as NSX-V VXLAN destination port

Products

• VMware NSX

Issue/Introduction

Symptoms:

• 3rd party VXLAN packets are lost when using the same destination port as the NSX-v VXLAN Destination port at an upstream link of ESXi host.
  
  For Example:
  Source VM > DLR > ESG > TOR > Destination. => Packets are lost at the ESG egress point ( ESXi host pNic).
  Source VM1 > TOR > Destination => Packets are lost at the VM egress point ( ESXi host pNic).
  
  Note: Packets addressed to unknown VTEPs are filtered out to secure the environment and prevent malicious VMs from participating in the VXLAN network.
   
• Running the command net-vdl2 -S -s {NSXpreparedDVS}, you see that tx.drop.guestTag of the VXLAN switch is increasing.
  
  > net-vdl2 -S -s {NSXpreparedDVS}

tx.passThrough: 0
tx.vxlanTotal: 0
tx.clone: 0
tx.tso: 0
tx.csum: 0
tx.drop.invalidFrame: 0
tx.drop.guestTag: 46228
tx.drop.noResource: 0
tx.drop.invalidState: 46
rx.passThrough: 0
rx.vxlanTotal: 0
rx.clone: 0
rx.drop.invalidFrame: 0
rx.drop.notExist: 928
rx.drop.noResource: 0
forward.pass: 0
forward.reject: 0
forward.rpf: 0
arpProxy.reply.total: 1
arpProxy.reply.fail: 0
arpProxy.request.total: 2
arpProxy.request.fail: 0
mcastProxy.tx.total: 0
mcastProxy.tx.fail: 0
mcastProxy.rx.total: 0
mcastProxy.rx.fail: 0
cdo.replicate.tx.total: 0
cdo.replicate.tx.fail: 0
cdo.mcastProxy.tx.total: 0
cdo.mcastProxy.tx.fail: 0

Environment

VMware NSX for vSphere 6.4.x

Resolution

This is expected behavior and as per the design.

 

To work around the issue, follow either of these options:

• Change the NSX-v VXLAN port. For more information, see NSX for vSphere Documentation.
  NSX for vSphere 6.4.x: Change VXLAN Port
   
• Change the 3rd party VXLAN port.
   
• Change the egress port group to a non VXLAN prepared distributed switch.