VXLAN traffic may be discarded by vnic if using non-default port

Products

• VMware NSX
• VMware NSX

Issue/Introduction

This article to intended to help diagnose VXLAN-related issues and provide one possible reason for the drop of VXLAN traffic on VNIC in non-NSX environment.

 

Symptoms:

• VXLAN traffic (usually generated by container deployed on virtual machines) is dropped by the VNIC
• network traffic captured on network card within the guest os shows there're tcp and icmp traffic (though there's no response to tcp traffic so it's retransmitted a few times): 
  
  
  
  
• Network traffic captured on switch port VnicTx shows that only ICMP traffic is transmitted but not TCP traffic: 

• This is noticed from the captured packet that VXLAN traffic is sent via port 4789
• The environment is using a VDS/VSS without NSX-T installed
  
  

 

Script Settings

Minutes to Remind:
Hours to Remind:
Auto Refresh Interval (seconds):
Auto Refresh:
Auto Check-In:
Check-In Time:
Check-Out Time:
Template File:
Save

Open Trumbowyg

▶

Script Settings

Minutes to Remind:
Hours to Remind:
Auto Refresh Interval (seconds):
Auto Refresh:
Auto Check-In:
Check-In Time:
Check-Out Time:
Template File:
Save

Open Trumbowyg

◀

Cause

The default ports for VXLAN traffic on ESXi: 

• 4789 in NSX-T environment 
• 8472 in non-NSX-T environment

Logic on VNIC will check the port of VXLAN traffic. If the traffic is sent via non-default port, VNIC will drop it.

Script Settings

Minutes to Remind:
Hours to Remind:
Auto Refresh Interval (seconds):
Auto Refresh:
Auto Check-In:
Check-In Time:
Check-Out Time:
Template File:
Save

Open Trumbowyg

▶

 

Resolution

This issue is resolved in ESXi7.0P08 (in which version both ports are added as default port for VXLAN traffic).

Workaround:

1. If possible, ask customer to use the default port for VXLAN traffic.
   
   
2. Disable checksum offload within guest OS but this would bring negative impact to performance at the same time.

Script Settings

Minutes to Remind:
Hours to Remind:
Auto Refresh Interval (seconds):
Auto Refresh:
Auto Check-In:
Check-In Time:
Check-Out Time:
Template File:
Save

Open Trumbowyg

▶

 

Additional Information

Impact/Risks:
VXLAN traffic will be dropped by VNIC

Script Settings

Minutes to Remind:
Hours to Remind:
Auto Refresh Interval (seconds):
Auto Refresh:
Auto Check-In:
Check-In Time:
Check-Out Time:
Template File:
Save

Open Trumbowyg

▶