TCP packets dropping over L2VPN not reaching to remote SDDC VM in NSX-T

Products

• VMware NSX

Issue/Introduction

This article provides information on how to avoid an outer-fragmented packet drops in the public network.

Symptoms:

• TCP packets dropping over L2VPN not reaching to remote SDDC VM in NSX-T.
• SDDC-1 is sending TCP packets over L2VPN but remote VM does not receive it (Packets gets dropped in between).

Environment

VMware NSX-T Data Center 2.x
VMware NSX-T Data Center

Cause

This behavior can be seen with NSX-T Data Center releases prior to 2.5.0. 

This issue occurs because the source virtual machine's MTU is 1500, which exceeds the fragment size in L2VPN tunnel (about 1390).

Up until NSX-T Data Center 2.5.0, PMTU discovery was not supported in L2VPN. The L2VPN behavior is when packet from the source VM reaches to L2VPN tunnel, GRE/IP encapsulation happens to the packet with DF bit unset (irrespective of DF bit value in the inner packet). Based on L3VPN PMTU and IPsec overhead, the oversized GRE/IP packets are fragmented before ESP encapsulation. So, technically there is no need to adjust the MTU on source VM, unless a packet drop is observed due to excess fragmentation-reassembly.

VMware recommends to reduce the MTU of source VM by 150 bytes (For example, set it to 1350 for default MTU on the Edge) to avoid fragmentation, at least at the NSX-T Edge level.

Resolution

To resolve this issue, set the MTU size to 1350 or less at the Source and Destination VM's interface (Uplink to L2VPN).