Large packets sent by an NSX-T Bare Metal Edge are truncated and dropped

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• NSX-T Data Center 3.2.x
• Bare Metal Edge with Mellanox NICs
• Jumbo Frames configured with MTU bigger than mbuff size of 2048 bytes
• Traffic from the T0 SR to an IP address north of the Edge fails at packet sizes above 2020 bytes
• Traffic from IP address north of the Edge to the T0 SR IP works as expected at higher packet sizes
• VMs configured with jumbo frames also have connectivity failures for traffic transiting the Bare Metal Edge
• Physical NICs have an MBUFSIZE which is smaller than the size of the packet being transmitted

> get physical-port | find NAME|MBUFSIZE

MBUFSIZE             : 2048
NAME                 : fp-eth0
MBUFSIZE             : 2048
NAME                 : fp-eth1
MBUFSIZE             : 2048
NAME                 : fp-eth2

• BGP flapping may be observed if BGP Update packets are large in size

Environment

VMware NSX-T Data Center

Cause

Network capture shows that these larger packets are truncated as the leave the Bare Metal Edge server.
The destination drops the packet as the value of the frame length is shorter than the value of the IP length due to packet truncation.
Packets that are larger than the default memory buffer size need to be distributed across multiple non-contiguous memory buffers.
Truncation of packet occurs because of incorrect handling when transmitting these larger packets.
Instead of chaining the buffer contents together to form the larger packet, the packet is transmitted based on the content of only one buffer resulting in truncation.

Resolution

This issue is resolved in VMware NSX-T 3.2.2 (build number 20737185)

Workaround:
To workaround the issue, an MTU lower than the mbuff size should be configured on the Bare Metal Edge.