Large packets bypass fragmentation over ECMP paths within NSX Edges
search cancel

Large packets bypass fragmentation over ECMP paths within NSX Edges

book

Article ID: 448075

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Inconsistent packet fragmentation behavior over forwarding paths configured with Equal-Cost Multi-Path (ECMP) routing (such as Tier-0 ECMP to the north side, or Multi-VTEP to the south side) may occur intermittently.

  • Some packets within a flow are fragmented properly while others are not.
  • When you review traffic flows with packet captures, certain large packets exceeding the configured Maximum Transmission Unit (MTU) are forwarded out directly without being fragmented.

For instance, a Tier-0 gateway may be configured with a northbound interface of 8000 MTU, while the Global Logical Interface MTU is set to 1500. This 1500 value is then applied to downlinks from the Tier-0 gateway. Therefore, the Tier-0 gateway may receive traffic up to 8000 MTU and should fragment it to 1500 MTU (unless the "Do Not Fragment" flag is set on the received traffic). However, you may observe packets leaving the downlink at a size higher than the 1500 limit.

This issue occurs under the following packet sequence:

  1. A network flow is initiated, and a fast-path flow cache entry is created by a small-sized packet (where the packet length is smaller than the egress MTU of the logical router ports).

  2. Subsequent larger packets matching this existing flow cache entry are forwarded without fragmentation.

  3. Large packets routed through the slow path (for which no fast-path flow cache entry has yet been established) undergo the standard fragmentation process successfully and are sent out as correctly MTU-sized fragments.

This behavior may lead to Virtual Machines (VMs) receiving packets larger than expected, which can cause packet drops due to MTU RX checks as detailed in Packet Drops Observed on VMXNET3. In the example above (MTU 8000 T0 uplink, 1500 on the logical router, and a VM with 1500 MTU), the VM may receive packets larger than it's own MTU (>1500 bytes) and drop them. This can be validated by checking the KB and validating for 'pkt length exceeds vNic mtu' within the VMXNET 3 stats of the VMs.

Environment

VMware NSX

Cause

This issue is caused by a flow cache processing limitation.

  1. When a network flow is established using a packet smaller than the logical router port's egress MTU, a fast-path flow cache entry is created.
  2. When subsequent larger packets belonging to the same flow arrive and match this pre-existing entry, they are processed and forwarded directly via the fast path.
  3. During this fast-path processing, the logical router bypasses the MTU verification and fragmentation checks, resulting in oversized packets being transmitted unfragmented intermittently.

Resolution

This is a known issue impacting VMware NSX. Subscribe to this KB article to be updated if a fix version becomes available. 

Workaround

  1. To address and mitigate packet fragmentation issues in your VMware NSX environment, reduce fragmentation requirements by aligning the MTU across the environment. Fragmentation, even when handled correctly, reduces overall network performance:

    • Verify and Align MTU Settings: Ensure that you configure the MTU consistently across your entire end-to-end network path. This includes:

      • Physical switches
      • Uplink profiles
      • Logical router interfaces

    • Refer to Official Documentation: For comprehensive step-by-step guidance on how to set and manage the Maximum Transmission Unit throughout your environment, refer to the Broadcom MTU Guidance documentation.

    • Adjust Specific Configurations: As referenced in the earlier example, increasing the VM and logical router MTU to 8800 (to align with the Tier-0 uplink) prevents fragmentation from being necessary.

  2. If drops on VMs due to checks as described in KB Packet Drops Observed on VMXNET3 then the KB can be followed to disable the MTU check at a VM/Host level.
  3. Disabling flowcache within the Edge node will prevent this issue occurring however will massively impact performance. If Edges are at high utilisation (High CPU usage reported inside edge CLI) with flowcache enabled, disabling it is not recommended as the network performance will be degraded. Disabling steps are documented in - Edge Flowcache

Additional Information

Subscribe to this Article - To get updates on this issue, Subscribe to this knowledge article to receive notifications if a fix or permanent patch becomes available.

Powered by Wolken Software