Virtual machines residing on NSX Overlay Segments are experiencing ICMP packet drops when the Guest OS vNIC MTU is configured to 9000

Products

• VMware NSX

Issue/Introduction

• Virtual machines residing on NSX Overlay Segments are experiencing ICMP packet drops when the Guest OS vNIC MTU is configured to 9000 bytes.
• Packet fragmentation or drops occur because the Geneve encapsulation process adds an overhead (typically 50 bytes) to the original packet.
• When the Guest OS sends a 9000-byte frame, the resulting encapsulated packet becomes 9050 bytes, which exceeds the 9000-byte MTU limit configured on the Distributed Virtual Switch (DVS).

• Symptoms include:

  • ICMP packet drops or timeouts when testing with large payload sizes (e.g., ping -f -s 8972).

  • Inability to establish high-throughput connections using jumbo frames over overlay networks.

  • Packet loss specifically isolated to overlay segments, while VLAN-backed segments function normally.

Environment

VMware NSX

Cause

The physical infrastructure and DVS MTU are not configured with the mandatory overhead padding required for Geneve encapsulation. In an NSX overlay environment, the ESXi host encapsulates virtual machine traffic. If the source frame is already at the maximum MTU supported by the DVS (9000), the additional Geneve header causes the frame to exceed the MTU of the transport path. Standard NSX architecture requires the physical infrastructure and DVS to support an MTU at least 50 bytes larger than the virtual machine's MTU to accommodate this header without fragmentation.

Resolution

Implement one of the following three options to remediate the issue:

1. Lower Guest MTU: Reduce the Guest OS vNIC MTU inside the affected virtual machines to 8900. This ensures the total encapsulated packet (8900 payload + 50 Geneve overhead) remains within the existing 9000-byte DVS MTU limit.

2. Increase DVS MTU: Increase the DVS MTU and physical network infrastructure MTU to 9100 or higher to comfortably accommodate the 9000-byte payload plus the encapsulation overhead.

   • Note: Increasing the DVS MTU requires a scheduled maintenance window as physical and virtual uplinks will flap during configuration changes.

3. Network Reconfiguration: Migrate the impacted virtual machine workloads from the overlay-backed network to a VLAN-backed segment or standard Distributed Portgroup. This removes the encapsulation requirement entirely, allowing 9000-byte frames to pass natively across the network.