VM network connectivity fails for packets exceeding 1500 MTU in NSX Overlay
Article ID: 447403
Updated On:
Products
Issue/Introduction
Virtual machines on an NSX overlay segment experience connectivity failures when communicating with external networks or different hosts, while local East-West traffic on the same host remains functional. This occurs when large packets (such as those used for authentication or large file transfers across host to edge) are dropped by the physical underlay.
Example Symptoms
- VMs cannot reach domain controllers for authentication.
- Standard pings (payload 32 bytes) succeed, but large pings (e.g.,
vmkping -d -s 1473) fail. - Traffic clips exactly at the 1500-byte physical link limit once Geneve encapsulation overhead (~50-68 bytes) is added.
- Applications like core policy admin servers or database synchronization tools show timeouts or hung states.
Environment
VMware NSX
Cause
A physical network device or routed hop is configured with a 1500-byte MTU. Because Geneve encapsulation adds overhead to the original frame, a 1500-byte guest packet becomes ~1560 bytes. If the physical path cannot accommodate this size and the 'Don't Fragment' (DF) bit is set, the packet is dropped.
Resolution
The physical network infrastructure must be updated to support the encapsulation overhead.
- Identify the bottleneck: Run a TEP-to-Gateway ping test from the ESXi host:
vmkping ++netstack=vxlan -I vmk10 -d -s 1572 [Gateway_IP] - Adjust Physical MTU: Coordinate with the network team to increase the MTU on the physical switch ports and routed interfaces.
- Minimum recommended MTU: 1600 bytes.
- Best practice MTU: 9000 bytes (Jumbo Frames).
- Confirm Resolution: Once the gateway passes the large payload test, verify connectivity to the Edge Node TEP and finally the guest VM destination.
Additional Information
Feedback
