Packets sent with jumbo frame MTU is lost after increasing the Gateway MTU on NSX
Article ID: 440517
Updated On:
Products
Issue/Introduction
- The NSX Global Gateway Configuration MTU is changed from 1500 to 8900 to support Jumbo Frames
- Following the MTU increase on the underlay and gateway, workload VMs experience local packet drops when attempting to send frames larger than the original 1500 MTU, disrupting network connectivity without explicit network interface state changes.
Environment
VMware NSX
Cause
The packet drops are caused by the Guest OS Path MTU (PMTU) cache. Prior to the MTU change, the Guest OS received ICMP "Fragmentation Needed" errors from the Gateway interface (which was operating at 1500 MTU) and cached this lower path MTU limit. Even after the NSX Gateway MTU was increased to 8900, the Guest OS continued to enforce the cached 1500 PMTU limit.
Resolution
To immediately permit Jumbo Frames without waiting for the PMTU cache to naturally expire or requiring a VM power cycle/network interface toggle, disable Path MTU Discovery locally on the affected Guest OS instances. This is achieved by modifying the sysctl parameter ip_no_pmtu_disc to bypass the cached ICMP fragmentation restrictions. Setting ip_no_pmtu_disc forces the Guest OS to ignore cached ICMP fragmentation-needed errors and implicitly sets IP_PMTUDISC_DONT on created sockets (if set to mode 1 or 2). This allows the OS to immediately utilize the newly configured 8900 MTU provided by the NSX Global Gateway Configuration, eliminating the local packet drops and achieving a seamless transition to Jumbo Frames.
Additional Information
Feedback
