VMs experiencing packet drops on internal network with jumbo frames (MTU greater than 1500)

Products

• VMware NSX

Issue/Introduction

• The newly deployed VMs are experiencing packet drops. VMs in the same hosts have no packet drops, but VMs from one host to the next have packet drops.
• Large packets (Jumbo Frames) fail to reach the destination application.
• Standard MTU (1500) traffic functions normally.
• vmkping -d -s 8972 fails between endpoints, while vmkping -d -s 1472 succeeds.
• Packet drops occur at the physical layer.

Environment

VMware NSX

Cause

Physical network infrastructure is not configured to support the MTU size defined on virtual switches or VMkernel adapters. End-to-End MTU is to be consistent and set to allow jumbo frames. 

 

Resolution

1. Verify the MTU setting on all physical switch ports connected to ESXi uplinks.
2. Configure L3 routers and firewalls in the path to support Jumbo MTU frames.
3. Use pktcap-uw on the ESXi host to verify large frame arrival at the physical NIC (vmnic).
   
   Post physical network adjustment to enable Jumbo Frames on the ESXi host uplinks connected interface, re-testing connectivity using vmkping with higher MTU size shows success.

Additional Information

Refer below documents for more information 

Guidance to Set Maximum Transmission Unit
Enabling Jumbo Frames on virtual switches