SMB connection drops on NSX Overlay Segments due to Geneve MTU overhead

Products

• VMware NSX

Issue/Introduction

Virtual machines (VMs) residing on NSX Overlay segments experience SMB connection drops when communicating across different ESXi hosts. This occurs because the combined size of the SMB packet and the Geneve encapsulation header exceeds the MTU limits of the transport network.

Symptoms

• SMB connections are disconnected between VMs on the same segment but different ESXi hosts.
• Windows Event Logs show: A network connection was disconnected. Instance name: \Device\LanmanRedirector.
• Packet captures indicate that traffic reaching a length of 9000 bytes is retransmitted and eventually terminated with a RST (Reset) packet.
  
• Connectivity is stable if the VMs are located on the same ESXi host (no Geneve encapsulation required).

Environment

VMware NSX

Cause

The issue is caused by the Guest OS MTU being set too close to the physical/virtual network MTU limit (e.g., 8986). In an NSX Overlay environment, Geneve encapsulation adds 50 bytes of overhead to the original frame.

When the Guest OS sends an 8986-byte frame, the resulting encapsulated packet becomes 9036 bytes. If the physical switch or Distributed Virtual Switch (DVS) is configured with a 9000-byte MTU, these packets are dropped at the uplink, leading to SMB timeouts and disconnections.

Resolution

To resolve this issue, adjust the Guest OS MTU to accommodate the encapsulation overhead or increase the infrastructure MTU.

Option 1: Reduce Guest OS MTU (Recommended)

Reduce the MTU within the Windows Operating System to the default 1500 (or 8900 if Jumbo Frames are required).

1. Open PowerShell as Administrator on the affected VM.
2. Run the following command to set the MTU to 1500: Set-NetIPInterface -InterfaceAlias "Ethernet" -NlMtuBytes 1500
3. If performance requirements necessitate Jumbo Frames, ensure the MTU does not exceed 8900 to leave overhead for Geneve.

Option 2: Increase Infrastructure MTU

If the Guest OS must maintain an MTU of 9000, you must increase the MTU of the entire physical path and the Distributed Virtual Switch (DVS).

1. Configure the DVS MTU to at least 9100.
2. Ensure all physical switch ports and routers in the transport path support a minimum MTU of 9100.

Additional Information

KB 440206 — Virtual machines residing on NSX Overlay Segments are experiencing ICMP packet drops when the Guest OS vNIC MTU is configured to 9000.