Risks and potential impacts of changing the MTU on a virtual switch

Products

VMware vCenter Server

Issue/Introduction

Effect of the MTU (maximum transmission unit) setting on a virtual switch

The MTU (maximum transmission unit) setting limits the size of Ethernet frames (carrying network packets) that will be accepted by a network device.

When a device desires to send a packet with a size that exceeds the MTU of a link on the network, devices on the network path may be able handle this process through fragmentation. However, packets that have a do not fragment bit set to 1 will not be fragmented and will be discarded if they exceed the maximum size.

Effects of changing the MTU setting

Changing the MTU on a virtual distributed switch will cause the uplinks to be brought down and up. This will result in a brief network interruption. Refer to Enabling Jumbo Frames on virtual switches for additional information on this process.

Additionally, changing the MTU on a virtual distributed switch will cause the change to be reflected on all hosts using this switch.

Risks and impacts of changing the MTU settings

Some ESXi services such as vMotion, vSAN and iSCSI storage send network traffic with the do not fragment bit set to true. These services honor the MTU setting on the virtual switch.

If the MTU setting on the virtual switch is set higher than or equal to the MTU of the uplink path, adverse effects are not expected to occur within the ESXi virtual switch, other than performance impacts from additional overhead, because the virtual switch will accept smaller frames. However, network devices external to the ESXi host may drop traffic.

If the MTU setting on the virtual switch is set lower than the MTU at the uplink switch or an intermediary switch along the network path, traffic for services such as ESXi storage may be dropped by the virtual switch. This can result in an outage.

Resolution

Configure network devices before changing the MTU on a virtual switch

To avoid potential issues, physical switches connected to the ESXi host and other switches along the network path should be configured to match or exceed the desired MTU setting prior to changing the setting on the virtual switch. Refer to vendor documentation for instructions on viewing and changing this setting.

Options for testing and validating MTU

To test the supported MTU on an uplink, the vmkping command can be used as follows:

vmkping -I <vmkernel> <target IP> -s <size> -d

Replace <vmkernel> with a VMkernel name, <target IP> with a valid target IP, and <size> with the packet size you wish to test. To allow for headers, for the standard MTU of 1500 the size should be 1472 and for the jumbo frame MTU of 9000 the size should be 8972.

If the VMkernel is on a IP stack other than the default TCP/IP stack, specify it with the -S option.

A result similar to the output below indicates that jumbo frames are supported on the path to the target IP using the specified VMkernel.
# vmkping 192.168.0.1 -I vmk2 -s 8972 -d
PING 192.168.0.1 (192.168.0.1): 8972 data bytes
8980 bytes from 192.168.0.1: icmp_seq=0 ttl=64 time=0.011 ms
8980 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.012 ms
8980 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.012 ms
Results similar to this output indicate that jumbo frames may not be supported on a device along the network path, or the target device is not replying to ping requests.
# vmkping 192.168.0.1 -I vmk2 -s 8972 -d -S defaultTcpipStack
PING 192.168.0.1 (192.168.0.1): 8972 data bytes
--- 192.168.0.1 ping statistics --- 3 packets transmitted, 0 packets received, 100% packet loss
This output indicates that the MTU of the virtual switch was exceeded.
# vmkping 192.168.0.1 -I vmk2 -s 8972 -d -S defaultTcpipStack
PING 192.168.0.1 (192.168.0.1): 8972 data bytes
sendto() failed (Message too long)
sendto() failed (Message too long)
sendto() failed (Message too long)

The vSphere Distributed Switch Health Check feature can test the maximum supported MTU size on uplinks. Refer the article Enabling vSphere Distributed Switch health check in the vSphere Web Client for more information. Note that this feature will test the MTU setting currently configured on a virtual switch.

Additional Information

The MTU for a standard size Ethernet frame is generally 1500 bytes or larger to account for overhead, while the MTU for jumbo frames is generally 9000 bytes or larger.

The Maximum Transmission Unit (MTU) on Distributed Switches that are configured with a Link Aggregation Group (LAG) can also be adjusted to 9000 bytes.