vCenter and/or VMs down after changing MTU

Products

• VMware vSphere ESXi

Issue/Introduction

A change was made to the MTU of a vSwitch (vSphere Standard Switch, vSS, or vSphere Distributed Switch, vDS) or vmkernel adapter (vmkX), and vCenter and/or VMs are now disconnected or unresponsive.

The hosts may still be accessible via the host web UI directly.

Storage on the hosts may be showing "0" for storage size and utilization information.

The MTU was changed for a storage vmkernel adapter or a vSwitch that a storage vmkernel adapter uses.

Storage is network based (iSCSI, NFS, vSAN, FCoE), not direct-attached (DAS, fibre channel).

Environment

VMware ESXi

VMware vCenter

VMware vSAN

Cause

Changing the MTU of something that generates traffic, such as a vmkernel adapter, to be larger than the MTU of whatever passes that traffic, such as the vSwitch or a physical switch, will lead to fragmentation and drops.

For this issue, since the packet fragmentation is specifically related to storage traffic, storage connectivity is lost and as a result anything using that underlying storage, for example VMs, are subsequently impacted as well.

Resolution

Correct the MTU configuration to pass traffic normally, either by reverting the MTU change in CLI or via the host web interface (for vSS MTU or vmkernel MTU), or addressing the MTU bottleneck in the physical network, as applicable.

See Configuring Standard vSwitch (vSS) or virtual Distributed Switch (vDS) from the command line in ESXi for CLI commands related to MTU or other ESXi-level network configurations.

MTU must be changed in a specific order to allow traffic to pass without drops.

For example, if the vSAN vmkernel adapter on a host is changed to 9000 MTU while the underlying vSwitch is only configured to handle traffic of 1500, vSAN traffic from that host will now be dropped until the vSwitch MTU is increased or the vmkernel MTU is reverted to 1500.

Generally, the MTU of whatever passes traffic (e.g. virtual switch, physical switch, etc.) should be set to an MTU value at or higher than the MTU value of whatever is passing traffic.

Therefore if MTU is being increased in an environment, the physical and virtual switch MTU should be increased first, followed by the vmkernel adapter.

Conversely, if MTU is being lowered, the vmkernel adapter MTU should be lowered before decreasing the virtual switch and physical switch MTU.

Additional Information

Please note that most vendors require the physical network MTU to be set higher than the vSphere environment to allow for overhead, for example the physical switch MTU may need to be set to 9216 for a vSphere MTU value of 9000. Please refer to vendor documentation for specific values based on the infrastructure in-use.