vCenter and/or VMs down after changing MTU

Products

• VMware vSphere ESXi

Issue/Introduction

• A change made to the MTU of a vSwitch (vSphere Standard Switch, vSS, or vSphere Distributed Switch, vDS) or vmkernel adapter (vmkX), and vCenter and/or VMs are disconnected or unresponsive.
• VM power-on operation is stuck at 0% with no possible error on the GUI.
• The hosts will still be accessible via the host web UI directly.
• Storage on the hosts might show  "0" for storage size and utilization information.
• Storage is network based (iSCSI, NFS, vSAN, FCoE), not direct-attached (DAS, fibre channel).

Environment

VMware ESXi 8.x

VMware vCenter 8.x

VMware vSAN

Cause

Changing the MTU of component that generates traffic, such as a vmkernel adapter, to be larger than the MTU of whatever passes that traffic, such as the vSwitch or a physical switch, will lead to fragmentation and drops.

 

Resolution

• Correct the MTU configuration to pass traffic normally, either by reverting the MTU change in CLI or via the host web interface (for vSS MTU or vmkernel MTU), or addressing the MTU bottleneck in the physical network, as applicable.
• See Configuring Standard vSwitch (vSS) or virtual Distributed Switch (vDS) from the command line in ESXi for CLI commands related to MTU or other ESXi-level network configurations.
• MTU must be changed in a specific order to allow traffic to pass without drops.
• For example, if the vSAN vmkernel adapter on a host is changed to 9000 MTU while the underlying vSwitch is only configured to handle traffic of 1500, vSAN traffic from that host will now be dropped until the vSwitch MTU is increased or the vmkernel MTU is reverted to 1500.
• Generally, the MTU of whatever passes traffic (e.g. virtual switch, physical switch, etc.) should be set to an MTU value at or higher than the MTU value of whatever is passing traffic.
• Therefore if MTU is being increased in an environment, the physical and virtual switch MTU should be increased first, followed by the vmkernel adapter.
• Conversely, if MTU is being lowered, the vmkernel adapter MTU should be lowered before decreasing the virtual switch and physical switch MTU.

Additional Information

Please note that most vendors require the physical network MTU to be set higher than the vSphere environment to allow for overhead, for example the physical switch MTU may need to be set to 9216 for a vSphere MTU value of 9000. Please refer to vendor documentation for specific values based on the infrastructure in-use.