Cluster Performance Degradation and Cluster Outage due to MTU Mismatch

Products

• VMware vCenter Server

Issue/Introduction

 

• Virtual machines (VMs) appear with "stale" or disconnected network interfaces after a host reboot or site power restoration.

• The issue intermittently resolves and recurs without manual intervention.

• Severe latency spikes and packet loss in the vSAN network

 

Environment

• VMware vSphere
• VMware vCenter Server

Cause

Th default vSwitch/vDS Failback policy in conjunction with unstable physical uplinks.

1. Failback Policy: When using the "Route Based on Originating Virtual Port" algorithm, the Failback setting is "Yes" by default. If an active NIC (e.g., vmnic2) fails and then recovers, the ESXi host immediately moves traffic back to it.

2. Path Isolation Failure: If the recovering NIC has an underlying MTU mismatch (e.g., 1500 on the vDS vs. a different setting on the physical switch) or is connected to an unstable port, the traffic is "preempted" back into a failing path.

3. Boot Sequencing: Following power restoration, ESXi hosts typically boot faster than enterprise physical switches. If the switch port is not in PortFast/Edge mode, the physical port remains in a "Listening/Learning" state while the VM attempts to join the network, leading to failed port allocation and disconnected vNICs.

Resolution

To resolve this issue and prevent future occurrences, implement the following changes:

1. Disable Network Failback

To prevent traffic from automatically returning to a recently recovered (but potentially unstable) NIC:

1. Navigate to the vSphere Distributed Switch (vDS) or Standard Switch.

2. Select the vSAN Port Group > Configure > Policies > Teaming and Failover.

3. Change Failback from Yes to No.

4. Repeat for the Management and vMotion port groups if stability issues persist.

2. Physical Switch Configuration (PortFast)

Ensure that all physical switch ports connected to ESXi hosts are configured as "Edge" ports:

• Enable Spanning Tree PortFast (Cisco) or Edge Port (Arista/Dell/HP).

• This allows the port to transition immediately to a "Forwarding" state when the link comes up, preventing "stale" vNIC connections during host reboots.

3. MTU Consistency Validation

Perform an end-to-end MTU audit using the following commands on the ESXi CLI:

• Check vSwitch MTU: esxcfg-vswitch -l

• Check VMkernel MTU: esxcfg-vmknic -l

• Validate vSAN Connectivity: Use vmkping to test large packet sizes without fragmentation:

  Bash

   

  vmkping -I vmkX -d -s 1472 <Target_vSAN_IP>
  

  (Note: Use 1472 for a 1500 MTU or 8972 for 9000 MTU).

4. Optimize Load Balancing 

If using vSphere Distributed Switches (vDS), change the Load Balancing policy to Route Based on Physical NIC Load.

• This policy monitors actual NIC throughput and only shifts traffic if a NIC exceeds 75% utilization, preventing unnecessary traffic reshuffling.

Additional Information

 

• VMware KB 1004088: NIC Teaming in ESXi

• NIC team failover, failback, or vMotion results in loss of network connectivity