vSAN Skyline Health Alert: "vSAN: MTU check (ping with large packet size)" for few hosts on the cluster due to Witness Traffic Mismatch.

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• vSAN Skyline health report error "vSAN: MTU check (ping with large packet size)".
• Clicking the Troubleshoot option for the alert confirms only few data nodes failing to communicate with the witness node using MTU 9000.
  
  

Environment

VMware vSAN 7.x
VMware vSAN 8.x

VMware vSAN 9.x

Cause

This issue occurs when the Witness Traffic tag is not configured on any of the VMkernel adapters on an affected data node.

When no dedicated Witness Traffic tag exists, the ESXi host defaults to routing its witness traffic over the management VMkernel adapter. If the management VMkernel adapter on the data node utilizes jumbo frames (MTU 9000) but the companion interface on the Witness Node is set to standard frames (MTU 1500), large packets sent by the data node drop at the destination interface, triggering the MTU health check alert.

• Verification from the Witness Node:
  
  
  • Log in to the witness node via SSH using administrator credentials.
  • Run the command "esxcli vsan network list". Note down the VMkernel adapter tagged for vSAN traffic.
    
    Example: 
    
    esxcli vsan network list
Interface
   VmkNic Name: vmk2
   IP Protocol: IP
   Interface UUID: ######-####-####-####-############
   Agent Group Multicast Address: ###.#.#.#
   Agent Group IPv6 Multicast Address: ####::#:#:#
   Agent Group Multicast Port: #####
   Master Group Multicast Address: ###.#.#.#
   Master Group IPv6 Multicast Address: ####::#:#:#
   Master Group Multicast Port: #####
   Host Unicast Channel Bound Port: #####
   Data-in-Transit Encryption Key Exchange Port: #
   Multicast TTL: 5
   Traffic Type: vsan
    
    
  • Run the command "esxcfg-vmknic -l". Note down the MTU configured.
    
    esxcfg-vmknic -1
Interface   Port Group/DVPort/Opaque Network IP Family IP Address                              Netmask           Broadcast         MAC Address       MTU     TSO MSS   Enabled Type                NetStack
vmk2        24                               IPv4      ###.#.#.#                             ###.###.###.###     ###.###.###.###   ##:##:##:##:##:## 1500    #####     true    STATIC              defaultTcpipStack
    
    In this example, the VMkernel adapter on the witness node configured for vSAN traffic is set to an MTU of 1500.
    
    
• Verification from Working host: 
  
  
  • Log in to the witness node via SSH using administrator credentials.
  • Run the command "esxcli vsan network list". Note down the VMkernel adapter tagged for witness traffic.
    
    Example: 
    

    esxcli vsan network list
Interface
   VmkNic Name: vmk2
   IP Protocol: IP
   Interface UUID: ######-####-####-####-############
   Agent Group Multicast Address: ###.#.#.#
   Agent Group IPv6 Multicast Address: ####::#:#:#
   Agent Group Multicast Port: #####
   Master Group Multicast Address: ###.#.#.#
   Master Group IPv6 Multicast Address: ####::#:#:#
   Master Group Multicast Port: #####
   Host Unicast Channel Bound Port: #####
   Data-in-Transit Encryption Key Exchange Port: #
   Multicast TTL: 5
   Traffic Type: vsan

    Interface
   VmkNic Name: vmk0
   IP Protocol: IP
   Interface UUID: ######-####-####-####-############
   Agent Group Multicast Address: ###.#.#.#
   Agent Group IPv6 Multicast Address: ####::#:#:#
   Agent Group Multicast Port: #####
   Master Group Multicast Address: ###.#.#.#
   Master Group IPv6 Multicast Address: ####::#:#:#
   Master Group Multicast Port: #####
   Host Unicast Channel Bound Port: #####
   Data-in-Transit Encryption Key Exchange Port: #
   Multicast TTL: 5
   Traffic Type: witness

  • Run the command "esxcfg-vmknic -l". Note down the MTU configured.
    
    Example: 
    
    esxcfg-vmknic -l
vmk2       ####                                    IPv4      ##.##.###.##                            ###.###.###.###  ##.##.###.###    ##:##:##:##:##:##  9000    #####     true    STATIC              defaultTcpipStack
vmk0       ###                                     IPv4      ##.##.###.##                            ###.###.###.###  ##.##.###.###    ##:##:##:##:##:##  1500    #####     true    STATIC              defaultTcpipStack
    
    This validation confirmed that witness traffic tagged on vmk0 and vmk0 is set to an MTU of 1500.
    
    
• Verification from the Non working host:
  
  
  • Log in to the witness node via SSH using administrator credentials.
  • Run the command "esxcli vsan network list". Note down the VMkernel adapter tagged for witness traffic.
    
    Example: 
    
    esxcli vsan network list
Interface
   VmkNic Name: vmk2
   IP Protocol: IP
   Interface UUID: ######-####-####-####-############
   Agent Group Multicast Address: ###.#.#.#
   Agent Group IPv6 Multicast Address: ####::#:#:#
   Agent Group Multicast Port: #####
   Master Group Multicast Address: ###.#.#.#
   Master Group IPv6 Multicast Address: ####::#:#:#
   Master Group Multicast Port: #####
   Host Unicast Channel Bound Port: #####
   Data-in-Transit Encryption Key Exchange Port: #
   Multicast TTL: 5
   Traffic Type: vsan

  • Run the command "esxcfg-vmknic -l". Note down the MTU configured.

    esxcfg-vmknic -l 
vmk2       ####                                    IPv4      ##.##.###.##                            ###.###.###.###  ##.##.###.###  ##:##:##:##:##:## 9000    #####   true    STATIC              defaultTcpipStack
    
    vmk0       ###                                     IPv4      ##.##.###.##                            ###.###.###.###  ##.##.###.###  ##:##:##:#:##:##  1500    #####   true    STATIC              defaultTcpipStack
    
    
    This validation confirmed that
    • This validation confirms that Witness traffic was not separated. Only vmk2 was flagged for vSAN traffic. When witness traffic is not explicitly separated, by default, witness traffic flows through the same VMkernel adapter on which standard vSAN traffic is enabled (in this case, vmk2)
    • vmk2 (MTU 9000) is used to communicate with the witness node (MTU 1500), causing large packet drops and triggering the MTU health check alert.

Resolution

• Enter the affected ESXi host into maintenance mode with Ensure Accessibility.
• Navigate to Host > Configure > VMkernel adapters.
• Click the ellipses (three dots) next to the target VMkernel adapter.
  
• Click Edit.
• Select the checkbox for vSAN Witness.
• Click OK.
• Exit the host from maintenance mode.