vSAN Health Service - Network Health - Hosts small ping test (connectivity check) and Hosts large ping test (MTU check)

Products

• VMware vSAN

Issue/Introduction

This article explains the "Network Health - Hosts small ping test (connectivity check)" and "Network Health - Hosts large ping test (MTU check)" check in the vSAN Health Service, and provides details on why they might report errors.

Environment

VMware vSAN 6.x
VMware vSAN 7.x

VMware vSAN 8.x

Resolution

Q: What does the Network Health - Hosts small ping test (connectivity check) check and Hosts large ping test (MTU check) check do?

• While most network-related vSAN health checks assess various aspects of the network configuration, this health check takes a more active approach. Because vSAN lacks visibility into the physical network configuration, it verifies IP connectivity by having every ESXi host ping every other ESXi host on the vSAN network.
• Hosts Small Ping Test (Connectivity Check) - This health check automates the ping process between all ESXi hosts in the vSAN cluster. In this test, all nodes actively ping all other nodes to ensure basic end-to-end IP connectivity exists across the vSAN network.
• Hosts Large Ping Test (MTU Check) - This check complements the basic connectivity test. Increasing the Maximum Transmission Unit (MTU) size improves network performance. Incorrectly configured MTUs frequently do not trigger a vSAN network partition alarm; instead, they cause performance issues, I/O errors in individual objects, or virtual machine deployment failures. The success of this large ping test is critical for vSAN cluster stability.
• While the basic test uses small packets, the MTU check uses large packets (9000 bytes), commonly known as jumbo frames. If the small ping test succeeds, the large ping test also succeeds, provided the MTU size remains consistently configured across all VMkernel adapters (vmknics), virtual switches, and physical switches.
  
  Note:
  
  
  • If the source vmknic has an MTU of 1500, it fragments the 9000-byte test packet. These fragments travel normally along the network to the destination ESXi host for reassembly. The test passes as long as all network devices in the path use an MTU of 1500 or higher.
  • If the vmknic has an MTU of 1500 and the physical switch has an MTU of 9000 (for example, to support iSCSI traffic concurrently), no issue occurs, and the test passes.
  • A failure occurs if the vmknic has an MTU of 9000 but the physical switch enforces an MTU of 1500. Because the source does not fragment the packet, the physical switch drops it.
    

     

Q: What does it mean when it is in an error state?

• A failure in the small ping test indicates a network misconfiguration. The test sends three pings and considers a single lost ping a failure. This issue stems from various factors within either the virtual network (vmknic, virtual switch) or the physical network (cable, physical NIC, physical switch). Examine the other network health check results to identify the root cause. If the remaining health checks show a healthy ESXi configuration, the problem likely resides in the physical network.
• This ping test uses very small packets to ensure basic connectivity.
• A failure in the large ping test indicates an MTU misconfiguration somewhere in the vSAN network. Resolving this requires tracing the source of the misconfiguration to the VMkernel adapters, virtual switches, or physical network switches.

1. 1. Ensure the MTU is consistently configured across the entire cluster.
   2. If the data nodes or the witness appliance retain the default MTU of 1500, the error message indicates a failure when the test sends a 9000-byte packet over the network. When the MTU is 1500 and this test fails, it means a device somewhere in the network has the "Don't Fragment" (DF) flag set. Applications send packets of any size, and the network is responsible for delivering them. Normally, the DF flag remains unset. In a standard configuration, oversized packets fragment into MTU-sized pieces and successfully reassemble at the destination. However, if the DF flag is set, oversized packets cannot fragment and fail to transmit. To resolve this, clear the Don't Fragment flag across all network devices. If clearing the flag is not an option, contact VMware Support for further evaluation before silencing the health check.

Q: How does one troubleshoot and fix the error state?

1. Identify the VMkernel port (vmknic) used for vSAN traffic by running the following command: esxcli vsan network list

2. Perform a small packet ping test. Ping another vSAN node in the cluster using the vmknic identified in step one: vmkping -I vmk# <vSAN Node>

3. Perform a large packet ping test: vmkping -I vmk1  -s 8972 <vSAN Node>

Note: If the vSAN traffic MTU is 1500 and this test fails, it indicates a device on the network has the "Don't Fragment" (DF) flag set. Clear the DF flag everywhere along the network path. If clearing the DF flag is not an option, collect support bundles for vCenter, the ESXi hosts, and NSX (if applicable), and contact VMware Support for further evaluation before silencing the health check.

4. If using jumbo frames, test the "do-not-fragment" (-d) switch; otherwise, skip this step: vmkping -I vmk1 -d -s 8972 <vSAN Node>

Note: The -d flag sets the "do not fragment" option on the vmkping command. If you omit this option, the packet fragments and does not provide valid results.

Q: Will TCP/IP packets become fragmented on communication between the vSAN witness node and the vSAN data nodes?

• The vSAN data nodes typically communicate with the witness over the Management network on vmk0 (MTU=1500). The witness communicates with the data nodes on either vmk0 or vmk1 (both usually MTU=1500), depending on the witness network configuration. Meanwhile, the vSAN data nodes communicate with each other using whichever vmk# is configured with the vSAN tag. Because the vSAN data nodes do not communicate with the witness on the vSAN-tagged vmk# (usually MTU=9000), packet fragmentation does not occur despite the typically different MTU sizes.