Lost communication between ESXi hosts in vSAN cluster.

Products

• VMware vSAN

Issue/Introduction

vSAN: MTU CHECK (PING WITH LARGE PACKET SIZE) alert is reported on the Skyline health.

Multiple other alerts about the following are reported cause of the mtu check alert:

- Witness host not found.

- vSAN: Basic (unicast) connectivity check

- vSAN object health.

 

Environment

vSAN OSA.

Version - 8.0 U3.

 

Cause

Hosts cannot communicate among themselves via vmk, which is used for vSAN communication.

 

Resolution

Check the ping among esxi hosts in the cluster with vmk used for vSAN communication on the esxi host.

 

• Run the following command to identify the VMkernel port used for vSAN, and copy the output for later use: 

esxcli vsan network list

[root@server name:~] esxcli vsan network list
Interface
   VmkNic Name: vmk1
   IP Protocol: IP
   Interface UUID: ########-####-####-####-############
   Agent Group Multicast Address: 
   Agent Group IPv6 Multicast Address: 
   Agent Group Multicast Port: 
   Master Group Multicast Address: 
   Master Group IPv6 Multicast Address: 
   Master Group Multicast Port: 
   Host Unicast Channel Bound Port: 
   Data-in-Transit Encryption Key Exchange Port: 
   Multicast TTL: 
   Traffic Type: vsan

            Note: Take note of the VmkNic Name - in the above output it's "vmk1"

 

• Ping another vSAN node in the cluster using the vmknic found in step one.

vmkping -I vmk# <vSAN Node IP>

 

• Perform a packet ping test, if MTU is 1500 use 1472, if MTU is 9000 use 8972.

vmkping -I vmk# <vSAN node IP> -d -s 8972

 

Action plan:

If the Ping test is not successful with packet loss, customer need to involve their internal network team to fix the configuration.