vSAN skyline health reports errors: vSAN: Basic (unicast) connectivity check and vSAN: MTU check (ping with large packet size)

Products

• VMware vSAN

Issue/Introduction

• vSAN Skyline health check reports below errors:
  vSAN: Basic (unicast) connectivity check.
vSAN : MTU check (ping with large packets) 
Vsan object health
• To validate the faulty host, click on the Troubleshoot option for the primary issue "vSAN: Basic (unicast) connectivity check".
  • This will ping between vSAN vmkernels on hosts in the vSAN cluster, and gives a report that will tell if any hosts are unreachable. It also shows which vmkernel is used for vSAN, and what size packet (MTU) was used for the ping test. 
• Validate MTU for the vSAN vmkernel is the same as the MTU configured on the port group that vmkernel is assigned to on each host. 
  
  • In vSphere select a host from inventory > Configure > Networking > VMkernel adapters > select the VMkernel with vSAN service enabled > Properties > NIC settings > MTU
  • In vSphere select a host from inventory > Configure > Networking > Virtual switches > expand the virtual switch used by the vSAN VMKernel > three dots to the far right > View Settings  > MTU 
  • Verify that the MTU in these two places match. 
• Ping the faulty host from a working host using a 1500 MTU and 9000 MTU. (See Testing VMkernel network connectivity with the vmkping command)
  • Example output of failed vmkping tests:
    PING xx.xx.xxx.xx ( xx.xx.xxx.xx): 1472 data bytes

---  xx.xx.xxx.xx ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

PING xx.xx.xxx.xx ( xx.xx.xxx.xx): 8972 data bytes

---  xx.xx.xxx.xx ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

Either of the above tests failing confirms there is an issue preventing network connectivity between vSAN vmkernels on different ESXi hosts. 

Environment

• VMware vSAN 6.x
• VMware vSAN 7.x
• VMware vSAN 8.x
  

Cause

The hosts are unable to communicate with each other over vSAN traffic due to an NIC issues

Resolution

 

• SSH to the ESXi host that is not reachable per the vSAN Skyline Health Check > esxtop > press 'n' key 
• Identify the Active vmnic serving vSAN Traffic from ESXTOP data.
  
  
  In the esxtop output the vmnic in the TEAM-PNIC column across from the vmkernel for the vSAN service is the active vmnic serving vSAN traffic. 
  
  
• Enter the host into Maintenance mode to ensure all active VMs migrated to other hosts in the cluster.
• Bring down the active vmnic, if there is more than one vmnic in active or standby in the vSAN vmkernel port group's teaming settings. (See How to configure NIC teaming in ESXi and ESX)
  
  • esxcli network nic down -n vmnicX (where vmnicX is the vmnic being used by the vSAN vmkernel)
• Observe the vmnic failover using ESXTOP data by confirming there is a different vmnic under the TEAM-PNIC column after administratively downing the active vmnic. 
  
  
  
• After the vmnic failover, perform the same vmkping tests between affected hosts.
  
  1. "vmkping -I vmkX -d -s 1472 <ip address>"
  2. "vmkping -I vmkX -d -s 8972 <ip address>"
• If one NIC fails to serve traffic and the other does not, involve internal networking team or hardware vendor to investigate and resolve the issues with vmnic.
• If only one host is unreachable over the vSAN network, it is most likely an issue with the upstream physical switchport configuration(s). 
  • Another item to check when looking at a host in vSphere would be Configuration > Networking > Physical Adapters > expand the vmnic(s) used by the vSAN vmkernel and check the CDP/LLDP tab. This information can help determine if the correct VLAN is configured on the upstream physical switch port, or if the vmnic has been physically connected to the wrong physical switch port.