vSAN Errors - "vSAN: Basic Connectivity Check" "vSAN cluster partition" "vSAN object health" after physical network change.

Products

• VMware vSAN

Issue/Introduction

After physical network change Multiple vSAN errors are present in the vSAN skyline health. 

 

Environment

VMware vSAN (all versions)

Cause

Cause 1 :

Witness host became network isolated from the rest of the environment after a change was made on the tunnel that connects the physical site with the witness site. 

When viewing the "vSAN cluster partition" the witness host is network partitioned. 

Per the "vSAN basic connectivity check" the witness host is currently isolated form the rest of the cluster via network isolation. 

The Basic connectivity check for vSAN, also known as a unicast or small ping test, automates the pinging of each ESXi host from each of the other ESXi hosts in the vSAN cluster and ensures that connectivity between all the ESXi hosts on the vSAN network exists. In this test, all nodes ping all other nodes in the cluster.

If the Basic connectivity tests fail, it indicates that the network is misconfigured. The test sends 3 pings. If one ping is lost, the check considers this a failure. This could be caused by many factors, and the issue may be in the virtual network (vmknic, virtual switch) or the physical network (cable, physical NIC, physical switch). 

Cause 2 :

The IP address assigned to vSAN VMK has been used for other node in the network , causing the duplication of IP address

The ESXI host logs /var/run/log/vobd.log

vobd.log: [netCorrelator] 1810650199us: [vob.net.vmknic.ip.duplicate] A duplicate IP address was detected for <IP ADDRESS> on interface < VSAN VMK>. The current owner is ( MAC address xx:xx:xx:xx:xx:xx.)
vobd.log: [netCorrelator] 1803764829us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for <IP ADDRESS> on interface < VSAN VMK>, current owner being ( MAC address xx:xx:xx:xx:xx:xx.)

 

Resolution

Work with your network vendor to determine what change was made between the physical hosts and witness that resulted in the loss of network communication.

Reference kb 

vSAN Health Service - Network Health - Hosts small ping test (connectivity check) and Hosts large ping test (MTU check)