Direct connect vSAN 2-node stretched cluster is network partitioned and has basic unicast connectivity error in skyline health

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Direct connect 2-node vSAN Stretched cluster appear partitioned, despite no networking changes after initial setup, or a host reboot or a cluster shutdown and restart.

• The two data nodes are partitioned and the alert for "vSAN Basic (unicast) connectivity" fails between them.

Environment

VMware vSAN (All Versions)

Cause

• With direct connect 2-node vSAN stretched clusters, if NIC teaming and failover settings have multiple uplinks set to active, it's possible for each host to power back up using different vmnics.

• This will result in the directly connected hosts to not be able to communicate and partition, since vSAN only uses one NIC.

Resolution

There are 2 options to correct this issue a resolution and a workaround. 

Resolution:

• Change NIC teaming and failover settings to Active/Standby so both hosts use the same vmnic as Active, and a same alternate vmnic as standby.

• This will ensure that the hosts are always utilizing the same vmnic and will avoid this issue.  Please see: NIC teaming in ESXi and ESX and process to configure.

Workaround:

• This is a temporary solution and it can reoccur on any host reboot. 

• If changing the vmnics from Active/Active to Active/Standby is not possible at the moment, validate which vmnic is being utilized via "esxtop" command from SSH of the host, press option N on both hosts. Once the the current vmnic in use on both hosts is identified down the active vmnic on one host  to force the traffic across the same vmnics. 
          esxcli network nic down -n <vmnicx>

• Once it has been validated that the vSAN cluster has formed bring up the vmnic which was marked down. 
          esxcli network nic up -n <vmnicx>