vCenter VM fails to power on following vSAN witness connectivity loss

Products

• VMware vSAN

Issue/Introduction

• vCenter Server VMs fail to power on.
  
  
• vSAN objects are in a "Reduced availability with no rebuild" state.
  
  
• vSAN Witness components are marked as Stale.
  
  
• Cluster is fully formed.
  
  

Environment

VMware vSAN 7.x
VMware vSAN 8.x

Cause

Some data nodes cannot reach the witness node over vmkping on vmkernal port which is tagged for witness traffic.

• vmkping -I vmk# <Witness-IP> from some vSAN data nodes results in 100% packet loss.

 

From witness node ping tests over vSAN vmk are unsuccessful for some data nodes.

• vmkping -I vmk# <Data-Node-IP> from the vSAN Witness Appliance is successful for some data nodes and results in 100% packet loss for other nodes.

Resolution

Remove the Witness traffic tagging from the dedicated interface on all data nodes to force Witness traffic over the standard vSAN path as this is reachable.

esxcli vsan network ip interface tag remove -i vmk# -t VSANWitness

Repeat these steps on all affected data nodes in the cluster.

Verify that vmkping to the Witness node is now successful over the default vSAN path.

See vSAN Witness Traffic Failure Due to vmk0# Network Communication Loss