vSAN in errored state, unable to manage VM's or hosts.

Products

• VMware vSAN

Issue/Introduction

• Some VMs will show inaccessible even though all the objects in vSAN Skyline Health is showing available 
• Over all cluster performance seems to be degraded
• You might see Resyncs happening but the resync rate is slow 

Environment

7.x

Resolution

Validate if the issue is the result of an ongoing networking issue:

 

1. Validate in the vmkernel logs of host to see if there are any heartbeat timeout happening on vSAN objects
    [vmfsCorrelator] 1330954078562us: [vob.vmfs.heartbeat.timedout] ########-########-####-############ ########-####-####-####-############
   

   [vmfsCorrelator] 1330954078562us: [vob.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume ########-########-####-############ (########-####-####-####-############): [Timeout]

2. Do a vmkping test across hosts on the vSAN kernel interface:
   vmkping -I vmk2 192.168.#.# -d -s 8972
3. If ping fails, put the host in maintenance mode or bring down a specific vmnic (if multiple are available) to test traffic on another
   esxcli network nic down -n vmnic0
4. Validate the performance of the cluster 

 

 

 

Additional Information

• More storage troubleshooting steps can be checked on Troubleshooting vSAN OSA disk issues
• More Network troubleshooting steps can be checked on Troubleshooting vSAN Networking