Several vsan alarms are intermittently triggered

Products

• VMware vSAN

Issue/Introduction

These three vsan health status changed to unhealthy status repeatedly.

# 1
MM DD hh:mm:ss ##### vpxd[54296]: Event [3430475] [1-1] [YYYY-MM-DDThh:mm:ss.#####Z] [vim.event.EventEx] [error] [vSAN Health] [#####] [3430475] [vSAN Health Test 'vSAN host daemon liveness' status changed from 'green' to 'red']
MM DD hh:mm:ss ##### vpxd[54296]: Event [3430475] [1-1] [YYYY-MM-DDThh:mm:ss.#####Z] [vim.event.EventEx] [error] [vSAN Health] [#####] [3430475] [vSAN Health Test 'vSAN host daemon liveness' status changed from 'red' to 'green']

# 2
MM DD hh:mm:ss ##### vpxd[54296]: Event [3430463] [1-1] [YYYY-MM-DDThh:mm:ss.#####Z] [vim.event.EventEx] [warning] [vSAN Health] [#####] [3430463] [vSAN Health Test 'vSAN configuration consistency' status changed from 'green' to 'yellow']
MM DD hh:mm:ss ##### vpxd[54296]: Event [3430463] [1-1] [YYYY-MM-DDThh:mm:ss.#####Z] [vim.event.EventEx] [warning] [vSAN Health] [#####] [3430463] [vSAN Health Test 'vSAN configuration consistency' status changed from 'yellow' to 'green']

# 3
MM DD hh:mm:ss ##### vpxd[54296]: Event [3430462] [1-1] [YYYY-MM-DDThh:mm:ss.#####Z] [vim.event.EventEx] [warning] [vSAN Health] [#####] [3430462] [vSAN Health Test 'Host issues retrieving hardware info' status changed from 'green' to 'yellow']
MM DD hh:mm:ss ##### vpxd[54296]: Event [3430462] [1-1] [YYYY-MM-DDThh:mm:ss.#####Z] [vim.event.EventEx] [warning] [vSAN Health] [#####] [3430462] [vSAN Health Test 'Host issues retrieving hardware info' status changed from 'yellow' to 'green']

Environment

VMware vSphere ESXi 7.0.x
VMware vSAN 7.x

Cause

This could happen because of the unstable vSAN node due to vSAN Management (vsanmgmt service) memory leak in ESXi 7.0 U1 and 7.0U2.

Resolution

Based on the vSAN Management (vsanmgmt service) memory leak in ESXi 7.0 U1 and 7.0U2, restart vsanmgmt service on a faulty node. 

You could find a faulty node base on the vmkernel.log like below.

YYYY-MM-DDTHH:MM:SS cpu2:2102824 opID=c61309af)Admission failure in path: host/vim/vmvisor/vsanperfsvc:vsanmgmtd.2102775:vsiHeap.2102775