'Stats primary election', 'vSphere cluster members match vSAN cluster members' and 'Hosts with connectivity issues' vSAN Skyline Health alert triggers frequently and auto resolves

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• The vSAN performance service alarm 'Stats primary election' 'vSphere cluster members match vSAN cluster members' and vSAN network alarm 'Hosts with connectivity issues' are triggered together and auto resolves.

• These alerts keep repeating.

• The host reporting the 'Hosts with connectivity issues' issue keeps changing.

• There are no issues over the management network between the hosts and vCenter. 

• Restarting vsanmgmtd service on master node may help resolve the issue temporarily.

• Recreating the performance stats object does not help resolve the 'Stats primary election' alert as per vSAN Health Service - Performance Service - Stats master election check.

Environment

Vmware Aria Operations 8.16.x

Vmware Aria Operations 8.17.x

VMware vSAN 8.x

Cause

• The 'Stats primary election' alert is triggered because the vsanmgmtd service on the vSAN hosts go to a not responding state frequently.

• This is due to continuous API calls with unsupported parameters, which run into an error while fetching the details being made on the service, thereby overloading it.

• From the ESXi host's /var/run/log/vsanmgmt.log the unsupported parameter call can be seen:

  YYYY-MM-DDTHH:MM:SS.SSSZ error vsand[2110597] [opID=########-#### statsdb::Run] When run command execute for mode normalMode, met exception in DB thread data processing: no such column: throughputDevRead, out is no such column: throughputDevRead Traceback (most recent call last):   File "/usr/lib/vmware/vsan/perfsvc/statsdb.py", line 4535, in Run sqlite3.OperationalError: no such column: throughputDevRead

• The /var/log/vmware/vsan-health/vmware-vsan-health-service.log file on the vCenter report below mentioned errors:
  
  YYYY-MM-DDTHH:MM:SS.SSS+SS:SS WARNING vsan-mgmt[#######] [VsanVcPerformanceManagerImpl::processException opID=66187cdb] Exception: (vmodl.fault.SystemError) {
  msg = "Received SOAP response fault from [<<io_obj p:0x00007fdc30972588, h:##, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-##/vsan>]: queryVsanPerf\nno such column: throughputDevRead",
  reason = "OperationalError('no such column: throughputDevRead')"
}
Traceback (most recent call last):
  File "bora/vsan/perfsvc/vpxd/vpxdPyMo/VsanVcPerformanceManagerImpl.py", line 898, in _RunRegularQuery
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 598, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 388, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
PyCppVmomi.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
  msg = "Received SOAP response fault from [<<io_obj p:0x00007fdc30972588, h:##, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-##/vsan>]: queryVsanPerf\nno such column: throughputDevRead",
  reason = "OperationalError('no such column: throughputDevRead')"
}

• These API calls are coming from VMware Aria Operations.

• Due to these calls the hostd service on the host also goes down causing the 'Hosts with connectivity issues' alert.

• This is a known issue found in VMware Aria Operations 8.16 (prior to Hot Fix 2) & VMware Aria Operations 8.17 (prior to Hot Fix 1) where it queries the vSAN hosts with an unsupported parameter.

Resolution

To resolve this issue:

1. Upgrade VMware Aria Operations to either 8.16 Hot Fix 2 or 8.17 Hot Fix 1.

2. Delete and re-create the vsan stats object- SAN Health Service - Performance Service - Stats master election check.