vSAN Health Service - Performance Service - Stats primary election check

Products

VMware vSAN

Issue/Introduction

This article explains the Performance Service - ‘Stats primary election’ check in the vSAN Health Service and provides details about why it might report an error.

There should be only 1 host acting as the stats primary keeping track of the performance of the entire vSAN cluster. The alert will show when there are no hosts acting as the stat primary.

Impact/Risks:
Performance stats are not being collected for the cluster.

Environment

VMware vSAN (All Versions)

Resolution

Q: What does the Performance Service - Stats primary election check do?

A: This check verifies that a Stats primary exists in the cluster. The Stats primary is responsible for collecting statistics from agent hosts.

In a healthy vSAN cluster with the performance service enabled only one Stats Primary exists. The Stats primary must be a CMMDS primary. In a partitioned vSAN cluster, multiple CMMDS primary's can be present, but only one Stats primary is allowed.

Q: What does it mean when it is in an error state and how do I troubleshoot the issue?

A: This check is in a Red error state if there is no Stats primary elected.

Ensure vCenter is not down rev of ESXi as vCenter should always be equal to or newer than the ESXi version to ensure proper communication, otherwise it can lead to false positives or API calls failing. See KB vCenter version to ESXi version for more information on this.

Check the health of the vSAN cluster by looking at vSAN Health Service ensuring all other health tests or green healthy state. Also, make sure that the performance service is enabled at vSAN> Configure> Services.

Try restarting the vSAN management service on all hosts in the cluster by running:
/etc/init.d/vsanmgmtd restart

If the cluster is healthy and not actively troubleshooting a performance issue you can try disabling/re-enabling vSAN Performance Service. This can be done via the vCenter Web UI for 7.x

Note: Disabling/re-enabling Performance Service will wipe out any existing performance data.

In vCenter 8.0, to disable and re-enable vSAN Performance service, follow the below steps:

Log into RVC via vCenter SSH.
1. Open vCenter SSH.
2. Log into shell
3. Run the command: rvc
4. Login with administrator@<domain>@localhost
cd into localhost > datacenter > computers > vSAN Cluster
Then run the command to delete the vSAN performance object: vsan.perf.stats_object_delete .
Once deleted, proceed to create the vSAN performance object by using the command: vsan.perf.stats_object_create .

Note: You can also re-enable Performance Service via the vCenter Web Client once the .vsan.stats object has been deleted.

Ensure there is only one .vsan.stats folder on the vSAN datastore as there should ONLY be one .vsan.stats folder/object per vSAN cluster. If there is more than one .vsan.stats folder/object this can negatively impact the Stats Primary election process. See KB vSAN -- vSAN Health Service - Performance Service - Stats DB object conflicts to delete the additional .vsan.stats objects. Normally this other alert would be triggered if multiple .vsan.stats objects are detected, however this may not trigger if the path for the object is listed as (Missing)

If all of these checks are confirmed healthy and the above troubleshooting steps doesn't clear the alert then collect the logs for the entire cluster including vCenter logs, file a Support Request and upload the logs for analysis.

Q: How to locate the elected stats primary ESXi host in vCenter Web-Client?

A: Open the VCenter Web-Client by login as [email protected].

Host and Cluster View >> Monitor >> vSAN Health >> Performance Service > Stats Primary Election
The right side pane will display the host which is currently working as stats primary