vSAN Health Service - Performance Service - All hosts contributing stats
search cancel

vSAN Health Service - Performance Service - All hosts contributing stats

book

Article ID: 326431

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

This article explains the performance service - All hosts contributing stats check in the vSAN Health Service and provides details about why it might report an error.

Environment

Resolution

Q: What does the Performance Service - All hosts contributing stats check do?
 
A: This check verifies that all hosts which are currently part of the same network partition are contributing statistics to the collection. Any host that is not in the same network partition will not be checked. Use the general network health checks to assess this aspect of overall cluster health. This check is used to verify that the vSAN performance service can communicate with the host and retrieve statistics.

Q: What does it mean when it is in an error state?
 
A: If this check is Yellow, it means some hosts are not contributing performance statistics.
 
For more information about this issue, refer to the following article: vSAN Host Not Contributing Stats reports with SSL error.

Q: How does one troubleshoot and fix the error state?
 
A: There are no known causes for this health check to fail. If it fails consistently, contact VMware Support and file a Support Request. Provide support bundles for the entire cluster, including the vCenter Server and all hosts.
  • If we have set the parameter to ignore the member list updates from vCenter, vCenter fails to fetch the stats from the hosts. 

To review and address this issue, follow the below steps.

To check if the value is set on hosts, we need to run the below command on all the hosts.

# esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates

If we see any host set the above parameter as 1 it means hosts are not allowing any updates from vCenter.

To fix this, please modify the parameter to 0 which is default, using below command on the hosts where we see it is set to 1.

# esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates

After fixing this parameter, we will need to perform re-test on vSAN health and check if we still see the issue.

  • You can also try to disable and then re-enable the performance service on the vSAN cluster (only applicable for ESXi version 7.x): 
    1. Navigate to the vSAN cluster.
    2. Click the Configure tab.
    3. Under vSAN, select Services.
    4. Click the Performance Service Edit button.
    5. Click to enable vSAN Performance Service.
    6. Select a storage policy for the Stats database object.
    7. (Optional) Click to enable the verbose mode. This check box appears only after enabling vSAN Performance Service. When enabled, vSAN collects and saves the additional performance metrics to a Stats DB object. If you enable the verbose mode for more than 5 days, a warning message appears indicating that the verbose mode can be resource-intensive. Ensure that you do not enable it for a longer duration.
    8. (Optional) Click to enable the network diagnostic mode. This check box appears only after enabling vSAN Performance Service. When enabled, vSAN collects and saves the additional network performance metrics to a RAM disk stats object. If you enable the network diagnostic mode for more than a day, a warning message appears indicating that the network diagnostic mode can be resource-intensive. Ensure that you do not enable it for a longer duration.
    9. Click Apply.

         Refer the link: Configure vSAN Performance Service

  • In some cases 'Stats Primary' host (https://knowledge.broadcom.com/external/article/385769) might log (in /var/run/log/vsanmgmt.log): 'VMK vmkX can not connect to host', example:

    YYYY-MM-DDT15:05:02.517Z error vsand[##########] [opID=vsan-############ statscollector::RetrieveRemoteStats] VMK vmk3 can not connect to host ##.##.##.##. Traceback (most recent call last):   File "/usr/lib/vmware/vsan/perfsvc/statscollector.py", line 1117, in RetrieveRemoteStats   File "/lib64/python3.8/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>   File "/lib64/python3.8/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod   File "/lib64/python3.8/site-packages/pyVmomi/SoapAdapter.py", line 1523, in InvokeMethod   File "/lib64/python3.8/site-packages/pyVmomi/SoapAdapter.py", line 1602, in GetConnection   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1520, in __call__   File "/lib64/python3.8/http/client.py", line 1259, in request   File "/lib64/python3.8/http/client.py", line 1305, in _send_request   File "/lib64/python3.8/http/client.py", line 1254, in endheaders   File "/lib64/python3.8/http/client.py", line 1014, in _send_output   File "/lib6
    2025-09-11T15:05:02.517Z warning vsand[2231522] [opID=vsan-############ statscollector::RetrieveRemoteStats] No available vmknic to retrieve remote stats.

    If seen, user could solve it by logging into non-contributing host using SSH and refresh ESXi firewall rules by running:

    esxcli network firewall refresh

Additional Information

 

 

Powered by Wolken Software