vSAN Host Not Contributing Stats reports after Cert change due to SSL certificate errors

Products

VMware vSAN

Issue/Introduction

1. You see a Warning in Cluster > Monitor > Virtual SAN > Performance service > All hosts contributing stats.

2. One or more ESXi hosts are listed in the Hosts Not Contributing Stats field.

3. Unable to place ESXi hosts into Maintenance Mode. vCenter Server fails to issue the EnterMaintenanceMode task with generic security or communication errors.

Environment

VMware vSAN (All Versions)

Cause

Following scenarios can cause this error -

1. When there is an issue with the CA certificate of the ESXi host that causes the SSL connection between hosts to fail.

2. When you use VCF Operations to change the certificate of ESXi from self signed to Microsoft CA

3. Manually generating a new RSA key (rui.key) or CSR directly on the ESXi filesystem without using the vSphere Client certificate management tools. This creates an immediate SSL thumbprint mismatch between the host's actual identity and the records held in the vCenter database and vSAN Cluster Member Manager (CMMDS).

NOTE: This failure will prevent the collection of vSAN stats metrics from the disconnected hosts.

vsanmgmt.log:

2026-01-16T20:02:00.263Z Wa(12) vsand[6173677]: [opID=vsan-3d87f8e2-64886aca167c0 statscollector::RetrieveRemoteStats] Error happened during RetrieveRemoteStats of hostIps ['10.#.#.#'], type: <class 'ssl.SSLCertVerificationError'>, message: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1006)
2026-01-16T20:02:00.300Z Wa(12) vsand[6173707]: [opID=vsan-5d561145-64886aca167bf statscollector::RetrieveRemoteStats] Error happened during RetrieveRemoteStats of hostIps ['10.#.#.#'], type: <class 'ssl.SSLCertVerificationError'>, message: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1006)
2026-01-16T20:02:00.581Z Wa(12) vsand[6173693]: [opID=vsan-5b8615b8-64886aca167c2 statscollector::RetrieveRemoteStats] Error happened during RetrieveRemoteStats of hostIps ['10.#.#.#'], type: <class 'ssl.SSLCertVerificationError'>, message: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1006)
2026-01-16T20:02:00.596Z Wa(12) vsand[6173696]: [opID=vsan-3ffc11f9-64886aca167c1 statscollector::RetrieveRemoteStats] Error happened during RetrieveRemoteStats of hostIps ['10.#.#.#'], type: <class 'ssl.SSLCertVerificationError'>, message: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1006)

This issue might occur when there is an issue with the CA certificate which cause the SSL connection between hosts to fail. This failure will prevent the collection of vSAN stats metrics from the unconnected hosts.

To verify if this is the issue, enable debug logging for vsanmgmt on the StatsMaster node (or/and one of the Not Contributing nodes).

To check which host is the StatsMaster:

Log in as root to the ESXi.
Run the following command if using a RAID1 storage policy to check the Perf Service Node information:

#esxcli vsan debug object list | grep -B46 "Directory Name: .vsan.stats" | head -4

Object UUID: ########-####-####-####-########6050

Version: 7

Health: healthy

Owner: host.example.com >> StatsMaster Host

Or

# python /usr/lib/vmware/vsan/perfsvc/vsan-perfsvc-status.pyc svc_info

--------Perf Service Stats Object Dir Content--------
total 780288
-rw-r--r-- 1 root root 28672 Apr 10 05:15 config.db
-rw-rw-rw- 1 root root 0 Apr 10 05:15 stats.db.lck
.
.
--------Perf Service Node Information--------
(vim.cluster.VsanPerfNodeInformation) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
version = '6.5.0',
hostname = <unset>,
error = <unset>,
isCmmdsMaster = true,
isStatsMaster = true,
vsanMasterUuid = '########-####-####-####-########23ea',

Or

localcli vsan cluster get | awk -F "Sub-Cluster Master UUID:" '{print $2}' | sed -e 's/^[ \t,^$]*//' | sed '/^$/d' |while read i ; do clear ; echo "Following is vSAN Master node also serving as Statsmaster " ; echo " " ; cmmds-tool find -f json -t HOSTNAME |grep -E "uuid|content"|sed 'N;s/\n/ /'|awk -F \" '{print $10": " $4}'|sort | grep $i ; echo " " ; done ;

To change the log level:

Log in as root to the ESXi.
Go to /etc/vmware/vsan/:

cd /etc/vmware/vsan/
Edit the vsanperf.conf file:

vi vsanperf.conf
Modify the following entries (Esc and :wq! to save and quit):

loglevel = debug
logrotate = 10
Restart the vsanmgmt service:

/etc/init.d/vsanmgmtd restart

In the /var/log/vsanmgmt.log file, you will see the following error:

VSANMGMTSVC: DEBUG vsanperfsvc[Collector-0] [statscollector::SampleHostStats] collecting remote stats for host 192.#.#.# from VSI
VSANMGMTSVC: DEBUG vsanperfsvc[Collector-1] [statscollector::RetrieveRemoteStats] Unexpected error during RetrieveRemoteStats:<class 'ssl.SSLEOFError'>
VSANMGMTSVC: "/build/mts/release/bora-4192238/bora/build/esx/release/vmvisor/sys/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1005, in __call__
File "/build/mts/release/bora-4192238/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/ssl.py", line 911, in wrap_socket
File "/build/mts/release/bora-4192238/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/ssl.py", line 579, in __init__
File "/build/mts/release/bora-4192238/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/ssl.py", line 808, in do_handshake
SSLEOFError: EOF occurred in violation of protocol (_ssl.c:590)

Note: Ensure to change the log level to its previous value.

Resolution

To resolve the issue, renew the host certificate for each problematic node:

Log in to the vCenter Server using the Web Client.
Go to the Host and click > Manage/Configure > Certificate under System.
Click Renew.
Log in to the vCenter Server and restart the vsan-health service. This will renew and publish the certificate.
- On Windows vCenter Server, use Service Manager.
- On vCenter Server Appliance, run the command:
  - service-control --stop vmware-vsan-health
  - service-control --start vmware-vsan-health
SSH into the host(s) and ensure that the size of the CA file /etc/vmware/ssl/castore.pem is not zero byte in size.

The host must be Contributing Stats (may take up to 5 minutes as per collection interval).

Reset the alarm status to GREEN & then Reboot all hosts in the cluster by putting them in Ensure Accessibility Mode (vSAN services will not update the certificate until a reboot is performed).

Recovery when user Manually generated new RSA key

If manual changes were made, restore the original rui.key and rui.crt from backup.
Restart all management agents to reload the management agents with the valid certificate.
If Manually generation and replacement of certs is required place host in to MM before starting the replacement process.

Additional Information

vSAN Health Service - Performance Service - All hosts contributing stats (326431)