vSAN Host Not Contributing Stats reports with SSL error

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• You see a Warning in Cluster > Monitor > Virtual SAN > Performance service > All hosts contributing stats.
• One or more ESXi hosts are listed in the Hosts Not Contributing Stats field.

Environment

VMware vSAN 6.1.x
VMware vSAN 6.0.x
VMware vSAN 6.2.x
VMware vSAN 6.5.x
VMware vSAN 6.6.x

Cause

This issue might occur when there is an issue with the CA certificate which cause the SSL connection between hosts to fail. This failure will prevent the collection of vSAN stats metrics from the unconnected hosts.

To verify if this the issue, enable debug logging for vsanmgmt on the StatsMaster node (or/and one of the Not Contributing nodes).

To check which host is the StatsMaster:

1. Log in as root to the ESXi.
2. Run the following command to check the Perf Service Node information:
   
   #esxcli vsan debug object list | grep -B35 "Directory Name: .vsan.stats" | head -5 ;

Object UUID: 009fce5b-c0c3-774e-8e3e-ecf4bbec6050

Version: 7

Health: healthy

Owner: is-tse-d155.isl.vmware.com     >>    StatsMaster Host 
                                                                                                                                                                                                                                                

Or
 

# python /usr/lib/vmware/vsan/perfsvc/vsan-perfsvc-status.pyc  svc_info

--------Perf Service Stats Object Dir Content--------
total 780288
-rw-r--r--    1 root     root         28672 Apr 10 05:15 config.db
-rw-rw-rw-    1 root     root             0 Apr 10 05:15 stats.db.lck
.
.
--------Perf Service Node Information--------
(vim.cluster.VsanPerfNodeInformation) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   version = '6.5.0',
   hostname = <unset>,
   error = <unset>,
   isCmmdsMaster = true,
   isStatsMaster = true,
   vsanMasterUuid = '548b5e07-28cd-1415-b0c7-3417ebe523ea',
 

Or

 

localcli vsan cluster get | awk -F "Sub-Cluster Master UUID:" '{print $2}' | sed -e 's/^[ \t,^$]*//' | sed '/^$/d' |while read i ; do clear ; echo "Following is vSAN Master node also serving as Statsmaster " ; echo " " ; cmmds-tool find -f json -t HOSTNAME |grep -E "uuid|content"|sed 'N;s/\n/ /'|awk -F \" '{print $10": " $4}'|sort | grep $i ; echo " " ; done ; 

To change the log level:

 

1. Log in as root to the ESXi.
2. Go to /etc/vmware/vsan/:
   
   cd /etc/vmware/vsan/
    
3. Edit the vsanperf.conf file:
   
   vi vsanperf.conf
    
4. Modify the following entries (Esc and :wq! to save and quit):
   
   loglevel = debug
   logrotate = 10
    
5. Restart the vsanmgmt service:
   
   /etc/init.d/vsanmgmtd restart
   
   In the /var/log/vsanmgmt.log file, you will see the following error:
   
   VSANMGMTSVC: DEBUG vsanperfsvc[Collector-0] [statscollector::SampleHostStats] collecting remote stats for host 192.168.0.11 from VSI
   VSANMGMTSVC: DEBUG vsanperfsvc[Collector-1] [statscollector::RetrieveRemoteStats] Unexpected error during RetrieveRemoteStats:<class 'ssl.SSLEOFError'>
   VSANMGMTSVC: "/build/mts/release/bora-4192238/bora/build/esx/release/vmvisor/sys/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1005, in __call__
   File "/build/mts/release/bora-4192238/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/ssl.py", line 911, in wrap_socket
   File "/build/mts/release/bora-4192238/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/ssl.py", line 579, in __init__
   File "/build/mts/release/bora-4192238/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/ssl.py", line 808, in do_handshake
   SSLEOFError: EOF occurred in violation of protocol (_ssl.c:590)

Note: Ensure to change the log level to its previous value.

Resolution

To resolve the issue, renew the host certificate for each problematic node:

1. Log in to the vCenter Server using the Web Client.
2. Go to the Host and click > Manage/Configure > Certificate under System.
3. Click Renew.
4. Log in to the vCenter Server and restart the vsan-health service. This will renew and publish the certificate.
   • On Windows vCenter Server, use Service Manager.
   • On vCenter Server Appliance, run the command:
     • service-control --stop vmware-vsan-health
     • service-control --start vmware-vsan-health
5. SSH into the host(s) and ensure that the size of the CA file /etc/vmware/ssl/castore.pem is not zero byte in size.

The host must be Contributing Stats (may take up to 5 minutes as per collection interval).

Additional Information