vCenter showing few hosts not contributing stats in a stretched cluster

Products

• VMware vSAN

Issue/Introduction

Symptoms:
You see the below entries in /var/log/vsanmgmt.log from ESXi  
----------------------------------------------------------------------------------
2019-07-22T02:50:21Z VSANMGMTSVC: DEBUG vsanperfsvc[Collector-Main] [statscollector::_FetchAndCalculateStats] waiting for stats readiness
2019-07-22T02:50:25Z VSANMGMTSVC: DEBUG vsanperfsvc[MainThread] [statsdaemon::HandleRoleChanges] detecting role transition... (newRole = master, oldRole = master)
2019-07-22T02:50:25Z VSANMGMTSVC: DEBUG vsanperfsvc[MainThread] [statsdb::GetStatsDBDir] VDFS is disabled
2019-07-22T02:50:25Z VSANMGMTSVC: DEBUG vsanperfsvc[MainThread] [statsdb::OwnerLock] StatsDB is locked
2019-07-22T02:50:25Z VSANMGMTSVC: WARNING vsanperfsvc[Collector-2] [statscollector::RetrieveRemoteStats] Error happened during RetrieveRemoteStats of host 10.9.24.11, type: , message: timed out

2019-07-22T02:50:25Z VSANMGMTSVC: DEBUG vsanperfsvc[Collector-2] [statscollector::SampleHostStats] Got stats sample
2019-07-22T02:50:25Z VSANMGMTSVC: DEBUG vsanperfsvc[Collector-2] [statscollector::SampleHostStats] waiting for collection task...
2019-07-22T02:50:25Z VSANMGMTSVC: DEBUG vsanperfsvc[Collector-Main] [statscollector::_FetchAndCalculateStats] waiting for stats readiness
2019-07-22T02:50:25Z VSANMGMTSVC: WARNING vsanperfsvc[Collector-3] [statscollector::RetrieveRemoteStats] Error happened during RetrieveRemoteStats of host 10.9.24.14, type: , message: timed out

Ping tests between vSAN hosts are successful 

esxcli network firewall ruleset allowedip list shows as "All", then no traffic should be blocked

Use the below link to validate that vSAN ports are not blocked.
https://ports.esp.vmware.com/home/vSAN

Environment

VMware vSAN 6.0.x

Cause

Perfsvc chose the witness network to collect metrics instead of the vSAN network

The vSAN perf service doesn't differentiate vSAN witness and vSAN data network, so this is the reason the "host not contribute" warning happened.

Resolution

This issue has been fixed in 6.7U1 and higher.

Workaround:
1. Write down all configuration of the vSAN traffic interface and witness traffic interface(e.g. in host spt-svr-vx1-esx01, vsan traffic interface is vmk3 and witness traffic interface is vmk5), the configuration includes the "Network label", "Switch", IP address, etc.

2. Remove the vsan traffic interface and witness traffic interface, for example, in host spt-svr-vx1-esx01, vsan traffic interface is vmk3 and witness traffic interface is vmk5.

3. Add the interface which will enable vsan traffic, configure this interface, and enable vsan traffic, wait several seconds until the vsan traffic enabling operation completed.

4. Add the interface which will enable witness traffic interface, configure this interface, and enable witness traffic.