After upgrading the vSAN cluster to 8.0U2, vSAN Health may report warning "Hosts with connectivity issues." and/or "All hosts contributing stats."
[root@esxihost1:/etc/vmware/ssl] esxcli vsan health cluster get -t "Hosts with connectivity issues"Hosts with connectivity issues redHosts with communication issuesHost---------------10.2#.##.310.2#.##.2
socket.timeout messages are seen, such as:
YYYY-MM-DDTHH:MM:SS.MSZ Er(11) vsand[18496974] [opID=vsan-###################### statscollector::RetrieveRemoteStats] VMK vmk3 can not connect to host 10.xxx.xxx.1. YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] Traceback (most recent call last): YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/usr/lib/vmware/vsan/perfsvc/statscollector.py", line 1192, in RetrieveRemoteStats YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/lib64/python3.8/site-packages/pyVmomi/VmomiSupport.py", line 598, in <lambda> YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/lib64/python3.8/site-packages/pyVmomi/VmomiSupport.py", line 388, in _InvokeMethod YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/lib64/python3.8/site-packages/pyVmomi/SoapAdapter.py", line 1527, in InvokeMethod YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/lib64/python3.8/site-packages/pyVmomi/SoapAdapter.py", line 1611, in GetConnection YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1770, in __call__ YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/lib64/python3.8/http/client.py", line 1259, in request YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/lib64/python3.8/http/client.py", line 1305, in _send_request YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/lib64/python3.8/http/client.py", line 1254, in endheaders YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/lib64/python3.8/http/client.py", line 1014, in _send_output YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/lib64/python3.8/http/client.py", line 954, in send YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1914, in connect YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/lib64/python3.8/http/client.py", line 1421, in connect YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/lib64/python3.8/http/client.py", line 925, in connect YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1906, in vsanperf_create_connection YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1869, in VsanPerfCreateConnection YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1860, in VsanPerfCreateConnection YYYY-MM-DDTHH:MM:SS.MSZ Er(11)[+] vsand[18496974] socket.timeout: timed out
VMware vSAN 8.0U2 and higher
Prior to vSAN version 8.0 U2 the vSAN master host retrieves remote host stats via port 80 and from vSAN 8.0 U2 and later builds, port 443 is used.
If ESXi host firewall has blocked port 443 port (ruleset vSphereClient) for vSAN network. It does not populate the vSAN IPs under allowed IP list.
The below output shows the vSAN vmkernel ports are not added under vSphere client allowed IP list.
[root@EX2:~] esxcli network firewall ruleset allowedip listRuleset Allowed IP Addresses--------------------------- --------------------sshServer All
updateManager AllfaultTolerance AllwebAccess AllvMotion AllvSphereClient 19#.16#.#.###,19#.16#.#.###,19#.16#.#.###,19#.16#.#.###,19#.16#.#.### >>>>>>>>> Missing vSAN vmk IPs <<<<<
To add the vSAN IPs to vSphereClient Rule via vCenter see Add Allowed IP Addresses for an ESXi Host by Using the VMware Host Client.
From ESXi command line do the following:
Output should be as below.
[root@EX2:~] esxcli network firewall ruleset allowedip listRuleset Allowed IP Addresses--------------------------- --------------------vSphereClient 19#.16#.#.###, 19#.16#.#.###, 19#.16#.#.###, 19#.16#.#.###, 19#.16#.#.###, 10.2#.##.1, 10.2#.##.2, 10.2#.##.3
Restart the vSAN Health Service on vCenter by running "service-control --stop vmware-vsan-health && service-control --start vmware-vsan-health".
Also, restart the vsanmgmt service on the impacted as well as the performance master node using the below command:
/etc/init.d/vsanmgmt restart
More information on Configuring the ESXi Firewall