"Hosts with connectivity issues" or "All hosts contributing stats." vSAN Skyline health warning
search cancel

"Hosts with connectivity issues" or "All hosts contributing stats." vSAN Skyline health warning

book

Article ID: 376822

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms: 

  • After upgrading the vSAN cluster to 8.0U2, vSAN Health may report warning "Hosts with connectivity issues." and/or "All hosts contributing stats."

  • vSAN nodes may not communicate to each other. Sample manual health check output show as below.

[root@esxihost1:/etc/vmware/ssl]  esxcli vsan health cluster get -t "Hosts with connectivity issues"
Hosts with connectivity issues        red

Hosts with communication issues
Host
---------------
10.2#.##.3
10.2#.##.2

  • Referring to collected vsanmgmt.log on the vSAN master node, socket.timeout messages are seen, such as: 

2024-08-28T05:55:56.044Z Er(11) vsand[18496974] [opID=vsan-23581f33-620b795481de0 statscollector::RetrieveRemoteStats] VMK vmk3 can not connect to host 10.xxx.xxx.1. 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974] Traceback (most recent call last): 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/usr/lib/vmware/vsan/perfsvc/statscollector.py", line 1192, in RetrieveRemoteStats 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/lib64/python3.8/site-packages/pyVmomi/VmomiSupport.py", line 598, in <lambda> 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/lib64/python3.8/site-packages/pyVmomi/VmomiSupport.py", line 388, in _InvokeMethod 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/lib64/python3.8/site-packages/pyVmomi/SoapAdapter.py", line 1527, in InvokeMethod 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/lib64/python3.8/site-packages/pyVmomi/SoapAdapter.py", line 1611, in GetConnection 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1770, in __call__ 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/lib64/python3.8/http/client.py", line 1259, in request 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/lib64/python3.8/http/client.py", line 1305, in _send_request 

2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/lib64/python3.8/http/client.py", line 1254, in endheaders 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/lib64/python3.8/http/client.py", line 1014, in _send_output 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/lib64/python3.8/http/client.py", line 954, in send 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1914, in connect 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/lib64/python3.8/http/client.py", line 1421, in connect 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/lib64/python3.8/http/client.py", line 925, in connect 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1906, in vsanperf_create_connection 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1869, in VsanPerfCreateConnection 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974]   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1860, in VsanPerfCreateConnection 
2024-08-28T05:55:56.044Z Er(11)[+] vsand[18496974] socket.timeout: timed out  

Environment

VMware vSAN 8.0U2 and higher

 

Cause

  • Prior to vSAN version 8.0 U2 the vSAN master host retrieves remote host stats via port 80 and from vSAN 8.0 U2 and later builds, port 443 is used.

  • If ESXi host firewall has blocked port 443 port (ruleset vSphereClient) for vSAN network. It does not populate the vSAN IPs under allowed IP list.

  • The below output shows the vSAN vmkernel ports are not added under vSphere client allowed IP list.

[root@EX2:~] esxcli network firewall ruleset allowedip list
Ruleset                      Allowed IP Addresses
---------------------------  --------------------
sshServer                    All
updateManager                All
faultTolerance               All
webAccess                    All
vMotion                      All
vSphereClient                19#.16#.#.###, 19#.16#.#.###, 19#.16#.#.###, 19#.16#.#.###, 19#.16#.#.###  >>>>>>>>> Missing vSAN vmk IPs <<<<<

  • So the vSAN master host cannot retrieve remote stats from other hosts.

Resolution

To add the vSAN IPs to vSphereClient Rule via vCenter see Add Allowed IP Addresses for an ESXi Host by Using the VMware Host Client.

From ESXi command line do the following:

  1. To confirm if port 443 is being blocked for vSphereClient run the below command to see the current settings
    esxcli network firewall ruleset allowedip list|grep vSphereClient
  2. Manually update the vSAN IPs into the vSphereClient allowed IP list for all hosts in the vSAN cluster.
    esxcli network firewall ruleset allowedip add -r vSphereClient -i vSAN_IP1,vSAN_IP2,vSAN_IP3, ect.../<subnet mask>
  3. Check the Allowed IP list again using the command from step 1 to confirm the applied change.

Output should be as below.

[root@EX2:~] esxcli network firewall ruleset allowedip list
Ruleset                      Allowed IP Addresses
---------------------------  --------------------

vSphereClient                19#.16#.#.###, 19#.16#.#.###, 19#.16#.#.###, 19#.16#.#.###, 19#.16#.#.###, 10.2#.##.1, 10.2#.##.2, 10.2#.##.3

Restart the vSAN Health Service on vCenter by running "service-control --stop vmware-vsan-health && service-control --start vmware-vsan-health".

Also, restart the vsanmgmt service on the impacted as well as the performance master node using the below command:

/etc/init.d/vsanmgmt restart

Additional Information

More information on Configuring the ESXi Firewall

Powered by Wolken Software