vSAN perf service fails to start on vSAN node

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• vSAN performance service on one of the host fails to start.
  
  
• Running command : '/etc/init.d/vsanmgmtd restart', momentarily start the vsanperf service but it crashes immediately.
  
  
• hostd.log of the ESXi reports following errors:

2025-08-06T00:16:00.873Z warning hostd[2101592] [Originator@6876 sub=HttpConnectionPool-000000 opID=########-##-##-####] Failed to get pooled connection; <cs p:000
0005d624f7a50, TCP:localhost.localdomain:80>, (null), duration: 1msec, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
--> [context]zKq7AVICAgAAAFw4RQEKaG9zdGQAANJCF2xpYnZtYWNvcmUuc28AAK5ZIQCf0BoA8yEzAE9WLgDMxy0AVAMuAAIRPwE7fQBsaWJwdGhyZWFkLnNvLjAAAm3RDmxpYmMuc28uNgA=[/context]
2025-08-06T00:16:00.873Z warning hostd[2101587] [Originator@6876 sub=HttpConnectionPool-044765 opID=########-##-##-####] Failed to get pooled connection; <cs p:000
0005d624f7a50, TCP:localhost.localdomain:80>, (null), duration: 1msec, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)

• --> [context]zKq7AVICAgAAAFw4RQEKaG9zdGQAANJCF2xpYnZtYWNvcmUuc28AAK5ZIQCf0BoA8yEzAE9WLgDMxy0AVAMuAAIRPwE7fQBsaWJwdGhyZWFkLnNvLjAAAm3RDmxpYmMuc28uNgA=[/context]
2025-08-06T00:16:00.876Z error hostd[2101587] [Originator@6876 sub=vmomi.soapStub[44764] opID=########-##-##-####] Resetting stub adapter: service state request failed; /vsanperf, <cs p:0000005d624f7a50, TCP:localhost.localdomain:80>, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)


• Placing host in maintenance mode fails until a reboot is performed of the host.
  
  
• vSAN Skyline Health flags an alert on "All hosts contributing stats" with affected host listed under 'Hosts Not contributing Stats'

Environment

Vmware vSAN 7.0

Cause

• Found the issue occurring due to corrupted vSAN perf config file (/etc/vmware/vsan/vsanperf.conf)
  
  
• Observed that the vsanmgmt.log has not updated for several months and last few logs were as follows:
  
  

  2024-12-10T03:49:19.746Z error vsand[2117020] [opID=vsan-6288e8db3b338 statscollector::RetrieveRemoteStats] VMK vmkX can not connect to host #.#.#.#. Traceback (most recent call last):   File "/usr/lib/vmware/vsan/perfsvc/statscollector.py", line 1117, in RetrieveRemoteStats   File "/lib64/python3.8/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>   File "/lib64/python3.8/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod   File "/lib64/python3.8/site-packages/pyVmomi/SoapAdapter.py", line 1523, in InvokeMethod   File "/lib64/python3.8/site-packages/pyVmomi/SoapAdapter.py", line 1602, in GetConnection   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1520, in __call__
  File "/lib64/python3.8/http/client.py", line 1259, in request   File "/lib64/python3.8/http/client.py", line 1305, in _send_request   File "/lib64/python3.8/http/client.py", line 1254, in endheaders   File "/lib64/python3.8/http/client.py", line 1014, in _send_output   File "/lib
64/python3.8/http/client.py", line 954, in send   File "/lib64/python3.8/http/client.py", line 925, in connect   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1602, in vsanperf_create_connection   File "/usr/lib/vmware/vsan/perfsvc/VsanHealthUtil.py", line 1593, in vsanperf_create_connection socket.timeout: timed out

  2024-12-10T03:49:19.746Z warning vsand[2117020] [opID=vsan-6288e8db3b338 statscollector::RetrieveRemoteStats] No available vmknic to retrieve remote stats.

Resolution

1. Replaced the /etc/vmware/vsan/vsanperf.conf  file (copying it from a working vSAN node) on the affected host.
   
   
   
2. Restarted vsanmgmt service and now vsanperf is up and running.
   Command to run:
   # /etc/init.d/vsanmgmtd restart
   
   Note : vsanmgmt.log starts to log after performing these steps.