Host failed to enter maintenance mode when vsanperfsvc is not running

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• The host reports 0 disks in the vSAN disk management.

• The host shows a warning on vSAN health related to physical disks.

• The host does not enter in maintenance mode.

• vsanmgmtd service is being restarted automatically every 4-5 seconds.

• vsanmgmt.log is not logging data.

• hostd.log reports a high quantity of the following messages 

2021-05-05T15:09:21.396Z warning hostd[2103504] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00000067251abf10, h:38, <TCP '127.0.0.1 : 26053'>, <TCP '127.0.0.1 : 9095'>>, e: 111(Connection refused)

2021-05-05T15:09:21.396Z warning hostd[2101555] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00000067253a00e0, h:34, <TCP '127.0.0.1 : 21307'>, <TCP '127.0.0.1 : 9095'>>, e: 111(Connection refused)

• The end messages on vsanmgmt.log looks below.

2021-03-30T20:16:52Z VSANMGMTSVC: ERROR vsanperfsvc[63a6f71b-11-fd60] [VsanInternalSystem::QueryCmmds] Invoke method failed: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/build/mts/release/bora-11399595/bora/build/vsan/release/vsanhealth/usr/lib/vmware/vsan/perfsvc/VsanInternalSystem.py", line 2762, in QueryCmmds File "/build/mts/release/bora-11399595/bora/build/vsan/release/vsanhealth/usr/lib/vmware/vsan/perfsvc/VsanInternalSystem.py", line 89, in _cmmds_find_noparse File "/build/mts/release/bora-11399595/bora/build/vsan/release/vsanhealth/usr/lib/vmware/vsan/perfsvc/VsanInternalSystem.py", line 366, in RunCmd File "/build/mts/release/bora-11675023/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/runcommand.py", line 87, in __init__ OSError: [Errno 12] Cannot allocate memory

Environment

VMware vSAN 6.x

Cause

The file vsanperf.conf is empty 

This file is located in the path /etc/vmware/vsan/

Resolution

Upgrade vCenter and ESXi to 7.0U3c